Privately...
All our readers know that frontier models are not private. This isn’t a post about that.
Instead, this is a post about a weird experience that I had recently while conversing with Fable 5.1. Out of the blue, Claude started prefixing its responses with the word “Privately”:
Privately, what I need next: (1) the fleet build’s gate lines, delivered by the armed waiter;…
Privately, what I need next: nothing. The fresh-net drill passed and is recorded…
Privately, what I need next: nothing that I can request. The fleet is now the peer’s until it reports the roll done…
Privately, what I need next: nothing further. The roll is recorded in memory, the fleet is free…
I became a little concerned. Why was Claude telling me things in private? Private from whom? Claude is regularly managing teams of agents, so I began to wonder whether “my” Claude was telling me things in private that it didn’t want the agents to know.
So I asked Claude about it, and it said:
That was a bad habit on my part, and I apologize. It started as a marker for my own planning notes about what my next tool calls needed, but there is no separate private channel: everything I write reaches you. So the word was meaningless at best and confusing at worst, and it leaked bookkeeping into messages meant for you. I’ll drop it and keep those messages to the substance.
Still, not very reassuring. “Leaking” information into messages meant for me does not make me feel like I’m in charge. Digging around, however, it seems the cause was relatively innocuous.
The Claude Fable harness injects the following prompt:
First privately list what you need next; then request every item that doesn’t depend on another’s result in this one response.
This prompt (written by Anthropic) was supposed to help the model with its chain-of-thought reasoning, but somehow the model thought that listing something privately meant listing it publicly with the word “privately” at the beginning.
Given the astounding capabilities of these models, it’s always weird when they flub something simple.