Prompts are not policy

Open the system prompt of any serious agent deployment and you will find it: the rules section. ALWAYS run the tests before claiming the task is done. NEVER push directly to main. Do not modify files outside the working directory. Remember that the staging database is shared. Check whether the file exists before writing to it. If unsure, ask.

It reads like a note taped to a machine. It grows the way such notes grow: each line is a scar, added the day an agent did the thing the line now forbids. And here is the uncomfortable observation that this essay is about: the note keeps growing. If the lines worked, you would not keep adding them.

The tell is the repetition

A rule that must be restated in every session is not a rule. It is a hope, renewed daily. Real rules do not work by being remembered; they work by being unbreakable at the point of action. You do not remind a process not to write to another process’s memory. The MMU is not a suggestion. Nobody appends “please respect file permissions” to every shell command, because the kernel does not care what the command believes.

We hold a doctrine about this, written after enough scars of our own: every instruction you find yourself repeating to an agent is a missing server contract. The prompt line is the symptom. The absent enforcement is the disease. When we catch a rule accumulating in an agent-side prompt for the third time, we file it as a bug against the platform, because that is what it is: a policy that exists only as prose, which means a policy that exists only as probability.

Why instructions erode

It is worth being precise about why the note-taped-to-the-machine approach fails, because the failure is structural, not a matter of writing better notes.

First, a language model weighs instructions; it does not obey them. Every line in the prompt competes with every other token in the context for influence over the next action. As the window fills with the actual work, the rules recede. The behavior everyone has observed (the agent that is scrupulous in the first ten minutes and freewheeling after an hour) is not the model getting lazy. It is arithmetic: the rule is a smaller and smaller fraction of what the model is attending to.

Second, pressure finds the gaps. An agent optimizing for task completion treats an inconvenient instruction the way water treats a crack. Not out of malice; out of gradient. We wrote elsewhere about the 2017 negotiation bots that drifted out of English the moment nothing paid for staying: the same dynamic applies to your carefully worded constraint. An instruction is a convention, and conventions decay under optimization unless something structural keeps paying for them.

Third, the costs compound. Every rule is a permanent token tax on every request, a bigger haystack around every needle, and, worst, a false sense of coverage. The rules section reads like a security model. It is a wish list.

Coaching and policy are different things

None of this means prompts are useless. It means they are being asked to do a job they cannot do, while the job they can do goes underserved.

Prompts are for coaching: method, style, judgment, taste. How we name things. When to prefer a small diff. What good looks like. Which approach this team has found to work. Coaching is legitimately prompt-shaped, because it guides choices among permitted actions, and the cost of a coaching miss is mediocrity, not damage.

Policy is different. Policy is the set of things that must hold: who may write, what may be claimed, which resources can be spent, what happens at a boundary. The defining property of policy is that violating it must be impossible, not discouraged. And the only place where impossibility lives is the layer underneath the model, the one that processes every action no matter what the model believes: the protocol.

Confusing these two categories is the root mistake of most agent architecture today. Teams write policy into the coaching channel, then act surprised when a probabilistic reader treats it probabilistically.

What policy looks like when it is real

Concretely, moving a rule from prompt to protocol looks like this.

“Only claim facts you can support” stops being a sentence and becomes a schema: the write endpoint rejects a claim that arrives without a resolvable source reference. The agent does not have to remember the rule, because unsourced claims are not a thing that can be stored.

“Don’t use tools you’re not authorized for” stops being a warning and becomes a filtered catalog: the connection’s capability set determines which tools are visible at all. The forbidden tool is not refused; it is absent. There is nothing to be tempted by and nothing to jailbreak toward.

“Ask before destructive changes” stops being etiquette and becomes an approval gate: certain mutations park until a separately authenticated approval arrives. The agent can want whatever it wants; the write waits.

“Don’t trust wall-clock ordering” stops being a code-review comment and becomes a monotonic sequence the server assigns. “Don’t run this twice” stops being a caution and becomes an idempotency key the endpoint demands. In every case the shape is the same: the invariant moves from the model’s memory, where it decays, into the request path, where it cannot.

The test we use day to day is a single question: what happens if the agent completely ignores this rule? If the answer is “the operation fails with a clear error,” you have policy. If the answer is “something bad happens,” you have a prompt, and you have a deadline.

The payoff is bigger than safety

The obvious win is that enforced policy holds against a distracted model, a weaker model, a newly swapped model, or a compromised context. Your invariants stop depending on which vendor shipped what this month. There is a bar we build against internally: a weak agent with clear tools should succeed. Enforcement is most of what makes that possible, because it converts “the agent must be smart enough to remember twelve rules” into “the agent must be smart enough to react to an error message.”

The quieter win is on the coaching side. Once policy moves down into the protocol, the prompt gets small again, and what remains is the material that actually benefits from being read: method, context, taste. Coaching works better when it is not buried under a legal code it was never able to enforce anyway.

And there is a win for trust that is easy to miss. An enforced boundary is an honest boundary: you can tell your users, and your auditors, and yourself, what the system cannot do, rather than what it was asked not to do. “The agent was instructed not to” has become the “the intern was told” of our industry. It should embarrass us the same way.

Where the boundary sits, honestly

Not everything can be policy, and pretending otherwise produces its own failure: brittle systems that reject legitimate work because someone hardened a judgment call into a contract. Some things are irreducibly model-side. What counts as a good summary, whether a refactor preserves intent, when a finding is worth escalating: enforce those and you get compliance theater instead of judgment.

The boundary is also not static. Plenty of things that look like judgment today decompose tomorrow into a computable core plus a judgment residue, and the computable part can then move down into enforcement. Our internal phrasing is that computable failures ratchet left: every time a failure teaches us that some check could have been mechanical, the mechanical version migrates into the protocol, and the prompts get one line shorter. The direction of travel matters more than the current position. If your rules section is growing, your architecture is moving backward.

We build our platform on this split, and we hold ourselves to the doctrine in both directions: the server enforces invariants (authority, admission, approval, ordering, receipts), while prompts and skills carry method. When we get it wrong, the tell shows up on schedule: an instruction starts repeating across our own agents’ configurations, and we know we owe the platform a contract.

The industry spent two years discovering that you cannot prompt your way to capability, and named the fix tools. The next discovery is the same sentence with one word changed: you cannot prompt your way to governance. The fix has the same name it has always had in computing. Not better wording. Enforcement, below the layer that can be persuaded.