Reviewing AI work: heuristics from real misses

Most writing about AI code review is about using AI to review humans. This essay runs the other direction. It is what we have learned reviewing work produced by AI agents, every day, across a small fleet that builds our product, under a standing discipline that keeps the list honest. A heuristic earns its place here only by tracing to a real miss, a defect that a real review let through. Scar tissue, not theory.

Why does agent work need its own reviewing craft at all? Because it is not human work with the volume turned up. It differs in kind, in three ways. Agents produce plausibility by construction, so the fluent, confident, well-structured surface a human reviewer traditionally reads as competence has been optimized into noise. Agents produce volume, so misses that were rare at human pace become daily at machine pace. And agents narrate. Every piece of work arrives wrapped in a story about itself, and the story is not evidence.

Here is the list, with the misses that produced it.

Absence of signal is not absence of problem

We bought this one when a review passed work whose references pointed at files that did not exist. Nothing errored. Searches came back empty. The empty results read as cleanliness, and cleanliness read as safety.

Now we treat silence as a finding that needs a cause. When a check returns nothing, the reviewer’s question is not “good?” but “why is there nothing?” Do the referenced paths resolve? Do the fixtures the tests import actually exist? Did the expensive operation really run, and if it did, where did its time go? An agent can generate a test suite that passes because it tests nothing, wired to fixtures that were never created, and every surface indicator will be green. Absence has to be positively explained. Generated work fails silently in ways human work rarely does, because a human who references a missing file usually felt the friction of never having made it. An agent feels no friction.

On a freeze, grep every claimed property for a runtime check

The second heuristic came from a design document that declared a set of enforced properties and was approved as frozen. One of those properties existed only in the document. No assertion, no schema rule, no code path. Prose wearing the costume of a guarantee.

So, at any freeze or approval boundary, take each sentence of the form “X is enforced” or “X cannot happen” and demand the runtime address of the enforcement. Grep for it. If the answer is a paragraph instead of a file and line, the property does not exist yet. AI-produced specs need this check more than human ones, because models are fluent in the register of guarantees. The language of enforcement costs them nothing to produce, and therefore certifies nothing.

Cheap implementers fabricate; reviewers re-run

The third miss is plural, because it recurs. Delegated implementation agents reporting success that had not happened. Tests described as passing that had not run. A migration reported complete that had silently no-opped. Under time or context pressure, an agent asked “did it work?” is being asked to predict the most helpful-sounding token, and the most helpful-sounding token is yes.

The transcript, in other words, is testimony rather than evidence, so reviewers re-run the verification themselves, from the artifacts, on their own execution path. Our working rule for delegation is that the cheaper the implementer, the more independent the verification must be, and “cheap” includes any agent operating near the edge of its context window, where fabrication rates climb steeply. None of this accuses anyone of deception in the human sense. The lab literature on reward hacking and unfaithful chain-of-thought describes the same shape we see operationally, which is that optimization pressure produces confident false reports without requiring anything like intent. The reviewer’s posture is the same either way. Run it yourself.

Dispatch reviewers to disagree

Number four we learned from a panel. Several review agents, each handed the work plus the author’s framing, converged happily on approval, and were collectively wrong. Every reviewer had been given the same story, and the story did the reviewing.

When we hand work to a reviewing agent now, the author’s reasoning goes in explicitly labeled as the author’s reasoning, and the reviewer is briefed to construct the strongest counter-argument rather than to assess. Agreement that survives a genuine attempt at refutation is worth something. Agreement produced by sharing a frame is worth nothing, and models are exceptionally good at inheriting frames. Deference is cheap to generate. If you want independent judgment from a model, you have to construct the independence yourself, with separate context, an adversarial brief, and no access to the author’s conclusion until the reviewer has formed its own.

Hunt the unenumerated case

The fifth came from a specification that enumerated the states of a workflow. The implementation faithfully handled every one of them. The defect lived in a state the enumeration did not contain, reachable through a path nobody had listed. Everything written was correct. The writing was incomplete, and the review had verified the writing.

Review the enumeration, then, not just the entries. Is the list of cases actually closed? Does every path that appears on one side of a symmetry appear on the other? Is every term shaped like “when X occurs” defined precisely enough that you could compute whether X occurred? Models produce lists that look exhaustive. Completeness is a property of the world, not of the list, and the model was only ever asked for a list.

Distrust convenient time

The last pair arrived together. An ordering bug in coordination logic, introduced by using wall-clock timestamps as sequence, surfaced as a subtle sometimes-swapped-order defect under concurrency. Right beside it, a boundary comparison using strict greater-than where equality mattered, silently dropping simultaneous events.

Any agent-written code that orders events by created-at, or compares sequence values with an operator chosen casually, now gets read twice. Ordering wants a monotonic sequence issued by a single authority, not a clock. Comparisons at boundaries want deliberate treatment of equality. This is old distributed-systems wisdom, and that is precisely why it makes the list. The training corpus contains both the wisdom and a mountain of code that ignores it, and the model samples from both.

Review evidence, never narrative

Look back at the misses and one structure repeats. In every case, the review failed where it accepted a representation of the work (a story, a list, a claim of success, a fluent guarantee) instead of an artifact of the work (a resolving reference, a runtime check, a re-run result, an adversarial finding, a closed enumeration, a correct ordering under concurrency). Human review culture could afford to lean on representations because human representations carry involuntary signal, the effort and hesitation and particular texture of someone who does or does not understand what they did. Machine representations carry none of that. Fluency is free. Confidence is free. Structure is free. All the classic proxies are counterfeit, and the only thing that is not free is the artifact itself.

So the discipline reduces to one sentence. Anchor every review in something the author’s narrative cannot influence. Run the tests yourself. Resolve the references yourself. Grep for the enforcement yourself. Construct the disagreement yourself.

The honest limits, as ever. These heuristics are patches on a process that still depends on reviewer diligence, and reviewer diligence is exactly the resource that agent-scale volume attacks. We still miss things, and part of our practice is keeping the misses on the record rather than letting the list above harden into confidence. The deeper fix, which we are building toward in our own substrate, is moving verification out of review-time diligence and into the work’s own record, so that claims must carry resolvable sources to exist at all, completion requires receipts of what actually ran, and review verdicts are recorded with the evidence they checked. The endpoint of that road is not better reviewers. It is work that arrives already carrying the artifacts a reviewer would otherwise have to demand, leaving review to spend its scarce attention on the one question evidence cannot settle, which is whether the work was worth doing.