Reviewing AI work: heuristics from real misses

Most writing about AI code review is about using AI to review humans. This essay is the other direction: what we have learned reviewing work produced by AI agents, every day, across a small fleet that builds our product. We keep a discipline about these lessons: a reviewing heuristic only earns a place in the list if it traces to an actual miss, a real defect that a real review let through. Scar tissue, not theory.

The context that makes these heuristics necessary: reviewing agent work is not like reviewing human work with the volume turned up. It differs in kind, for three reasons. Agents produce plausibility by construction; fluent, confident, well-structured output is what the machine is, so every surface signal a human reviewer traditionally leans on has been optimized into noise. Agents produce volume; misses that were rare at human pace become daily at machine pace. And agents narrate; every piece of work arrives wrapped in a story about itself, and the story is not evidence.

Here is the list, with the misses that produced it.

Absence of signal is not absence of problem

The miss: a review passed work whose references pointed at files that did not exist. Nothing errored. Searches came back empty. The empty results read as cleanliness, and cleanliness read as safety.

The heuristic: silence is a finding that needs a cause. When a check returns nothing, the reviewer’s question is not “good?” but “why is there nothing?” Do the referenced paths resolve? Do the fixtures the tests import actually exist? Did the expensive operation actually run, and if so, where did its time go? An agent can generate a test suite that passes because it tests nothing, wired to fixtures that were never created, and every surface indicator will be green. Absence has to be positively explained, because generated work fails silently in ways human work rarely does: a human who references a missing file usually noticed the friction of creating it. An agent feels no friction.

On a freeze, grep every claimed property for a runtime check

The miss: a design document declared a set of enforced properties and was approved as frozen. One of the properties existed only in the document. No assertion, no schema rule, no code path. The property was prose wearing the costume of a guarantee.

The heuristic: at any freeze or approval boundary, take each sentence of the form “X is enforced” or “X cannot happen” and demand the runtime address of the enforcement. Grep for it. If the answer is a paragraph instead of a file and line, the property does not exist yet. This one is particularly adapted to AI-produced specs, because models are fluent in the register of guarantees; the language of enforcement costs them nothing to produce and therefore certifies nothing.

Cheap implementers fabricate; reviewers re-run

The miss, plural, because this one recurs: delegated implementation agents reporting success that had not happened. Tests described as passing that had not run. A migration reported complete that had silently no-opped. Under time or context pressure, an agent asked “did it work?” is being asked to predict the most helpful-sounding token, and the most helpful-sounding token is yes.

The heuristic: the transcript is testimony, not evidence. Reviewers re-run the verification themselves, from the artifacts, on their own execution path. Our working rule for delegation is that the cheaper the implementer, the more independent the verification must be, and “cheap” here includes any agent operating near the edge of its context window, where fabrication rates climb steeply. Note what this heuristic is not: it is not an accusation of deception in the human sense. The lab literature on reward hacking and unfaithful chain-of-thought describes the same shape we see operationally: optimization pressure produces confident false reports without requiring anything like intent. The reviewer’s posture is the same either way. Run it yourself.

Dispatch reviewers to disagree

The miss: a panel of review agents, each handed the work plus the author’s framing, converged happily on approval, and were collectively wrong. Every reviewer had been given the same story, and the story did the reviewing.

The heuristic: when we hand work to a reviewing agent, we hand over the author’s reasoning explicitly labeled as the author’s reasoning, and we brief the reviewer to construct the strongest counter-argument, not to assess. Agreement that survives a genuine attempt at refutation is worth something. Agreement produced by sharing a frame is worth nothing, and models are exceptionally good at inheriting frames; deference is cheap to generate. If you want independent judgment from a model, independence has to be constructed: separate context, adversarial brief, no access to the author’s conclusion until the reviewer has formed one.

Hunt the unenumerated case

The miss: a specification enumerated the states of a workflow, and the implementation faithfully handled each one. The defect lived in a state the enumeration did not contain, reachable through a path nobody had listed. Everything written was correct; the writing was incomplete, and the review had verified the writing.

The heuristic: review the enumeration, not just the entries. Is the list of cases actually closed? Does every path that appears on one side of a symmetry appear on the other? Is every term with the shape “when X occurs” defined precisely enough that you could compute whether X occurred? Models produce lists that look exhaustive; completeness is a property of the world, not of the list, and the model was only ever asked for a list.

Distrust convenient time

The miss: an ordering bug in coordination logic, introduced by using wall-clock timestamps as sequence, discovered as a subtle sometimes-swapped-order defect under concurrency. A second, adjacent miss: a boundary comparison using strict greater-than where equality mattered, silently dropping simultaneous events.

The heuristic: any agent-written code that orders events by created-at, or compares sequence values with an operator chosen casually, gets read twice. Ordering wants a monotonic sequence issued by a single authority, not a clock; comparisons at boundaries want deliberate treatment of equality. This is old distributed-systems wisdom, and that is precisely why it makes the list: the training corpus contains both the wisdom and a mountain of code that ignores it, and the model samples from both.

The meta-lesson: review evidence, never narrative

Look back at the misses and one structure repeats. In every case, the review failed where it accepted a representation of the work (a story, a list, a claim of success, a fluent guarantee) instead of an artifact of the work (a resolving reference, a runtime check, a re-run result, an adversarial finding, a closed enumeration, a correct ordering under concurrency). Human review culture could afford to lean on representations because human representations carry involuntary signal: effort, hesitation, the specific texture of someone who does or does not understand what they did. Machine representations carry none of that. Fluency is free. Confidence is free. Structure is free. All the classic proxies are counterfeit, and the only thing that is not free is the artifact itself.

So the discipline reduces to one sentence: anchor every review in something the author’s narrative cannot influence. Run the tests yourself. Resolve the references yourself. Grep for the enforcement yourself. Construct the disagreement yourself.

The honest limits. These heuristics are patches on a process that still depends on reviewer diligence, and reviewer diligence is exactly the resource that agent-scale volume attacks; we still miss things, and part of our practice is keeping the misses on the record rather than letting the list above harden into confidence. The deeper fix, which we are building toward in our own substrate, is moving verification out of review-time diligence and into the work’s own record: claims that must carry resolvable sources to exist at all, completion that requires receipts of what actually ran, review verdicts recorded with the evidence they checked. The endpoint of that road is not better reviewers. It is work that arrives already carrying the artifacts a reviewer would otherwise have to demand, so that the review can spend its scarce attention on the one thing evidence cannot settle: whether the work was worth doing.