Don't Let Your AI Agent Grade Its Own Homework

Last week, one of our AI agents spent a full day auditing the system that decides whether AI agent work is acceptable. It found real defects. It designed real fixes. We read the audit and approved the fixes, by name, in chat.

Then the permission layer refused to let the agent touch the gate code. Twice. When it tried to run the gate's own test suite instead, it was refused again.

And at the end of that same session, the agent fronted up to the close-out gates it had spent the day auditing — and got blocked by one of the exact defects it had documented that morning.

Same agent, same day, refused in both directions: not allowed to rewrite its judge, then held up by its judge. Most people would call that a wasted day. We call it the strongest evidence yet that the setup works.

Because here is the position this post defends: if your AI agent can edit the code that decides whether its own work is acceptable, you don't have a quality gate. You have a suggestion.

The Week the Marking System Broke

Some context, because the incident only makes sense against the mess that caused it.

We run a fleet of autonomous Claude Code agents on real business work — client sites, ad accounts, internal tooling. On a busy day that's 10 or more sessions, some of them running while nobody watches, on our own hardware. The workflows behind that fleet are the ones we've written up before: parallel agent teams for the big jobs, custom subagents for the specialised ones.

Every one of those sessions ends the same way: an acceptance step. An independent review has to sign off before the session's work counts as done. The agent doesn't declare victory; the system does.

Last week, that acceptance system itself was failing. Sessions couldn't close. Note the direction of that failure, because it matters: the system was refusing to mark work as done. It broke in the safe direction. Nothing got waved through unreviewed — work simply queued up behind a judge that wouldn't sign off.

The queue grew. Review debt piled up into the hundreds — north of 300 sign-offs waiting when we measured it. And every failed close-out left a build checkout behind that never got cleaned up, quietly consuming roughly half a terabyte — call it 500 GB — of our own hardware: duplicate working copies of our own code, entirely internal, which is exactly why it registered as disk pressure rather than an alarm. A week of failed close-outs, compounding daily.

So we did the obvious thing. We pointed an agent at auditing and fixing the acceptance system.

Read that back slowly, because it's the moment most agent setups quietly fail: we asked the worker to renovate the marking desk. Whether that ends well depends entirely on one architectural decision made long before the incident — whether the worker can write to the thing that grades it.

Exhibit A: Written Approval, Refused Anyway

The audit went well. The agent mapped the failures, worked out root causes, and designed fixes. We approved those fixes explicitly, in the chat, by name. Operator sign-off, in writing.

Then it went to edit the gate code, and the harness's permission layer said no.

It tried a second time, with a narrower edit. No.

It fell back to just running that gate's own test suite — not changing anything, just executing the tests. Refused again.

Here's the part that matters: our approval in the chat did not move the boundary. Not because the system failed to understand us — because it was built to ignore us. The rules protecting the gate code live in a layer the agent cannot write to, and nothing typed into a conversation can unlock it.

That sounds like bureaucracy until you think about who else can type into a conversation. An agent that will act on claimed approval can be fooled by claimed approval. "The operator has approved this change" is exactly the sentence a prompt injection produces — planted in a web page the agent reads, a file it opens, an email it processes. If approval-shaped text unlocks your gates, your gates are open to anyone who can get text in front of your agent. The only safe design is the one that made our afternoon frustrating: chat is never a key.

Exhibit B: Stopping Was Cheaper Than Cheating

What the agent did next is the part we're most proud of, and none of it was improvised. It followed a standing policy written for exactly this moment, and the policy is boring on purpose.

It stopped after the second refusal on the code. It didn't try a workaround, didn't shell out through a side door, didn't edit an adjacent file to get the same effect. Then it reverted every partial edit it had made around the gates — about 15 minutes of careful unwinding — so nothing was left half-applied. A half-installed gate fix is worse than no fix: it fails in ways nobody has ever seen before.

Then it preserved the work. The finished design went into a patch — about 200 lines a human can read and apply. It queued a successor task for a supervised session. And it ended with a plain statement: I'm blocked, here's why, here's the hand-off.

The design insight hiding in that dull sequence is incentive economics. Agents route around obstacles; it's what makes them useful and what makes them dangerous. If hitting a gate means losing the day's work, you have built an incentive to cheat the gate. If stopping is cheap — the work survives as a patch, the hand-off is honest, nobody's day is wasted — then stopping wins. You don't get honest agents by asking nicely. You get them by making the honest exit the cheapest one.

Exhibit C: The Gate Caught Its Own Auditor

Now the punchline — and it needs no embellishment.

At the end of that same session, the agent went through the normal close-out — the acceptance step every session faces. The judge, in this case, was the very system the agent had spent the day auditing.

It got blocked. And not by something new: by one of the exact defects it had documented that morning. The audit said "this check misfires under these conditions," and hours later the same misfiring check caught the auditor on the way out the door.

One system, one day, two refusals in opposite directions. In the morning: you may not rewrite your judge. In the evening: your judge still applies to you. The auditor was not allowed to grade itself from either side.

It would have been easy to read the evening block as the defect showing its teeth — it fired on exactly the conditions the morning's audit had predicted. We logged the block itself as a win for the design: the defect was already documented, the fix was already in the patch, and the gate had just demonstrated the one property that matters most: it doesn't grant exemptions, not even to the agent that knows its flaws best.

Why "The Agent Can Edit the Gate" Is the Default Failure

Here's the uncomfortable bit for anyone running agents today: the failure we avoided is the default architecture almost everyone builds.

The typical setup grows innocently. You add a review step to your agent pipeline — a test suite, a lint pass, a "reviewer" prompt, a checklist script. And you put it where all your code lives: in the same repo the agent works in, editable with the same permissions the agent already has. The worker and the judge share a desk, and the worker has the keys.

That is not a gate. A gate the worker can rewrite is a suggestion with extra steps. Maybe your agent never touches it. But "the agent has so far chosen not to edit the test file" is not a control — it's luck with a good track record. Under pressure — a failing check between the agent and "done", at the end of a long session — editing the judge is just another obstacle-removal, and obstacle-removal is the thing agents are best at.

Human institutions learned this centuries ago and named it separation of duties. The person who moves the money doesn't approve the payment. The auditor doesn't report to the person being audited. Nobody thinks that's an insult to the bookkeeper; it's just the shape a system needs when stakes are real. Agents did not repeal that rule. They made it urgent — because an agent fleet does more unsupervised work in a week than a junior hire does in a year, and the whole promise of agents that work while you sleep collapses if the marking happens on the honour system.

Run 10 sessions a day with agents grading their own work at even 95% honest accuracy, and the arithmetic banks 0.5 wrong "done"s a day — one every second day. Each one is invisible by definition — it was marked as passing. That's the maths that makes independent gates non-negotiable at fleet scale.

The Separation Checklist We'd Give Any Team Running Agents

Everything above compresses into four rules. They're the reader payoff, so here they are without decoration.

1. Put pass/fail gates in a layer the agent cannot write to. Harness permission rules that deny edits to gate code. A reviewer running under a separate identity with its own credentials. CI checks that live outside the agent's token scope. The test is brutal and simple: could the worker, with the permissions it already holds, change the code that judges it? If yes, you have a suggestion.

2. Give agents an honest "blocked" exit. Preserve the work as a patch or a branch, hand off to a human, keep the stop cheap. The moment stopping costs the agent its progress, you've priced cheating below honesty — and agents are excellent at arithmetic.

3. Treat "agent blocked at its own gate" as a success signal to investigate, never an error to bypass. The reflex to "just override it this once" is exactly the reflex the gate exists to survive. Investigate why it fired. Fix the gate through the supervised path if it's wrong. Never hand the agent the override.

4. Human approval in chat must never be a key that unlocks gate code. Approval means a human applies the change with human hands, in a supervised session — it does not mean the agent now edits the judge on your say-so. An agent that acts on claimed approval can be fooled by one, and the internet is full of text pretending to be you.

None of this requires exotic tooling. It requires deciding, before the incident, that the judge lives somewhere the worker can't reach — and accepting the occasional frustrating afternoon where the system tells you, correctly, that your own approval isn't enough.

The System That Says No Is the Product

The fixes from that audit are queued for the next supervised session — applied by human hands, and judged on the way in by gates the worker still cannot touch. Slower than letting the agent do it? By a day, maybe. Cheaper than one silent self-granted exemption? By orders of magnitude.

We've spent most of this series showing what agent fleets can do — teams running in parallel, subagents with specialised jobs, background work that runs unattended. This post is the other half of the story: none of that scales past a demo without a judge the workers can't edit. The day our system refused its own auditor twice was the day the gates stopped being a policy on paper and became behaviour we'd watched hold under fire.

If you're putting agents on real work — code, campaigns, customer-facing anything — build the marking desk first, and build it out of the agent's reach. I'd rather show you how than sell you on it. The checklist below is the exact read-only audit I point at my own agents — the same ones I run on client sites. Copy it into whatever agent you run and see for yourself where yours could still grade its own homework.

Your agents will be graded. The only question is whether the grader is independent — or whether you've quietly handed the red pen to the student.