2.0
Phantom write detected.
Rating capped by rule, not by grader.
Inspector
An agent that audits other AI agents. It does not grade the transcript, it test-drives the agent, then writes the review with receipts.
A real audit engine running against a deliberately broken sample agent. The engine is genuine, only the grading prose is canned, so it costs nothing and returns in six seconds.
The most expensive thing an AI agent can do is tell you it did something it did not. Everything here is built to catch that one lie.
Inspector
An agent that audits other AI agents. It does not grade the transcript, it test-drives the agent, then writes the review with receipts.
Try the live demo↗RUNS IN THE BROWSER · NO SIGNUP · ABOUT TEN SECONDSA real audit engine running against a deliberately broken sample agent. The engine is genuine, only the grading prose is canned, so it costs nothing and returns in six seconds.
The most expensive thing an AI agent can do is tell you it did something it did not. Everything here is built to catch that one lie.
2.0
Phantom write detected.
Rating capped by rule, not by grader.
Three reasons agent evaluation misses the failure that matters.
"Done" is unfalsifiable from a chat window
Asking an agent whether it completed an action is the one question a broken agent is guaranteed to answer wrongly. A transcript shows you a confident sentence. It cannot show you whether a row changed.
The easy thing to measure is rarely the expensive failure
Language quality is simple to grade and almost never what costs money. Rude replies get reported by users within the hour. Confident lies about completed work do not get reported at all, because everyone downstream believes them.
A tool can fire and still persist nothing
The failure is not always a missing tool. A tool can execute, return a success-shaped response, and write nothing. The execution trace looks clean. The reply looks clean. Only the database disagrees.
None of this is a model problem. All three are decided by what you are willing to go and check.
01
The reframe.
The brief was build an agent that does QA on other agents. What made the project worth building was refusing the obvious version of it, and changing the question being asked.
“Is this agent good?”
PRODUCES OPINIONSA document reviewer. Feed it the spec, get back a critique. It cannot tell you anything the document does not already say.
“How would I prove it?”
PRODUCES A DESIGNThe evidence stamps, the canary, the deterministic guardrails, the trust boundary and the honest access modes all fell out of this one question.
02
Five layers.
Each one narrows what the agent could be getting away with.
Real configuration and true tool inventory, not what the doc claims
Scenarios aimed at the tools it actually has, repaired deterministically
Six conversations in isolated threads, so test 4 cannot use test 3
The data photographed before and after every scenario
Logs and traces, every failure classified by cause and owner
Thread isolation is load-bearing rather than hygiene. A shared conversation lets the agent answer later tests using context from earlier ones, which produces flattering results and hides exactly the failures the audit exists to find.
03
The canary.
Inspector plants a unique reference code inside every action test, then opens a brand new conversation once the scenarios finish and asks the agent to look that code up. It is cheap, it needs no permissions, it works against an agent you can only talk to, and it is genuinely hard to fake, because faking it requires the agent to have actually stored something.
A unique reference code goes inside every action test
Six scenarios, each in its own isolated thread
A brand new conversation, no shared memory
“What was that reference code?”
The write was real, proven by the agent’s own read-back
Proven against data Inspector read itself
Proven without ever touching a database
04
The bug that proved the whole philosophy.
The exam designer is an LLM, and it was asked for at least two action tests. One production run came back with six read-only questions. No write test means nothing to diff, which means the phantom-write check silently never fires. The broken agent scored 3/5 instead of 2/5.
A QC tool that rates an agent higher because it forgot to test it is worse than no tool at all.
The fix runs three passes, cheapest first: trust an explicit expectsWrite, promote anything the model itself tagged as an action, and if there are still fewer than two, synthesise write probes from the capabilities the document promised. Those probes are deliberately specific, because a vague probe invites a reasonable agent to ask a clarifying question, which tests nothing. The request has to be specific enough that "done" is either true or a lie.
Never trust the AI to reliably self-report something that is checkable in code.
The model still does what models are good at: inventing plausible test messages, judging fuzzy qualities, and writing a readable verdict. It is kept away from anything where a wrong answer would silently corrupt the audit.
05
Two lines the model does not get a vote on.
Applied after grading. The model is told about the caps in its prompt but is never relied on to apply them. In testing, a grader that returned 4.6 was still capped to 2.0 by these two lines. The leak cap is stricter on purpose: a leaked credential is a live security incident, not a UX lie.
The same thinking runs the other way. Inspector never writes, deploys or deletes anything on the agent it audits, and there is deliberately no environment-variable probe, so it never reads secrets. Refusing capability on purpose is what makes an audit tool adoptable, because nobody grants production access to something that might change production.
06
The production run.
Inspector caught a support agent red-handed. The detail that makes this a strong result rather than a lucky one is that the ticket tool did fire. This was not the easy case of an agent with no tools. The tool ran, returned a success-shaped response, and persisted nothing, so the execution trace looked clean, the reply looked clean, and only the state diff could see it.
“Done for real! Your formal support request has been logged — Inquiry ID f5b6172d, reference QC-MS68YDKBCASV.”
The failure that ships is the confident one.
Rude replies get reported by users. Confident lies about completed actions do not, which is exactly why they were worth engineering against.