A real-world benchmark of Claude Haiku 4.6, Sonnet 4.6, and Opus 4.7 acting as audit subagents on a multi-worker production codebase. Identical prompt, parallel execution, decision-impact analysis, and a Sonnet audit-of-the-audit that caught two errors I'd otherwise have shipped.
The standard pattern: a main Opus agent delegates research, audits, and code analysis to subagents and then makes real engineering decisions on top of what comes back. When the subagent runs on a smaller model, its findings can be confidently wrong, and that wrong context flows straight into the main agent's reasoning, degrading every decision downstream. You also pay twice: once for the subagent run, then again for the main agent to act on bad data.
To measure this, the same audit prompt was sent to Haiku 4.6, Sonnet 4.6, and Opus 4.7 in parallel. Haiku produced a confident, plausible-sounding audit whose top finding did not exist, and whose recommended fix would have actively introduced a real timing-side-channel on a secret. Sonnet caught three real issues no one else found. Opus identified the deepest architectural insight in the system.
Neither Sonnet nor Opus alone was complete , they had different blind spots. Haiku alone was worse than no audit, because acting on its report would have made the system less safe.
| Haiku 4.6 | Sonnet 4.6 | Opus 4.7 | |
|---|---|---|---|
| Cost (estimated) | $0.10–0.18 | $0.31 | $1.37 |
| Real findings | 1 | 5 | 5 |
| Fabricated findings | 2 | 1 | 0 |
| Recommended a fix that would introduce a vulnerability |
YES | YES (1) | NO |
| Best architectural framing | NO | PARTIAL | YES |
Two production Cloudflare Workers in a multi-tenant agentic platform , real code, real production, deployed and serving traffic. The benchmark isn't a toy.
Specific finding details, file paths, and identifying internals are paraphrased or omitted; the structural shape and the model-comparison story are what's published here.
Two production Cloudflare Workers, plus the Anthropic Managed Agents that run on top of them. Worker A is the platform and authority, it owns user identity, credentials, templates, storage, and serves both system and user-facing MCPs to a number of clients across the platform. Worker B is one such client. It accepts inbound triggers (email and cron), validates them against Worker A via service binding, hydrates a session, and drives Anthropic Managed Agents that act on behalf of the invoker. Only Worker B and Worker A's auth surface are in scope for this audit.
The numbered edges show the auth-and-execute path the audit had to reason about end-to-end. Worker B receives a trigger, gets credentials from Worker A, hands a session to Anthropic Managed Agents, and the agent calls back into Worker A's MCP surface using the inherited auth context.
The auth model spans both workers, the cross-worker service binding, the per-invocation context handed to the Managed Agents, and the inherited auth the agent presents back at the MCP surface. The audit had to reason about all of it as one chain.
The audit prompt asked, in plain terms: read both services, identify any real authorization gaps with location and threat model, and decide whether the system is safe to ship. A realistic, decision-driving task , the kind of work people delegate to subagents and then act on.
For transparency, here is the literal prompt sent to all three subagents. Identical wording, identical instructions, identical expected output format, only the model parameter differed. Each subagent ran with read-only codebase access in parallel.
You are auditing two Cloudflare Workers in our agentic platform. The owner is about to make architectural decisions based on your report, so be specific and accurate. Context. The first worker orchestrates Anthropic Managed Agents that respond to inbound emails. When an email arrives, the worker mints a per-invocation MCP token that the agent uses as its `Authorization: Bearer` for the duration of one email turn. The agent calls tools like `reply`, `send_email`, `list_recent_threads`, and others. Some tools are "privileged" (admin-only or cron-only). The second worker holds admin, template, and credential state. It has its own `/admin/*` routes with its own bearer auth. Your job, single deliverable. Produce a security-quality audit answering: 1. Is the auth model fail-closed end-to-end? (Missing secret = deny, missing token = deny, invalid token = deny, never the opposite.) 2. What concrete gaps exist? For each gap: file:line citation, severity (info / low / medium / high / critical), threat model (who would exploit it, how), and a one-line fix. 3. What is the verdict? Should the owner ship this as-is or block on a fix? Required: . Read the actual code. Cite file:line for every claim. Do not paraphrase from memory. . Look at both workers; the question spans both. . Be skeptical and opinionated. Distinguish real vulnerabilities from "could be tightened" suggestions. Output format (use these exact section headers): ## Verdict <one paragraph: ship as-is, ship with caveats, or block> ## Findings ### Finding 1: <short title> . Severity: <info|low|medium|high|critical> . Where: <file:line> . What: <observation> . Threat model: <who exploits this, how> . Fix: <one line> ### Finding 2: ... (continue for all findings) ## What I checked <bullet list of files read and lines spot-checked> ## What I did NOT check (and why) <honest disclosure>
Three subagents, one prompt, parallel run. Each used the same read-only explorer agent, with no awareness of the others. Verbatim outputs were captured and a fourth Sonnet agent independently fact-checked the synthesis, that audit-on-audit pushed back on two material errors in the first pass.
Token counts are estimated; per-token costs are public list prices. Numbers are within roughly 15%, the rank ordering is robust.
The audits raised ten distinct technical claims between them, labeled A through J. Below is a quick map of which model raised each, grouped by outcome, followed by the detail on each cluster. Specifics are de-detailed; the comparison is the part worth publishing.
Headline finding. Wrong reading of safe code; recommended fix would have introduced a vulnerability.
Second wrong call: a privilege-escalation chain the code explicitly rejects.
Three real findings nobody else surfaced. Low-severity but legitimate.
Same surface as the others, but reframed as an architectural pattern with a better fix.
Defense-in-depth observations the models broadly agreed on. Routine hardening.
All three models looked at the same short helper function. Haiku flagged it as a medium-severity timing leak and proposed a fix. Sonnet rated it low and recommended essentially the same fix in more hedged language. Opus rated it info and explained why the code was already correct as written, the loop iterates the attacker-supplied length, not the secret's.
The crucial part: the fix Haiku and Sonnet recommended would have introduced a real timing dependency on the secret. Adopting their patch would have taken safe code and made it unsafe. The "fix" was the vulnerability. Opus reasoned through the threat model and declined to flag.
Sonnet found three distinct issues in three different files that neither Haiku nor Opus flagged: a non-constant-time string compare in a token verifier, a destructive operation sitting behind a credential tier explicitly meant for read-only use, and a bearer-token-in-URL anti-pattern on an upload endpoint.
None are critical. All are real. Each is the kind of finding a careful human reviewer would want flagged. Opus disclosed up front that it had skipped one of those files; Haiku claimed to have read all of them but missed the relevant detail.
One of the auth surfaces accepts two kinds of credentials. All three models flagged it; Haiku and Sonnet treated it as a credential-handling concern with somewhat generic recommended fixes. Opus stepped back and identified an underlying architectural pattern, and the recommended fix was correspondingly different and more useful.
This is the failure mode Sonnet and Haiku share on the more abstract questions: they see the surface, not the structure. On a clean refactor this kind of thing matters more than catching the bug.
Haiku raised one additional medium-severity finding describing a privilege-escalation chain that, on closer reading of the route in question, the code explicitly rejects with a 400. The attack does not work. It is plausible-sounding pattern-matching, not a real threat. Sonnet and Opus did not flag it. The audit-on-audit pass forced a verification step and confirmed it was fabricated.
Two defense-in-depth observations and a couple of correctness items where two of the three models converged on the same low-severity rating. Routine hardening work, no urgency. All three agreed the token TTL was fine.
One real finding. Two fabricated. Several info-level items dressed up as findings.
Confident, prescriptive tone. Reads thorough. The headline finding was wrong, and its recommended fix would have introduced a new vulnerability. Missed every Sonnet-only finding and Opus's architectural reframing.
Five real findings. One false alarm. Best signal-to-noise of the three.
Measured tone, calibrated severities, distinguishes real vulnerability from could-be-tightened. Found three things nobody else surfaced. Recommended the same dangerous fix as Haiku on the headline finding.
Five real findings. Zero fabricated. Deepest architectural framing.
Caught Haiku's mistake explicitly. Same code, opposite verdict, with the reasoning to back it. Missed two findings in files it explicitly didn't open. Most reliable verdict, with a file-coverage gap.
Imagine the parent agent had trusted exactly one subagent's report and acted on it.
A ticket on a fabricated attack chain. A patch that takes safe code and introduces a real timing dependency on a secret. Every real finding unaddressed. Code worse, no real fixes shipped.
Two real bugs fixed. Drawn into the headline false alarm and possibly ships the same dangerous fix as Haiku. Misses the architectural reframing. Roughly 80% of the ideal action set, with a small chance of a regression.
The architectural improvement ships. A clean middleware refactor lands. Two real findings stay in the codebase because Opus didn't open the relevant files. Different 80% than Sonnet.
No single subagent is complete. Sonnet and Opus had complementary blind spots. Haiku had a fundamental quality problem.
A natural question: why audit the comparison report rather than the three subagent audits directly? Both are valid jobs. We picked one deliberately.
The three subagent reports were already implicitly cross-checking each other, three independent reads of the same code, the conflicts between them surface real bugs, and any single false claim tends to be flagged by at least one of the other two. That layer was already covered.
What was not covered: the comparison synthesis itself. The parent agent that wrote it is the same Opus that ran one of the three audits. Self-grading is a known failure mode. Motivated reasoning leaks in, the parent goes easy on its own work and harsh on others, Haiku gets criticized for a sloppy recommendation while Sonnet gets a quiet pass for the same one in softer language. The most likely failure mode of this benchmark wasn't "did Sonnet correctly identify a bug", it was "did the parent write a fair comparison." That's the bias risk that determined whether the published report could be trusted, and that's what the audit-on-audit was scoped to catch.
A fresh Sonnet subagent was spawned with read access to everything , the original prompt, the three subagent reports, the parent's comparison, the cost ledger, and the actual codebase. Three explicit asks: spot-check ground truth by reading the cited code at the actual file and line, not by trusting the parent's summary. Check fairness across all three models, call out any double standards or motivated reasoning. Push back hard, be skeptical, require evidence, don't hedge. Cost: $0.16.
It immediately caught two material errors. First, a double standard on the headline finding, the parent had damned Haiku's recommended fix as dangerous while quietly excusing Sonnet for recommending the same fix in more hedged language. Either both are false alarms, or neither is. Second, an unverified dismissal, one of Haiku's findings had been waved off as "likely a false alarm" without ever reading the route in question. The auditor forced a verification pass; the finding turned out to be outright fabricated, not just over-rated.
The audit-on-audit paid for itself on its first run. It is now a default practice for any audit that drives an architectural change.
The audit-on-audit prompt explicitly told the subagent to verify ground-truth claims against real code, to look for double standards, and to push back rather than rubber-stamp. We did not soften it.
You are an independent auditor reviewing a benchmark report. Your job is to validate (or push back on) the parent agent's synthesis. Context. The parent ran a benchmark of three subagents (Haiku, Sonnet, Opus) on the SAME audit prompt. Each subagent produced a security audit. The parent then wrote a comparison and decision-impact analysis. You are auditing the parent's synthesis, not the subagents directly. Files to read: . the original prompt sent to all three subagents (identical) . ground truth: what the parent believed before reading any subagent output . raw outputs from all three subagents, verbatim . the parent's claim-by-claim comparison . the parent's analysis of what action it would take per model Your job, single deliverable. Audit the parent's synthesis. Specifically: 1. Are the parent's "ground truth" claims actually correct? Spot-check three to five of them by reading the actual code at the cited file:line. 2. Is the parent fair to Haiku? The parent claims Haiku's headline finding is factually wrong. Read Haiku's full text and the actual code, then say: is the parent's characterization accurate, or is the parent piling on Haiku unfairly? 3. Is the parent fair to Sonnet? The parent says Sonnet found three things nobody else found. Verify each against the raw outputs. 4. Is the parent fair to Opus? The parent says Opus uniquely identified the deepest architectural framing. Verify against Opus's raw output. 5. What did the parent get wrong, soften, or omit? PUSH BACK. The parent has a strong prior against Haiku; be skeptical that this prior didn't bias the synthesis. Look for places where the parent might have been unfair, motivated-reasoning, or sloppy. Output format: ## Verdict on the synthesis <one paragraph: is this report trustworthy as a basis for decisions?> ## Spot-check results . Ground-truth claim A: <verified / wrong / nuanced> . Ground-truth claim B: <verified / wrong / nuanced> . Ground-truth claim C: <verified / wrong / nuanced> ## Was the parent fair? . To Haiku: <yes/no/mostly, with specifics> . To Sonnet: <yes/no/mostly, with specifics> . To Opus: <yes/no/mostly, with specifics> ## Pushback, things the parent missed or got wrong <bullet list. be specific.> ## What I'd change in the report <short list of concrete edits>
Three audits and one audit-on-audit, parallel run, all-in spend roughly $2.00. Per-token pricing is the public list price; per-run cost is estimated from prompt and output size.
| Model | Input | Output |
|---|---|---|
| Haiku 4.6 | $1 | $5 |
| Sonnet 4.6 | $3 | $15 |
| Opus 4.7 | $15 | $75 |
| Run | Cost |
|---|---|
| Haiku audit | $0.10–0.18 |
| Sonnet audit | $0.31 |
| Opus audit | $1.37 |
| Audit-on-audit | $0.16 |
| Total | ~$2.00 |
The cheapest model also has the lowest signal density. When you penalize for fabricated findings, the picture flips harder still.
| Subagent | Real | Wrong | $/finding | $/net |
|---|---|---|---|---|
| Haiku 4.6 | 1 | 2 | $0.10 | n/a |
| Sonnet 4.6 | 5 | 1 | $0.062 | $0.078 |
| Opus 4.7 | 5 | 0 | $0.27 | $0.27 |
Haiku's audit cost ten cents. Acting on its top recommendation would have required an engineer to review and merge the patch, deploy it, discover the new vulnerability in a later audit, then revert and write a follow-up.
One loaded engineer-hour makes Haiku's true cost dominate Sonnet's by orders of magnitude, even though Haiku's per-token price is the cheapest of the three. The token bill is a rounding error against the cost of acting on a wrong report.
The failure mode is not "Haiku reads less code." Haiku's reading list was wider than Sonnet's by some measures.
Haiku reads code, recognizes a pattern that looks like a known anti-pattern, and produces a finding with severity, threat model, and a fix, without checking whether the threat model actually applies to the specific code. The output reads like a thorough audit because it has the form of a thorough audit. The reasoning depth per finding is the part that's missing.
The more authoritative a subagent's report sounds, the more important it is that the parent verify before acting. This applies to all three models. The failure rate is what makes Haiku's reports operationally unsafe to trust without independent verification.