Does Haiku Fan-Out Replace Sonnet? Two Experiments in Code Review and Solution Discovery

The claim going around: replace one expensive model instance with several cheap ones and get the same quality for less money. Haiku 5.5 launched on October 7 at $0.10 input and$0.50 output per million tokens. Sonnet 5.5 is $2 and$10.^[Prices from the Anthropic Haiku 5.5 announcement, for prompts up to 100k tokens. Above 100k, Haiku 5.5 is $0.50 and$2.50. The page does not list Opus 5.5 pricing.] That is a 20x gap on paper.

Two sources set the context. The Anthropic announcement says Haiku 5.5 "pairs well with Opus 5.5 and Sonnet 5.5 as a subagent on coding work," and keeps Sonnet and Opus as the better choice for complex agentic coding. Addy Osmani's LinkedIn post says his team uses it "as a subagent paired with Opus" for "summaries, compactions, or database queries."

Neither source says to replace your code review hunters with a swarm of Haiku. That extension came from the community. My review pipeline has eight parallel hunters, so I tested it: first on code review, then on design decisions.

What the Research Says

I ran a deep research workflow first.^[It is the adversarial research pipeline from my earlier post. 15 sources fetched, 129 claims put through 3-vote verification. 69 claims survived and 60 were refuted. The verification phase hit a rate limit at 62% of its agents, and the report lists 82 claims as unverified.] The benchmark evidence favors Haiku:

On a second benchmark, SWE-PRBench, Haiku detected slightly more issues (0.306 vs 0.297) with a higher false positive rate (0.346 vs 0.227). Haiku sits on the cost-quality Pareto frontier. Sonnet is dominated.

The report also lists what the benchmarks do not cover:

• All the code review data is on the Haiku 4.5 and Sonnet 4.6 generation. None tests Haiku 5.5 against Sonnet 5.5.
• Every benchmark is single-pass review of a diff. None tests agents that read files, grep, and navigate a repo, which is what my hunters do.
• All models collapse from synthetic to real PRs. Sonnet's F1 drops 94%.
• Weaker models show stronger correlated failures, so running several copies of one model gives less diversity than the count suggests.

The report recommended replacing two or three of the five Sonnet hunters and keeping the rest. The pilot I ran used the existing --hunter-model haiku flag, which replaces all five Claude hunters. The three Codex hunters stayed.

Experiment 1: Code Review

Method. I took 14 merged PRs where a review had flagged a must-fix defect and a later commit fixed it. The fix commits gave me ground truth: 26 verified defects, each recorded as file, function, and issue class. PR #69 had no verified defect, so the recall numbers use the other 13 PRs.

Each PR was reviewed twice at the pre-fix SHA in a detached worktree. Both runs used the same dual-stack pipeline. The only difference was the five Claude hunters: Haiku in one run, Sonnet in the other. Both used the same three Codex hunters and the same cross-family validation. A finding counted as found if it matched the defect's location and class.

Results, 13 PRs with verified defects:

Recall is a tie. Each config found defects the other missed. Haiku found both defects in PR #38 where Sonnet found none and got one partial. Sonnet found one extra defect each in PRs #44 and #47 that Haiku missed.

Haiku cost more on 13 of 14 PRs. The ratio ranged from 0.92x to 1.68x.

(PR #69 is left out of the table and the totals: it has no verified defect. Haiku raw 14 and $17.0, Sonnet raw 8 and$10.5. "p" means partial match.)

The cost inversion

The Haiku config produced 61% more raw findings (226 vs 140 across the 13 scored PRs) and used 28% more tokens. The mechanism is the validation stage. Every finding goes through cross-family validation, and the validators are not Haiku. Across all 14 PRs the Haiku config ran 1,259 agents against 981 for Sonnet. Both configs have eight hunters (five Claude plus three Codex), so the extra agents are validators. Validated findings rose from 128 to 211. Validators dismissed 24% of Haiku's findings (51 of 211) and 16% of Sonnet's (20 of 128). A cheaper hunter that produces more noise sends more work to the expensive stage.

One limit on this number: the pilot priced every run at a flat $3 per million tokens, whatever the model. So the 28% is a token result. It says nothing about dollars at Haiku's real price. Dollars depend on what share of the tokens the five Haiku hunters used, and I did not log tokens per role. Under this simplified model, at a 10x price gap, the Haiku config breaks even if the Haiku hunters account for about 25% of all tokens in a run. With 80 validator agents next to 5 hunters, I expect the share to be lower, but I did not measure it.

Experiment 2: Solution Discovery

The same idea applies to design work. My /solution-design skill dispatches three Sonnet agents with different lenses (minimal, pattern-following, idiomatic), scores their proposals, and runs one Sonnet agent to attack the winner. I tested 8 Haiku advocates plus one Opus adjudicator against it.

Method. Three past design decisions, each run twice:

• RIG-9: suppressing dismissed review findings.
• RIG-79: detecting runaway agent sessions.
• RIG-67: a two-pass ticket risk classifier.

The agents read a checkout from just before each feature was built, so they could not see the shipped code. I stripped the chosen design from the tickets. The skill has no model override, so the Haiku run was a custom workflow with eight lenses: the three standard ones plus data-first, safety-first, operability, incremental, and contrarian. The Opus adjudicator merged the proposals, checked conflicting claims against the code, and scored them with the skill's own matrix.

Both configs reached the same design family on all three decisions — the same core architecture, the same primary data structures. The details differed materially. For RIG-9 the Sonnet run picked a tiered rule table with an audit log, and the Haiku run picked an event log with a rule engine and a shadow-mode flag. Haiku with Opus produced 12 distinct approaches against 7, and used 3.3x the tokens. The wall-clock time for RIG-67 is an estimate that includes a delay: the 20-subagent concurrency cap refused my last four launches.

Cost in dollars. At list prices, with Opus 5.5 assumed at $5 and$25 per million, the Sonnet run costs $0.81 to$1.78 for all three decisions. The Haiku run costs $0.91 to$2.92. The range depends on whether the Haiku prompts were under or over 100k tokens, and how much was cached. That is 1.05x to 1.74x the Sonnet cost, and no scenario comes out cheaper. The 24 Haiku advocates cost no more than the 9 Sonnet advocates in any scenario I computed ($0.13 to$1.30 against $0.63 to$1.39). The Opus adjudicator is what pushes the total above the Sonnet run.

Accuracy. The Haiku advocates repeated wrong claims about the code. On RIG-9, five of eight assumed that a new column on review_findings would be wiped on re-ingest. One of the three Sonnet advocates assumed the same. Two Haiku advocates assumed there was no per-finding severity, which was wrong, and two picked migration numbers that collide with existing files. On RIG-79, two assumed role data reaches the digest row, which was wrong. The sample is small, so these are counts and not rates.

Why the Benchmarks Were Wrong for Me

The benchmarks measure a different job. They give a model a diff and score one pass. My hunters read files, search the repo, and form hypotheses over many steps, and no benchmark in the report tests that.

The second difference is validation. In a single-pass benchmark, a hunter's false positives are a score penalty. In my pipeline, each one is a paid validation call. More raw findings does not mean more distinct defects. The Haiku hunters raised raw findings by 61% and found a similar number of known defects (20 vs 21 of 26).

The third is correlated failure. Eight Haiku lenses per decision gave me 12 distinct approaches in total but not a different answer. That fits the report's finding that same-family fan-out gives less diversity than the instance count suggests. I did not isolate the effect.

What Worked: The Verifier

The most useful part of the Haiku run was the Opus adjudicator, not the fan-out. It read the code behind each advocate's claims and found facts the Sonnet run missed:

• In the pre-RIG-79 snapshot, FleetIngestionService is never constructed, so the scheduled ingestion every advocate assumed does not run. The identity lookup also selects columns that do not exist, so role and node ID are NULL.¹
• gate-predicate.ts is a stall watchdog, not a validation gate. Two different GATE_ROLES constants exist with different members.
• The verdict field already carries MUST_FIX, so an advocate's proposed severity migration was unnecessary.

The Sonnet run had its own strength. Its challenger step found three flaws the adjudicator did not look for: the exact-hash match tier will almost never fire because the hash covers raw model prose, a second writer to the digest table skips the sample insert, and the module-counting rule does not fit the real directory layout. The Haiku run had no challenger.

The best design process I saw combines both: independent proposals, a verifier that checks claims against the code, and a challenger that attacks the winner.

Caveats

• One run per cell. I did not repeat any run, so variance is unmeasured. A Sonnet control on PRs #44 and #74 found every known defect at any tier (5 of 5), but agreed with the original review's severity tier on only 3 of 5. Tier assignment is unstable between runs, so I compare any-tier recall.
• Experiment 1 cost is estimated, not billed. It is subagent tokens times a flat $3 per million. Spend was about$280 of a $300 cap, and the run stopped at budget.
• No false positive rate. I did not adjudicate non-baseline findings. At least one Haiku must-fix on PR #76 looks like a real bug missing from the baseline, so Haiku's extra findings are not all noise.
• No zero-must-fix PRs ran. The budget ran out first.
• Baseline limits. The baseline contains only bugs that someone found and fixed. The mapping for PR #80 is approximate. The validators ran on pre-fix code but could read later repo state through web and gh lookups in places. I did not control for that.
• Model version. The research evidence is on Haiku 4.5 and Sonnet 4.6. My runs used the haiku alias, and my results files do not record which snapshot it resolved to.
• Experiment 2 is confounded. The Haiku run has 8 lenses and the Sonnet run has 3. It uses an Opus adjudicator where the Sonnet run uses an orchestrator and a Sonnet challenger. I cannot separate model tier from lens count from the adjudication step. The cost range assumes Opus 5.5 pricing.

Additional Takeaways

A few things the data shows that are independent of the Haiku-vs-Sonnet question:

Both configs are weak at severity classification. Both Haiku and Sonnet recovered only 14 of 26 defects at must-fix tier (54%). Any-tier recall (77-81%) is the more stable metric — tier assignment varied 40% even between two Sonnet runs on the same PR. If your pipeline gates on must-fix severity, you will miss nearly half the real defects regardless of which model hunts.

Sonnet also makes code-fact errors. The post's accuracy section counts Haiku's errors because there were more of them, but one of three Sonnet advocates on RIG-9 also assumed the wrong re-ingest behavior. The error rate difference is directional, not categorical.

Both arms include Codex hunters. Three of the eight hunters in each config are Codex (GPT-6 Astra), and their findings are pooled. A defect "found by the Haiku config" may have been found by a Codex hunter. I cannot isolate Haiku-only recall from this data.

Duplicate findings are an optimization opportunity. Haiku's 61% more raw findings did not produce 61% more distinct defects — most of the extra volume is duplicate or overlapping findings that validators spend tokens dismissing. A pre-validation dedup step (by file + line + issue class) would reduce validator cost for any hunter model.

What I'm Doing

I am keeping Sonnet hunters for now. In this pilot, Haiku showed similar recall but no cost advantage in a pipeline that validates every finding — the extra noise shifts cost to the validators. The data shows near-parity, not Haiku inferiority, but I need a reason to switch, and this pilot did not establish one.

Haiku fits where Osmani and Anthropic put it: bounded subagent work such as summaries, compaction, and lookups. The next test is a mixed config: three Sonnet lenses, three Haiku lenses (safety-first, incremental, contrarian), an Opus verifier, and a challenger. I will run each cell at least twice, so I can tell whether the extra lenses or the verifier is what helps.