The question arrives in every architecture review now: should we build a fixed workflow of small, cheap agents that each do one defined thing — or hand the whole job to the most capable model and let it decide how to split the work, spin up its own subagents, and judge when it's done? This page is the proof behind the field note. Every number on it comes from a real, separately billed model run against ground truth written before any model saw the task.
fixed_haiku on a vague bug-diagnosis task failed 2 of 11 hidden tests, and the trace shows why -- plan correctly diagnosed the real bug, implement applied the exact correct fix, then review invented a nonexistent issue and broke previously-correct code fixing it. Every other run passed everything. Cost decided almost everything else: at the same model tier, the autonomous agent was cheaper than the fixed pipeline in 32 of 36 head-to-head comparisons, usually by more than half, because every fresh invocation pays a fixed overhead that the pipeline pays three times and one autonomous call pays once. Autonomy went almost entirely unused -- zero subagents in 35 of 36 autonomous runs, holding even when a task was explicitly built to invite decomposition (round 4b) or had six genuinely independent pieces to analyze (round 5's forecasting task). The one exception is the most interesting result in the study: auto_opus on the incident-diagnosis task spawned a fresh-context adversarial reviewer against its own fix and test suite, which caught two real gaps a first pass missed -- decomposition used for self-critique, the one place in 36 autonomous runs the model reached for help at all.How to read this
Fixed means a human-designed three-step pipeline — plan, implement, review — each step a fresh model call with the Task tool disallowed, so it cannot improvise. Autonomous means one call to a single model with explicit permission to decompose the work, spawn subagents, and set its own acceptance criteria. Tasks 1-7 are graded by a pre-written pytest suite the model could read. Tasks 8-9 (round 5) remove one shared boundary each: task 8 is a new domain (revenue forecasting) still graded deterministically (MAPE against synthetic ground truth); task 9 is a new grading methodology (a vague bug report, no visible test file, graded by an 11-case hidden suite applied only after the run plus a judged root-cause-diagnosis score). Cost is USD from the CLI's own accounting; time is wall-clock. Win rules were locked and SHA-256 hashed before each round ran.
Pass rate — every condition, every task
Cost by model tier on the hardest task
Same model, different architecture
What the numbers don't show
Autonomy was reached for exactly once, and it wasn't to split work. Zero subagents across 35 of 36 autonomous runs, including a task (round 4b) explicitly worded to say the model could parallelize three independent pieces, and a task (round 5) with six genuinely different independent product lines to forecast. The one exception -- auto_opus on the incident-diagnosis task -- used a subagent as a fresh-context adversarial reviewer against its own work, not to divide labor. Anthropic's own write-up of its multi-agent research system says decomposition pays on parallelizable research and is 'less effective for tightly interdependent tasks such as coding' -- this one exception is arguably still consistent with that: verification is closer to research than to writing the code in the first place.
The review step changed code 3 times in 36 fixed-pipeline runs -- two real catches, one real regression. Round 3 (Fable, cron task): caught a spec bug the test suite itself had missed. Round 5 (Fable, forecast task): caught a real NaN-propagation defect in a seasonal-decomposition edge case the given data never happened to trigger. Round 5 (Haiku, incident task): the one failure. Plan correctly diagnosed the bug. Implement applied the textbook-correct symmetric fix. Review then claimed a nonexistent 'boundary condition' problem and changed a correct strict inequality to a non-strict one, breaking previously-working adjacent-booking behavior. The judge still scored this run's diagnosis pass/1.0, because the stated root cause was correct -- the failure was introduced two steps later, by a step with no ground truth of its own to check against.
Per-invocation overhead is the mechanism, not architecture. A one-word Opus reply cost $0.41 before producing any output -- roughly 40,000 tokens of system prompt and tool schema. On small tasks that fixed cost, paid once versus three times, explains most of the 32-of-36 pattern above; where it doesn't (the four exceptions), each has its own concrete story rather than reverting to noise.
The floor was not found on eight of nine tasks. Cron calculators, event-sourced ledgers with planted bugs, six-unit revenue forecasting, a vague incident report over a real bug -- the cheapest tier matched the most expensive one on all of them. It failed on exactly one, and the trace shows the failure was a process defect (review overreaching) rather than a comprehension defect (the model not understanding the task).
n = 1 per cell. No variance estimate. Nine tasks, one vendor's model family, one CLI's overhead profile, one judge backend for the diagnosis-quality score. Real and falsifiable; not a benchmark.
Method, pre-registered
Round 1 compared a fixed Haiku pipeline with an autonomous Fable agent on three easy tasks. Round 2 crossed the design — fixed Sonnet, autonomous Haiku — to separate architecture from price. Round 3 added three harder tasks and completed the 2×4 matrix on the hardest one, then round 4 filled the remaining deferred cells, completing a full 48-cell matrix. Round 4b tested whether structural breadth alone (not difficulty) triggers subagent use — it didn't. Round 5 removed two different boundaries at once: task 8 is a new domain (forecasting) under the same deterministic-grading discipline, calibrated against a naive baseline and a real statistical reference before locking; task 9 is a new grading methodology (hidden tests plus a judged root-cause score) under a vague incident report instead of a written spec, calibrated against a known buggy file and a known-correct fix, and the judge itself spot-checked against a known-good and known-bad diagnosis before being trusted on real model output. Every round's win rules were written and hashed before its runs; every mid-round addition or correction (task 6, task 7's harness bug, a session-rate-limit corruption in round 4, round 5's task designs) is disclosed in the round it happened. Reference solutions and held-out ground truth were kept outside the run tree during every run. The orchestrator is a Python script shelling out to claude -p; it never used the Agent tool itself.
Scope and limits
This measured cost and pass rate on tasks with clear ground truth — deterministic tests for tasks 1-8, a hidden deterministic test suite plus one LLM-judged secondary score for task 9 — and nothing else. It did not test the sampling-and-voting ensembles of Li et al. (More Agents Is All You Need), a genuinely different 'many cheap agents' design. It did not test open-ended, genuinely parallelizable research tasks, the shape of work where Anthropic's own write-up reports real multi-agent gains — task 8's six independent forecasting units came closest and still didn't trigger it. The verdict is real for these 72 runs under these rules; generalize with that in view.