Many Cheap Agents, or One Smart One?

The question shows up in almost every architecture review now, and it always has the same shape. One camp wants a fixed workflow: small, cheap models, each doing one explicitly defined job — plan, then implement, then review — wired together by a human who decided the steps in advance. The other camp wants to hand the whole job to the most capable model available and let it decide how to split the work, spin up its own subagents if it wants them, and judge for itself when the job is done. Cheap and explicit, or expensive and autonomous. Both camps have a story. Neither, in any room I’ve been in, has had a number.

So before writing a word of this, I ran the bakeoff: two architectures, four model tiers, nine tasks, each one with ground truth written and checked before any model saw it. Seventy-two recorded runs across five rounds, about ninety-four dollars all in, every one of them a real, separately billed model call. Every win rule was written down and hashed before the runs it governed. What follows is what the data said, including the part where it refused to answer the question I asked, and the one run, out of 72, where something actually went wrong. The full source, raw results, and everything needed to run it yourself are public: github.com/90rdon/agent-architecture-bakeoff.

The first result was clean, and wrong

Round one was the question as posed. A fixed three-step pipeline on the cheapest tier, Haiku, against one autonomous agent on the most capable, Fable. Three tasks: a string function, a two-file feature, a bug fix in existing code. The result could not have been cleaner. Both hit 100% of their tests on every task. The pipeline cost 17 to 26% of what the autonomous agent cost, on every task, far outside the tie band. Three for three. The headline wrote itself: the cheap explicit pipeline wins.

It was also almost meaningless, and the reason is worth sitting with because it’s the reason most architecture bakeoffs mislead. The comparison had moved two things at once. The architecture changed — fixed versus autonomous — and so did the price of the model running it. A clean win told me nothing about which of those did the work. So round two crossed the design: the same fixed pipeline on Sonnet, the same autonomous agent on Haiku. Now the architecture could be read on its own, with the model held constant.

At matched tier, the result reversed. On the same three tasks, on the same cheap model, the autonomous agent cost 41%, 42%, and 62% of what the fixed pipeline cost — and was faster every time, at the same 100% quality. The fixed pipeline wasn’t the efficient architecture. It was the architecture that happened to be wearing the cheaper model in round one.

The mechanism is unglamorous. Every fresh invocation of a model through an agent harness carries a fixed overhead before it does any work — the system prompt, the tool schemas, the context that has to be loaded whether the task is one line or a thousand. I measured it directly: a one-word reply from Opus cost $0.41 and produced zero output tokens, because roughly forty thousand tokens of setup were billed first. A three-step pipeline pays that three times per task. An autonomous agent pays it once. On tasks small enough to finish in a few minutes, that overhead is the difference between the architectures, and no amount of clever step design gets it back.

The cheapest configuration on every task was the one the original question didn’t offer: one autonomous agent on the cheapest model.

Then I tried to break the cheap model

A result that says “cheap is enough” is only useful if you know where enough ends. Round three was built to find that edge. I added an expression evaluator with the precedence rules people get wrong — right-associative exponents, unary minus binding looser than a power — and a four-module event-sourced ledger with a planted bug that only shows up when a snapshot is taken mid-transaction. Both saturated instantly: 100% at Haiku, under both architectures. So I built a third, on purpose, to be brutal: a cron “next fire time” calculator with the full Quartz extensions — last-day-of-month, nearest-weekday, nth-weekday-of-month, name ranges, step syntax — and the Vixie rule that when both day fields are restricted, either one matching fires. Forty-five tests. No library shortcut exists. It is the kind of spec where a small model usually gets nine things right and the tenth silently wrong.

Haiku, with no structure at all, scored 45 of 45. For forty-eight cents. I checked the test file was byte-identical, checked the solution used nothing but the standard library, and re-ran the suite myself. Then I ran the same task through every other condition.

Every cell is 100%. Six tasks, one of them designed specifically to fail the cheapest model, and the cheapest model passed all of them under both architectures. The frontier tiers passed too — Opus at four times the cost and twice the time, Fable at three and a half times the cost and, to its credit, faster than Haiku — for the identical score. I want to state the limit of that plainly, because it’s easy to read as a triumph and it isn’t one: I could not construct a coding task with a deterministic test suite that Haiku failed. That is a fact about my task set, not a proof that no such task exists. But it moves the floor. If your mental model is that routine work needs a mid-tier model and anything with real logic needs the frontier, this data says that model is a tier or two too pessimistic, on any work a test can grade.

What autonomy did with its freedom: nothing

Every autonomous run was told, in plain words, that it could split the work however it liked, spawn other agents to help, and set its own idea of when the job was done. Across the original six tasks, that adds up to twenty-four autonomous runs — every tier, every task. Zero subagents spawned. Not on the trivial tasks, not on the cron calculator, not at Opus, not at Fable. Each one read the spec, wrote the code, ran the tests, and stopped.

That isn’t the models being lazy. It matches what the people who built them say. Anthropic’s own engineering write-up of its multi-agent research system — an orchestrator spawning three to five subagents in parallel — reports that the multi-agent version beat a single Opus agent by 90% on their research evaluations, and also that it burned about fifteen times the tokens of a normal chat, that token spend alone explained roughly 80% of the performance variance, and that the approach is “less effective for tightly interdependent tasks such as coding.” The models in my bakeoff, offered decomposition on coding tasks, declined it. Their vendor would have told them to.

But that result only ever varied one thing: how hard the task was. It never varied whether the task actually had a shape worth splitting up. So I built a fourth round to ask that directly. The new task was three small utility modules — a text-slug function, a check-digit algorithm, a Roman-numeral converter — that share no code and no state with each other. Nothing about solving one helps with either of the others. The instructions said, word for word, that the model could build them “in any order, in parallel, or however you judge best.” If any task in this whole study was going to make a model reach for a subagent, this was it: easy, obviously divisible, and told in advance that splitting was fine.

It still didn’t. All four tiers, zero subagents, all forty-one tests passed by every one of them, working the three modules in a single uninterrupted pass. That brings the total to twenty-eight autonomous runs across seven tasks, at every price point from cheapest to most expensive, and not one of them ever used the tool it was explicitly handed. This is the part of the original question that dissolves on contact with data. “Let the smart model build its own subagents” assumes there is a decomposition worth building, or at least worth the model’s trouble. On self-contained coding work, even work built to invite it, there mostly isn’t. A model good enough to be handed the job seems to know that a straight line through three easy files beats the overhead of managing helpers for them.

Then, once, it wasn’t nothing

Both round 4b tasks I’d used to test decomposition were still coding: write files, satisfy a spec. So round five asked two questions that had nothing to do with difficulty and nothing to do with breadth. Could a genuinely different domain — not code, a business forecast — get a model to reach for help? Six product lines, forty-eight months of revenue each, real trend and seasonality and one line with a planted mid-history break, forecast the next six months, no library shortcut for “figure out what kind of line this is.” Six independent units to think about separately if a model wanted to. It didn’t. All four tiers, zero subagents, every unit passed the accuracy bar. Autonomy’s answer to a new domain was the same answer it gave to new difficulty and new breadth: no.

The second round-five task asked something else: could genuine ambiguity do it, where difficulty and breadth hadn’t? No spec this time, no visible test file — a vague incident report, the way an on-call engineer actually gets one. “Room 4B got double-booked twice this week, no error in the logs, didn’t happen for every overlapping pair, just some of them. Figure out why and fix it.” The real cause was a genuinely subtle bug: the booking system’s overlap check only worked in one direction, so whether a conflict got caught depended on which of two people happened to book first. Three tiers solved it in a single pass, no subagent. At Opus, once, something different happened: the model wrote a fix, wrote its own test suite, ran sixteen mutation tests against its own code — and then spawned a second, fresh-context copy of itself to attack what it had just built. That reviewer found two real problems the first pass had missed: a test that claimed to be exhaustive but only ever checked two bookings, not three, and a subtler bug where handing back a live reference to internal state let a caller silently corrupt a booking after it had already passed validation. Both were real. Both got fixed. This is the only subagent spawned in 36 autonomous runs across the whole study.

The shape of it matters as much as the fact of it. This wasn’t decomposition — there was nothing to split, one bug, one fix. It was verification: a model using a second, disinterested copy of itself to argue against its own conclusion before trusting it. That lines up with what Anthropic’s own write-up says almost too neatly — decomposition earns its keep on tasks that look like research, and this is the one task in the study that actually required investigation before a fix was possible, not just implementation of one already known. It cost real money to find out: this run was $2.87 and took just over ten minutes, against $0.17 to $2.00 and roughly one to two and a half minutes for the tiers that solved the same bug without asking for a second opinion. A subagent is not free, and the one time it showed up, it was buying something specific — not more hands, a harsher reader.

Where structure actually showed up

Every task in the study ran at every tier under both architectures — not most, all of it, once round four filled what earlier rounds had deferred and round five ran full matrices from the start. That means I could compare fixed against autonomous at the same model, on the same task, thirty-six separate times — the fairest test available, because nothing but the architecture differs. The autonomous agent was cheaper in thirty-two of those thirty-six. Usually by a lot: 35 to 65 cents on the dollar for the identical result.

The four exceptions don’t reduce to one story. On the expression-evaluator task, at Opus, the pipeline came in a single cent under the autonomous agent — $2.587 against $2.691, close enough to call it noise. On the cron task, at Haiku, the pipeline cost $0.35 against the autonomous agent’s $0.48 and finished faster — the densest spec in the set, the cheapest model, a plan-first step that seems to have kept the implementer from thrashing. The other two are new, from round five, and each has its own concrete cause rather than reverting to noise. On the forecasting task, at Haiku, the autonomous agent cost nearly double the pipeline — not a mechanism failure, just genuinely more work: the transcript shows fifty conversation turns of the model trying approaches, checking residuals, and iterating on the forecast before it was satisfied, where the fixed pipeline’s single plan-then-write pass got there in one. And on the incident task, at Opus, the autonomous agent cost more than the pipeline for a reason I’d already have guessed by this point in the piece: that’s the run that spawned a subagent. A second opinion isn’t free, and its cost shows up exactly where you’d expect it to.

Move one tier up and the advantage is gone. At Sonnet the pipeline cost $1.48 to the autonomous agent’s $0.90; at Opus, $2.88 to $2.09; at Fable, $3.84 to $1.70. Same 100%. The overhead of three invocations scales with the price of the model making them, and above the cheapest tier it swamps whatever the plan step saves.

The review step deserves its own reckoning, because it’s the step everyone insists on. Across all thirty-six fixed-pipeline runs in the study, it changed the code exactly three times. The other thirty-three: it read the code, confirmed the tests passed, and stopped — a third invocation’s worth of cost for a sentence of reassurance. A separate, preserved attempt sits outside that count and is worth naming for a different reason: a Sonnet implement step once hit a time ceiling and left nothing on disk, and the review step wrote the whole solution itself, correctly. That’s a real property — a multi-step pipeline has somewhere to recover that a single call doesn’t — but it’s recovery, not quality control, so it isn’t counted as review catching anything.

Two of the three real changes are worth the price of the whole experiment on their own. Fable’s implement step, on the cron task, wrote a parser that passed all forty-five tests and rejected WED as an invalid day — a substring check meant to catch the W extension had caught the W in Wednesday. No test exercised it. Fable’s review step found it, fixed it, and explained why. I went back afterward and probed every other cron solution with five spec cases the suite never covered; all of them passed, Haiku’s included, so the top tier’s first draft had made a mistake the cheapest tier hadn’t, and the top tier’s own second look was what caught it. The second genuine catch showed up in round five, on the forecasting task: Fable’s review step found a real division-by-zero hiding in the seasonal math — a phase that happened to be zero in every year of the given data would have produced NaN, and a clamp meant to guard against bad output couldn’t save it, because NaN poisons a maximum the same way it poisons everything else. None of the six product lines actually hit that phase, so the tests never would have found it. Review did, unprompted, and fixed it with a one-line floor on the index before it can hit zero.

The third change is the only failure in the entire study, and it happened at the step with no ground truth of its own to check against. On the incident task, at Haiku: the plan step correctly diagnosed the real bug — an overlap check that only worked in one direction. The implement step applied the textbook-correct fix, the same symmetric formula my own reference solution used. At that point the code was right. Then the review step read it again, decided on its own that it had found a “boundary condition” problem that did not exist — back-to-back bookings that touch but don’t overlap were never the bug — and changed a correct strict inequality to a non-strict one to “fix” it, which broke exactly the case it invented a problem for. Two of eleven hidden tests failed as a direct result. The judge I used to separately score the model’s own diagnosis still rated it correct, because the stated root cause genuinely was correct — the failure came two steps later, from a step reasoning freely with nothing to check itself against.

What I make of all three together: on work the tests already cover, review is a third invocation usually looking for something the second one didn’t leave behind, and most of the time it finds nothing because there’s nothing to find. But twice in this study it found something real precisely because the tests didn’t cover it — a spec case no one wrote a check for, a data shape the given inputs never happened to trigger. And once, on the one task in the study that gave review room to reason about something no test could verify either way, it invented a problem and broke a correct answer fixing it. Those aren’t three separate lessons. They’re one: review is only as trustworthy as whatever catches review’s own mistakes. Twice, nothing did, and it got lucky — or rather, it was right. Once, the hidden test suite did, and that’s the only reason this failure is a data point in a field note instead of a bug that shipped.

The rule I run now

In Stop Using Your Biggest Model for Everything I argued that model tier should be chosen by complexity and stakes, not by whatever feels safe. This bakeoff adds the other half: decomposition should be chosen by the shape of the work, not by faith in agents or faith in autonomy. Work that splits into parallel, independent strands — research across many sources, say — is where Anthropic’s own numbers show multi-agent paying off, at a real token premium. Work that is one interdependent thing, which is most coding, gets a single agent, and the tier question gets settled by the test suite, not by reputation.

Concretely: one autonomous agent, on the cheapest tier that passes the evaluation gate for that class of task. Explicit steps get added only when they do something the tests can see — a genuine second opinion, a recovery path, a verification the first pass can’t perform on itself — not because a diagram with three boxes looks more rigorous than a diagram with one. Frontier reasoning is paid for by exception, when the gate says the cheap tier failed, not by default. And the gate is the point — not because nothing ever goes wrong, but because when something does, the gate is what tells you. Across 72 runs, one thing went wrong: a review step overreached and broke a correct fix. The reason it’s a sentence in this piece and not a shipped defect is that every run ended in a test that could say no, and this one did. The same distinction I drew in Guardrails Are Not Governance holds here one level down: the score doesn’t own the outcome, but without a score there is nothing to own. It’s also the mechanism behind the loaded-cost argument in The ROI Number in Your AI Business Case Is Probably Wrong: the denominator stays honest only if something is measuring what each extra invocation actually bought.

What this does not show

It’s one run per cell, so there is no variance estimate — the one failure in the study could be a coin that comes up differently on a re-run, and I have no second flip to check it against. It’s one vendor’s model family and one harness’s overhead profile — a harness that front-loads less context would shrink the gap between the architectures. Seven of nine tasks are Python with pytest as ground truth; the eighth, forecasting, swaps the domain but keeps deterministic grading; the ninth, the incident task, is the one place in the study grading isn’t purely deterministic — a hidden test suite decides pass or fail, but the secondary diagnosis-quality score comes from an LLM judge, spot-checked against one known-good and one known-bad answer rather than calibrated against a real human-labeled set. Treat that one score as directional. Task 8’s six independent forecasting units are the closest thing in the study to open-ended, genuinely parallelizable work, and even that didn’t trigger decomposition; it still isn’t the kind of sprawling, many-source research task where Anthropic’s own numbers show multi-agent work paying off. And none of this tested the genuinely different “many cheap agents” design: Li and colleagues showed in More Agents Is All You Need that simply sampling several small models and voting can beat one larger model, with the gain growing as tasks get harder. That’s not a pipeline and not autonomy; it’s redundancy. On a task set where the cheap model almost never failed, there was nothing for a vote to fix. On the one task where it did, that’s the experiment I’d run next — would three cheap votes have caught the same regression the hidden test caught?

What I’d actually ask

Not “should we use agents or a big model.” The better questions:

• Is there a test that decides when this task is done — or is “done” a feeling the model reports?
• What does one invocation cost before it does any work, and how many invocations does our workflow make per task? Multiply. That number is probably the architecture decision.
• When the capable model was given room to decompose the work — even work built to invite it — did it? If it didn’t, what is the orchestration layer buying us?
• Every discretionary step in our pipeline — every review, every second pass — what does it check itself against? If the answer is “its own judgment,” that step can make things worse, not just fail to help.
• What would the cheapest tier have scored on the last ten tasks we ran on the most expensive one — and does anyone know?

The question was “many cheap agents or one smart one.” The data answered a different one: one cheap agent, and a test that can say no.

Part of a larger framework — The System Around the Algorithm