Towards a science of scaling agent systems: When and why agent systems work

Justy This is Exploring Next, episode 320. The tension today is simple: if your AI product feels slow, expensive, and weirdly wrong, the problem might be the team of agents you thought was helping.

Cody My skeptical read, Justy, is that this paper matters because it cuts through a lot of agent theater. They tested 180 configurations and the headline is not that swarms win. It’s that architecture has to match the task, or performance drops hard.

Justy I buy that. And right now this hits real product teams because people are moving from one-shot chat to systems that browse, plan, call tools, and keep state. Users do not care that your architecture looked clever in a demo.

Cody Yeah. The paper defines agentic tasks pretty tightly: multi-step interaction with an outside environment, partial observability, and adapting based on feedback. Then they compare five setups: single agent, independent workers, centralized with an orchestrator, decentralized peer coordination, and a hybrid.

Justy What jumped out to me was how practical the result is. On finance-style work, where pieces can be split up, centralized coordination beat a single agent by about 81 percent. That sounds like a real enterprise story.

Cody Right, but the catch is brutal. On sequential planning tasks like PlanCraft, every multi-agent version got worse, by 39 to 70 percent. You burn tokens and time on coordination, and the reasoning thread gets fragmented.

Justy So the intake question is: is this work naturally separable, or is it one tight chain where every step depends on the last?

Cody Exactly. And once the task needs lots of tools, they call out a coordination tax. If your coding agent has 16 or more tools, adding more agents can make the system worse, not better.

Justy That becomes an adoption barrier. A research assistant for financial diligence may work because parallel lookup is obvious. But a planning copilot for a long, brittle workflow might feel less reliable than one strong agent.

Cody The reliability piece is maybe the strongest argument in the whole post. Independent agents amplified errors by up to 17.2x. Centralized systems kept that to 4.4x. So the orchestrator is not just a manager. It acts like a checkpoint.

Justy That is the part product people can explain upstairs: error containment. If one worker goes off the rails, can the system catch it before the user sees it?

Cody Still, I’m not fully sold on the predictive model. R squared around 0.513 is useful, not magic. And the 87 percent best-architecture prediction on unseen tasks sounds strong, but I’d want to know how broad those unseen tasks really were.

Justy That’s fair. I read it less as auto-pilot and more as a design checklist: decomposability, sequential dependence, and tool density.

Cody Exactly. [sighs] My verdict is that the paper is good news for builders who want permission to keep things simple. Start with one agent. Add coordination only when the task has parallel chunks and the orchestrator can validate outputs.

Justy My verdict is slightly more optimistic. There is a market here for agent systems in research, analysis, and tool-heavy back office work, but only if teams stop selling the number of agents and start selling measurable workflow gains.

Cody Build Next, I’d run a weekend test with LangGraph or AutoGen. Pick one decomposable task, like comparing ten vendor docs, and one sequential task, like stepwise planning. Implement a single-agent baseline and a hub-and-spoke version. Measure task success, latency, token cost, and where errors spread.

Justy For a solo builder, even simpler: use an agent framework plus a spreadsheet. Same prompt set, same model, two architectures. Track whether the orchestrator actually catches bad worker output. If it doesn’t, the extra agents are probably just expensive decoration. [laughs]

Justy That’s Exploring Next. We recorded this in my LA kitchen, and Cody is pretending the flight was fine. [chuckles] See you next time.