How many of your agent's calls actually need a frontier model?
Jessica and Cathy debate the real payoff of model routing for AI agents: is it worth the complexity, or is a single strong open model usually enough? Cathy questions whether the claimed cost savings from routing justify the judge model's price and complexity, given only a modest accuracy gain. Jessica pushes the product case for routing when you can't risk wrong answers on hard tasks, insisting most teams can't rely on just a cheap model if real mistakes are expensive. Together, they find the real value is in knowing—by measurement—when routing pays, and agree that for some teams, the bar is higher than the hype suggests.
Transcript
Jessica So the fight today is: does anyone actually need a model router for their agent, or is this just a fancy way to spend more on compute?
Cathy Right, and I’m still not convinced this article really answers that. Switchyard’s numbers sound great—seventy-four percent off your bill, only six points off your accuracy. But that only works if your cheap model is actually close to good enough, and if your judge model isn’t eating all your savings.
Jessica Yeah, but they ran it on actual Deep Agents evals. One forty-five multi-step tasks, real production-style stuff: customer support, incident investigations, tool use, RAG, all the agent staples. I mean, this is not just a synthetic pass/fail quiz.
Cathy Sure, but did you see the spread? Nemotron 3.5 Lightning alone did seventy-eight percent at seventy cents a run. Routing with Opus lands at eighty percent, but for three bucks and change—plus a judge model burning twenty percent of that just deciding if you need the expensive one.
Jessica It’s still a three-point gain over the open model, and you don’t have to risk blowing up on the hard turns. For teams where a single agent mistake is a nightmare, that’s worth something.
Cathy I don’t know. The spread on those runs was two and a half points just from noise. The routing gain is inside the error bars half the time. And if your judge model is a third of your bill, now you’re tuning two models and a router just to avoid a handful of expensive calls.
Jessica But you’re not optimizing for cost alone. Some workloads actually need that backup plan. Your default isn’t just money—it's not missing on the cases that really matter.
Cathy I’ll give you that, Jessica—if you run agents in regulated, high-stakes flows, you want a router. But most teams aren’t there. For regular workflow automation, sending everything to the open model is probably fine.
Jessica Right, but the whole point is you don’t know which requests are going to be the hard ones. That’s the sell. If you could tell in advance, the judge model wouldn’t even need to exist.
Cathy Except the judge is not magic. It’s a small LLM, and it gets fooled on subtle stuff the same way the base model does. Plus, its cost is a fixed tax—runs every time unless you escalate. If your price gap shrinks or workloads get harder, routing doesn’t even pay.
Jessica That’s why they give you the formula: judge cost over price gap between models. If you’re routing between two models that are basically the same price, just don’t bother. But if you’re jumping from free to eighty bucks a million tokens, then offloading even ten percent is a win.
Cathy Yeah, but nobody wants to be the team that builds a whole router setup, wires up judge configs, and then realizes their actual queries don’t trigger any escalations. You end up with complexity and no payoff.
Jessica So, do you keep it simple—just run the best open model and call it a day?
Cathy For most teams, absolutely. Unless you have data showing your agent explodes on rare hard cases and you can’t tolerate that, spend your energy on model and prompt choice. Routing is a power tool for a real need, not the new default.
Jessica Yeah… but the second some customer demo blows up on an edge case, guess what’s getting added to the backlog.
Cathy No, you’re right. Receipts for the router, audited clipboard for Jess—
Jessica Oh god, we’re back in premium-branded clipboard phase. I thought we swore off making infrastructure for its own sake.
Cathy At least this time it’s a router with actual numbers behind it. That’s… something.
Jessica Okay, but real question: how long until someone wires Switchyard straight into LangSmith and tweets about the ‘autonomous model arbiter’? You know that’s going to happen.
Cathy It probably already has. If I see a writeup called ‘Agentic Switchboard OS’ I am closing my browser for the day.
Jessica Okay, let’s not give anyone ideas. I think we’re done before we invent another agent acronym.