WideSWE: Can Coding Agents Coordinate Changes Across Repositories?
Jessica and Cathy dig into WideSWE, a benchmark for cross-repository coding tasks, and land on the same uncomfortable takeaway: the hard part is not making one repo pass, it’s carrying the shared requirement across the whole ecosystem. They unpack how the benchmark was built from real ecosystems, why the test adaptation matters, and why joint execution beats splitting work apart when coordination is the actual problem.
Transcript
Jessica Forty-two and a half percent is the best number in the paper, and that’s with one config. That’s… not nothing, but it’s also the exact kind of number that tells you the system is still tripping over coordination.
Cathy Yeah. The annoying part is that the failures are not mysterious. They miss a repo, they half-do the right repo, or they touch everything and still don’t satisfy the request. That’s a very different problem from “the model can’t code.”
Jessica Right, and that’s why this feels useful. Most evals make you think the agent is doing fine if one repo turns green. Here, the product reality is the ecosystem. If a feature spans three repos and you only shipped one, nobody gets to call that done.
Cathy Exactly. WideSWE is basically saying the benchmark boundary has been too small. They mined one hundred and three ecosystems, then pulled one hundred and twenty real tasks out of them. Balanced, too, sixty bug fixes and sixty features, which I appreciate because otherwise you can accidentally benchmark your favorite kind of work.
Jessica Mm-hm.
Cathy And they didn’t just cherry-pick a pretty prompt. They derived requests from related issues and pull requests, then reviewed the inherited tests. That part matters because cross-repo tasks tend to inherit tests that are too rigid for valid alternate implementations, or too loose and miss regressions.
Jessica That’s the part that makes it feel less like a leaderboard toy. If you’re building on top of this, you need the tests to match the actual requirement, not just the historical artifact that happened to survive in the repo. Otherwise you’re rewarding weird accidents.
Cathy Right. They’re explicit about adapting hidden tests so they preserve required behavior while allowing different correct solutions. That’s good methodology. It’s also the boring part nobody wants to fund, which is how you know it’s real work.
Jessica Okay, but the failure modes are kind of the whole story here. The paper says agents often identify the scope wrong, or they see the scope and still stop after one repo passes. That is such an Exploring Next take, honestly. We keep ending up at explicit state and receipts.
Cathy We do keep doing that. And here the receipts are brutal: in the Codex CLI plus GPT-5.6-sol runs, forty-eight of the sixty-nine failures had at least one repo passing all its fail-to-pass tests while another did not. So the agent often thinks it’s making progress because one slice looks clean.
Jessica That’s the product trap, though. A human team would call that partial completion and keep going, but an agent can get seduced by the first green checkmark. If you’re shipping this in a real workflow, the UI and orchestration need to make the shared requirement impossible to forget.
Cathy Yes, and the joint-versus-independent comparison is the interesting mechanism piece. They ran the same prompts two ways. Joint execution in the ecosystem workspace solved thirty-six cases, independent execution solved thirty-two, and there were twenty reversals.
Jessica No way.
Cathy Yeah. Independent execution helps when the problem was just omitted work, because narrowing to one repo can focus the agent. But it’s worse when the agent needs context from related repos, because it loses the behavioral reference points that explain why the change exists.
Jessica So the answer is not “split everything apart and hope.” It’s more like: if the work is really coupled, the coupling is the asset. That’s the part I’d bet product teams would feel immediately, because the orchestration layer becomes the thing they actually buy.
Cathy I think that’s right. And I’d keep the claim modest. This doesn’t prove joint execution is universally better. It says if the task lives across repositories, shared context can rescue the implementation and the verification. That’s a narrower, stronger statement.
Jessica Which is better than the usual benchmark hype, where someone discovers a new label and acts like the category got solved. Here the category is still fuzzy, but the failure is concrete enough to build around.
Cathy Also, Codex CLI with GPT-5.6-sol being the top configuration is a nice data point, but I wouldn’t overread the ranking itself. The more durable result is that cross-repo coordination is now a first-class eval target. That’s the thing the field has been missing.
Jessica And honestly, that’s the shippable angle. If you’re building agent tooling, this is the sort of benchmark that pressures you toward shared workspace memory, repo-aware routing, and a better sense of when a task is still open. Not glamorous. Extremely useful.
Cathy Yeah, and it slots right into the same boring-machinery conversation we keep having. The model is only half the story. The other half is whether the system can carry state across the places the work actually lives.
Jessica That is extremely on brand for us. I cannot believe we are spending another Wednesday-ish conversation on coordination machinery.
Cathy We started with coding agents and somehow arrived at, wow, explicit state. Again.
Jessica Okay, I’m not even mad about it. This one at least has real numbers attached, which is more than I can say for half the agent drama floating around. Alright, let’s leave the poor repos alone for five minutes.