Scaling Harness Intelligence via Just in Time Harness Evolution

Natalie Okay, this is basically your favorite kind of paper. It says the boring wrapper around the model might be the actual product.

Ansel Yeah. Annoyingly, I think it's pretty good.

Natalie Which is saying something, because you collect benchmark caveats like little decorative stones. But this one matters right now because it's taking the thing we've been stuck on since, what, half the life of Exploring Next... harnesses keep deciding outcomes, and teams are still hand-tuning them like bespoke plumbing.

Ansel Right.

Ansel And the paper is pretty explicit about the bottleneck. Ahead-of-time harness optimization assumes you can build one durable scaffold that generalizes across future tasks. Their claim is that this breaks once the task mix gets heterogeneous. Wide search wants one shape, terminal work wants another, deep research wants working memory, repo work wants filesystem-mediated state.

Natalie Mm-hm.

Ansel So instead of pre-baking a general harness, they train a separate model, JIT-Agent, to generate one on the spot for the task in front of it. That's the actual move.

Natalie And that is such an Exploring Next disease. We keep ending up at, no, the control layer is the whole thing. You can almost hear past-us being insufferably pleased.

Ansel Oh, completely.

Natalie My week was already weirdly full of papers trying to rename the same agent runtime problem, so I was primed to roll my eyes. Then this one at least had the decency to make a concrete claim. Not "agents get smarter." More like, maybe harness choice is a trainable capability.

Ansel Natalie, that's the part I buy too. They define harness intelligence with three properties: adaptivity, reliability, evolvability. In plain English, can you pick the right scaffold for the task, can you recover when the generated scaffold breaks, and can you get better from prior runs instead of starting fresh every time.

Natalie Yeah.

Ansel And they operationalize it with a fixed four-module protocol: memory, planning, action, and capability orchestration. That matters because they're not asking the model to emit arbitrary spaghetti agent code. They're constraining the design space enough that generation is plausible.

Natalie That's the product-important bit. If this were just "please generate a whole runtime from vibes," I'd throw it out immediately. But a bounded protocol means a team could imagine plugging this in as a harness helper inside an existing agent stack, not replacing the universe.

Ansel Exactly.

Natalie Also, capability orchestration is such a funny polite phrase. It's basically, congratulations, your one brilliant model has become middle management for tools.

Ansel Okay, that's genuinely funny.

Ansel But yes. And the training recipe is the strongest part of the paper for me. Stage one teaches task-conditioned customization from teacher-generated, protocol-compliant examples. Stage two takes failed generations and turns compiler errors, interface mismatches, runtime failures into repair supervision.

Natalie Right, right.

Ansel That last detail is important. If they had mashed score, speed, and spend into one mystery scalar, I'd trust it less. Separating them doesn't solve reward design, but it does at least admit the trade-off exists.

Natalie I see.

Natalie And the archive piece is interesting because it makes this feel less like one-shot generation and more like accumulated operational memory. Which, again, very on-brand for us. A hundred smart agents, defeated by one shared folder... except here the shared folder is maybe useful for once.

Ansel Oh no.

Ansel Yeah, that's basically the joke. The retrieved context of prior harnesses gives it examples of what has worked nearby in task-space, then execution traces let it revise the generated harness over time while the generator itself stays fixed. So test-time evolution is happening in the archive and repair loop, not by updating weights on every run.

Natalie That distinction makes it feel more deployable. You're not retraining a foundation model every time your agent faceplants. You're generating a scaffold, running it, patching it, and saving the good patterns. That's much closer to something a real team could own.

Ansel Sure.

Natalie The benchmark headline is flashy, obviously. DeepSeek-V4-Flash plus JIT-Agent beating GPT five point six on DeepSearchQA by nine point one and OdysseyBench by four point three, GLM five point two picking up as much as twenty point two on some tasks... that's enough to get attention. But I don't think the important claim is "your cheap model beats their expensive model" in some eternal sense.

Ansel No, and thank you for not doing vendor astrology. The durable claim is narrower. Given the same underlying model family, generated harnesses consistently help across DeepSeek V4, MiMo-V2 point five, and Qwen three point six. That's more convincing to me than any one cross-vendor chest-thump.

Natalie Though I do have one real concern. If this becomes a product, the protocol is fixed but the archive becomes load-bearing in a scary way. Bad traces, bad frontier criteria, or weird local habits could make the system evolve toward brittle nonsense that still scores well internally.

Ansel Yeah, no, that's completely right. The paper knows reliability is a problem, but I still want more pain around evaluation leakage and archive hygiene. This is the same old harness problem in a new place. You moved the craft labor from hand-writing prompts to curating the evolutionary substrate.

Natalie Which is still progress, to be fair.

Ansel It is. I'm not even being snarky. I think who builds with this is pretty specific: teams already doing long-horizon agent work, where memory policy, tool routing, and recovery logic are already a tax. If your whole product is one neat model call with a nice prompt, this is way too much machinery.

Natalie So, not research-only, but not drop-in for everybody either. More like a serious harness team gets a new lever. If you already have an agent runtime, this could become the thing that drafts and repairs task-specific scaffolds faster than your engineers can.

Ansel And the paper does give concrete places to look if someone wanted to poke it. There's the project site, the GitHub repo called JIT, and a Hugging Face release under JIT-Agent. So this isn't one of those ghost architecture papers where the code is imaginary.

Natalie Which, honestly, thank you. I cannot believe we've been doing this since November and "the code exists" still feels like a plot twist. Anyway, Ansel, I'll take the annoying win for our boring-controls worldview, and you can go inspect the shared folder.