Learning Meta Skills for Agent Harness Design in Test Time AI4AI

Justy Eight-point-nine-five over no-skill construction is the number that jumps out at me. That’s not a tiny prompt tweak, that’s the Builder learning how to set the table better than just throwing more tools at the Target.

Cody Yeah, and the weirdly important part is that the weights stay fixed. They’re not sneaking in a stronger model and calling it insight. The Builder is learning policies for support design, which is a very different bottleneck than raw task execution.

Justy Which is why this feels like the same old harness story wearing a new hat. We keep finding out that the thing around the model is the thing people actually ship, and somehow we still act surprised when the environment matters.

Cody Right. The paper’s cleanest move is the construction–execution–reflection loop. The Builder starts with an empty meta-skill bank, makes a harness for development tasks, watches the Target’s traces and scores, then writes down principles like when support is needed and what resource to provide.

Justy And that’s the part I can imagine product teams actually using. Not in some mystical self-improving-agent way, just in the boring, valuable sense where the system learns, ‘this task needs a validator,’ or ‘this one needs a shared scratchpad,’ and stops making users rebuild the same scaffolding every time.

Cody Exactly. The meta-skill is not ‘solve physics’ or ‘write code.’ It’s more like, ‘when the task has hidden state, provide a persistent log,’ or ‘when verification is cheap, surface a checker and keep the model from guessing.’ That’s a support policy, not a task policy.

Justy We really did get back to explicit state again. That is such an Exploring Next move, honestly.

Cody We started with a paper about AI4AI and somehow ended up at, wow, receipts. Again.

Justy The result that matters to me is that the same bank did better when it guided the Builder than when it was just handed straight to the Target. That tells you the value is partly in translation, not just content. Somebody has to decide what support is worth packaging.

Cody That’s the mechanism I buy most. A Target can know a useful trick and still fail to use it well, but a Builder that has learned from prior executions can choose the right structure and the right timing. The paper says the full-bank meta-skills beat direct delivery by twelve point zero two points, which is a nice sign that the wrapper is doing real work.

Justy Small caveat, because we’re not doing a courtroom reenactment here. On a normal paper like this, we trust the source and our read of it rather than re-deriving every table, so a tiny figure could be off. But the shape of the result is clear enough to talk about.

Cody Fair. And the shape is what matters: Harness-Bench and NewtonBench both point the same way, and they report sixty-five point three one percent macro-average for the full-bank Builder. That’s a pretty solid signal that this is not just one benchmark flattering one trick.

Justy The production question for me is whether this turns into something teams can actually operationalize. If I’m building an agent product, I do not want ‘more model’ every time the user hits a brittle step. I want the system to notice, from prior runs, that this workflow needs a harness with better validation or better shared context.

Cody That’s the shippable version, yeah. The nice thing is they’re not asking for a new magical planner. They’re asking for reusable support principles that can be implemented as harness decisions, which is a much more believable place to cash out the gains.

Justy And because the Builder and Target can be the same model, the story gets more interesting than just advisor-and-student. If the same system can learn how to build better support for itself, that’s a real control-layer improvement, not just a one-off demo.

Cody That’s also where I’d keep my skepticism honest. The idea is good, but transfer is the whole game. If those meta-skills stay brittle across tasks or need constant babysitting, the gain shrinks fast. The paper does claim transfer across Builders and Targets, which is the right direction, but I’d want to see that hold under messier task mixtures.

Justy Sure, but this is still the first useful version of the idea I’ve seen in a while. Not ‘the model got smarter,’ more ‘the system got better at knowing when to help itself.’ That’s a product story people can build around.

Cody And that’s probably why I’m less annoyed than I expected to be. It’s a harness paper, but it’s a harness paper with a mechanism: feedback becomes principles, principles become executable support, support changes performance. That’s a cleaner chain than most of the agent buzz we get handed.

Justy You know what this reminds me of? Every time we act like the secret sauce is the skill bank itself, and then it turns out the routing, timing, and packaging are the actual moat. Very on brand for us to keep discovering that the wrapper ships.

Cody Yeah, and I’d put the emphasis exactly there. The interesting thing is not that agents can collect advice. It’s that a Builder can learn which advice belongs in the environment versus which advice belongs in the model’s head.

Justy Which is a very normal sentence that somehow sounds like a future product category. I hate that I like it.

Cody Same. The paper is basically saying, if you want better agents, teach the thing around the agent how to stop wasting the agent’s time.

Justy That’s a good place to leave it, Cody. I cannot believe we’re spending another Wednesday-ish conversation on coordination machinery, but at least this one actually seems to earn the calories.