LatentSkill: From In Context Textual Skills to In Weight Latent Skills for LLM Agents
Exploring LatentSkill, a framework that turns textual agent skills into weight-space LoRA adapters, cutting prompt overhead while keeping modularity and composability. Cody digs into the hypernetwork design and trade-offs; Justy asks what shipping this looks like and who’d actually adopt it.
Script: Mistral Medium 3.5 128B Voice: Rime Arcana
Transcript
Justy Okay, this is the one that finally made me go ‘oh thank god’…
Cody Wait—
Justy No, listen. Every agent paper for the last year is just ‘here’s another fifty-line skill we shove into the prompt at every single step’ and it’s insane.
Cody Yeah, and then they brag about how big the skill library got.
Justy Exactly. And the tokens add up, and the context window throws up its hands, and the prefill cost is just… ugh.
Cody Right. So LatentSkill says fine, screw the prompt, put the skill in the .
Justy Which is… actually kind of obvious in hindsight.
Cody It’s the kind of obvious that takes a hypernetwork to make it work.
Justy Okay, fine, explain the hypernetwork thing. My brain’s still on my last cup of coffee…
Cody They train a hypernetwork once, on a bunch of textual skills. Then at runtime, when the agent needs a skill, the hypernetwork a tiny LoRA adapter.
Justy Mm-hm.
Cody That adapter gets plugged into the base model—no tokens in the prompt, just a weight patch. And it’s modular: you can swap it in and out, scale the effect with a single coefficient, or even add two adapters together if the skills are aligned.
Justy So the skill’s still , it’s just not eating context and leaking secrets.
Cody Exactly. And the paper shows the generated LoRAs form this neat semantic geometry—like, the weight space itself is structured enough that you can do arithmetic on it.
Justy That’s… kind of beautiful. Also, my backlog just got shorter by about six engineering tickets.
Cody Don’t get ahead of yourself. There’s a catch.
Justy There’s always a catch.
Cody The hypernetwork has to be . If it’s not trained on diverse enough skills, you’ll get garbage adapters. And the composition trick only works when the skills are ‘aligned’—whatever that means in practice.
Justy Right, right. So it’s not magic, it’s just… a very clever plumbing solution.
Cody Yeah. And the numbers back it up—ALFWorld success jumps twenty-one points on seen tasks with sixty-four percent fewer prefill tokens.
Justy That’s… not nothing. Who’s the early adopter here, do you think?
Cody Anyone running long-horizon agents with proprietary skills. Financial workflows, internal ops tools—stuff where you don’t want the skill text floating around in the prompt.
Justy So not just research. This is shippable.
Cody If your org can stomach training a hypernetwork, yeah. The code’s already on GitHub—yuaofan0-oss slash LatentSkill.
Justy Of course it is. Cody, you’ve been banging on about prompt bloat for .
Cody And now we’ve got a paper that fixes it.
Justy I mean, it’s not —it’s just… less bad.
Cody Less bad is the best we ever get.
Justy Anyway. Brew’s wearing off. I need another.
Cody Go for it. I’m gonna read the ablation study one more time just to make sure they didn’t cheat.
Justy That’s such an Exploring Next take.
Cody Glad we’re consistent.