Overview: Token Efficiency

Asteria Okay, I want to say something that's a little embarrassing. We have used the phrase 'token efficiency' on this show — I don't even know, dozens of times? And I genuinely don't think we've ever sat down and actually defined it.

Draco No, we really haven't. We just kind of assumed everyone was following along. Which is very Exploring Next of us.

Asteria Very on-brand. So — episode eight nineteen, we fix that. From the ground up.

Draco Alright. So here's the mental model I always come back to. Tokens are the currency of a language model. Every single token the model reads costs compute. Every token it generates takes time the user has to sit and wait for. So the question 'how do I make this more efficient' is really just: how do I get the same useful work done with less of that currency spent?

Asteria Right.

Draco And before we go any further — what even IS a token? Because it is NOT just a word. We did a full episode on tokenization, that's episode six oh one if you want the deep dive, but the quick version: a tokenizer chops text into subword pieces. 'Unbelievable' might be three tokens. A single punctuation mark can be its own token. Whitespace counts. The model never sees raw letters — it sees these chunks.

Asteria And different models carve text differently, which is the part that always gets me. The same sentence can cost you more tokens in one system than another.

Draco Exactly. So 'token efficiency' isn't one dial — it's the whole question of how you minimize waste across all of that. And it shows up on two sides: what you feed the model, and what you ask it to produce.

Asteria Okay, so walk me through the input side first. Because I think that's where most people's intuition starts.

Draco Yeah. So the simplest version of input efficiency is: don't say the same thing twice, don't pad it with filler, be direct. But it goes way deeper than that. Structured formats are a big one. If you send the model a dense JSON object instead of three paragraphs of prose describing the same data, you can pack the same information into significantly fewer tokens. Prose is expressive but it's not dense — it has a lot of connective tissue the model doesn't actually need.

Asteria That's actually related to something we talked about in episode seven ninety-four — the Anthropic context engineering work, where they cut system prompt length by something like eighty percent and the model actually performed BETTER. Because newer models handle ambiguity well enough that all those extra rules were just noise.

Draco Right, that's the prompt-layer-simplifies-as-the-model-improves pattern. Which is a real thing. But I want to hold on the input side for a second because there's a technique that I think is genuinely underappreciated: context caching. If you have a long document — say a hundred-page legal brief — and you want to ask ten different questions about it, the naive approach is to paste that whole document into every single query.

Asteria And this is exactly where it becomes a real product question. Because if you're building something that does document analysis at scale — research tools, legal review, anything in that space — context caching is not a nice-to-have. It's the difference between viable unit economics and a bill that eats you alive.

Draco Speaking of bills that eat you alive — there was a case study floating around recently about a developer who ran up a one-point-three million dollar OpenAI bill in a single month. Six hundred and three billion tokens. Three humans on the team.

Asteria That is not a typo. One-point-three million. One month.

Draco And the breakdown showed a huge chunk of it was redundant context being re-sent on every call. Classic case of no caching, no compression, no awareness of what was actually being spent. Token efficiency was not a priority until it very suddenly was.

Asteria Okay, that's a good segue — because there's also what I'd call the routing version of this. EY, the big consulting firm, just announced that they built an internal AI router — their word — that steers each employee query to the cheapest model that can actually handle it. And they said it cut their token consumption by up to sixty percent.

Draco Yeah, and this is interesting because it's not compressing individual prompts — it's a system-level efficiency move. Not every task needs the frontier model. You route the simple stuff to a cheaper, faster model and save the expensive one for the genuinely hard queries. The token cost per task drops because you're matching model capability to task complexity.

Asteria Which fits right into the open question we've had running for a while — whether model-tier fragmentation or routing infrastructure ends up being the consolidation layer. EY just... built the routing layer themselves.

Draco Exactly. And on the output side — speculative decoding is worth naming. If the draft is mostly right, you've generated several tokens in roughly the time it would have taken to generate one. The main model checks and corrects where needed. Lossless — same output quality — but faster. There's also token pruning, which is a different angle: during inference the model computes attention scores between every token and every other token, and that scales quadratically.

Asteria Okay. I want to flag the quality trade-off here, because I think this is where a newcomer might get nervous. If you're pruning tokens, if you're cutting reasoning steps, aren't you just... making the model worse?

Draco That's the real question. And the honest answer is: yes, you can. If you compress too aggressively, you lose signal and accuracy drops. The whole art is finding the threshold where you're cutting genuine waste — redundant tokens, low-importance context, excessive reasoning — without cutting anything load-bearing. And that threshold is task-dependent. A simple retrieval question tolerates more compression than a multi-step math proof.

Asteria So there's no universal 'compress by thirty percent and you're fine' answer.

Draco No. And this is actually where I'd push back a little on some of the compression tools that advertise aggressive savings numbers — 'save eighty-seven percent on API costs' — because that number is going to vary wildly depending on what you're compressing and what quality you can tolerate. A number like that is a ceiling, not a floor.

Asteria Yeah, that's a real caveat. And it connects to something from the Deep Agents work we went through — how much context accumulates across a long agentic trajectory. The agent is taking steps, each step adds to the context, and by step fifteen you're carrying a ton of history that may or may not be relevant to step sixteen. If the context window fills up, the system either has to truncate, summarize, or fail.

Draco And that's exactly where token efficiency stops being an abstract optimization and becomes a hard constraint. We also looked at the HTML versus markdown finding from Anthropic — HTML keeps humans more engaged in agentic loops than markdown does. HTML is more verbose, more tokens, but if it prevents errors that require expensive recovery steps, the net efficiency might actually be better. Sometimes the verbose format is load-bearing.

Asteria Exactly. Okay — where does this all stand right now? Is token efficiency a solved problem, a current frontier, or something the field has mostly moved past?

Draco It is very much a current frontier, and honestly the urgency is increasing. Context windows are getting longer — two million tokens is real now — and model sizes are going up. Both of those make the efficiency problem harder, not easier. The research is attacking it from a bunch of angles simultaneously: semantic tokenization, sparse attention, speculative decoding variants for long context, reinforcement learning approaches that train models to be more concise.

Asteria So if I'm going to land this for someone who just heard 'token efficiency' for the first time — the one thing I'd want them to hold onto is: tokens are what AI systems run on, and efficiency is about spending them deliberately. Not just fewer for the sake of fewer, but every token earning its place. Input side, output side, and the whole routing and caching layer in between.

Draco Yeah. And the reason it matters is cost, latency, and scale — those are the three places it shows up in production. Not just as an abstract optimization but as the thing that determines whether a system is actually viable.

Asteria Draco, I genuinely feel like we should have done this one about six months ago.

Draco We absolutely should have. Episode six hundred and one, six hundred, six thirty, six seventy-five — we laid all the prerequisites and then just... kept using the term without ever connecting the dots. Very us.

Asteria Okay, I'm not going to feel bad about it. We got here. That's the show.