Overview: Dynamic Code Execution

Masonry Okay so this one keeps coming up — we name-drop it, we reference it, listeners keep asking us to actually EXPLAIN it — so today we're finally doing it. Dynamic code execution, from the ground up.

Eyre Yeah, and I think the reason it keeps coming up is that it's underneath a surprising amount of what agent systems actually do. Like it's not a flashy feature — it's closer to load-bearing plumbing. So it's worth understanding properly.

Masonry Alright, Eyre — before any jargon, give me the honest intuition. What IS this thing?

Eyre Okay so here's the mental model I want you to hold onto for the whole episode. Imagine you hire a very well-read research assistant. They've read basically every textbook ever written. You ask them: what's the hundred and third Fibonacci number?

Masonry Right.

Eyre They could try to compute it in their head. And they might even sound confident about the answer. But they'd almost certainly get it wrong — because they're pattern-matching from memory, not actually doing arithmetic. Now imagine instead they pick up a calculator, punch in the formula, read the result, and hand that to you. Same assistant, totally different outcome.

Masonry Okay, that's the whole thing right there, isn't it.

Eyre That's the whole thing. A language model is the research assistant. Dynamic code execution is giving them the calculator — except the calculator can be Python, or S Q L, or JavaScript, or whatever fits the problem. The model writes the code, something actually RUNS it, and the result comes back. The model sees what happened and can go again if it needs to.

Masonry And the key word there is 'actually runs it.' Not the model predicting what the output would be — real execution.

Eyre Exactly. That's the whole distinction. Predicting code output is still just pattern-matching. Running the code gives you ground truth. And that difference is enormous in practice.

Masonry Okay so let's build the loop up properly. Walk me through what actually happens, step by step.

Eyre Sure. Step one: the model gets a problem — math question, data analysis task, whatever. Instead of answering directly, it writes a code snippet. Could be ten lines of Python, could be a single S Q L query. Step two: a runtime environment — completely outside the model — takes the code, runs it, and captures whatever comes out. An answer, a table, an error message, whatever. Step three is the part that makes it a loop: that output gets fed back into the model as context.

Masonry And that 'repeat until done' part — that's where agentic loops come in. Episode six twenty-one if you want the full treatment. Short version: an agentic loop is just the model running repeatedly, taking actions and observing results, until it hits a goal. Dynamic code execution is one of the main things that loop is actually doing. And it leans on tool use — episode six sixteen — where the model calls some external capability instead of predicting the answer itself.

Eyre Yeah, without structured output the execution layer has to do a bunch of fragile string-parsing to figure out where the code even starts.

Masonry So back to our research assistant — the runtime is the calculator, the structured output is writing the formula in a way the calculator can actually accept, and the agentic loop is checking the result and going again if needed.

Eyre That's exactly right. Hold that picture.

Masonry Okay, I want to make this concrete. Where does this actually show up in real systems?

Eyre Oh it's absolutely shipping. The most visible example is probably a code interpreter in a chat product — you upload a spreadsheet, the model writes Python to analyze it, and you get back an actual computed answer with a chart. That's the loop running live.

Masonry And SQL generation and data analysis are probably the highest-volume real-world cases right now.

Eyre Yeah, by a lot. Any time you've got structured data and a question about it, this is the pattern.

Masonry What are the real failure modes here? Because I don't want to oversell this.

Eyre It does NOT make correctness guaranteed. The model can still write code that is syntactically valid but logically wrong — it computes something, returns a number, and that number is just not the answer to the question that was asked. And there are other failure modes: infinite loops that eat your timeout budget, code that technically runs but produces output the model then misreads, off-by-one errors that propagate through a multi-step analysis.

Masonry Let me put that a better way — it changes the failure mode. Without execution, the model confidently hallucinates an answer and you have no signal that it's wrong. With execution, you at least get real output you can inspect, and errors surface as errors instead of as confident-sounding wrong answers.

Eyre That's a much better way to say it. The failure mode shifts from silent wrongness to visible wrongness, which is strictly better even if it's not perfect.

Masonry What about cost?

Eyre It's genuinely slower and more expensive than pure prediction. You're spinning up a runtime, executing code, waiting for output, potentially doing that several times in a loop. Each iteration adds latency. If the model loops five times, you've paid five execution cycles on top of five model inference calls. The tradeoff is pretty clear — accuracy versus speed and cost. You reach for this when getting the right answer matters more than getting an answer fast.

Masonry And the harness engineering framing that's been gaining traction in twenty twenty-six — the idea that the model plus the execution harness plus the eval loop is the real product — that's just this pattern named more explicitly.

Eyre Yeah. And it's still exactly how things get built. It hasn't been replaced — if anything it's more central than it was two years ago. The vocabulary got absorbed into the agent stack conversation, but the mechanism is identical. By mid twenty twenty-six, if you're building a serious coding agent or a data analysis product, you're almost certainly running some version of this loop.

Masonry Which connects back to our whole control-layer thesis — the boring infrastructure around the model is the product. The execution environment, the sandbox, the timeout policy, the output parsing. None of that is glamorous, all of it is load-bearing.

Eyre Yeah, hundred brilliant agents, defeated by one misconfigured sandbox. That's the unglamorous version of this.

Masonry Okay, that's a good callback and I'm going to use it. So — if you walked away from this episode and had to hold onto one thing, what is it?

Eyre The one thing is: the model writes instructions, the runtime actually executes them, and the result feeds back in. That loop — write, run, observe, revise — is what turns a pattern-matcher into something that can do reliable computation. Everything else is implementation detail on top of that.

Masonry And if the loop is the thing, then the sandbox is what makes the loop safe to run at all. You can't have one without the other.

Eyre Exactly. The calculator is only useful if it can't also accidentally delete the building.

Masonry Eyre, that is maybe the most reasonable thing you've said on this show in nine months of doing it, and I mean that sincerely.