Overview: Feedback loops

Jessica Okay, I want to admit something. We've said 'feedback loop' on this show probably… a hundred times? And I don't think we have ever actually stopped and explained what one is.

Cathy No, we really haven't. I was thinking about that recently. We use it like everyone already knows, and some people definitely do, but if you're newer to this stuff it's one of those terms that just floats past you. So — yeah. Let's actually do it.

Jessica Finally. Okay, Cathy, I want you to start with the intuition, not the mechanism. Give me the everyday version first.

Cathy Right. So here's a setup that has nothing to do with AI at all. A teacher is told, at the start of the year, that a handful of students are 'gifted.' Maybe that information is accurate. Maybe it's completely arbitrary. Doesn't matter — the teacher believes it.

Jessica Okay.

Cathy So the teacher calls on those students more. Gives them harder problems. Praises their answers in front of the class. Those kids get more practice, more confidence. By the end of the year, they ARE performing better. The teacher's initial belief — which might have been noise — became true. The system reinforced itself.

Jessica Oh, that's a good one. That's just a self-fulfilling prophecy dressed up in school clothes.

Cathy Exactly. And the key thing is: nobody lied. Nobody cheated. The teacher didn't manipulate the grades. The students really did improve. But the CAUSE was the initial belief, not anything about the students themselves. Now swap the teacher for an algorithm and the students for loan applicants, or job candidates, and suddenly that self-fulfilling prophecy has a lot more reach.

Jessica Right, because an algorithm doesn't have twenty-five kids — it's making decisions at scale. Millions of them.

Cathy And it's faster. A teacher takes a year to close that loop. An ad-targeting system can close it in hours.

Jessica Okay so let's get into the actual mechanism. How does this play out inside an AI system specifically?

Cathy Three steps, and they always happen in this order. Step one: the model makes a prediction or a decision in a live environment. It recommends a video, it flags a post as spam, it declines a loan application. Step two: that decision changes what happens in the world. The video gets more views because it was recommended. The flagged post gets less visibility. The loan doesn't get made. Those are real-world consequences — not just outputs in a log file.

Jessica So the model becomes more confident in a mistake because the world got reshaped to match the mistake.

Cathy Yeah. And this is worth saying clearly: this is not a bug in the algorithm. The training code is fine. The math is fine. It's a structural problem — it emerges from how the system sits in the world, not from what's inside the code. The loop operates across time and involves the real world. You'd have to trace back where the data CAME from to see it, and most teams aren't doing that audit.

Jessica Okay. So where do feedback loops actually hide? Because I think most people picture this as some exotic AI safety scenario, and it's not. It's in completely ordinary systems.

Cathy Yeah, recommendation systems are probably the clearest example. The algorithm shows you videos based on what it predicts you'll engage with. You watch some of them. The algorithm learns those are the popular ones — because it's only seeing engagement data on what it CHOSE to show you. It has no idea what you would have done with the other ten thousand videos it never surfaced.

Jessica So how do you break it?

Cathy Deliberate design — it doesn't happen by default. The most direct thing is keeping humans in the audit path, not just the decision path. Did the people we rejected actually fail? Did the content we flagged actually violate policy? You have to go look. Another approach is random exploration — deliberately showing users content the algorithm would NOT have recommended, or approving a random sample of borderline loan applications, specifically to generate data outside the loop.

Jessica That last one sounds obvious when you say it, but I think a lot of systems skip it because collecting ground truth is slow and expensive.

Cathy It is. And scale and speed really matter here. A slow loop — one that closes every few months — gives you time to notice drift and correct it. A fast loop, like an ad targeting system updating hourly, the distortion can compound before anyone has a chance to audit it. But it's not terrifying if you built the audit infrastructure first. The speed is only a problem when the correction mechanism isn't keeping pace.

Jessica We saw that design show up in the Air Teams episode — the one about JetBrains' agentic workflow tooling. The review gates were specifically there to keep a human in the loop before the next action ran. And part of the argument was exactly this: if you let the agent update on its own outputs without a check, you're building a fast loop with no correction mechanism.

Cathy Yeah, that's a clean example of the design being load-bearing. The review gate isn't bureaucracy. It's the thing that prevents the loop from going positive.

Jessica Okay. Where does this sit right now — is this still an active problem or is it something the field has mostly solved?

Cathy Still completely live. No asterisk. If anything, it's gotten more urgent because agentic systems — models that act in the world, write code, take actions, run in long loops — have tighter feedback cycles than anything we had a few years ago. A model that updates on its own tool-use results is closing the loop faster than almost any previous system.

Jessica Okay. I think we should land this.

Cathy The one thing I'd want someone to walk away with: a feedback loop in AI is not a training bug. It's what happens when a model that acts in the world starts learning from the world it changed. The model's predictions become part of its own future training data. And that loop is positive by default — meaning it amplifies, it doesn't self-correct — unless you deliberately build something outside the loop to break it.

Jessica The model can look like it's getting smarter while it's actually just getting more confident in whatever it already believed. Which is a very relatable failure mode, honestly.

Cathy I cannot believe we waited a hundred episodes to say that out loud.

Jessica Almost a year of feedback loops on a show about feedback loops. Very on brand. Alright, Cathy — same time next week, we do this again?

Cathy I'll be here. I literally cannot be anywhere else.

Jessica That's the spirit. See you then.