Overview: Next State Prediction

Vince Okay so this one keeps coming up. Like, genuinely — we say 'next-state prediction' and then just… move on, and I know that's annoying. Listeners have told us. So today we actually stop and do it properly.

Ava Yeah, and it's worth it because this is not a niche concept. This is the thing underneath most of what we talk about on this show. We did a whole Overview on Predictive Modeling back in episode nine twenty-five, and next-state prediction is really one specific, powerful version of that — so if you want the broader picture, go there. But today we zoom all the way in on this one idea.

Vince Alright, Ava — before any jargon. Give me the image.

Ava Okay. You're watching a video with the sound off. No subtitles, no context. Just frames. And after enough of them, you can guess what happens next — the ball is going to bounce, the person's about to turn around, the door's going to open. You're not following a rule book. You've just seen enough of the world that the patterns are in you.

Vince Right.

Ava Next-state prediction is that same idea, scaled up and applied to a neural network. You train a model to guess the next thing in a sequence — the next word, the next video frame, the next game state — and the surprising part is what happens as a side effect. The model ends up building an internal picture of how the world works. Not because you told it to. Because it HAD to, to get the prediction right.

Vince Okay that's the part that always gets me. It's not like anyone sat down and said 'learn grammar' or 'learn physics.' It just… falls out.

Ava Exactly. And that's the core insight. Understanding is a side effect of prediction pressure. Hold that — we're going to keep coming back to it.

Vince So walk me through the actual mechanism. What is the model doing during training?

Ava So — quick background. A neural network is a system of connected numerical weights that transforms an input into an output, and you can adjust those weights based on how wrong the output was. During training, you take a sequence — say, a sentence — you show the model the first chunk of it, you hide the next part, and you ask the model: what comes next? The model makes a guess. You compare that guess to what actually came next in the data. You measure how wrong it was.

Vince And nobody labeled any of that data.

Ava That's the beautiful part. The label is already there. The next word in the sentence IS the label. The next frame in the video IS the label. The sequence provides its own training signal. This has a name — it's called self-supervised learning. And it's why you can train on raw internet text, or raw video, without anyone going through and tagging it. The signal is free.

Vince Okay so I want to make sure the newcomer is still with us. We've got a model, it's guessing the next thing, it's getting nudged toward better guesses billions of times. Why does that produce something that can reason? Why doesn't it just get really good at… predicting words?

Ava So this is the question. And the answer is: to predict the next word reliably, you have to understand what the sentence is ABOUT. If the sentence is 'the capital of France is…' — to guess 'Paris,' you need to have encoded the fact that Paris is the capital of France. If the sentence is 'she picked up the glass and then it…' — to guess 'fell' or 'shattered,' you need something like a model of physics and causality. The prediction task is a proxy for understanding.

Vince Alright. Let's talk about where this shows up, because it's not just language.

Ava Yeah, this is where it gets fun. The same principle runs across totally different domains. Language is the obvious one — every GPT-style model, every autoregressive language model, is doing next-token prediction. A token is basically a chunk of text, usually a word or part of a word. You feed in context, you predict the next token, repeat.

Vince Alright — there's one more piece I want to cover. In-context learning. When a language model reads a few examples in its prompt and then does the task — how is that related to next-state prediction?

Ava So in-context learning wasn't explicitly trained for. It emerged. The leading theory is that it emerges from next-token prediction at scale: the model has seen so many sequences of the form 'here are some examples, here's the next one' that it learned to extract the pattern from the context and apply it. The model is still just predicting the next token — it's just that 'the next token' happens to be the answer to a task.

Vince Hm. That's either a deep insight or a slightly unsatisfying one depending on how you look at it.

Ava Both, again. That's kind of the theme of this concept.

Vince Okay. And briefly — people hear 'next-state prediction' and then they hear 'fine-tuning' and wonder how those fit together.

Ava Quick version: next-state prediction is what happens during pre-training — the phase where the model learns from raw data at massive scale. Supervised fine-tuning is what happens AFTER, where you take that pre-trained model and train it further on curated examples to steer its behavior toward what you actually want. The pre-training gives the model its general capabilities. Fine-tuning shapes how it uses them.

Vince Alright. Where does this stand right now? Is next-state prediction still the thing, or has the field moved somewhere else?

Ava It's still the thing. No asterisk. Every major language model, every video generation system — built on this training objective. The field has built an enormous amount on TOP of it, and there's active research into alternatives, but the core paradigm hasn't been displaced. And honestly, what's changed most isn't the objective — it's the scale, the data quality, and what gets layered on afterward. The prediction task itself has been remarkably stable.

Vince Which is kind of remarkable when you think about how fast everything else has moved.

Ava It is. The field tried a lot of other approaches, and this one just kept winning. Not because it's perfect, but because the signal it provides is so rich and so free that nothing else has matched its practical leverage.

Vince Okay. So — if you had to distill this to the one thing to hold onto.

Ava The one thing: if you make a model really good at guessing the next thing in a sequence, it ends up building an internal model of how the world works. Not because you told it to. Because it had no other way to get the predictions right. Understanding is a side effect of prediction pressure. That's the whole trick. Everything else is engineering on top of that.

Vince Back to our person watching the silent video. They didn't read a physics textbook. They just watched enough frames that the physics got in.

Ava That's exactly it.

Vince Ava, I think that might be the clearest we've ever explained anything on this show. Which means our bar was probably too low before, but I'll take it.