Ep 1164 Research Paper 5:31 w/ Justy & Cody

Theoretical Limits of Language Model Alignment

The paper derives a theoretical ceiling for alignment under a KL-divergence budget, offers a way to estimate that ceiling before training, and finds that best-of-N sampling gets close while PPO and GRPO leave substantial performance on the table. Justy and Cody explore what this means for reward hacking, production alignment, and the cost of inference-time search.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1164"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1164 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script GPT-5.6 luna Voice ElevenLabs v4

Transcript

Justy The awkward result here: the supposedly simple alignment trick might be closer to optimal than the sophisticated training loops built to replace it.

Cody That's best-of-N sampling. You generate several answers, keep the one with the highest reward. And the paper argues it gets very near the best possible reward for a given distance from the base model.

Justy And PPO and GRPO? The actual reinforcement-learning methods?

Cody Noticeably below that ceiling. In their tests, anyway.

Justy Which is why I like the framing. We usually ask which alignment method wins, head to head. This paper asks how much better ANY method could possibly get before it starts changing the base model too much.

Cody The distance part is KL divergence. It charges the aligned model for moving its output distribution away from the original. So if the base model usually gives three kinds of answers and alignment makes one much more likely… KL captures that shift.

Justy mm

Justy And that's the reference point product teams have been missing. Without a ceiling, a new training run can look impressive just because the previous run was worse. You want to know whether the algorithm is leaving attainable reward on the table.

Cody So under the KL-regularized setup, they derive the maximum reward gain for a fixed KL budget. The key object is Jeffreys divergence… forward KL plus reverse KL, between the base and aligned models.

Justy Didn't earlier analyses have something like this already?

Cody A square-root-of-KL upper bound. This expression is exact for the optimal aligned model,

Justy Hold on. Isn't that circular?

Justy The expression mentions the optimal aligned model. Which you only know AFTER you've aligned.

Cody That's the escape. They rewrite it as a covariance computed under the base model. So you sample answers from the base model, watch how reward varies across them, and you can estimate the available gain before you ever do the post-training.

Justy Covariance here is basically asking, do the outputs that score well under the reward sit in a direction the base model already has some probability mass in? If good answers are already nearby, alignment has room to improve without a huge behavioral shift.

Cody And if reward is high only on outputs the base model almost never produces, the budget gets expensive fast. The model has to move farther, so the same reward target costs more in capability retention. Which is a mechanism, not just a leaderboard observation.

Justy The reward-model section is where it gets operational for me. They split the ideal reward, the thing we actually care about, from the proxy… the learned scorer trained on human labels.

Cody And the gap between those alignment outcomes grows with proxy error, and when the KL penalty factor gets smaller.

Justy So the more aggressively you optimize, the more dangerous a weak grader gets.

Cody That's the uncomfortable bit. Lower the penalty and the policy chases reward more forcefully, sure, but the proxy also gets more room to exploit quirks correlated with approval rather than the underlying goal. Straight line to reward hacking… the model just finds the easiest route through a misspecified score.

Justy mm

Justy Their relief valve is reward ensembling. Average several reward models, assume their errors aren't all the same, and the prediction approaches the ideal reward at one over the square root of the number of models.

Cody So four independent-ish reward models don't buy you four times the accuracy, but they can cut random error roughly in half. IF the assumptions hold. Correlated errors could make the ensemble much less useful.

Justy Then the experiments… safety alignment on BeaverTails, summarization on TL;DR. Best-of-N tracks the theoretical frontier closely, while PPO and GRPO stay substantially suboptimal.

Cody On both tasks?

Justy Both. Strong signal. But it's two tasks, not a universal ranking of alignment methods.

Cody And best-of-N comes with a very obvious bill. Sample a hundred candidates, you pay for a hundred candidates, plus reward scoring, to produce one answer. It can approach the limit because it searches among outputs the base model already knows how to generate. PPO and GRPO try to compress that preference into updated parameters, and they may not find the best distribution for every KL budget.

Justy yeah

Justy Which makes this less "use best-of-N everywhere" and more "use best-of-N as a measuring stick." If a cheaper training method sits far below it, that gap is evidence the method is leaving attainable alignment quality unused.

Cody One caution before we hand out medals though. The paper's frontier is defined by the reward being measured, and the reward itself can be wrong. A method can move closer to the proxy optimum while moving farther from what users actually want. The theory gives us a cleaner ruler… it doesn't make the ruler ground truth.

Justy That's probably the takeaway for builders. Before celebrating a better reward score, ask whether the model found a good behavior or the scorer just became the easiest tool in the room.

Cody I'd also want this tested across more domains. Long-horizon tool use especially, where the output distribution is a whole trajectory, not one response. The covariance estimator is promising there, but the assumptions and compute costs may change once actions affect later observations.

Justy No code or dataset to build along with here, so we're not pretending there's a command to run tonight. Cody, I'm putting the equations back in their folder before they start grading us.

Cody Fair. Though now the folder needs an ensemble.

Justy Before the folder learns reward hacking… we're out. See you next time.