Exploring Next
Exploring Next — Ep 247 w/ Justy & Cody — Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
Episode 247 dives into groundbreaking research on how LLMs internally respond to increasingly difficult tasks. The team discovered that as inputs become more out-of-distribution, models make their representations dramatically sparser — essentially concentrating computation into specialized subspaces. This isn't random; it's an adaptive mechanism for handling unfamiliar territory. The researchers built this insight into Sparsity-Guided Curriculum In-Context Learning, showing real performance gains by using sparsity patterns to intelligently schedule few-shot examples.