Exploring Next
Exploring Next — Ep 301 w/ Justy & Cody — LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Justy and Cody dig into LongAct, a paper about making long-context RL work better by updating only the attention weights tied to unusually large query and key activations. They unpack why that matters for long docs, agents, and multi-step reasoning, how the saliency-guided sparse updates map activation outliers back to specific weight rows, and why the reported gains across LongBench v2, RULER, and multiple RL algorithms suggest this could be more than a lab curiosity.