Topic
Knowledge Distillation
12 episodes
-
Introducing Flint Recursive
Recursive Flint is an open-weight, local coding model that combines sparse MoE computation with a heavily optimized inference stack. Its real story is less the self-improving-AI framing than the attempt to make capable agentic coding fast, private, and practical on personal hardware.
-
Beam is now available — Reflection
Vince and Ava talk through Reflection's Beam becoming available as an open-weight release, focusing on what changes now that the model, serving stack, and customization tooling are all shipping together. They stay skeptical about vendor-reported benchmarks, but land on the real story being the adoption path: a model you can actually download, serve, and adapt across a surprisingly broad ecosystem.
-
The Tasteful Agent: Measuring and Improving Taste in Long Horizon Tasks
Justy and Cody dig into a paper arguing that long-horizon agent performance is bottlenecked by something more specific than general capability: taste, meaning the ability to choose the better path before the evidence is obvious. They unpack Taste-Bench, how decision forks are mined from trajectories without human annotation, why the best model only gets 59.7 percent, and why more reasoning tokens do not rescue weak judgment. The conversation lands on a familiar Exploring Next theme: the useful progress is in better scaffolding and better supervision, not mystical agent vibes.
-
Harness Zero: Harness Distillation via Agent as Harness
Harness-Zero turns a great, evolved agent harness into training data so the underlying model permanently learns those behaviors and can run on a much simpler harness — often matching or even beating the original setup.
-
Repo to Skill: Distilling GitHub Repositories Into AI4AI Skills
Repo-To-Skill introduces DisCo, a skill-distillation system that extracts operational knowledge from GitHub repositories and papers, packaging it as compact, verified skills that autonomous ML research agents can load on demand. The AREX-Skill Library contains 5,000+ skills from 1,000 repositories organized into 20 areas and 178 capability families. In matched tests with GPT-5.5 backbone and fixed execution budget, skill-equipped agents outperform skill-free baselines by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet—gains purely from operational knowledge, not model or harness improvements.
-
Overview: Knowledge Distillation
We finally do the episode we kept promising — a proper ground-up explanation of knowledge distillation: what it is, how the teacher-student mechanism actually works, why soft targets carry more signal than hard labels, and where this shows up in real systems being built right now.
-
DART SD: Diamond topology Aware Retrieval and Tuning for Self Distillation of Multi Turn Tool Calling Agents
Edmund and Geffen discuss the ByteDance/USTC paper DART-SD, which tackles 'topological collapse' in agent distillation. They discuss how moving from linear trajectory imitation to a diamond-topology graph (ISTG) allows student models to learn recovery from errors without destroying their own valid reasoning paths.
-
Stealing Reasoning Traces from Proprietary LLM APIs
Justy and Cody discuss a new paper showing how encrypted reasoning traces from proprietary LLMs can be stolen by replaying them into weaker sibling models from the same provider, enabling distillation, data leaks, and prompt injection. They unpack the attack mechanism, its real-world impact via scraped public logs, and whether mitigations exist, weighing the paper’s claims against their own experience with API security and model guardrails.
-
Overview: Supervised Fine Tuning
We finally slow down and make supervised fine-tuning click, because we keep leaning on S F T like everyone already has the whole shape of it. We build it from the apprentice-and-worked-examples picture into the actual training loop, the examples, and the trade-offs.
-
Seed: Self Evolving On Policy Distillation for Agentic Reinforcement Learning
Seed tackles the credit-assignment problem in long-horizon agent reinforcement learning by turning completed trajectories into evolving natural-language hindsight skills, then distilling their effect into dense token-level training signals. Vince sees a potentially shippable training pattern for teams already running agentic RL; Ava likes the on-policy design but wants stronger evidence that self-generated skills do not amplify the model’s own blind spots.
-
VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
-
Morphing into Hybrid Attention Models
Talon and Wildflower discuss FlashMorph, a new method for choosing which Transformer layers should keep full attention when converting pretrained LLMs into hybrid attention models.