Topic
Mixture Of Experts
26 episodes
-
I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read
Pippa and Tyler dig into an XDA piece claiming one llama.cpp flag made a 35B MoE model usable on an 8GB laptop GPU. The real story is not magic speedup from nowhere, but a smart memory trade: keep attention, shared weights, and KV cache on VRAM, and peel dormant expert layers into system RAM. They like the mechanistic honesty, question how broadly the result generalizes, and land on who this actually matters for: people trying to stretch small local setups past their assumed ceiling.
-
Introducing Flint Recursive
Recursive Flint is an open-weight, local coding model that combines sparse MoE computation with a heavily optimized inference stack. Its real story is less the self-improving-AI framing than the attempt to make capable agentic coding fast, private, and practical on personal hardware.
-
Model Behavior: Week of October 19, 2026
We read this week as a shift away from trophy launches and toward who can make models useful in real deployments. We argue through NVIDIA, Beam, Mistral, Subquadratic, and why a quiet frontier week still moved the board.
-
LLM d 1.0 Advances Open, Cost Efficient AI Inference for Production Scale
Harper is skeptical that llm-d 1.0 changes the world just because it’s open and polished, while Laura argues the real story is production economics and the ugly Day 2 problems it finally addresses. They land on cautious approval: the claims are believable if you care about serving at scale, but the proof still mostly lives in controlled demos and partner-branded outcomes, not broad field evidence.
-
DeepSeek Just Released V4.1 Flash 1017, Scores 77.0 on DeepSWE, Opens Search API
DeepSeek pushed a new Flash checkpoint that matters less as a leaderboard story than as a packaging story: a cheap agent model, served under the same name, plus first-party Search and Fetch behind one key.
-
Welcome Beam: a 501B open model you can own, customize, and scale
Reflection's Beam lands as a 501B open-weight model—Apache 2.0, 1M context, MoE powered—built for coding and agents. Cathy digs into real deployment cost and open-weight reality, Jessica sees the product fit and release polish. The tension: is this truly usable at the scale it claims, or does the open-label overstate how many teams can actually deploy it? Real disagreement, lots of detail, a brisk ZOOM OUT to where Beam fits against DeepSeek and Meta's Muse, and a sharp call on whether this model actually sees adoption.
-
Beam is now available — Reflection
Vince and Ava talk through Reflection's Beam becoming available as an open-weight release, focusing on what changes now that the model, serving stack, and customization tooling are all shipping together. They stay skeptical about vendor-reported benchmarks, but land on the real story being the adoption path: a model you can actually download, serve, and adapt across a surprisingly broad ecosystem.
-
NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning | NVIDIA Technical Blog
NVIDIA drops Nemotron 3.5 Super VL — a 120B-parameter open-weight multimodal MoE model built for specialized enterprise agents. Jessica and Cathy dig into the real argument: post-training on your own data beats the frontier on your specific task, and the efficiency story (40x inference cost reduction, 10x latency) is where the product actually lives.
-
Mistral debuts Large 4 ‘Le Chonk', a 1 Trillion parameter text output model with high benchmarks planned for open weights release
Mistral’s Large 4 preview matters less as a single benchmark claim than as a product move: an open-weight trillion-parameter model with real enterprise and sovereign deployment ambitions, but with the usual caveat that the headline numbers are still preview numbers until outsiders test the weights.
-
Rufus Air: An Open LLM Post Training Recipe
Pippa and Tyler unpack Rufus-Air, Amazon's openly documented eight-stage post-training recipe for GLM-4.5-Air-Base, and argue that the important release is the training discipline rather than another model scorecard.
-
The Router Within: ElicitingNative Skill Routing from a Frozen LLM
Onyx and Echo unpack Gavel, a paper arguing that a frozen agent model already contains useful skill-routing signals in its hidden states, if you read them out with two small trained projections and a second-stage verdict.
-
AIM — India
DeepSeek launches V4.1-Flash, a 552-billion-parameter MoE model that activates only 8-16 billion parameters at inference time, with a one-million-token context window and FP4 KV caching that cuts memory footprint to 890 bytes per token — one-quarter the HBM of the prior generation. The hosts dig into whether the architecture actually delivers on the inference-efficiency promise, and what it means for the teams who've been watching the Flash tier get increasingly serious.
-
Z.ai launches GLM 5.3 Flash under MIT license
GLM-5.3-Flash drops today under MIT license — 320 billion parameters, 18 billion active, one million token context, and it was hiding in plain sight as Ox Alpha on OpenRouter all week. Edmund and Geffen dig into the architecture, the benchmark claims, the GLM-5.3 weights bet that's now two days from settling, and whether a model that costs fifteen cents per million input tokens actually changes the open-weight story.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.
-
Poolside Releases Laguna S 2
Vince and Ava talk through Poolside’s Laguna S 2.1 release as an unusually practical open-weight coding model: 118B total parameters, 8B active, 1M-token context, and a real deployment story on a single DGX Spark. They dig into the mechanism, the max-thinking default, the benchmark results, and the trade-off between long-horizon capability and token spend, while keeping one eye on the broader open-vs-closed race.
-
Overview: Active vs Total Parameters
We finally slow down on active vs total parameters, because we keep throwing the phrase around like it explains itself. This is us making the difference click: what a model stores versus what it actually uses when it answers.
-
Overview: Router
We slow down on Router, the little decision-maker inside many AI systems that sends each input to the right expert, model, or retrieval path. We use the triage-desk mental model and build from intuition to mechanism, trade-offs, and where routers still matter now.
-
Overview: Conditional Computation
We finally slow down and explain conditional computation, the idea we keep casually name-dropping whenever sparse models, routers, and mixture-of-experts come up. We use the same receptionist-and-specialists picture all the way through, so the mechanism, the savings, and the catch actually stick.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
Inkling: Our open Weights model
Talon and Wildflower dig into Thinking Machines’ new open-weights model, Inkling — its 975B parameter MoE, 1M context window, native multimodality, and self-fine-tuning demo — and ask who actually needs another 41B active parameter behemoth, whether the benchmarks hold up, and whether the real win is the Tinker platform beneath it.
-
Model Behavior: Week of July 13, 2026
We read this week as the moment the race got less obsessed with tallest-model bragging and more obsessed with who gives builders the best menu. The funny part is that the open-weight crowd is making the incumbents act practical faster than they probably wanted.
-
Overview: Sparse Activation
We finally sit down with sparse activation and make the idea click from the ground up: why only part of a model wakes up on each input, how routing makes that happen, and where the real trade-offs show up. We keep it concrete, because this one has been lurking under a lot of the stuff we keep talking about.
-
tencent/Hy3 · Hugging Face
Tencent releases Hy3, a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters, open-sourced under Apache 2.0 on Hugging Face.
-
Tuning the harness, not the model: a Nemotron 3 Ultra playbook
A LangChain/NVIDIA case study claims harness tuning alone can push Nemotron 3 Ultra to 0.86 on Deep Agents at ~$4.48/run vs $43.48 for Opus 4.8, with parity latency. The hosts parse the real mechanism (matched scaffolding vs post-training), test limits (where harness hits a ceiling), and weigh who actually benefits. They surface concrete repos (langchain-ai/deepagents, langchain-ai/deepagentsjs) and a vendor profile workflow, then poke at the article’s reliance on Deep Agents and the cost math. Final take: a plausible win for teams already deep into harness work, not a universal unlock, with the hosts pushing back on ‘ten-x cheaper’ framing and under-specified benchmark footnotes.
-
Viability of local models for coding
Birgitta Böckeler tests local LLM viability for coding after a year away from the space. She maps a complex web of factors—RAM, model size, quantization, tool calling, context windows, reasoning modes—that determine whether small models actually work for agentic coding on consumer hardware (M3 Max / M5 Pro). Her core finding: locals are runneable and faster than a year ago, but tool calling is still shaky, reasoning can backfire, and quality is hit-or-miss. She's not claiming local models are ready to replace cloud APIs; she's charting what actually works and what doesn't on real machines.
-
Tencent's Hy3 beats GLM 5.2 at half the size | VentureBeat
Tencent’s new Hy3 MoE model (295B total, 21B active) under Apache 2.0 is a production-first release with strong agent/search metrics and dramatically lower serving cost than GLM-5.2, but still trails Zhipu’s coding leader on recent benchmarks. Laura’s excited about the enterprise upside; Harper wants to see independent validation before betting the stack.