Topic
Quantization
19 episodes
-
I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read
Pippa and Tyler dig into an XDA piece claiming one llama.cpp flag made a 35B MoE model usable on an 8GB laptop GPU. The real story is not magic speedup from nowhere, but a smart memory trade: keep attention, shared weights, and KV cache on VRAM, and peel dormant expert layers into system RAM. They like the mechanistic honesty, question how broadly the result generalizes, and land on who this actually matters for: people trying to stretch small local setups past their assumed ceiling.
-
Introducing Flint Recursive
Recursive Flint is an open-weight, local coding model that combines sparse MoE computation with a heavily optimized inference stack. Its real story is less the self-improving-AI framing than the attempt to make capable agentic coding fast, private, and practical on personal hardware.
-
Model Behavior: Week of October 19, 2026
We read this week as a shift away from trophy launches and toward who can make models useful in real deployments. We argue through NVIDIA, Beam, Mistral, Subquadratic, and why a quiet frontier week still moved the board.
-
NVIDIA Dory 1.0 4B on a Mac: 3.7GB Fits 16GB, Fork Build Needed
Dory's headline is not that a four-billion-parameter NVIDIA model fits on a Mac. It's that memory is basically solved at this size, but software support is not. One small GGUF fits even on an eight-gigabyte machine; the real blocker is that stock llama.cpp, Ollama, vLLM, and MLX don't properly support the architecture yet, so the only working Apple Silicon path is a pinned fork build.
-
Welcome Beam: a 501B open model you can own, customize, and scale
Reflection's Beam lands as a 501B open-weight model—Apache 2.0, 1M context, MoE powered—built for coding and agents. Cathy digs into real deployment cost and open-weight reality, Jessica sees the product fit and release polish. The tension: is this truly usable at the scale it claims, or does the open-label overstate how many teams can actually deploy it? Real disagreement, lots of detail, a brisk ZOOM OUT to where Beam fits against DeepSeek and Meta's Muse, and a sharp call on whether this model actually sees adoption.
-
NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning | NVIDIA Technical Blog
NVIDIA drops Nemotron 3.5 Super VL — a 120B-parameter open-weight multimodal MoE model built for specialized enterprise agents. Jessica and Cathy dig into the real argument: post-training on your own data beats the frontier on your specific task, and the efficiency story (40x inference cost reduction, 10x latency) is where the product actually lives.
-
NVIDIA's Nemotron 3.5 Super VL with post training beats Claude Opus 5, GPT 6.1 Sol on specialized task benchmarks
Edmund and Geffen dig into NVIDIA’s Nemotron 3.5 Super VL launch, with Geffen skeptical of benchmark-cherry-picked claims and Edmund focused on the product reality: a multimodal model that can be post-trained on enterprise docs, tools, and policies, plus local Bolt variants for smaller deployments. They land on the idea that the real story is not “it beats Claude or GPT” in the abstract, but that NVIDIA is selling an adaptable stack with deployment control, while the public benchmark chart still needs independent verification.
-
My 8GB GPU shouldn't run flagship local LLMs, but this workflow makes it work anyway
The episode explores how an 8GB GPU—well below today’s flagship local LLM hardware—can still deliver useful local AI with careful model selection, quantization, and load tuning. The hosts dig into the article’s real-world workflow: mixing medium and tiny models (Qwen 3.5 9B, Gemma 4 E4B, LFM2.5 1.2B) at specific quant levels, using GPU offload, and matching each model to the right job. They debate whether these clever workarounds change anything practical for most users, and where the technical limits still bite.
-
PrismML — Introducing Bonsai 2 27B: Near Lossless Compression in a 9x Smaller Footprint
Onyx and Echo discuss PrismML's Ternary Bonsai 2 27B launch, focusing on near-lossless low-bit compression, local deployment, throughput, energy efficiency, and where the benchmark claims still need real-world validation.
-
AIM — India
DeepSeek launches V4.1-Flash, a 552-billion-parameter MoE model that activates only 8-16 billion parameters at inference time, with a one-million-token context window and FP4 KV caching that cuts memory footprint to 890 bytes per token — one-quarter the HBM of the prior generation. The hosts dig into whether the architecture actually delivers on the inference-efficiency promise, and what it means for the teams who've been watching the Flash tier get increasingly serious.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Overview: Knowledge Distillation
We finally do the episode we kept promising — a proper ground-up explanation of knowledge distillation: what it is, how the teacher-student mechanism actually works, why soft targets carry more signal than hard labels, and where this shows up in real systems being built right now.
-
Overview: Inference Optimization
We finally give inference optimization its own episode — the idea that's quietly under half the stories we cover. We walk through what it actually means to make a trained model run faster and cheaper, from caching to quantization to batching, and why it matters more than almost anything else once a model ships.
-
1Aggregate results comparing DFM Mimir 1B against the HRM Text 1B, Qwen 3.5 2B and Gemma 4 E2B, displaying highly competitive performance across 20 benchmarks.
Pippa and Tyler examine DFM Mimir v1, a one-billion-parameter HRM trained from scratch with permissible post-training data, and ask whether its strong benchmark results make it genuinely useful for low-resource language work.
-
Overview: Quantization
We finally sit down with quantization and make it concrete: why shrinking a model’s numbers can speed inference up without wrecking the thing. We walk through the trade-offs, the common formats, and where quantization still matters in practice.
-
Poolside Releases Laguna S 2
Vince and Ava talk through Poolside’s Laguna S 2.1 release as an unusually practical open-weight coding model: 118B total parameters, 8B active, 1M-token context, and a real deployment story on a single DGX Spark. They dig into the mechanism, the max-thinking default, the benchmark results, and the trade-off between long-horizon capability and token spend, while keeping one eye on the broader open-vs-closed race.
-
How to Run Open Source AI Models
Sid Saladi argues that frontier AI vendors (Claude, GPT) bundle model, compute, access, and application into one proprietary stack—trapping users in unpredictable pricing and competitive capture. The counter: open-weight models like GLM-5.2, DeepSeek V4, Qwen, and Kimi are now frontier-adjacent in capability (GLM-5.2 beats GPT-5.5 on coding benchmarks, matches Opus 4.8 on others) and cost roughly one-sixth as much. The real problem isn't model quality anymore; it's that companies like Tesla, Uber, and Meta are hemorrhaging money on metered AI because they can't decouple the stack. The guide walks four layers—model, compute, access, harness—and shows how to own each one deliberately instead of letting a vendor own all four by default.
-
Hot French startup ZML releases free product to speed inference across lots of AI chips | TechCrunch
Laura and Harper dig into ZML's new free inference server and the bigger claim underneath it: that the real leverage now is software that decouples models from chip vendors. Harper likes the direction but doubts the article proves the hard part, while Laura thinks the product story is strong even if the benchmarks are still missing.
-
Viability of local models for coding
Birgitta Böckeler tests local LLM viability for coding after a year away from the space. She maps a complex web of factors—RAM, model size, quantization, tool calling, context windows, reasoning modes—that determine whether small models actually work for agentic coding on consumer hardware (M3 Max / M5 Pro). Her core finding: locals are runneable and faster than a year ago, but tool calling is still shaky, reasoning can backfire, and quality is hit-or-miss. She's not claiming local models are ready to replace cloud APIs; she's charting what actually works and what doesn't on real machines.