Topic
KV Cache
31 episodes
-
I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read
Pippa and Tyler dig into an XDA piece claiming one llama.cpp flag made a 35B MoE model usable on an 8GB laptop GPU. The real story is not magic speedup from nowhere, but a smart memory trade: keep attention, shared weights, and KV cache on VRAM, and peel dormant expert layers into system RAM. They like the mechanistic honesty, question how broadly the result generalizes, and land on who this actually matters for: people trying to stretch small local setups past their assumed ceiling.
-
LLM d 1.0 Advances Open, Cost Efficient AI Inference for Production Scale
Harper is skeptical that llm-d 1.0 changes the world just because it’s open and polished, while Laura argues the real story is production economics and the ugly Day 2 problems it finally addresses. They land on cautious approval: the claims are believable if you care about serving at scale, but the proof still mostly lives in controlled demos and partner-branded outcomes, not broad field evidence.
-
NVIDIA Dory 1.0 4B on a Mac: 3.7GB Fits 16GB, Fork Build Needed
Dory's headline is not that a four-billion-parameter NVIDIA model fits on a Mac. It's that memory is basically solved at this size, but software support is not. One small GGUF fits even on an eight-gigabyte machine; the real blocker is that stock llama.cpp, Ollama, vLLM, and MLX don't properly support the architecture yet, so the only working Apple Silicon path is a pinned fork build.
-
NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning | NVIDIA Technical Blog
NVIDIA drops Nemotron 3.5 Super VL — a 120B-parameter open-weight multimodal MoE model built for specialized enterprise agents. Jessica and Cathy dig into the real argument: post-training on your own data beats the frontier on your specific task, and the efficiency story (40x inference cost reduction, 10x latency) is where the product actually lives.
-
Introducing Claude Haiku 5
Jessica and Cathy dig into Anthropic’s Claude Haiku 5.5 launch as a real fast-tier product, not a benchmark trophy: cheaper, much quicker, and aimed at repetitive agent work, with an adjustable effort setting and a sharper place in the Claude stack.
-
My 8GB GPU shouldn't run flagship local LLMs, but this workflow makes it work anyway
The episode explores how an 8GB GPU—well below today’s flagship local LLM hardware—can still deliver useful local AI with careful model selection, quantization, and load tuning. The hosts dig into the article’s real-world workflow: mixing medium and tiny models (Qwen 3.5 9B, Gemma 4 E4B, LFM2.5 1.2B) at specific quant levels, using GPU offload, and matching each model to the right job. They debate whether these clever workarounds change anything practical for most users, and where the technical limits still bite.
-
Model Behavior: Week of September 28, 2026
We read this week as a shift away from one giant model race and toward tiered defaults, workflow ownership, and the places agents actually run. The weird part is that the capability jumps are real, but the competitive win may belong to whoever makes the capable-enough model feel like the normal choice.
-
Introducing GPT 6.1 Sol
GPT-6.1 Sol drops today — near-Astra performance on coding, computer use, and professional work at one-fifth the price, with cached input at ten cents per million tokens. Justy and Cody dig into whether the benchmark story holds up, what the cache pricing actually unlocks for agent builders, and where this lands in the broader cost-curve race.
-
Memory Attention
Jessica and Cathy dig into Memory Attention, a paper that swaps the usual value projection for a token-indexed memory table plus the contextual key. Their take: the real argument is not "memory" in the long-context product sense, but a capacity-placement move inside attention that might buy extra parameters and even CPU offload without wrecking latency. Cathy likes the mechanism more than the headline, but both note the core caveat: the reported gains are under matched token budgets with many more parameters, so the paper does not cleanly prove the architecture itself is better than simply adding capacity.
-
Claude Opus 5.5, GPT 6 Sol, GPT 6 Luna, and a new price war
Jessica and Cathy dig into Simon Willison’s read on Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna as a pricing war with real product consequences. They focus on the practical shift: cheaper default models, cache economics, and whether Opus 5.5’s communication improvements matter more than its benchmark aura. They also poke at the weird failure mode where “max” thinking literally breaks on a silly SVG prompt.
-
Introducing GPT 6 Sol and Luna
Edmund and Geffen dig into OpenAI’s GPT-6 Sol and Luna launch as a cost-curve move rather than a pure capability leap. They focus on the practical user story, the benchmark shape, the caching changes, and whether cheaper frontier-ish models actually change what ships.
-
Introducing Claude Opus 5
Pippa and Tyler unpack Anthropic’s Claude Opus 5.5 launch — a cheaper, faster, Fable-level flagship tuned for long coding and agentic work, with heavier safety and alignment testing — and what it really changes for teams already on Opus 5 or Fable 5.1.
-
Your Agent Is Only As Good As Your Infrastructure
Justy and Cody dig into CoreWeave's argument that agent quality in production is mostly an infrastructure story once workflows get long, bursty, and tool-heavy. They mostly buy the core claim, but Cody pushes on how much of this is genuine systems insight versus a cloud vendor setting up next week's pitch on KV caching and routing.
-
AIM — India
DeepSeek launches V4.1-Flash, a 552-billion-parameter MoE model that activates only 8-16 billion parameters at inference time, with a one-million-token context window and FP4 KV caching that cuts memory footprint to 890 bytes per token — one-quarter the HBM of the prior generation. The hosts dig into whether the architecture actually delivers on the inference-efficiency promise, and what it means for the teams who've been watching the Flash tier get increasingly serious.
-
NVIDIA Brings Simplified Local AI Support To NVIDIA GPUs Carrying 24+ GB VRAM While vLLM & & llama.cpp Optimizations Boost Compute By Up To 1
NVIDIA ships kernel optimizations for local AI inference on RTX and DGX platforms, delivering up to 1.9x performance gains through vLLM and llama.cpp, paired with one-click agent setup (Perplexity Portable Computer, Hermes Agent, OpenClaw) for GPUs with 24+ GB VRAM. The move collapses the friction between local-first and cloud-capable workflows.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Z.ai launches GLM 5.3 Flash under MIT license
GLM-5.3-Flash drops today under MIT license — 320 billion parameters, 18 billion active, one million token context, and it was hiding in plain sight as Ox Alpha on OpenRouter all week. Edmund and Geffen dig into the architecture, the benchmark claims, the GLM-5.3 weights bet that's now two days from settling, and whether a model that costs fifteen cents per million input tokens actually changes the open-weight story.
-
Overview: Inference Optimization
We finally give inference optimization its own episode — the idea that's quietly under half the stories we cover. We walk through what it actually means to make a trained model run faster and cheaper, from caching to quantization to batching, and why it matters more than almost anything else once a model ships.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.
-
Introducing Cursor Router · Cursor
Cursor Router is Cursor's new Teams and Enterprise model-routing layer, using a classifier trained on more than six hundred thousand live requests to select models by task, context, complexity, and domain. Jessica sees a clean adoption story for teams stuck paying frontier rates for routine coding work; Cathy likes the production-oriented evaluation and cache-aware accounting, while keeping an eye on how much trust enterprises place in Cursor's routing judgment.
-
Overview: Context Window Management
We finally slow down and explain Context Window Management from the ground up, because we keep hand-waving it whenever agents, memory, cost, and long tasks come up. The whole thing is the fixed-desk problem: what stays on the desk, what gets compressed, and what falls off.
-
Kimi K3 Kimi API Platform
Two friends unpack the Kimi K3 API docs, debating its 1M‑token claim, hybrid attention, and tool dynamics, and weigh who should pay the price for the hype.
-
Model Behavior: Week of July 13, 2026
We read this week as the moment the race got less obsessed with tallest-model bragging and more obsessed with who gives builders the best menu. The funny part is that the open-weight crowd is making the incumbents act practical faster than they probably wanted.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Long Horizon Terminal Bench: Testing the Limits of Agents on Long Horizon Terminal Tasks with Dense Reward Based Grading
Laura and Harper dig into Long-Horizon-Terminal-Bench, a new benchmark exposing the gap between short-task agent demos and real multi-hour workflows. They break down the dense-reward grading system, the staggering token costs (9.9M per task), and why current models are failing at sustained execution despite high step-level competence.
-
Overview: KV Cache
We finally sit down with KV cache and make the whole thing click: what it stores, why generation gets faster, and why memory is the bill you still have to pay. We keep coming back to the same working-memory picture so the mechanics stay grounded instead of turning into acronym soup.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
OpenAI Releases GPT 5.6 (Sol, Terra, Luna): A Three Tier Model Family With Programmatic Tool Calling in the Responses API
OpenAI's GPT-5.6 family — Sol, Terra, and Luna — introduces three permanent capability tiers with distinct cost profiles and a new cache billing model, plus a multi-agent Ultra mode that runs four agents in parallel by default.
-
Hot French startup ZML releases free product to speed inference across lots of AI chips | TechCrunch
Laura and Harper dig into ZML's new free inference server and the bigger claim underneath it: that the real leverage now is software that decouples models from chip vendors. Harper likes the direction but doubts the article proves the hard part, while Laura thinks the product story is strong even if the benchmarks are still missing.
-
Viability of local models for coding
Birgitta Böckeler tests local LLM viability for coding after a year away from the space. She maps a complex web of factors—RAM, model size, quantization, tool calling, context windows, reasoning modes—that determine whether small models actually work for agentic coding on consumer hardware (M3 Max / M5 Pro). Her core finding: locals are runneable and faster than a year ago, but tool calling is still shaky, reasoning can backfire, and quality is hit-or-miss. She's not claiming local models are ready to replace cloud APIs; she's charting what actually works and what doesn't on real machines.
-
nvidia/Nemotron TwoTower 30B A3B Base BF16 · Hugging Face
Justy and Cody dig into NVIDIA’s Nemotron-TwoTower-30B-A3B-Base-BF16 and whether block-wise diffusion decoding is a real systems win or just a benchmark-shaped detour. Cody is skeptical about the headline throughput claim and the way the model compares itself to a single autoregressive baseline, while Justy focuses on who actually benefits from faster generation without a big quality drop. They land on cautious interest: interesting infrastructure idea, but not a universal replacement for standard decoding.