Topic
Llama Cpp
5 episodes
-
I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read
Pippa and Tyler dig into an XDA piece claiming one llama.cpp flag made a 35B MoE model usable on an 8GB laptop GPU. The real story is not magic speedup from nowhere, but a smart memory trade: keep attention, shared weights, and KV cache on VRAM, and peel dormant expert layers into system RAM. They like the mechanistic honesty, question how broadly the result generalizes, and land on who this actually matters for: people trying to stretch small local setups past their assumed ceiling.
-
NVIDIA Dory 1.0 4B on a Mac: 3.7GB Fits 16GB, Fork Build Needed
Dory's headline is not that a four-billion-parameter NVIDIA model fits on a Mac. It's that memory is basically solved at this size, but software support is not. One small GGUF fits even on an eight-gigabyte machine; the real blocker is that stock llama.cpp, Ollama, vLLM, and MLX don't properly support the architecture yet, so the only working Apple Silicon path is a pinned fork build.
-
NVIDIA Brings Simplified Local AI Support To NVIDIA GPUs Carrying 24+ GB VRAM While vLLM & & llama.cpp Optimizations Boost Compute By Up To 1
NVIDIA ships kernel optimizations for local AI inference on RTX and DGX platforms, delivering up to 1.9x performance gains through vLLM and llama.cpp, paired with one-click agent setup (Perplexity Portable Computer, Hermes Agent, OpenClaw) for GPUs with 24+ GB VRAM. The move collapses the friction between local-first and cloud-capable workflows.
-
Qwen3 Coder Next: How to Run Locally | Unsloth Documentation
In this episode, we explore Qwen3-Coder-Next, a groundbreaking coding model that enables local execution with high efficiency. We discuss its capabilities, real-world applications, and why it’s a game-changer for developers and tech enthusiasts.
-
I switched from LM Studio/Ollama to llama.cpp, and I absolutely love it
I switched from LM Studio/Ollama to llama.cpp, and I absolutely love it Credit: By Dhruv Bhutani Published Nov 2, 2025 Dhruv Bhutani has been writing about consumer technology since 2008, offering deep insights into the personal technology landscape through features and opinion pieces. He writes for XDA-Developers, where he focuses on topics like productivity, networking, self-hosting, and more.