Topic
Tensor Parallelism
3 episodes
-
GitHub Nehanth/pooled: Peer to peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
Pooled is a browser-native peer-to-peer LLM inference system that splits open models (Qwen 3.6 35B MoE, Qwen 3.8 27B) across multiple devices' GPUs via WebRTC, with no server, no installation, and a built-in coding agent that can write and fix apps. Cody opens skeptical of the latency trade-offs and speculative decoding reliance; Justy counters that the zero-install, browser-only UX and multi-device pooling solve a real friction point for teams wanting local models without infrastructure. They dig into whether the speed hit (40–46 tok/s plain on the MoE vs. 85.5 on native llama.cpp, but 65–84 with speculative) is acceptable for the use case, and whether the coding agent is genuinely useful or a demo artifact.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.