GitHub Nehanth/pooled: Peer to peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
Pooled is a browser-native peer-to-peer LLM inference system that splits open models (Qwen 3.6 35B MoE, Qwen 3.8 27B) across multiple devices' GPUs via WebRTC, with no server, no installation, and a built-in coding agent that can write and fix apps. Cody opens skeptical of the latency trade-offs and speculative decoding reliance; Justy counters that the zero-install, browser-only UX and multi-device pooling solve a real friction point for teams wanting local models without infrastructure. They dig into whether the speed hit (40–46 tok/s plain on the MoE vs. 85.5 on native llama.cpp, but 65–84 with speculative) is acceptable for the use case, and whether the coding agent is genuinely useful or a demo artifact.
Transcript
Justy Okay, so there's this thing called Pooled that's basically asking: what if your laptop, your friend's desktop, and a phone all opened the same browser tab and pooled their GPUs to run a thirty-five-billion-parameter model together.
Cody Right, I saw the demo. And my first instinct is: latency is going to eat this alive.
Justy Why?
Cody Because every token has to bounce between devices. The hidden state is small—four to ten kilobytes—but you're sending it over WebRTC, and at fifty milliseconds of round-trip latency, you're looking at about a hundred and fifty milliseconds added to every single token. That's brutal.
Justy Mm-hm.
Cody They report forty to forty-six tokens per second on the MoE with plain decoding. Native llama.cpp on the same machine does eighty-five point five. That's more than half the speed just… gone.
Justy But they also have speculative decoding, right? The model drafts its own tokens and they collapse most of the latency penalty because seventy to eighty-five percent of the drafts are accepted on code. So you're actually at sixty-five to eighty-four tokens per second in practice.
Cody Okay, fair. Speculative helps. But that's the point—they're relying on speculative decoding to make the baseline speed acceptable. That means if you hit a workload where speculative doesn't work as well, you fall back to forty-six tokens per second. That's a cliff.
Justy Yeah, but here's the thing: who cares about raw tokens per second if the model's running on your devices with zero setup? No installation, no account, no server somewhere else. Friends open a link and they're pooling compute.
Cody Oh interesting.
Justy Like, the demo runs a coding agent—you ask it to build a Tetris game, it writes the files, serves them in a sandbox preview right there in the browser, reads its own console errors, and fixes them. Everyone in the room sees the same thing. That's not a speed demo. That's a control demo.
Cody Okay, I'll grant that the UX is slick. But slick UX doesn't ship if the speed is a non-starter. And for a coding agent that's actually useful—not a toy—you need throughput. The latency hit plus the prefill cost, which they report as one to two and a half seconds, means the time to first token barely moves with network latency, sure, but the token-by-token experience is going to feel slow.
Justy Right, right. Prefill is rough. But for a coding agent—especially one that's drafting and fixing incrementally—you're not running it in real-time chat mode. You're running it async. You ask for something, the agent thinks for a few seconds, and you get back code you can iterate on. That's a different use case than a live chat with a frontier model.
Cody Sure. And I'm reading the docs, not running it myself, so take this with a grain of salt—
Cody —but the real test is whether teams actually adopt this. Because the speed-to-UX tradeoff is real. Yes, zero installation is great. Yes, pooling is clever. But you can rent a GPU for thirty bucks a month and run a 35B model at native speed without waiting for your friend's phone to connect.
Justy Fair. But you're also introducing a new dependency—you need the money, you need the account, you need the internet to reach that server. Pooled is asking: what if you didn't? What if the compute was already sitting in the room?
Cody Exactly.
Justy That's the real unlock. It's not speed. It's friction. And if you're a small team—two, three people, each with their own machine—and you want a coding agent that you control completely, with no external server and no vendor lock-in… this is probably the first time that's been genuinely possible in a browser tab.
Cody Okay, so you're saying the adoption path is teams that have both a latency tolerance AND a control requirement.
Justy Yeah. And the coding-agent part is actually the killer feature, because it's the one workload where you might actually want all three of those things together: control, visibility, and the ability to iterate locally. Plus, the model can call tools, use JSON schemas, and serve its own API—the whole thing is wired up as an OpenAI-compatible endpoint.
Cody Right, and they built the Serve API so you can run it on your own machine with Node twenty-two and point any tool to it. That's good architecture.
Justy So the question is: does that narrow use case actually exist? Are there enough teams that want this badly enough to accept the speed hit?
Cody I think it does exist. But it's narrow. And the speed ceiling is real. The moment you need sustained throughput—like, running agents that chew through a lot of context or making rapid-fire requests—you're going to hit the latency wall and start looking for alternatives.
Justy Which is exactly why the roadmap has faster prefill as the top item. They know the bottleneck.
Cody Fair.
Justy So the real bet is: does Pooled ship as a product, or does it stay a clever demo? Because the infrastructure is genuinely there. The WebGPU kernels work, speculative decoding is validated against plain decoding, the room protocol is solid. But adoption is friction.
Cody I'd say… sixty-forty it finds a real user base within six months. The coding-agent story is compelling, and the zero-install UX is hard to compete with. But the speed hit is also hard to justify if you have any other option.
Justy I'd go the other way. Seventy-thirty it ships and gets real adoption. The friction it removes is so concrete—no accounts, no servers, everyone in the room can see what's happening—that I think teams will take the speed hit for it.
Justy To try it: clone the repo, run `npx -y serve -l 8080`, open `the link in the show notes No GPU required for the local setup—the tests run on SwiftShaker. If you want to hit real hardware, the e2e tests use a tiny synthetic model, but the golden tests run on actual Qwen if you have a WebGPU GPU and Deno two.
Cody And the whole thing is open source—Apache two-point-oh, I think—so if the speed ceiling becomes a real problem, people can actually fork and optimize.
Justy Yeah. Boring but works. Infrastructure you can inspect and modify if you need to.