NVIDIA Dory 1.0 4B on a Mac: 3.7GB Fits 16GB, Fork Build Needed
Dory's headline is not that a four-billion-parameter NVIDIA model fits on a Mac. It's that memory is basically solved at this size, but software support is not. One small GGUF fits even on an eight-gigabyte machine; the real blocker is that stock llama.cpp, Ollama, vLLM, and MLX don't properly support the architecture yet, so the only working Apple Silicon path is a pinned fork build.
Transcript
Vince The sneaky point here is that Dory on a Mac is not a memory story at all. It's a software support story wearing a memory headline.
Ava Yeah. And I think the article earns that. The smallest official GGUF is about three point six seven three gigabytes, so the weights fit easily on sixteen gigabytes of unified memory. Even eight gigabytes can technically hold the file.
Vince Right. Which is why this is useful for normalizing a thing people do constantly. They see NVIDIA, they see Mac, and they assume the blocker is some giant footprint. Here, no. The blocker is you can't just type an Ollama command and move on with your life.
Ava Exactly. Stock llama.cpp doesn't support the architecture yet. The article points to an open draft pull request for Dory support, and says Ollama has no library tag at all. MLX doesn't have an implementation either, so there isn't some cute side door where you avoid the fork.
Vince And NVIDIA's own answer is basically, sure, just clone this contributor fork, fetch this exact commit, build it yourself, verify the checksum, then run llama-server locally. Which is fine for you. It is not fine for, like, ninety percent of people who say they want to run something locally.
Ava Mm-hm. The technical part I liked is that the author actually checked the file sizes and the header values instead of hand-waving. They pull the exact GGUF byte count, note that the repo total includes cards and images, and separate that from the real download. That's the right level of annoying.
Vince That is your love language, by the way. Not flowers. Byte counts.
Ava Oh, stop it.
Vince No, come on, Ava, it's true. Somewhere out there is a tiny little shrine to exact blob sizes.
Ava Oof. But the article also doesn't overclaim the memory side. On eight gigabytes, yes, the weights fit under their seventy percent GPU-usable budget rule. But you only have about one point nine gigabytes left for KV cache and buffers, so they say short context only. That matters because the model card says up to two hundred fifty-six K context, and the launch recipe they quote is four thousand ninety-six.
Vince That's the actual consumer-protection bit. The product page number is two hundred fifty-six K. The lived number on a small Mac is four thousand ninety-six unless you're making tradeoffs and doing homework.
Ava Right.
Vince The looped part is the real model idea, though. One checkpoint, and you change dory.num_loops between one and four to trade compute for quality. That is a pretty good product shape, honestly, because it lets you tune for latency without swapping models.
Ava I think that's fair. Mechanistically, it's input block, shared recurrent block, output block, and the recurrent section runs multiple times. But the catch is KV cache mode. Per-loop caching grows memory with loop count. last_loop apparently avoids that overhead by overwriting, which is elegant, but then you're relying on support that isn't upstream yet.
Vince So who actually cares? To me, it's Mac local-inference people who are comfortable compiling forks, or teams testing weird architectures early. If you're an Ollama person, this changes nothing today.
Ava Yeah, that's my read too. I'm not the local-Mac-maximalist here, but the argument mostly holds. The only place I'd be careful is the phrase "fits on an eight-gigabyte Mac" because listeners will hear "usable" when it really means "loads, with tight context budget."
Vince No, that's good. And this is also not the tiny inference shrine situation from Muse Spark where people start calling a ridiculous desktop setup "home" compute. This one is actually modest hardware. It's just gated by plumbing.
Ava Okay, that's good.
Vince My bet, one for episode ten ninety-four: sixty-five thirty-five upstream llama.cpp support for Dory lands by November fifteenth. Ollama tag by then, lower. Maybe fifty-fifty.
Ava I'd go a little colder on Ollama. Maybe forty-five fifty-five by November fifteenth, just because they usually wait for the path to stop moving. But yeah, llama.cpp merging first is the visible thing to watch.
Vince That's it. Not whether it fits. Whether you can run it without becoming your own release engineer.