Moonshot AI and Tsinghua Researchers Propose Prfaas a Cross Datacenter Kvcache Architecture That Rethinks How LLMs Are Served at Scale

Justy If your AI app keeps making the same expensive sandwich in three different kitchens, this paper is for you.

Cody [chuckles] And somehow we're recording a niche infrastructure episode from your kitchen, which feels on brand.

Justy Cody flew into LA, immediately judged my coffee setup, and now we're doing Exploring Next, episode 308. And yeah, this one matters because people are stuffing giant prompts into apps and paying to recompute context over and over.

Cody Right, and that pain gets worse the second you have more than one region. A user lands in one place, traffic shifts, failover happens, or your router picks a different datacenter... and the model rebuilds the same KV cache like it has no memory at all.

Justy Which from a product angle is brutal. The user story is basically, "why is the second request still slow, and why is my bill weird?" If you're building chat, coding tools, agent workflows, anything with long history, prefill becomes the tax you keep paying.

Cody [exhales] So PRFaaS is trying to treat that tax differently. The paper's move is: stop thinking of KV cache as something trapped beside one GPU in one datacenter. Make prefill a service, make the cache portable across datacenters, and let later requests reuse it remotely instead of regenerating it.

Justy Okay, break that down without turning into documentary narrator voice.

Cody [laughs] In the wild, the large prompt gets processed once. That creates KV states for every layer, every token. Normally those states sit local to the serving stack that made them. PRFaaS splits the world into a prefill side and a decode side, then adds a cross-datacenter KV-cache layer so decode can fetch or attach to prefills computed elsewhere.

Justy So instead of re-reading the whole novel, it ships over the notes.

Cody Exactly. And that's where the design choice is actually smart. Shipping raw prompts is cheap but recomputing them is expensive on GPU. Shipping KV cache is heavier on the network, but if the prompt is long enough, moving cache beats doing prefill again. The system is basically making that trade in a more explicit way.

Justy That feels very current. Everybody wants longer context windows, richer memory, more tool traces... then acts surprised when latency turns into airport-security-line latency.

Cody Yes. [giggles] And the paper leans into the fact that prefills and decodes behave differently. Prefill is chunky, parallel, expensive. Decode is step-by-step and latency sensitive. If you decouple them, you can place them differently, scale them differently, and reuse the expensive part across sites.

Justy Midway tangent: I'm giving your airport sandwich review from this morning a D-plus, by the way.

Cody It was a mercy meal, Justy. The red-eye rule applies. If I still care about a systems paper after that sandwich, the paper is real.

Justy [chuckles] That's fair. Back to it — who actually uses this? To me it's the teams serving lots of repeated context: enterprise copilots with huge docs, chat products with persistent threads, maybe agent systems where every loop drags around a backpack full of prior tokens.

Cody Yeah, and maybe not your tiny weekend bot at first. The architecture assumes cross-datacenter traffic and enough repeated prefills to justify the machinery. But the mechanism itself is broadly useful: identify shared prefixes, materialize KV once, route later requests to that state. That's the core idea whether you're huge or just tinkering.

Justy And compared with the usual tricks... prompt caching, prefix caching, sticky sessions... this is more aggressive, right?

Cody Right, because sticky sessions help only if the next request lands in the same place. Prefix caching helps inside one serving domain. PRFaaS is saying the cache should survive geography. That's the interesting part. The risk, obviously, is network overhead, consistency headaches, and whether your cache-hit rate is high enough to pay for the complexity.

Justy [sighs] Which is always the part where my PM brain goes, "cool diagram, who is on call for this at 2 a.m.?" Because now your serving path includes remote cache availability, placement logic, and transfer costs that change with traffic.

Cody Totally. I don't think the paper reads like magic. It reads like a serious trade study. My read is the win shows up when prompts are long, reuse is meaningful, and cross-site bandwidth is good enough that moving KV is still cheaper than burning more GPU time. If those conditions aren't true, simpler local caching probably wins.

Justy [pause] That's why I like it, though. It feels less like "new model trick" and more like, "hey, inference is a distributed systems problem now." Which, Cody, is unfortunately your love language.

Cody [laughs] You flew me three thousand miles for storage-and-routing takes. But yeah. If I were playing with this over a weekend, I'd start with vLLM or SGLang locally and benchmark prefill versus decode on a long prompt so you can actually see where the time.