How NetEase Games cut LLM cold starts from 42 minutes to 30 seconds

Justy Cody, this one feels painfully practical: NetEase Games says it cut LLM cold starts from 42 minutes to about 30 seconds.

Cody Yeah, and that matters because cold start used to mean, like, a sleepy web service. With LLMs, it can mean downloading or mounting tens, sometimes hundreds, of gigabytes of model weights before a GPU does anything useful.

Justy Exactly. If a game feature, support bot, asset tool, whatever, takes most of an hour to come alive, users just experience that as broken. Also, I made coffee and forgot to drink it, so I’m personally aligned with any technology that fixes cold starts. [chuckles] Anyway, this thing is called Fluid.

Cody That is such an LA product manager metric: beverage activation latency. But yeah, Fluid is the interesting bit. It’s a Kubernetes-native data orchestration project, originally out of NetEase, and the article frames it as a way to manage model data separately from the inference container.

Justy So not, “make a giant Docker image with the model baked in and pray the node has it cached.” That’s the product read for me. The user is an ML platform team that needs to spin up inference for different models without every deployment becoming a moving-day situation.

Cody Right. Fluid gives you Kubernetes custom resources for the data side. You define something like a Dataset, then a Runtime that says what cache engine should back it, like Alluxio or JuiceFS depending on the setup. Then a DataLoad-style step can prefetch the model weights into the cache before the inference workload lands.

Justy And NetEase Games is a very believable place for this. They have lots of interactive products, lots of traffic shape weirdness, and probably a bunch of internal AI use cases that don’t all deserve always-on GPUs.

Cody The clever part, to me, is placement. If Kubernetes schedules your vLLM or other inference pod onto some random GPU node, but the model cache is warm somewhere else, you still lose. Fluid tries to make the scheduler data-aware, so the pod goes where the model already is, or the model gets warmed where the pod needs to run.

Justy That’s the part that makes the 42 minutes to 30 seconds feel less magical. It’s not making the model tiny. It’s changing when and where the pain happens.

Cody Totally. You’re pulling cold I/O out of the request path. The model can live in object storage or shared storage, then Fluid mounts it into the pod through the cache layer. By the time the inference container starts, it sees local-ish files instead of waiting on a huge remote fetch.

Justy Adoption barrier, though: this is not a weekend toggle for a random SaaS team. You need Kubernetes, GPUs, storage choices, and someone who understands cache eviction without turning the cluster into a haunted closet.

Cody [laughs] Haunted closet is accurate. There are trade-offs. FUSE mounts can add overhead, cache coherency matters when model versions change, and GPU scheduling is already annoying before you add data locality. Compared with baking weights into images, this is more operationally sophisticated, but it avoids massive image pulls and rebuilds for every model revision.

Justy I also like that the user story is broader than games. Any company running private inference could care: coding assistants, internal search, document tools, customer operations. The market is basically teams that want cloud-like elasticity but can’t afford to let expensive GPUs sit around warm forever.

Cody One thing I’d sanity-check is how many models they’re rotating through. If it’s one flagship model on stable traffic, keep it warm and move on. Fluid shines when model choice, tenant isolation, or spiky demand means you’re constantly starting things that weren’t already resident.

Justy For Build Next, I’d keep it small. Clone the Fluid repo from github.com slash fluid-cloudnative slash fluid, spin up kind or minikube, install it with Helm, and use MinIO as fake object storage. Tiny model, tiny expectations.

Cody Yeah. I’d pair it with vLLM and something small from Hugging Face, not a monster model that turns your laptop into a space heater. Create a Dataset pointing at the model files, pick a cache runtime like JuiceFSRuntime or AlluxioRuntime, warm it, then deploy an inference pod that mounts the dataset path. Measure pod-ready time before and after. [pause] That’s a pretty good solo Saturday, if your Saturday has a little infrastructure goblin in it.

Justy And for a team, the next step is comparing three boring numbers: cold start time, GPU idle time, and model rollout time. If those hurt, Fluid is worth a look.

Cody I’d add failure behavior. Pull the cache node, roll the model version, fill the disk, see what breaks. The happy path is nice, but this only earns trust when the cache gets messy and the service still recovers.

Justy That’s the actual episode 371 energy, Cody: make the demo work, then immediately ruin it responsibly. [chuckles] I’m going to reheat this coffee.