My colleague Jani Tarvainen and your truly did some testing over the past weekend. We managed to get our hands on a spare B200 GPU and Jani had the idea of trying out FreeToken and some models that are effectively too big for GPU's RAM. We tried DeepSeek V4 Flash but the bigger success came from GLM 5.3 Flash. Its checkpoint is 198GB in size whilst the GPU had about 191GB available. So too big, but FreeToken to the rescue.

GLM 5.3 Flash is a mixture-of-experts model: its config file says 42 layers of 288 small expert networks each, and every token uses 8 of the 288 experts in each layer. So a 313-billion-parameter model costs only about 17B parameters of work per token. Single token requires enough memory for that 17B slice, but all 313B have to sit somewhere accessible at the same time, and 304B of them are those experts. We used RedHatAI's NVFP4 build (https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4), which squashes the experts to 4 bits plus their scale factors. That is the reason the starting point is 198GB. For comparison, the original checkpoint from the creators, in 8 bits, is 328GB.

vLLM, which we typically use to run our LLMs, by default wants all of the model's weights on the GPU. There is an option to spill part of the model into system memory, but it does not really help us here: whatever sits in system memory is copied to GPU over the PCIe link again and again for every single token the model generates. Our arithmetic says even a modest 15GB of spilled weights would add roughly a quarter of a second to every token, so generation would crawl. vLLM's own recipe for this model needs 229GB of GPU memory at 4-bit, more than the 191GB our GPU had to start with. We found no published recipe anywhere that runs it on a single B200.

The vLLM number also counts the KV cache, where the engine keeps the work it already did on earlier tokens so it never has to re-read them. In a real world serving setup you want to leave real room for it. When we host Qwen3.8 Flash Next on a B200 we keep about 40% of the GPU memory free for the KV cache, and we serve 60 agents in parallel on that box. Fitting the weights is only half the budget, really.

FreeToken (https://github.com/FlashML-org/FreeToken) is an Apache-2.0 engine built for GPUs too small to hold the model. All 12,096 experts (42 layers times 288) live in system memory, and the engine keeps a cache of 10,094 of them, 83%, on the GPU. When a token needs an expert that is not there yet, FreeToken copies that one expert, about 14MB, over the link, which takes around a quarter of a millisecond on this machine, and it stays on the GPU for the tokens that follow. Every token consults 336 experts in total (8 per layer across those 42 layers), and later tokens keep picking the same popular ones, so an expert already on the GPU crosses the link no more. FreeToken decided its own settings; the startup log said: strategy: offload (auto). That was the trick. One engine moves the whole spilled part for every token; FreeToken moves one expert once, after which it stays on the GPU.

Loading took 9 minutes and 4 seconds, mostly reading 185GB from disk and building that cache.

Then we ran many agentic sessions on top of it. During the measured hour a single agent client sent 497 requests. Every one returned HTTP 200, and the engine never restarted. Generation ran at 79 tokens per second for a single request and 160 tokens per second in total at the default cap of four concurrent requests. The GPU drew 336 W out of its 1,000 W budget.

Quick word on those figures. This was by no means a controlled test, it was one hour of a single real client. The GPU can stream roughly 8TB/s, a token reads its 17B slice, roughly 21GB of weights, so it could in principle generate something like 375 tokens per second; we measured 79, about a fifth. The post is about fitting, not speed.

The detail I keep turning over: only the expert parameters, 304B of them, went to 4 bits. The attention, the shared expert every token uses on top of its 8, and the embedding layers, about 9B parameters, stayed 16-bit, which means two bytes per parameter. They alone weigh 18GB, and that is what overflowed the GPU. Everything that was quantized fit all along. The part that was never quantized is the part that did not.

So: one GPU does what the recipes said would take two, as a fallback, not as a fast path. FreeToken is made for consumer GPUs that cannot hold the model, and on a B200 it plays exactly that role. It makes the model run. It does not make it fast. Serving a whole team is a two-GPU problem, and our memory arithmetic lands on the same verdict as vLLM's own 229GB minimum.

Today the model is up and answering on one GPU, with plenty of headroom left on the power meter.