A B200 for 35% less, and an evening of agents got more out of it than the GPU did

We benchmarked a B200 next, the box after the B300 from last week's post. On Qwen3.8-Flash-Next it does the B300's job for 35% less money, so the B300 is released and the B200 is the one we keep. That took a morning. The evening is the part worth writing down: with a few hours of research and benchmarking, and the willingness to run the serving engine ahead of its release, the same GPU ended up doing 1.7x the work at 32 agents and holding 80 agents where it had collapsed at 48. The GPU you pick matters less than what you are prepared to do to the software on it.

How the B200 turned up

The polling script from the B300 post runs on halo, my Strix Halo box at home (an AMD Ryzen AI MAX+ 395 mini PC, the setup page has the details). It had logged B300 stock on most polls since the 15th and a B200 not once. On the 18th at 11:18 UTC one appeared. The account's GPU quota is two and the H100 and B300 were both alive, so the B300 had to go first; the second attempt took the B200 and the benchmark started on it at 12:05. It costs €4.50 an hour, about €3,290 a month, against the B300's €6.95.

For anyone arriving cold: Trail Openers runs a shared NVIDIA H100 (80 GB) in UpCloud's Helsinki data centre, serving Qwen3.8-27B to the team's coding agents through vLLM, the serving engine. The production post has that history, and the H100 keeps doing exactly that. The Blackwell boxes were about what the next generation does with the same work, and which one is worth paying for.

Same benchmark, three days apart

Same method as before: cold prefill (the model reading a prompt nobody has seen), single-stream decode (the model writing on its own), the production request mix and multi-turn 100K sessions at 16 agents, and the size of the KV cache, the working memory the GPU keeps for every open conversation. Flash-Next ran on the same vLLM nightly on both GPUs.

The B300 wins cold prefill at 100K, 47K against 34K, and that is the spec sheet showing through: it has about half again the dense 4-bit compute, and prefill is the compute-bound half of the job. Decode is bound by memory bandwidth, both have 8 TB/s, so decode is the same. At 16 agents the B200 is ahead, because at that load neither GPU is near busy and the engine sets the pace. The B300's extra memory shows only on the dense 27B, where its pool holds 71 agents at 100K against 44; Flash-Next's cache is small (36 of its 48 layers are linear attention, with a fixed-size state instead of one that grows with the conversation), so there the gap is 5%. Two GPUs, same machine, two prices.

Then the evening

I say "we" from here on purpose: I work through agents. A Claude Code session ran the benchmark and read the engine logs, and when I asked what else could be tuned it sent six research agents through vLLM's source, GitHub and the box itself, one area each, then had one agent rank the levers and another try to knock holes in the plan. I decided which experiments got a slot and what quality gate each had to pass: a needle hidden at ten depths in 100K and 200K of context, and a log-likelihood score on long code, for every change that could touch output quality. Every number below is from a sweep of 1 to 80 agents, each on its own 100K-token code context, the shape of the team's real traffic.

MTP first. Flash-Next carries a small extra layer that guesses the next few tokens and the main model checks the guesses in one step; k is how many it guesses. The MTP post covered it on the H100. On the B200, engine otherwise untouched, greedy sampling:

Up to 32 agents, more guessing is better. At 48 both MTP settings fall off a cliff, and the engine log says why in one line each: the KV pool is 6.56M tokens with MTP off, 5.52M at k=1, 5.07M at k=3. Each running request reserves working space that grows with k, out of the same pool. 48 sessions of 100K-plus tokens need about 5.3 to 6M to stay resident. Once they don't fit, the engine evicts the next agent's context, that agent re-reads 100K tokens from scratch, and the share of each prompt already in memory drops from 98% to 84%. MTP is a trade of memory for speed, and memory runs out first.

The lever nobody puts on a list. The agent reading vLLM's speculative-decoding code found that the engine drops the last block of cached prefix every turn when a drafter is on, deliberately, because reusing it can lower the drafter's accuracy. On a dense model that block is 16 tokens. On this hybrid model it is forced up to 3,200 to line up with the linear-attention state pages, so every turn of every agent re-read 3,200 tokens it already had, and at 32 agents each of those re-reads is a prefill chunk in the batch that every other agent's decode step waits behind. Keeping the block took the cache hit rate from 95% to 98.8% and, at 16 agents, added 27% to total throughput.

Then the state itself. Storing the linear-attention state in bf16 instead of fp32 halves it, which halves the block (3,200 to 1,664) and gives back about a third more cache. Quality was the risk, so it was gated: log-likelihood on the last 4K tokens of eight 100K-plus documents came out at 0.20437 nats against 0.20430 in fp32, noise, and the needle was 10 for 10. Add a cap on how many requests run at once (the rest queue with their context still cached, instead of evicting each other), memory utilisation 0.95, a small MTP index-sharing setting, and this is where the box stood by dinner:

That third column exists because the reviewer agent asked the plan an awkward question: what sampling does the benchmark use? Greedy, as benchmarks do. The model's own config says temperature 1.0, top-k 20, and under that the drafter's guesses are accepted 57% of the time instead of 96%. So MTP in real use is about 1.9x, not 3x, and from here on this blog quotes the default-sampling column. Every speculative-decoding number you read elsewhere has this problem unless it says otherwise.

Then the edge

64 agents still collapsed, because 64 sessions need about 6.7M tokens of cache and the tuned pool is 6.04M. The box has 235 GB of RAM and a PCIe Gen5 link, and vLLM can copy every cache block to host memory and bring a session back from there in tens of milliseconds instead of re-reading it in seconds. That is the fix, and on this model it has a bug: an open vLLM fix (#60861) exists because a reloaded session can resume from linear-attention state that was never written, which means silent corruption, no error. Unmerged.

We are going to be retuning this box nightly for a while anyway, so the agents mounted the patch over the running image and built a gate for it: continue eight 100K code documents from a cached prefix and record the model's confidence in its own tokens, any NaN, how many characters of the real next text it reproduces, and, per document, how many gigabytes actually came back from host memory, so a run where nothing reloaded cannot pass by accident (two did, and were thrown out). On the clean run every document reloaded 1.3 to 1.55 GB, no NaN, the same opening on all eight, six of eight reproducing the real code to the same character. A stronger variant first showed a confidence gap of 0.056 against a limit of 0.05, and then five repeat runs of the baseline alone spanned 0.116: the limit was tighter than the instrument's own noise. No NaN, no garbage, no gap beyond noise, on eight documents, which cannot exclude a subtle effect. That is the honest state of it. All of the following is with that unmerged patch overlaid, default sampling, running cap 32:

Offload costs 8 to 13% at the team's typical load and turns the cliff past 50 agents into a slope that still gives 39 tokens a second each at 80. The team's peak so far is 50, once, and contexts grow. I had asked for a balance between decode speed and the number of parallel agents; the agent chose this one, offload on, patch overlaid, cap 32, and it has been serving since Saturday 21:56 UTC with the no-offload config one file-copy away. I'd have chosen the same.

What I actually think

The morning's answer was easy: same speed on the model we run, 35% cheaper, kept. The evening's is the one I'd pass on. In a few hours, a GPU that was already the cheaper right choice went from 1,332 to 1,891 tokens a second at 32 agents at honest sampling, and from falling over at 48 agents to serving 80, and not one of those gains came from the hardware. They came from reading the serving engine's source, measuring instead of assuming, catching two of our own instruments lying (greedy sampling flattering the drafter, a log-likelihood check that silently bypassed the cache it was meant to test), and being willing to run a patch nobody has merged behind a gate we wrote ourselves. That is the price. If you are not prepared to benchmark, to read the engine, and to live a few weeks ahead of the release, you get the spec sheet's version of the GPU, and the spec sheet said the two were different.

Two production endpoints now, the H100 with the 27B and the B200 with Flash-Next, side by side. The first real peak is the next gate, and when #60861 merges the overlay comes off...