A B300 in Helsinki: room for 70 agents, 47K tokens a second of prefill, and worst cases that finally behave

I had been polling UpCloud's availability feed for a B200 for over two weeks. Zero sightings, while the smaller L40S plans came and went all day: the big GPUs were allocated, not briefly out of stock. Then Mikko, our contact at UpCloud, let me know they were about to get B200s and B300s in. So I reworked the polling script to take whichever of the two showed up first, and on Wednesday afternoon it took a B300. By Wednesday evening the box had run three models through my benchmark and was serving my own agent sessions. Still no B200. If one shows up I will grab that too, for the side-by-side. Here is what the B300 measured, what it means for the H100 it is meant to relieve, and which numbers you should not believe yet.

The problem is not throughput

Some background, briefly. Trail Openers runs one shared NVIDIA H100, a data-centre GPU with 80 GB of memory, in UpCloud's Helsinki data centre. It serves a 27-billion-parameter model, Qwen3.8-27B in a 4-bit INT4 format, through vLLM, the serving engine, to a handful of developers whose coding agents talk to it all day. The production post, the cost post and the MTP post have the history.

Before spending any money on a bigger GPU, the question was whether it was needed at all. So I sampled the H100's own metrics for 34 hours: 313 samples. On average 1.7 requests were running. In 51% of samples, none at all. The H100 is, most of the time, a GPU waiting for work, and by any cost-per-token measure a GPU at 2.5-4x the price is poor value. I knew that going in and wrote it down before buying anything, so I could not talk myself out of it afterwards.

The same metrics show the other half, though. One morning at 07:18: four requests running, nine waiting, and the reason logged for the wait was memory. To understand why four requests can fill an 80 GB GPU you need one piece of how these models work. While a model is reading or writing a conversation, it keeps a working memory of everything in that conversation on the GPU, called the KV cache, and its size grows with the length of the conversation. Our agents' conversations are long: half of all requests carry about 100,000 tokens of context, the top 5% carry 200,000 or more. Four of those at once and the H100's memory is full. Everyone else waits.

And the way agents work makes waiting hurt more than the averages suggest. A 45-75 minute agent session is not one long answer. It is hundreds of short turns in a row: the agent reads, calls a tool, reads the result, calls another, each turn producing maybe 100 tokens. When the GPU is full, every one of those hundreds of turns waits. When the GPU is busy, each turn streams at maybe 38 tokens a second. Slow per turn, times hundreds of turns, is the afternoon the team was complaining about.

So the case for a Blackwell GPU, NVIDIA's generation after the H100, was never "more tokens per hour". It was two things: how many of these 100,000-token agent sessions fit on the GPU at once, and how fast each one gets its tokens when they are all running. Everything below is measured against those two.

How I measure

Same method I use on halo, my Strix Halo box at home (an AMD Ryzen AI MAX+ 395 mini PC with 128 GB of memory, the setup page has the details), ported to vLLM and run on the B300 itself. Two words I will use throughout: prefill is the model reading the prompt, decode is the model writing the answer, and they behave very differently.

So, what actually runs. Prefill on real source code at 2K to 200K tokens with caching off, so the GPU has to actually read it. Decode on its own. Needle recall, where a fact is hidden deep in a long prompt and the model has to find it. Short requests fired in the middle of a 100K-token prefill, to see whether they get stuck behind it. Then 1 to 32 agents at once, each given its own secret word to repeat back, so if answers ever cross between agents it shows; and for each run I report both what one agent gets and what the GPU produces in total, because total going up while each agent slows down is exactly how a GPU looks busy and feels slow. Capacity gets two numbers: how many 100K-token conversations the engine's own KV cache holds, and a ramp that adds agents until the latency targets from production break (30 ms between tokens, and the slowest 1% of requests starting within 7.5 s). Last, two runs at 16 agents with prompts drawn from production's actual distribution: one where every prompt is cold and distinct, one multi-turn where each turn extends the previous one, so the cache works as it does in real use.

That last distinction matters because of a shortcut vLLM and every other serving engine takes: if the start of a prompt is identical to one it has seen recently, it reuses the work. Agents resend the same growing conversation turn after turn, so in production 97.7% of prompt tokens hit that cache. A benchmark that ignores this measures a workload nobody runs, and a benchmark where every request is secretly the same prompt measures a ~60-token cached read at every concurrency. Mine generates a distinct prompt per request from a fixed seed, so every GPU sees byte-identical cold work.

The box is one NVIDIA B300, 275 GB of memory. Three models. Qwen3.8-27B-INT4, exactly what the H100 runs today. The same 27B in NVFP4, a 4-bit number format the Blackwell tensor cores run natively, so the only thing that changes is how the weights are stored. And Qwen3.8-Flash-Next in NVFP4: a mixture-of-experts model with about 177 billion parameters in total but only about 6 billion active for any given token, which is what makes it cheap to run per token despite its size. Halo has run it since August. The two 27Bs ran on vLLM 0.30.0; Flash-Next would not start on that version, so it ran on a newer 0.32 build. Different number format, different engine build: keep both in mind when reading across the table.

Results

Three things I'd pull out of that.

The number format alone is worth 4x on prefill. INT4 to NVFP4 on the identical 27B: 7,200 to 28,900 tok/s reading a cold 100K prompt, 21,500 to 84,700 on the multi-turn run, decode 14% faster. Same weights, same engine, same GPU. The INT4 version has to unpack its weights into a format the GPU can compute with, on every layer, every token. The NVFP4 version is running on the tensor cores the GPU was built around. I'd say: if you move to Blackwell and keep an INT4 checkpoint, you have bought a very expensive H100.

Flash-Next is where the parallelism gets silly. 47,200 tok/s of cold prefill means a 116,000-token prompt nobody has seen before is read in 2.5 s. With 32 agents running at once, each one still streams at 59 tok/s, which is faster than the H100 gives one developer on a busy afternoon, with 31 other agents going. The two 27Bs with 32 agents: 19-20 tok/s each. The GPU alone does not fix per-agent speed under load. The model's active parameter count does, and 6 billion active against 27 billion is the whole difference between a GPU that scales and a GPU that merely holds more sessions.

Room for about 70 agents at 100K context, against roughly 16 on the H100. That is the question the whole exercise was about, and all three models answer it the same way, because the KV cache is a property of the 275 GB, not the model. Flash-Next's architecture helps too: 36 of its layers are a cheaper kind of attention that barely uses KV cache, so despite being six times the total parameters of the 27B it holds the same number of full-length conversations.

Short requests fired in the middle of a 100K prefill came back in 70-80 ms on all three, same as on an idle GPU. Nothing gets stuck behind a long prompt.

Two caveats before the next section. The ramp that adds agents until latency breaks also produced a figure, 0, 2 and 2 agents for the three models: ignore it. It reads cold 100K prompts and judges them against a time-to-first-token target taken from production, where 97.7% of the prompt is already cached. A cold 116K read on the INT4 model takes 14 s against a 7.5 s target: fail at one agent, done. Wrong test for this workload; the KV cache figure is the real capacity, and the warm numbers in the next section are the real latency. And quality on this box is untested, full stop. The needle-recall and secret-word checks did not produce usable results on any of the three, for the same reason the LFM post ran into last week: these models think before they answer, and the check reads only the first few tokens of the reply. Halo's build retrieved Flash-Next needles at 64K, 131K and 203K, but that is a different engine and a different quant. The real sessions a colleague and I are running on it until tomorrow are the quality test. One more on Flash-Next: part of its layers fell back to a slower generic kernel with a warning, so its 199 tok/s and 47K prefill may be a floor rather than a ceiling.

Then it got real traffic

Every headline number above is cold: distinct prompts, cache off, the opposite of production. So the better test started at 19:52 the same evening, when the box, running Flash-Next, was pointed at my own agent sessions. One hour later: 8,335 requests, 668 million prompt tokens in, 1.15 million generated. That is 135 requests a minute, and 98.0% of the prompt tokens were served from cache, against production's ~97%. Same shape as production too: median prompt about 100K tokens, 90th percentile 200K, median output about 100 tokens (my outputs ran a little shorter at the top end than the team's, p90 200 against 500, so not perfectly like for like). Average concurrency about 4.7 against production's 1.7, so a busier hour than the H100's typical one, if lighter than its worst. Peak seen: 12 requests running, none waiting, KV cache 14.4% full.

Against the H100's long-run production figures. Both sides come from vLLM's own latency histograms, which record counts per bucket rather than exact values, so each figure is an upper bound: "the slowest 1% started within 2.5 s", not "at 2.5 s". p50 is the median request, p99 the slowest 1%.

The typical turn is faster, and the slow end is much faster. Mean time to first token 0.57 s against 0.88 s. A whole typical request done in 2.67 s against 5.75 s, about half. That is with three times the H100's usual concurrency on the box. Then the slow end, which is what people actually feel: the slowest 1% of requests got their first token within 2.5 s against within 10 s, and finished within 40 s against 60 s. Median time between tokens under 10 ms against under 25 ms, which is the difference between ~100 tok/s and ~40 tok/s on your screen while a long answer streams. And twelve agents using 14% of the memory, where the H100 would be near its ceiling.

Which is exactly the shape of the original complaint. Nobody said the H100 was unusable. They said that some turns, at some times, took forever, and that an agent session is hundreds of turns. Halving the typical turn and cutting the worst ones by three quarters is the thing you notice. One hour, one evening, one person's traffic, so a direction rather than a result. It is also the first warm number in the whole exercise, and it points the same way as the cold ones.

What I actually think

The question I wrote down before buying was how many agents fit, and the B300 answers it: roughly four times the H100, at 100K context, with any of the three models. On the other question, how fast each agent goes when they all run, the honest reading is that the GPU helps a lot and the model helps more. Put the production 27B on the B300 and you get room for 70 agents of a model that still slows to 19 tok/s each when 32 of them run. Put Flash-Next on it and the same 32 agents each get 59.

My gut, and it is a gut until the quality checks say otherwise: Flash-Next is as good as the 27B or better on the coding work this endpoint sees. Halo has run it in production for two months. The 27B is on the H100 because it was the best model that fit in 80 GB with room left for conversations, not because it won a quality bake-off. If those sessions hold up, the B300 question answers itself. The cost: €6.95 an hour, about €5,070 a month, against €1.79 for the H100, so 3.9x the price. With Flash-Next on it, room for four times the agents at three times the speed each, the cost per agent is the same or better than today. With the 27B in INT4 on it, it would be worse: you would pay 3.9x for a GPU that is still limited by how slowly it reads prompts. And either way the honest framing is that the GPU will sit idle half the time, bought so that the other half of the time nobody waits. That is what headroom at the busiest hour costs. It is a latency purchase, not an efficiency one, and anyone presenting it should say so.

Still open: whether a B200 ever shows up so the two Blackwells can be set side by side, whether the ~70 holds when agents are actually added one by one on a warm cache, and whether more than one evening of traffic keeps the slow end where mine put it. The B300 goes back tomorrow. If a B200 turns up, that one we might keep. The polling is still running...