How good is an LLM in programming? Turns out, it is QUITE DIFFICULT INDEED to measure it
As you probably know, I run open-weight models on our company's rented GPUs for our own internal developers team of developers who work with long-running coding agents. We are currently using Qwen3.8-Flash-Next for most of our tokens but earlier this week I was benchmarking few alternatives, including NVIDIA Nemotron 3.5 Super VL: 120B total params, 12B active, a hybrid Mamba-Transformer MoE (mostly Mamba layers, whose memory use barely grows with context length). I used NVFP4, a 4-bit weight format, on vLLM 0.33.0, an open-source serving engine .. and a B200 GPU at UpCloud, Helsinki.
Nemotron was interesting for few reasons:
- Licensing is much more permissive than Qwen3.8-Flash-Next's Qwen Community License 1.0
- On paper, and in reality also, much faster
- more parallel long-context agents compared to Flash Next
.. however, having tried some of the earlier Nemotron releases and been disappointed by their coding capability after few hours in real world coding tasks this seemed to bite again. It seems to be no match for Flash Next. But how much poorer? How to find out?
Well, the obvious answer was benchmarks. Plenty of those to pick from. I went with SWE-bench Verified with mini-swe-agent.
What is SWE-bench Verified?
SWE-bench Verified is 500 real GitHub bug fixes that actual humans have checked are solvable. The agent gets the repository as well as the issue text and writes a patch, one shotting it. Then the official benchmark harness runs project's own tests to see if the patch passes. mini-swe-agent is the bare-bones agent for it, one bash tool and nothing else, so you can compare models and not some fancy scaffolds. We (me and ze agents) did not run all 500. The set was weighted toward long tasks because that is what most of our token consuption is .. agents writing or testing software in long running sessions. We took all 45 tasks that are rated at more than an hour of human work, plus 55 drawn at random from the 15 minutes to 1 hour tier. Four matplotlib tasks would not run on our rootless Podman setup for any model, which left 96 as the final tally.
Results?
In the benchmark runs Qwen3.8-Flash-Next solved 93 of the 96 tasks, including 43 of the 45 long ones. Nemotron solved 45. Massive gap. Agents and yours truly were looking at those figures and it was pretty clear that Flash Next's figures could not be real. I asked agents to have a look and they were able to quickly tell me that Flash Next agents had, for many tasks, landed on exactly on official solutions.
Median similarity between the agents' solutions and the official solutions was 0.81 for Qwen and 0.13 for Nemotron. 28 of Qwen's patches were byte-identical to the merged fix! Nemotron had some byte-identical patches too, 5 of 96, and 8 at 0.9 similarity or above, against 43 for Qwen.
The model has been trained on the tasks! Either on purpose or accidentally, we'll probably never know. The dataset has been public since 2024 after all. In the end the run was absolutely no use as a comparison.
Then we tried SWE-rebench
What we needed was tasks that Flash Next could not have seen in training. SWE-rebench is built for that: Nebius keeps collecting new issues and their fixes from GitHub as they appear, so a model trained before they existed should not have the answers memorised. In theory, at least.
We took the newest set in its Hugging Face dataset: 110 tasks from 79 repositories, created between March and May 2026. Same agent, same settings, and SWE-rebench's own harness for the scoring.
Nemotron solved 36 of the 110 (33%), Flash Next 89 (81%). And that 81% is above the best score on SWE-rebench's own leaderboard, about 64.5%. Different tasks and a different agent there, so you cannot compare the two directly, but it was a pretty good hint that something was off. The agents ran the same patch comparison here as well, and sure enough, 21 of Flash Next's patches were near-copies of the official fixes (0.9 similarity or above), one of them a 194-line fix that came back identical. Nemotron had none. Spring 2026 GitHub was evidently in Flash Next's training data too, so even tasks that fresh were not fresh enough.
We could still leave out the tasks Flash Next may have remembered: we dropped every task where its patch had a similarity of 0.6 or more to the official fix, 39 of them, which left 71. On those Flash Next solved 51 (72%) and Nemotron 26 (37%). Flash Next solved 28 tasks that Nemotron missed, and Nemotron solved 3 that Flash Next missed. The filter is picked from Flash Next's own output, so this is a rough correction and not a clean score. It cannot flatter Flash Next though, because it removes exactly the tasks it may have recalled.
I'd put it this way: Nemotron solves roughly half of what Flash Next does, and that is as close as I got. Even the benchmark that was built to avoid memorised tasks needed a correction, and I still do not have a clean number.
What Nemotron was good for?
So, in semi-empirical tests my agent teams' leader agents told me that "please can we switch back to Qwen3.8-flash-next for subagents, this Nemotron sucks" and I yielded under pressure. But I was still able to get some value out of the Nemotron trial.
It feld like a pretty good generic model for general office work: "fan out agents, use research skills and research topic X for me, ingest into wiki". Lighting fast.
Here is what it did on the B200 with simulated coding agents, each holding about 100k tokens of conversation:
| Agents | Nemotron: time to first token (median / slowest 5%) | Nemotron: speed per agent | Nemotron: all agents together | Qwen3.8-Flash-Next: speed per agent | Qwen3.8-Flash-Next: all agents together |
|---|---|---|---|---|---|
| 1 | 0.61 s / 2.22 s | 499 tok/s | 281 tok/s | 369 tok/s | 369 tok/s |
| 8 | 0.78 s / 2.44 s | 211 tok/s | 1,048 tok/s | 96 to 109 tok/s | about 800 tok/s |
| 32 | 0.94 s / 2.80 s | 71 tok/s | 1,566 tok/s | about 76 tok/s | about 2,400 tok/s |
| 64 | 1.03 s / 4.11 s | 41 tok/s | 1,614 tok/s | not tested | not tested |
No request failed at any level, 64 agents included. "Speed per agent" is how fast each answer is written, "all agents together" is the total output over the whole 12-minute test window. Past 32 agents the total hardly grows, so more agents just means slower agents: 71 tok/s each at 32, 41 at 64. The 64 is the limit set in the server. The GPU's memory could hold roughly 100 such conversations at once. Most of each prompt was already cached (87.6% by the engine's own counters), so the first-token times are for warm sessions, not for a fresh 100k prompt.
The Qwen3.8-Flash-Next columns come from our own tests on the production B200 before it went live, so it is a different test with different counters, and only indicative. Read that way, Nemotron is faster per agent for one to eight agents and level at 32 (71 against about 76 tok/s). Its total output is lower at one agent (281 against 369) and at 32 agents (1,566 against about 2,400), and higher at eight (1,048 against about 800).
But can it REALLY be used for coding? Nope.
What then?
Few months ago I burned some zillions of tokens to create our own benchmark tool, but then abandoned it as its figures would not have meant anything to anyone out there, and I would have had to burn even more zillions of tokens to build some kind of a leaderboard.
But was this the right decision? At least my tasks were tasks that have not been published online ...
If anyone has any good ideas on how to get some reliable metrics to compare models, please let me know!