As you know, if you have read my previous blog entry, we had a spare B200 on our hands last week and were pushing some tests through it. One of them was this rather exciting test of using Ripple 65B. I also had spotted that ppl in the interweb had been asking for a comparison between Ripple and Qwen 3.8, so we were happy to oblige.

What Ripple is?

It is a dense model of 65 billion parameters. Preview, they say. The vendor claims 76.5 on SWE-bench Verified, which sounds tempting. Unfortunately we did not run our benchmarks on Verified, but on the same SWE-rebench tasks that we ran the Nemotron in our previous post.

So continuing in the same vein of not doing as we're told by the vendors, we did not run it quite the same way as its makers do. Ricursive's own server setup is the full-size model on two B200 GPUs, with a context of one million tokens. We had one B200 at our disposal, so we went with the 4-bit build (the same model with its floats squeezed down, so it takes 57GB of memory) and the drafter model (a small helper model that guesses the next few tokens so the big one only has to check them). That fit on a single B200 just fine, and one agent got 190 tokens per second out of it in a single stream.

Now follows a personal opinion somewhat grounded in reality. So I do a lot of semi-autonomous or even fully autonomous agentic coding using various software factories, and whilst the manufacturers of Ripple claim that the model runs on a single DGX Spark, and the claims have been verified as far as I know, I think the decode speeds from a single DGX Spark are way too low to get any realistic software-factory-style work done. I have seen reports of 34 to 36 tokens per second when coding, or 52 tokens per second shared between three requests in parallel. Which is of course great, but this does not enable you to run parallel agents to build your software at any great speed.

Raw results

The same 110 SWE-rebench tasks as in the previous post, same agent and same settings. We also ran Qwen3.8-27B, the smaller sibling of Flash Next, through them. Ripple, the 27B and Nemotron were capped at a context of 131,000 tokens. Flash Next ran on our production server, where the cap for these runs was not recorded.

ModelSolved, out of 110
Qwen3.8-Flash-Next89 (81%)
Qwen3.8-27B79 (72%)
Ripple 65B40 (36%)
Nemotron 3.5 Super36 (33%)

Taken at face value Ripple lands next to Nemotron, at about half of what the two Qwens solve.

Results are not right for Ripple

Okay, so now we have to do a little bit of interpreting of the results. So 51 of the total of 110 runs ended with no patch for Ripple. 34 ran out of context, which was set to 131,000 tokens. 17 runs hit the 32,000-token output cap three times in a row while thinking. So 17 runs basically out-thought themselves. Ripple also seems to be keen on steps: about 93 per task, whilst the other models that we ran through the tasks took between 40 and 60. The good news is that it seems that there's no memorisation of the tasks.

Time for some cheating

So then we decided to do some cheating because we wanted to get some measurable results for Ripple. We decided to rerun the 34 overflow tasks at a larger context window. We reran them with a 262,000-token context window, which clearly helped. It managed to solve 20 of them. So we were up to 60 tasks solved out of total 110 tasks.

This enabled us to then step back and take the 59 tasks that we were relatively certain that neither Qwen3.8-27B nor Qwen3.8-Flash-Next had memorised as part of their training and look at the results. Now Ripple is performing much better.

ModelSolved, out of 59
Qwen3.8-Flash-Next44 (75%)
Qwen3.8-27B41 (69%)
Ripple 65B40 (68%)
Nemotron 3.5 Super21 (36%)

There's no statistically significant difference between Ripple and either Qwen model.

Important to also note here that only Ripple got the larger context. Qwen3.8-27B might have benefited also from a larger context window, as it got only 131,000 tokens for its runs. So not a fair comparison. Not an exact science this...

Conclusions, are there any?

So I'm not 100% certain on what I feel about this Ripple model. Whilst we were able to run it on a single B200, we only got enough oomph from the box for about five Ripple sessions in parallel, all capped at 131,000 tokens. At a 262,000-token context window we were able to get only three parallel sessions for agents. To me this sounds like quite an expensive model to get value out of. We get about 60 parallel agents running on the Flash Next model on the same hardware, so I'd say the value for money case is clearly on Flash Next's side. It would be interesting to hear if anybody has run it on the vendor's own two-B200 setup with full context?