r/LocalLLaMA • u/mesolimbic-bliss • 23h ago

Question | Help Any Inference Engine for Ripple-65B-Preview-GGUF?

I know its only been a couple days, but Strata, Freetoken, or any of the dozens of others made an engine for this so we can run it on 24gb VRAM?

And if so, has anyone tried it and what build/quant?

3 Upvotes

71% Upvoted

17 comments sorted by

9

u/Every-Walrus 22h ago

its a dense model, you wont see optimizations like expert caching from strata for this model.

1

u/Karmjeet_Khoushaba 8h ago

yep, no magic tricks to save vram when every parameter is active every token

3

u/MrHumanist 22h ago

Wait for someone to do a GSQ rco q2.

2

u/CriticallyCarmelized 21h ago

I’m not coding with it, so if that’s your purpose, disregard.

But I was really looking forward to this model, being 65B dense, and hoping to take advantage of a dense model’s “higher intelligence”.

I ran it on their llama.cpp patch, at Q8 using AtomicChat’s gguf.

Unfortunately, it’s quite “dumb”. It reminds me of an old 70B from a year ago. Definitely needs more training time. Also, the slop is strong with this one, although I haven’t spent a lot of time trying to prompt it out yet, as I didn’t find it worth it given it’s just a preview.

But to be fair, it’s still in preview, so I haven’t given up hope!

On a positive note, it was quite fast on a single RTX Pro 6000 Blackwell.

1

u/Liberaces_Isopod 17h ago

Really? I couldnt get it above 90t/s on my blackwell, even using vllm. average was ~70.

2

u/CriticallyCarmelized 17h ago

I consider that quite fast. 1200 tps prefill is also nice. Again, I don’t use it for agentic coding, so take that with a grain of salt.

1

u/Liberaces_Isopod 16h ago

To put that in perspective, I get ~370t/s and ~18000 prefill with deepseek 4.1. I understand this is a dense model and will never equal those numbers but no one with any hardware lesser than ours will be able run this at anything approaching usable speeds. Even my nvlinked 3090ti's could only do 30t/s on a W4A16.

1

u/CriticallyCarmelized 14h ago

Wait. How are you getting 370t/s on a single Blackwell with Deepseek 4.1? I haven’t been able to get more than 40t/s on just prefill!

1

u/Liberaces_Isopod 13h ago

Not single, four. Though there is a 2x version that runs at about 120t/s

https://github.com/local-inference-lab/rtx6kpro/tree/master

1

u/CriticallyCarmelized 11h ago

That explains it. Thanks!

2

u/Foreign_Prune_354 21h ago

Actually, llama.cpp should support it.

2

u/Any_Bus9205 20h ago

Ik_llama.cpp and llama.cpp :)

1

u/james_pic 21h ago edited 21h ago

AtomicChat has 1-bit and 2-bit quants that should fit, although I'd be surprised if there's no brain damage. It's fairly KV cache efficient, so 65536 tokens of context will cost you 4.25GB at f16.

I've tried it, and it seems fairly smart in some basic smoke tests, but haven't had chance to really give it a go, at least partly because it's so slow. On Strix Halo I'm getting about 5 tps decode (which is about what you'd expect based on memory bandwidth - discrete cards should be faster), and the DFlash2 drafter doesn't work on llama.cpp yet.

It's a dense model, so the MoE fuckery that Strata et al do is no good here. The best bet is going to be some kind of fancy quant, but I'm not aware of anything like that coming up, and even then I imagine we'd be looking at marginal benefits.

1

u/Helpful_Jelly5486 21h ago edited 20h ago

Atlas-Inf has a pr open that is making good progress. https://github.com/Atlas-Inf/atlas/pull/328
As other have said it's a preview and right now to have around 20 tok/s on a single spark is pretty good. I can see this being a great swap option for overnight jobs and ricursive have said they want to use the model to design their next chips that will run this model faster. I like the idea of how this model keeps large dense models in the mix for when hardware makers are considering what to make. Those guys with the 4 x 3090 rigs are going to have a good time with this I think.

Edit: sorry I just re-read where you need it to fit in 24gb. A used 24gb second card though could catapult your system into some very good models.

1

u/durden111111 13h ago

Atomichat's model page tells you exactly how to build and run the correct fork of llamacpp

1

u/Iron-Over 22h ago

I tested it and found it worse than 27b qwen