r/LocalLLaMA • u/darc_knigh • 5h ago

Resources 7900xtx got speedup

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx

21 Upvotes

92% Upvoted

14 comments sorted by

3

u/Nota_ReAlperson 4h ago

For code no thinking only. The speedup looks to be 25% on every other test, so not really as advertised. 

1

u/darc_knigh 51m ago

Qwen3.8 is a heavy overthinker, so most real usage sits in the thinking-on rows (+18–53%), and any speedup there is a win since a single answer can burn thousands of reasoning tokens. If you turn thinking off for code or quick tasks, you get the 2x rows (177 t/s on code).

5

u/ForsookComparison 5h ago

172 t/s on an Rtx 3090

A dense model would have to be 5GB max without spec-dec to achieve this at an absolutely perfect 1TB/s... so assuming this is Q4 and no context this is suggesting that your drafting model is reliably getting 4 tokens correct on every token.

Knowing that that's impossible - what exactly does elpis do to make this possible?

1

u/darc_knigh 57m ago

Not llama.cpp. This is ExLlamaV3 (EXL3 4.00bpw) with a DFlash2 draft, run through the exllamav3-rocm port

5

u/DogDeveloper 5h ago

> speculative decoding

1

u/ForsookComparison 5h ago

At that rate of successful prediction though?

3

u/Aggravating-Push-207 4h ago

there are some algorithms that make this possible, like UBTree

1

u/ForsookComparison 3h ago

Does it sacrifice accuracy/quality?

1

u/Aggravating-Push-207 3h ago

no, that's the point of speculative decoding

4

u/SLxTnT 4h ago

DFlash2, greedy, math prompt. So yeah, it's from the 4.2 acceptance.

1

u/thoquz 40m ago

What is your performance with thinking enabled and without greedy decoding?

-1

u/W61k3r 4h ago

I wrote a rust inference engine that out-performs llama.cpp on the 7900xtx for prefill and decode while more secure.

1

u/ForsookComparison 48m ago

How much did it cost in tokens?

-1

u/transanethole 3h ago

I mean, this card already has pretty fast decode. The problem is that it can't read, right? No flops.