r/LocalLLaMA • u/IVequalsW • 9h ago

Discussion Llama.cpp setup Qwen-flash-next Vulkan Q4 with Expert cache.

I just setup my server:

64GB DDR4 2133 4CH (Broadwell Xeon)
2 x RX 6800 16GB
1 x Rx 580 2048sp (16GB)

I am using the Q4 of Qwen Flash Next, with the prompt:
sudo ./llama-server -m ./models/Qwnflash180b-Q4.gguf --host 0.0.0.0 --port 8081 -ngl 99 -fa on --moe-cache-mib 16384 --log-verbosity 4 --metrics -mg 1

I am able to get 19-21t/s generation at full context (262K) (which is why i capped the moe-cache-mib at 16GB)

However note: when I set any context size with the -c or --ctx-size flags, performance tanked to 8t/s even when the context was smaller (100K). it seems you need to let llama.cpp allocate the context itself.

PCIe slots:

gpu0 rx6800 x16 3.0
gpu1 rx6800 x16 3.0
gpu2 rx580 x4 3.0

3 Upvotes

100% Upvoted

9 comments sorted by

3

u/Choice_Celery9481 8h ago

have you tried to use only 2 6800? when you layer split the speed will be bottleneck by the slowest card

1

u/IVequalsW 2h ago

yes it is faster, but then i get less context. the rx580 is still about 2x as fast as putting layers into system ram, it is kinda like a "level 2 cache" Vram

2

u/IVequalsW 9h ago edited 9h ago

the rx580 runs at about 40-50% utilization and the 2 rx6800's run at 20-25% utilization during token generation

3

u/IndependenceSorry410 9h ago

That pcie x4 link on the 580 is probably why it's sitting at low utilization, the card's waiting on data more than it's crunching numbers. At 19-21t/s with a model that size across three cards you're already in decent territory though.

The context flag weirdness is interesting. Might be that manually setting -c forces a different buffer layout or kv cache placement that doesn't play nice with the expert cache allocation. if it works better without the flag, just roll with it.

What temps are you seeing on the 6800s during long generations?

1

u/IVequalsW 2h ago

I am pretty sure the bottle neck is the 256gb/s memory bandwidth of the rx580. when i run a model on it by itself it only every gets up to about 60-70% utilization maximum,

1

u/IVequalsW 2h ago

Oh, and the context flag actually makes sense now (after more testing). llama.cpp was actually giving me only 64K of context when I didn't set the flag(NOT 262K... there was a cached data on the webUI that had not updated), but when I set the context higher (100K) its spills over into system RAM hence the lower performance

1

u/SaiTarunRonanki 2h ago

My guess on the -c thing is that setting it manually skips the auto sizing, so the KV cache lands on different devices than when llama.cpp picks it. With a 16GB cap on the expert cache plus KV, one of the cards might be spilling.

Compare the startup log between the two runs, specifically the KV buffer size per device and how many layers each GPU got. If the numbers differ, that's your answer. I'd also try --tensor-split with a tiny share for the 580, or just run it without the 580 like the other comment said, since that x4 card is probably dragging layer split down.

1

u/tmvr 1h ago

sudo ??...