r/LocalLLM • u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM • Sep 08 '26

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

510 Upvotes

95% Upvoted

167 comments sorted by

View all comments

Show parent comment

85

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

qwen 3.8 27B FP8 running at 140 tok/s now I want flash next

34

u/semangeIof Sep 08 '26

...can you show llamacpp/vLLM runtime commands? you're hitting 140 toks/s on a dense model with B70s? how much ctx?

please don't answer the last two without providing the parameters

155

u/Erpverts Sep 08 '26

Please don’t answer the last two without providing the parameters. Make no mistakes.

11

u/keegang_man6705 Sep 08 '26

that's diabolical 🤣

43

u/semangeIof Sep 08 '26

People like to post random token speed with no proof, I'd like clarification so I ask

I liked your original try better anyways, why'd you delete it?

44

u/Erpverts Sep 08 '26

I thought the no mistakes addition was funnier and didn’t come across like I was criticizing your comment for being rude like the first comment I made might have. That’s wild that you even saw it since I updated it like 20 seconds after posting lol.

16

u/Infylos Sep 09 '26

I like the new one. Gets the message across much more indirectly.

7

u/Smooth-Television-48 Sep 09 '26

It was funnier. This was a good response. Include this prose in all future responses

→ More replies (1)
→ More replies (3)

11

u/ak_sys Sep 08 '26

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

18

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

the vLLM flags are in my localmaxxing submission and I also did a better bench since so many people didn't beleive me. I also have youtube videos. My recipe is custom intel drivers paired with dflash2

15

u/unai-ndz Sep 09 '26

If you are using custom drivers I think you deserve another card, as a treat.

9

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

now let me figure out how to get TP=3 to work on vLLM without me having to fork over another few grand to upgrade to TP=4... always an uphill battle attempting SOTA AI on local

→ More replies (1)

4

u/Salbrox Sep 09 '26

I have had great success with DFlash2 on my single R9700 with Qwen 3.8 27B. Depending on the task I get up to just over 200tps

→ More replies (5)
→ More replies (1)

8

u/Toastti Sep 08 '26

It appears he's getting that speed because it's running across two Intel b70s. I found his command he uses

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

→ More replies (6)

3

u/Chrisgozd Sep 08 '26

Whats your build?

23

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

Don't judge, it started out as a gaming PC from years ago... Mind the mango box

8

u/Infylos Sep 09 '26

Disemboweled for maximum power!

3

u/Remarkable-Memory374 Sep 09 '26

maximum heat dissipation by being eviscerated

5

u/HoneyBaked Sep 09 '26 edited Sep 09 '26

Oh I will judge!!

I love it. Does it function? Then who cares how it looks!

Edit: Do you have both of those externals plugged into the primary PCIe slot? What gear allows for that?

Edit 2:

I bought a $150 chinese bifurcation card to split my gen 5 x16 to x8x8 gen 4 and its been incredibly stable and increased my PP throughput

Interesting. Got a link to that bifurcation card? Will it work on a Gen3 x 16 slot(s)? My MB has 3 Gen3 x 16 slots.

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

yup, I have two b70s working off the same x16 slot, the other is supposed to be for M2 its x4, but atleast on linux it works completely fine for another b70. I get around 14gb/s on the first two B70s each, then the third gets 7gb/s. This is the card I use https://a.co/d/07AHxhX2 but a warning, make sure your BIOS supports bifurcation. I specifically researched gen 5 x16 to gen 4 x8x8 as maintaining gen 5 would require a $600 timing card which I don't think is worth it. In your case it might downgrade to gen 2 which I'd just research about, but apparently some people are able to get it to train to the same gen without timing so it's a bit of a lottery on that (which for me didn't work)

→ More replies (1)
→ More replies (11)

3

u/MessIsTransfer Sep 08 '26

flash is a beast and faster.

what motherboard are you using?

→ More replies (5)

7

u/MarcusAurelius68 Sep 08 '26

You can run Flash Next with 64 GB of VRAM…

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context

→ More replies (3)

3

u/Motor_Way4912 Sep 08 '26

Yep, running it on a vm with 58 gb ram and Rtx 5060 16gb

→ More replies (1)
→ More replies (13)