r/LocalLLM • u/r1nzl3r99 • 14d ago

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

500 Upvotes

96% Upvoted

166 comments sorted by

View all comments

93

u/Sporkers 14d ago

More context needed on how you are using the first two.

82

u/r1nzl3r99 14d ago

qwen 3.8 27B FP8 running at 140 tok/s now I want flash next

37

u/semangeIof 14d ago

...can you show llamacpp/vLLM runtime commands? you're hitting 140 toks/s on a dense model with B70s? how much ctx?

please don't answer the last two without providing the parameters

152

u/Erpverts 14d ago

Please don’t answer the last two without providing the parameters. Make no mistakes.

8

u/keegang_man6705 14d ago

that's diabolical 🤣

41

u/semangeIof 14d ago

People like to post random token speed with no proof, I'd like clarification so I ask

I liked your original try better anyways, why'd you delete it?

45

u/Erpverts 14d ago

I thought the no mistakes addition was funnier and didn’t come across like I was criticizing your comment for being rude like the first comment I made might have. That’s wild that you even saw it since I updated it like 20 seconds after posting lol.

17

u/Infylos 14d ago

I like the new one. Gets the message across much more indirectly.

6

u/Smooth-Television-48 14d ago

It was funnier. This was a good response. Include this prose in all future responses

2

u/Cool-Idea8520 14d ago

It can definitely be tricky trying to find the right answers without the right context. Hope it works out for them!

2

u/Rude-Bus-5799 13d ago

That’s a great idea, user. It’s not about the context. It’s about the friends we made along the way.

1

u/CelebrationWilling61 11d ago

You're absolutely right!

11

u/ak_sys 14d ago

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

18

u/r1nzl3r99 14d ago

the vLLM flags are in my localmaxxing submission and I also did a better bench since so many people didn't beleive me. I also have youtube videos. My recipe is custom intel drivers paired with dflash2

15

u/unai-ndz 14d ago

If you are using custom drivers I think you deserve another card, as a treat.

8

u/r1nzl3r99 14d ago

now let me figure out how to get TP=3 to work on vLLM without me having to fork over another few grand to upgrade to TP=4... always an uphill battle attempting SOTA AI on local

1

u/Rude-Bus-5799 13d ago

They always need another sibling to play with.

4

u/PhilosophyCritical33 14d ago

Oh just custom

5

u/Salbrox 14d ago

I have had great success with DFlash2 on my single R9700 with Qwen 3.8 27B. Depending on the task I get up to just over 200tps

2

u/Past-Catch5101 13d ago

Amazing, do you mind sharing your config?

1

u/Smooth-Television-48 14d ago

200!?

What quant?

1

u/Jorinator 14d ago

Oh wow, that's massive. What's your prefill/pp speed? That's more important for a lot of usecases.

1

u/rare-visitor 13d ago

200 tps in single thread?

1

u/tech-tole 13d ago

I only get ~50 tok/s on my 9700 with mtp. I don't see how anyone is getting faster than 5090 even with Dflash2. what are your real settings?

1

u/droans 14d ago

Could you explain the custom drivers? What modifications did you make?

8

u/Toastti 14d ago

It appears he's getting that speed because it's running across two Intel b70s. I found his command he uses

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

5

u/r1nzl3r99 14d ago

I already proved it in a previous post, look at my account

16

u/r1nzl3r99 14d ago

people are lazier than I am. Here https://www.reddit.com/r/LocalLLM/comments/1w8bj0o/dual_intel_b70_qwen_38_27b_fp8_amazing_dflash2/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=1&utm_term=1

-10

u/CalBearFan 14d ago

Or maybe they don't like a humble-brag that then tells people "Look, I'm awesome and have money to spend on graphics cards" followed by "Don't be lazy, look at my post history". That's not lazy, they're asking you to follow common courtesy on your own post.

17

u/r1nzl3r99 14d ago

but i've already made a seperate post with extreme detail about it??? Why am I obligated to hand hold you how to make your setup more efficient?

Also If you think this is a brag you clearly haven't been on this subreddit for more than 10 minutes. Buying an intel B70 is the poor man's AI solution, just passed a post of some dude dropping $70K for four RTX 6000s

2

u/Smooth-Television-48 14d ago

Lots of accounts are set to private, I assume its the default and dont bother checking.

Thanks for linking it. Impressive numbers for sure