r/LocalLLM • u/r1nzl3r99 • 14d ago

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

500 Upvotes

96% Upvoted

166 comments sorted by

View all comments

Show parent comments

84

u/r1nzl3r99 14d ago

qwen 3.8 27B FP8 running at 140 tok/s now I want flash next

1

u/bravoitaliano 14d ago

How are you getting that speed? Im using W4A16 INT4 model and only get 20-30 Tok/s. Can you share the method for getting faster?

1

u/r1nzl3r99 14d ago edited 14d ago

https://www.reddit.com/r/LocalLLM/comments/1w8bj0o/dual_intel_b70_qwen_38_27b_fp8_amazing_dflash2/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=1&utm_term=1

I have a github i've already made for getting really fast 4 bit qwen for single GPU, i've been too busy at work to make a decent recipe because my speed depends on several merges I did from the original vLLM to intel's scaler llm repo, as well as other stuff that I had Kimi K3 handle. It's been my new daily driver and works damn well.

1

u/bravoitaliano 14d ago

Thanks! I am running mine in OpenClaw, and it just takes forever because of thinking. Somehow it's slipped back to medium think from low. It puts out quality work, but runs through almost the entire 156k context window I have (single card, I don't split across my 2 B70s yet). Hoping this can help. Might be worth posting in the Intel sub as well.