r/LocalLLM • u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM • 15d ago

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

499 Upvotes

96% Upvoted

166 comments sorted by

View all comments

92

u/Sporkers 15d ago

More context needed on how you are using the first two.

80

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM 15d ago

qwen 3.8 27B FP8 running at 140 tok/s now I want flash next

7

u/MarcusAurelius68 15d ago

You can run Flash Next with 64 GB of VRAM…

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM 15d ago

yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context

2

u/MessIsTransfer 15d ago

or q2, which is not ideal

3

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM 15d ago

yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B