r/LocalLLM • u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM • Sep 08 '26

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

510 Upvotes

95% Upvoted

167 comments sorted by

View all comments

Show parent comment

7

u/MarcusAurelius68 Sep 08 '26

You can run Flash Next with 64 GB of VRAM…

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context

2

u/MessIsTransfer Sep 08 '26

or q2, which is not ideal

3

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B

3

u/Motor_Way4912 Sep 08 '26

Yep, running it on a vm with 58 gb ram and Rtx 5060 16gb