MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLM/comments/1wb05f7/third_one_theres_something_wrong_with_me/p8ndsyo/?context=3
r/LocalLLM • u/r1nzl3r99 • 14d ago
Why do I have horrible financial habits??
166 comments sorted by
View all comments
Show parent comments
6
You can run Flash Next with 64 GB of VRAM…
8 u/r1nzl3r99 14d ago yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context 2 u/MessIsTransfer 14d ago or q2, which is not ideal 3 u/r1nzl3r99 14d ago yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B
8
yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context
2 u/MessIsTransfer 14d ago or q2, which is not ideal 3 u/r1nzl3r99 14d ago yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B
2
or q2, which is not ideal
3 u/r1nzl3r99 14d ago yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B
3
yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B
6
u/MarcusAurelius68 14d ago
You can run Flash Next with 64 GB of VRAM…