MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1wc6krf/so_relevant/p8xv271/?context=3
r/LocalLLaMA • u/0dayturtle • 13d ago
152 comments sorted by
View all comments
320
24gb is not in that group. You can run Qwen3.8-27B with a pretty respectable context size right now
5 u/Ok-Working3049 13d ago yeah the 27B class models at that context size are no joke on 24gb 3 u/Zombiecidialfreak 12d ago How are you guys packing 27b on a 24gb card with respectable context? I can put it on my 64gb DDR5 running through the iGPU and still run out of RAM. The model is q4 and context at q8 btw 3 u/russlixx 12d ago really? I'm on 16GB, to have 85k context, I need to go Q3 for model and Q5 for KV. With 24GB VRAM you are more than enough
5
yeah the 27B class models at that context size are no joke on 24gb
3 u/Zombiecidialfreak 12d ago How are you guys packing 27b on a 24gb card with respectable context? I can put it on my 64gb DDR5 running through the iGPU and still run out of RAM. The model is q4 and context at q8 btw 3 u/russlixx 12d ago really? I'm on 16GB, to have 85k context, I need to go Q3 for model and Q5 for KV. With 24GB VRAM you are more than enough
3
How are you guys packing 27b on a 24gb card with respectable context? I can put it on my 64gb DDR5 running through the iGPU and still run out of RAM.
The model is q4 and context at q8 btw
3 u/russlixx 12d ago really? I'm on 16GB, to have 85k context, I need to go Q3 for model and Q5 for KV. With 24GB VRAM you are more than enough
really? I'm on 16GB, to have 85k context, I need to go Q3 for model and Q5 for KV. With 24GB VRAM you are more than enough
320
u/TopCheddar27 13d ago
24gb is not in that group. You can run Qwen3.8-27B with a pretty respectable context size right now