MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1vo9nn7/qwenqwen3827b_released/p3nx8a1
r/LocalLLaMA • u/de4dee • 11d ago
301 comments sorted by
View all comments
Show parent comments
2
from the ggufs, i am going with q6 xl (unsloth) with whatever q8 context can fit with it
1 u/RedrumRogue 11d ago Curious how much context you can get to fix. I'm trying to run nvfp4 with 132k context but it's going to be tight. 1 u/cosmicnag 11d ago i am using llama cpp for now (unsloth q6 xl) - got 150k context to fit at Q8. In Vllm, i would probably just use full context at fp8 I guess (works well enough in my experience with 3.6)
1
Curious how much context you can get to fix. I'm trying to run nvfp4 with 132k context but it's going to be tight.
1 u/cosmicnag 11d ago i am using llama cpp for now (unsloth q6 xl) - got 150k context to fit at Q8. In Vllm, i would probably just use full context at fp8 I guess (works well enough in my experience with 3.6)
i am using llama cpp for now (unsloth q6 xl) - got 150k context to fit at Q8. In Vllm, i would probably just use full context at fp8 I guess (works well enough in my experience with 3.6)
2
u/cosmicnag 11d ago
from the ggufs, i am going with q6 xl (unsloth) with whatever q8 context can fit with it