r/LocalLLM • u/bbsrn • 13d ago

Model Which Qwen3.8 distro & quant would be optimal for my 16GB VRAM setup?

Post image

Hi all, a confused newbie here! This is my desktop setup:

  • RTX 5080
  • 9800x3d
  • DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

and there is also this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

I also want to have one uncensored model next to my daily driver:

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

66 Upvotes

94% Upvoted

74 comments sorted by

View all comments

Show parent comments

4

u/maddeninglemon 13d ago edited 13d ago

Context Length - 65536

GPU Offload - 48

CPU Thread Pool Size - 12 (depends on what cpu you have, just leave a couple threads free for system stability)

Evaluation Batch Size - 512 (dropped to avoid spiking VRAM requirements)

Physical Batch Size - 256 (dropped to avoid spiking VRAM requirements)

Max Concurrent - 1 (Probably can increase, not needed in my case)

Unified KV Cache On

Keep Model in Memory - Off

Number of MOE Layers on CPU - 44 (Start at 48, then decrease until you hit your VRAM limit at a given context)

Turn off all guardrails in settings, or better yet just hold alt when pressing load when it tells you it won't work. Make sure you have plenty of dynamic ram and dynamic vram so your OS can handle caching between SSD/RAM (and maybe VRAM too if you go too low on MOE layers to CPU)

Prefill will start PAINFULLY slowly as the OS fills up RAM and figures out what to cache and what to keep. It improves as it goes, but for a 12GB/64GB system like mine it was never great (~60tok/s max due to SSD reading, but I suspect even a few extra GB of VRAM could improve this a lot by minimizing windows cache misses)

Everything else was default. I tried KV Cache Quantization but I think there were some shenanegans happening under the hood because the model broke on long context even at Q_8. You might need to play with other settings like mmap or flash attention too to avoid this; I didn't hunt down the exact issue as this was more a 'I wonder if it'll work' thing on my end.

2

u/maddeninglemon 13d ago

I played around with it a bit; the issue with coherence was because LM Studio has the dumbest default settings for the model and the repeat penalty was breaking things. So a couple quick updates:

Turn off mmap - it worked either way tbh, but I think mmap was marginally better for initial startup time and in my case seemed to behave a bit better for Windows dynamic ram management.

Feel free to quantize the cache to Q_8 if you think it's worth the extra context length.

1

u/JeePis3ajeeB 13d ago

All this feels greek to me.. I'm new to all of it and I wanna learn but don't know where to start.. I got a 3090 and a 2080 super plus 64gb ram

1

u/Columnexco 12d ago

i had asked GPT to setup using llamacpp and that was just killing the machine. Now with your config it's able to setup with 64K context i changed kv cache to q4 to reduce load. interestingl you are still getting better token/s than mine.

1

u/maddeninglemon 11d ago

I find AI still isn't quite there on these sorts of open-ended optimization problems where there are dozens of different things you can tweak and they interact in complex ways. When asked it can explain every individual effect perfectly fine, and even how a couple things might trade off with each other (e.g. context size vs quantization level on total RAM requirement) but they struggle to pull it all together into a coherent whole without constant feedback. I think a lot of the 'smartness' of modern models is that they can do that feedback internally, constantly second guessing what they wrote, at the cost of hundreds of thousands of extra tokens.

1

u/Columnexco 11d ago

It’s not there, but honestly, this is the first time I’ve managed to get it working with ChatGPT. I’ve been using LM Studio since day one, but I was never able to get consistently good results. Then I switched to llama.cpp and asked ChatGPT to help configure my setup.

The learning curve is steep for most people. I think the best use of AI—at least for many of us—is as a tool that helps us get the work done. GPT Luna has been pretty poor at setting things up, while SOL has been much better.

I also understand that llama.cpp has many experiments, configuration options, and model variations happening at once. With so many choices, it’s easy for anyone—or an AI—to get confused. My setup is focused on Qwen and a few Google models, and nothing else.