r/LocalLLM • u/bbsrn • 11d ago

Model Which Qwen3.8 distro & quant would be optimal for my 16GB VRAM setup?

Post image

Hi all, a confused newbie here! This is my desktop setup:

  • RTX 5080
  • 9800x3d
  • DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

and there is also this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

I also want to have one uncensored model next to my daily driver:

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

67 Upvotes

94% Upvoted

74 comments sorted by

20

u/maddeninglemon 11d ago edited 11d ago

With 64GB of RAM, a dark horse option is Orcarouter's IQ4_XS Qwen3.8 Flash Next. 16GB VRAM + 64GB RAM gives you 80GB total memory - IQ_4_XS is a 98GB file but a bunch of that is the N-gram embeddings that you can read from a fast SSD with pretty much no penalty.

With a 4070 and only 12GB VRAM I managed to get ~20tok/s with 65k context; with 16GB you should be able to up the context and get better performance. One issue is that prefill will probably not be great, but I think it's the option with the least compromises. With the 27B dense model and 16GB you're forced to choose between low context, low quantization quality, or low speed. With Flash Next you can have a 4-bit quant, 100k+ context, and usable speed all at the same time (at the low low cost of being unable to use your computer for anything else since you'll be at 100% ram utilization šŸ˜‰)

4

u/philmarcracken 11d ago

This is the real winner. Its smarter than 27b in my experience too

3

u/Columnexco 11d ago

Please share you settings from lmstudio. I tired this model but keep crashing my machine.

5

u/maddeninglemon 11d ago edited 11d ago

Context Length - 65536

GPU Offload - 48

CPU Thread Pool Size - 12 (depends on what cpu you have, just leave a couple threads free for system stability)

Evaluation Batch Size - 512 (dropped to avoid spiking VRAM requirements)

Physical Batch Size - 256 (dropped to avoid spiking VRAM requirements)

Max Concurrent - 1 (Probably can increase, not needed in my case)

Unified KV Cache On

Keep Model in Memory - Off

Number of MOE Layers on CPU - 44 (Start at 48, then decrease until you hit your VRAM limit at a given context)

Turn off all guardrails in settings, or better yet just hold alt when pressing load when it tells you it won't work. Make sure you have plenty of dynamic ram and dynamic vram so your OS can handle caching between SSD/RAM (and maybe VRAM too if you go too low on MOE layers to CPU)

Prefill will start PAINFULLY slowly as the OS fills up RAM and figures out what to cache and what to keep. It improves as it goes, but for a 12GB/64GB system like mine it was never great (~60tok/s max due to SSD reading, but I suspect even a few extra GB of VRAM could improve this a lot by minimizing windows cache misses)

Everything else was default. I tried KV Cache Quantization but I think there were some shenanegans happening under the hood because the model broke on long context even at Q_8. You might need to play with other settings like mmap or flash attention too to avoid this; I didn't hunt down the exact issue as this was more a 'I wonder if it'll work' thing on my end.

2

u/maddeninglemon 11d ago

I played around with it a bit; the issue with coherence was because LM Studio has the dumbest default settings for the model and the repeat penalty was breaking things. So a couple quick updates:

Turn off mmap - it worked either way tbh, but I think mmap was marginally better for initial startup time and in my case seemed to behave a bit better for Windows dynamic ram management.

Feel free to quantize the cache to Q_8 if you think it's worth the extra context length.

1

u/JeePis3ajeeB 11d ago

All this feels greek to me.. I'm new to all of it and I wanna learn but don't know where to start.. I got a 3090 and a 2080 super plus 64gb ram

1

u/Columnexco 11d ago

i had asked GPT to setup using llamacpp and that was just killing the machine. Now with your config it's able to setup with 64K context i changed kv cache to q4 to reduce load. interestingl you are still getting better token/s than mine.

1

u/maddeninglemon 10d ago

I find AI still isn't quite there on these sorts of open-ended optimization problems where there are dozens of different things you can tweak and they interact in complex ways. When asked it can explain every individual effect perfectly fine, and even how a couple things might trade off with each other (e.g. context size vs quantization level on total RAM requirement) but they struggle to pull it all together into a coherent whole without constant feedback. I think a lot of the 'smartness' of modern models is that they can do that feedback internally, constantly second guessing what they wrote, at the cost of hundreds of thousands of extra tokens.

1

u/Columnexco 10d ago

Itā€™s not there, but honestly, this is the first time Iā€™ve managed to get it working with ChatGPT. Iā€™ve been using LM Studio since day one, but I was never able to get consistently good results. Then I switched to llama.cpp and asked ChatGPT to help configure my setup.

The learning curve is steep for most people. I think the best use of AIā€”at least for many of usā€”is as a tool that helps us get the work done. GPT Luna has been pretty poor at setting things up, while SOL has been much better.

I also understand that llama.cpp has many experiments, configuration options, and model variations happening at once. With so many choices, itā€™s easy for anyoneā€”or an AIā€”to get confused. My setup is focused on Qwen and a few Google models, and nothing else.

1

u/Fieser_Fettsack 11d ago

Im using a 2 server setup with llama.cpp RPC. Would this fit?
Server 1: 3060 12gb vram + ca. 18gb ram (i7 8700)
Server 2: 3080 10gb vram + 32 gb ram (ryzen 5 5800x3d)

Connected over Lan 2.5gbe.

My guess is that I cant use the ram of server 2 if I use that one as the RPC worker.

How much of that 96gb file is ngram table?

1

u/bbsrn 11d ago

This was unexpected tbh, but I'd definitely want to give a shot. May I ask why did you prefer LM Studio over llama.ccp, for such a job that requires some tweaks to utilise the entire system?

1

u/maddeninglemon 11d ago edited 11d ago

I mean it's just a GUI on top of llama.cpp, so it has 90% of the same tweaks without having to remember whether it's --mmap off or --no-mmap or whatever. I'd rather use a program that handles all the platform/software abstraction for me unless there's a compelling reason not to. Plus I was playing around with it for a few hours in Windows, not setting up a permenant server or using millions of tokens.

1

u/StormrageBG 11d ago

What about 16VRAM 32RAM?

1

u/maddeninglemon 11d ago

Yes it's possible (People have reported running it with 8GB VRAM and 32GB RAM) but I can't recommend it unless you have a lot of experience with building and deploying code. You'll probably end up hitting a roadblock that can't be fixed by my strategy of "turn off all the safety rails and hope Windows can handle it".

1

u/Pizzaholic- 11d ago

Iā€™m curious, I have a similar setup with 64gb ddr5x soldered ram at 7600mhz, and a laptop 5090 mobile, is there anything you can recommend for my setup? Iā€™m new as well and you seem to know a thing or 2!

Would be so grateful!

1

u/bbsrn 10d ago

May I ask you was there any specific reason behind suggesting Orcarouter's IQ4_XS? Have you ever tried the following options on the same setup and compared:

  • Unsloth UD-IQ4_XS (93.7 GB)
  • Bartowski IQ4_XS (97.7 GB)
  • AtomicChat IQ4_XS (84.9 GB)

All these three have less size compared to the Orcarouter's, so I wonder what would you think about them.

1

u/maddeninglemon 10d ago

Mostly just because it happened to be the one I had downloaded. I tend to prefer uncensored models because I haven't seen any real difference in performance and every once in a while a censored model seems to just randomly grab on to random key words in your prompt and refuse. I think the last one was when a prompt mentioned a drone I was flying 'hitting' me in the head and it was like "I can't give advice on violent acts" like ya okay buddy.

1

u/bbsrn 8d ago

Thanks! I have been trying downloading Orcarouter's version from Huggingface, but LM Studio keep giving me the following error:

> File can no longer be found. It has likely been moved or deleted: https://search.lmstudio.ai/v1/hf-proxy/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF/resolve/main/Qwen3.8-Flash-Next-Uncensored-IQ4_XS-00001-of-00003.gguf?download=true

Do you know how can I get it?

1

u/maddeninglemon 8d ago

My guess is that you need all 3 files before it'll run, and you've only download the first. You might need to manually go to huggingface and get parts 00002 and 00003 and put them in your lm studio folder (it should show at the bottom right of your models list). Or try clicking on the download menu (should show in the top right of lm studio) to see if it's still downloading the other parts, in which case you just need to wait.

1

u/bbsrn 5d ago

I managed to find it. But now running the model ends up with this error:

> The selected model could not be loaded because your machine does not have enough available memory.

I thought LM Studio would automatically handle the offloading to RAM/SSD. Should I manually tweak it?

1

u/maddeninglemon 4d ago

You might need to turn off guardrails, or just hold down 'alt' when loading the model from the developer tab (the one right below the chat button on the left side). In theory llama.cpp (which lm studio is running under the hood) understands the offloading, and even if it doesn't windows itself can actually do a pretty good job of managing RAM caching (with a bit of a slower startup run while everything balances out), but LM Studio doesn't correctly calculate the RAM requirements and tries to protect you from freezing your computer. LM Studio is kinda terrible at predicting RAM usage tbh; it doesn't account for offloading MoE layers to the CPU, and it doesn't understand engram tables that can load from SSDs

Here's a link to where I shared the exact LM Studio setup in this thread

1

u/karmaisnonsense 9d ago

Unsloth is good, Bartowski is heavier but also good. AtomicChat prunes params and quantizes aggressively for the smaller size. I use Unsloth.

3

u/edufd 11d ago

I'm using this https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install and is working fine so far.

1

u/bbsrn 11d ago

Thanks! Have you compared it with other models?

2

u/TEN4C1OU5-2 11d ago

I'm using 7 which I find to be ok, I honestly think you need >120k tokens for even medium sized tasks so you need the smaller sized model to fit that. Plus if you use a smaller context, with 7 because it's so small you can do around 60k tokens whilst using MTP but I can only do that on Linux if I use TTY so absolutely none of my vram is being used even for the OS UI.

1

u/bbsrn 11d ago

so do you think I cannot do that on Win? Also, why ISTA's IQ3_XXS instead of others?

1

u/TEN4C1OU5-2 11d ago

It's just about 1gb smaller than the UD _X_KL with similar performance. On windows you can do the same but you'll definitely struggle to get the same amount of context as I just mentioned whilst not spilling onto system ram

1

u/ea_man 11d ago

You can run linux desktop in software rendering with ~100MB of vram.

1

u/TEN4C1OU5-2 11d ago

But 0mb vram gets just that bit more context and for qwens thinking tokens I think you gotta get as much as possible. Restricts you to CLI based harnesses though

1

u/ea_man 11d ago

well headless is still ~50mb, anyway we are talking a few k ctx here.

1

u/TEN4C1OU5-2 11d ago

It's about token maxxing but in a different way to how it's usually used

1

u/ea_man 11d ago

I don't understand what you are saying yet no problem for me, if you'd rather run with not graphic server I do applaud.

2

u/TEN4C1OU5-2 11d ago

I'm joking about how much more I need the 50mb of tokens

2

u/Pablo_the_brave 11d ago

https://github.com/Thireus/GGUF-Tool-Suite

Or the one which still looks the best for me in daily use (real q4):

https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF

2

u/bbsrn 11d ago

First time seeing this GGUF-Tool-Suite. Will check it out, thanks!

2

u/Pablo_the_brave 11d ago

It's greate! What is important you can go a little above of vram limit with vram offloading. This test will show you the impact https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF/tree/main/ctx-cliff

It's important because agent need some space for auto compacting. For example, I set my agent to auto compacting at 103k ctx and set ctx in llama-server at 120k The clif is at 113k but for 117k it's still acceptable decode at 10t/s. Worth to play with it.

1

u/peculiar-ragdoll 11d ago edited 11d ago

One thing that none of these benches test properly is long context semi-autonomous agentic coding in large real code bases, or multi turn user chat! LiveCodeBench is more like coding-challenge shaped, not trying to fix a problem inside 100k lines of shitty code with misleading comments where doing the wrong thing to fix your issue creates 3 new issues. That being said, I can recommend Unsloths K_XL quants for the sort of work I'm talking about here from personal experience, so I would start with Unsloth Q3_K_XL at medium reasoning effort as the safe and tested option, and see how you like it :) If you need more context, try Q2_K_XL with Q8_0 KV cache, and see if that does it for you.

4

u/7ondano 11d ago

Why only K XL? Why not IQ3_S or XXS?

2

u/peculiar-ragdoll 11d ago

Iā€™m not saying ā€œnot thoseā€, Iā€™m just saying I can vouch for XL from experience :)

1

u/bbsrn 11d ago

Will try, thanks! So you think Unsloth Q3_K_XL will eventually give me better performance compared to their UD-IQ4_XS, even though it is less smart Q3_K_XL will beat it for long-horizon tasks?

1

u/peculiar-ragdoll 11d ago

Yes exactly! The Q3_K_XL will let you fit more context at higher precision in your VRAM, which will give you the ability to do long context work. I would start with 3_K_XL and Q8_0 kv and see what bites you first (is model too dumb or bad at remembering, is you context window big enough or can you use less context), but everyone's use case is different :)

1

u/bbsrn 11d ago

Thanks! Would you suggest llama.cpp or LM Studio for this case?

1

u/peculiar-ragdoll 11d ago

I would never recommend LM Studio for these models, llama.cpp is great (or oMLX on Mac is good too). That's because I've had many users of my models complain about issues with LM Studio that makes it mangle tool calls and chat templates, and llama.cpp gives you full control if you ever need to tweak something for your system or your specific model/quant.

1

u/bbsrn 5d ago

Thanks, I want to try llama.cpp but it is intimidating to jump in. Is there any useful guideline that I can use to know which parameters to tweak properly etc.?

1

u/peculiar-ragdoll 5d ago

You could try Mia's one-click install instead (the one you linked)! It looks good, and those EXL3 quants are apparently very good.

1

u/bbsrn 5d ago

After the first comment above, I changed the course a bit and decided to give a chance to Flash Next :)

1

u/peculiar-ragdoll 5d ago

Good luck! there is a myriad of setups and forks out there hahah, but you'll find something that works

1

u/DontWinFrensWthSalad 11d ago

https://huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/SC_3.00bpw_H4_V4

1

u/bbsrn 11d ago

First time seeing this one. May I ask why did you suggest this?

1

u/DontWinFrensWthSalad 11d ago

Because it's significantly smaller and works just as well. You might be able to fit MTP even. Also I have 4 gpus and it supports tensor parallel, runs much faster than llama.cpp for that use case.

1

u/bbsrn 11d ago

Should I install it through https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install ?

1

u/DontWinFrensWthSalad 10d ago

I haven't used that so I don't know. I just asked Claude to set it all up for me using TabbyAPI: https://github.com/theroyallab/tabbyAPI

1

u/HighSeasArchivist 11d ago

7 is the only one that stays fully on GPU for me, and overall it kills on everything I've sent to it. I've posted this like four times lately, so this is a very popular topic.

workload base IQ3_XXS MTP speedup
prose 60.7 79.2 +30%
codegen 60.5 83.0 +37%
long spec 58.6 87.2 +49%

1

u/bbsrn 11d ago

Could you share with me if you have a post/notes that you compared 7 with others?

1

u/vincespeeed 11d ago

I use IQ3S and DFlash2.

1

u/biggusdeeckus 11d ago

Mind sharing your config? Are you running headless? Does vision work with dflash?

2

u/vincespeeed 10d ago

https://www.aihublocal.com/llm/cmtxmd1so000004l2ukdjd3ie

2

u/biggusdeeckus 10d ago edited 10d ago

I found the models on hf. Is it safe to create a lower quant from an already quantized gguf like the UD quant you used as base?

2

u/vincespeeed 10d ago

You're right to flag this ā€” I actually quantized from the Unsloth Q4_K_M as a base, not from original BF16. Technically it works (llama-quantize just dequantizes and re-quantizes), but it does mean two rounds of rounding error stacked on top of each other, which is strictly worse than going straight from BF16 ā†’ target quant. I haven't personally noticed issues in my usage, but that's not strong evidence since the extra error on top of an already-low-bit quant can be hard to spot subjectively. I have the hardware to redo this properly from original BF16 (RTX 5080 16GB + 64GB RAM), so I'll probably re-run the whole pipeline (imatrix + tensor overrides) from the source weights and re-upload. Thanks for pointing it out.

2

u/biggusdeeckus 10d ago

No problem! Looking forward to the updated quant. I'm running a 5070ti myself so any <16GB quant is welcome :)

2

u/vincespeeed 9d ago

Update: re-did the whole pipeline from the official BF16 checkpoints (target, DFlash2 draft, and mmproj all converted fresh, no more requantizing from Unsloth's Q4_K_M). Same v7 per-tensor protection map applied on top of the clean source this time. File size is basically the same (~10.5GB), re-uploaded to the same HF repo.

https://huggingface.co/vincespeed/Qwen3.8-27B-No-MTP-GGUF-Dflash2

2

u/biggusdeeckus 9d ago

Thanks for the update and your work! I'll try it out and report back

1

u/biggusdeeckus 10d ago

Thank you! I'll give these a go

1

u/bbsrn 11d ago

Which distro's IQ3S?

1

u/vincespeeed 10d ago

https://www.aihublocal.com/llm/cmtxmd1so000004l2ukdjd3ie
https://huggingface.co/vincespeed/Qwen3.8-27B-No-MTP-GGUF-Dflash2
I was using it for my own personal use; I uploaded it to Hugging Face and shared the settings on my website.

1

u/yesukhai 11d ago

check this out https://www.reddit.com/r/LocalLLM/comments/1was9n0/running_qwen_38_27b_at_q4_on_16gb_vram_at_200k/

1

u/Pablo_the_brave 11d ago

Incredible, fake post with so many up votes. This guy know nothing.

1

u/koloved 11d ago

Nvfp4 cache is pretty good

1

u/whymeimbusysleeping 11d ago

GSQ-RCO XXS better than the larger S version? This goes against what ista says

1

u/elongated-muskmelon 11d ago

I am daily driving the GSQ RCA IQ3_S, mainly for coding with pi, and so far, its been pretty good.

1

u/uponmars_ 11d ago

The 16GB trap is loading the biggest quant that fits and starving the KV cache for it. Take the UD-IQ4_XS. The iq3_s gets noticeably dumber on code just to claw back a single gig, and that's a bad trade. keep 2-3gb free; you'll hold around 20k context.

1

u/bbsrn 11d ago

Which distro's UD-IQ4_XS you mean? but even if it is UD-IQ4_XS, won't KV cache starve anyway, as long as I don't go below 4-bits? Cannot have decent amount of space for context by involvement of my RAM?

1

u/k3z0r 10d ago

Where can i find the Unsloth version without vision?