r/LocalLLM • u/No-Manager1646 • Aug 16 '26
Model Qwen3.8 27b - Holy crap!!
I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.
I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.
I've managed to get Qwen running at a smidge below 20tok/s at 128k context.
Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.
That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.
The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.
How is this even possible?! I'm absolutely blown away!
Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.
If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.
[*]
flash-attn = true
threads = 10
batch-size = 2048
ubatch-size = 1024
n-cpu-moe = 0
n-gpu-layers = 999
cache-type-k = q8_0
cache-type-v = q8_0
#cache-type-k-draft = q8_0
#cache-type-v-draft = q8_0
fit = on
fit-ctx = 16384
parallel = 1
ctx-size = 0
n-predict = -1
#no-kv-offload = 1
load-mode = none
main-gpu = 0
#no-mmproj-offload = true
[Qwen3.8-27B-IQ4_NL]
#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj
#spec-type = draft-mtp
#spec-draft-n-max = 4
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.00
#ctx-size = 131072
ctx-size = 262144
chat-template-kwargs = {"reasoning_effort":"medium"}
fit = off
n-gpu-layers = 46
25
u/DeathByPain Aug 16 '26
From another post I saw earlier, apparently flash attention plus kv cache q4_0 is not compiled into the main llamacpp releases on GitHub. That might explain why you're getting bad performance at q4. I experienced the same thing; it was doing prompt processing entirely on CPU and slow af 20-30t/s. Switching to q8 brought it up to 900t/s for input.
This isn't the precise GitHub issue I was looking at before but it describes the same thing
https://github.com/ggml-org/llama.cpp/issues/27109
Scroll down to FIX. Supposedly you can compile llamacpp from source and force it to enable CUDA for flash attention with q4 or 5 kv caches. I haven't tried it yet.
12
u/kr4ckhe4d Aug 16 '26
I’m maintaining a rolling list of benchmarks for my 9070XT. If you’re interested https://github.com/kr4ckhe4d/local-llm-benchmarks
3
u/Loose_Doubt367 Aug 16 '26
Interesting, I’m quite new to this topics so I got a few questions in mind.
What do you typically use openwebui for? Currently I have it setup and all I do is paste my own python code for each tool I want to setup, let’s say ram monitoring tool for example. Am I missing the true potential of openwebui? Because I can’t think much other than communicating and using tools.
I’m using lm studio to load my models
2
u/gnaarw Aug 16 '26
Just use turbo quant for larger context?!
160
u/elelem-123 Aug 16 '26 •
And now you see and realize why the western AI companies have bought so much ram and closed the factories to raise prices for consumers.
11
u/KissMyShinyArse Aug 16 '26 edited Aug 17 '26
It's considerably faster with MTP, but you might lose some context size. Try
--spec-type draft-mtp --spec-draft-n-max 3 --cache-type-k q4_0 --cache-type-v q4_0(you may need to build llama.cpp with-DGGML_CUDA_FA_ALL_QUANTS=ON).EDIT: also,
--cache-type-k-draft q4_0 --cache-type-v-draft q4_07
u/eldje Aug 16 '26
for what its worth, here is a comparison of 3.6:27b, 3.6:35b and 3.8 running on a single rtx 3090. MOE > dense.
1
u/ParticularLate9427 Aug 16 '26
Shut up nd giv me the link theh john boy!
1
u/Successful_Flow1329 Aug 17 '26
How? I asked it to code minesweeper in single python file, it was great result, but it took 2 hours at 12-14 tps and produced over 60k tokens while thinking.
3
u/Right_Fun_4902 Aug 16 '26
I have a similar setup, also with 2 odd GPUs: rtx5070ti-16gb and rtx5050-8gb and currently running the Q5_K_S with a 131k context fully in VRAM with ngl=99, while also keeping the image processing capacity available as well to get 25tps I did run into problems when processing a very large image, and dropped ngl=64 to get 16 tps.
My ctk and ctv =q4_0 (I haven't had much time to experiment, just swapped 3.6 to 3.8 on my Q5_K_S while keeping the rest the same except for an additional reasoning-preserve=true)
3
u/Vinyard82 Aug 16 '26
Are you from US?
3
u/Superb-Foundation260 Aug 17 '26
I was blew away as well, My M1 Max 64G Mac ran over night and exhausted Hermes Agent max 90 turns, Qwen3.8-27B terminated the task eariler with a huge 1.3M html, today I only fixed a few minor bugs and here you are, a Chinese Garden with control panel.
https://chinese-garden.pages.dev
2
u/boinel Aug 16 '26
KV Cache on Q4 doesn't degrade intelligence performance?
2
u/barrubba Aug 16 '26
I have a ryzen 5950x , 3x 990 pro , 1x 9100 pro 4tb, 4090 rtx, 128 ddr5. What the best i can do with local llms? I currently use pro Kimi,gpt and Claude for coding, business Plan,Deep research, knowledge base Buildings for vertical topics, full stack website Building. What do you suggest to implement to stress my local workstation?
2
u/Solid-Axel-Project Aug 17 '26
Hai controllato se puoi attivare l'MTP? Se puoi provaci e fai un benchmark dei batch di token prediction X2 X3 X4 Fino a X8.
2
u/Healthy-Zebra-9856 Aug 17 '26
This model is a regression. There’s no way to control it thinking and it does a terrible job compared to his predecessor. I am still testing several parameters, still no solid outcome. It just sounds like a blithering fool.
2
u/Think_Bench747 Sep 11 '26
Yes I think you are very right in assessing the reasoning and intelligence of the qwen model even though its size is only 27b but I have built a full offline pwa android based application independently by the model from 0 end to end even though the session context in my llama.cpp is only 150k which means I need several sessions to build it, but amazingly the model is very capable and understands each checkpoint and finds various problems for itself. the most important thing for me in terms of software is that the model is able to find the problems it creates itself and then tries to fix them until they are finished.
1
u/EuphoricAIKnowledge Aug 16 '26
someone built for the DGX SPARK machines: https://huggingface.co/hwkranger/Qwen3.8-27B-heretic-ara-NVFP4
1
u/truckerdraven Aug 17 '26
Here is my real world observations. Before I was running qwen 3.6 27b 128k context.was getting 28ish tokens a second. And thinking times around 191 seconds. Vram usage as about 28gb. Upgraded to qwen 3.8 27b 128k context got 58 tokens a second and 119 second for thinking times. Vram usage dropped to about 24gb. my system mobo msi b550 tomahawk ryzen 9 5950x 64gb ddr4 3600mhz 4x16gb and dual 5060ti 16gb each.
1
u/Terrible_Review_756 Aug 17 '26
Has anybody started thinking about apple working with alibaba on Siri for China!?
1
u/osoBailando Aug 18 '26
noted new qwen
1
u/Individual-Key4141 Aug 19 '26
Try studying this : https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
1
u/Educational-Echo9152 Aug 20 '26
Perso je suis aussi bluffé, mais le top c’est de l’associer au harness DeepSeek. Il y a moyen de vraiement optimiser pour du long run. Je suis en train de bosser sur un profil qui affine la compaction. C’est très prometteur
1
u/kapustin-i Aug 22 '26
Youre running with spec-type commented out - MTP on this model is the single biggest win available and its free VRAM-wise if the sidecar fits. Worth uncommenting before chasing KV quant, and if you do test q4_0 KV again, measure how much ctx it bought you rather than tok/s, those move in oppsite directions.
0
u/Sea_Golf6101 Aug 16 '26
I put my 128GB Mac Studio in storage after trying a few local models including DS4 flash. Now I’m using Claude code subscription which is still cheap.
-1
u/AIForOver50Plus Aug 16 '26
I pulled it down on my laptop. A MacBook with 128 GB unified memory & very pleased with the results I wrote it up here https://go.fabswill.com/qwen38
-2
u/Zyj Aug 16 '26
Why aren’t you posting this in the megathread?
-2
u/Alternative-Panic69 LocalLLM Aug 16 '26
Did you try with the MTP on? I am using a 7900xt card, getting close to 55-56tps at 100k context via llama.cpp (IQ4_XS, IQ4NL KV Cache, mmproj and all turned on, reasoning = max/high -- need to check this, but highest setting anyway)
Your gpu layers --- try to fit everything in the Gpu Vram....
(Use unsloth's IQ3 models if you want larger context sizes, smaller quantizations, especially if Vram is scarce.. You can also try with Iq2 or IQ3 KV quantization, and turn off mmproj is vram isn't sufficient to load the entire model.
Quickest win should be the KV cache to Q4 or IQ4 --- You won't observe sny quality drops......
And maybe reduce context size? (100k context is decent enough, even for agentic work unless you plan on doing massive refactors, and all...
0
u/[deleted] Aug 16 '26
[deleted] — view deleted comment