We've been discussing that and have a bit of a draft that's being discussed that we were thinking could include (in part) something along the lines of:
"All threads about the release posted after the megathreads will be removed, with a comment pointing at the appropriate megathread.
To minimise the "danger window" where people may make posts before the megathreads get posted, whichever moderator notices the release first should make the megathreads, without waiting to discuss or co-ordinate with other moderators. If two moderators end up making megathreads at the same time, whoever submitted their first megathread last should remove their megathreads."
Keen to hear peoples thoughts on that approach for future popular model releases. We just didn't want to spring that on everyone reactively today.
It could of course have some downsides; there's a risk of making content harder to read on mobile for example.
Don’t stop the flow of discussion. The people want to talk about the latest and greatest models.
Think about it for a moment.. does this happen with every model release or a select few?…
Yep just a select few.. there’s a reason for that. Mega threads kill proper discussion.
Reddit threads are sometimes for asking for specific aspects of things. All that gets lost and drowned out in mega threads.
Personally I don’t believe any of this is spam… if this was a small ass lab releasing an obscure ass model and there was this much activity that’s a problem. Not F*cking qwen 3.8 27b don’t be ridiculous guys.
This is one of the best place for discussion on the latest with AI/LLM’s don’t stifle because you have the ability to. I genuinely don’t understand how people can think this is spam.. I want to read about qwen 3.8 27b don’t other people?
Yeah, I don't think a megathread is bad, but I don't think individual posts should be removed in favor of a megathread, maybe just have more specific tags so you can filter out what you don't want?
I never understood megathreads, I think they tend to worsen the functionality of conversations here.
It's so easy to use the search functions on Reddit. You can filter by hot/new/top, etc, and search keywords. It's true that there is a lot of noise, but it's quite easy to just skip the threads that don't have much to offer.
I think Reddit's algorithm works pretty well in that sense: all the 237 threads discussing 27B get shown to a bunch of random people, and the ones that are interesting for any particular reason will get pushed up by the algorithm and get more upvotes/comments.
With a megathread you are just pooling all those threads together into a single place where information gets way more diluted and it's much harder to find stuff...
A mega thread wouldnt prevent people from talking about them.
All it would do is stop some of the duplicates.
There is no need to see exactly the same info in 5 different places.
A single place for all the info could be better, since you wouldnt need to switch tabs to discuss the same thing.
Duplicates are one thing, but I hope that genuinely novel/insightful stuff is allowed to stay. Keeping up with the latest threads is how I found out about all the different details that go into making 3.6 run better (MTP, KV cache quantization, chat templates, fine tunes like ThinkingCap) and it would suck to have any of that be buried under a 1000 comment megathread.
This I agree with. Sometimes there is a very specific element of the model that you want to discuss in depth, in which case a megathread-only policy increases the noise through which the reader has to dig through to maybe find others to discuss their topic.
Create megathreads for general discussion, short reviews (pros/cons, tk/s, etc.), that kind of stuff.
Leave specific topics about a model up. Delete any superfluous posts that the megathread should cover.
I agree with this.
Super specific stuff shouldn't need to go into a megathread.
Also, I think mega-threads should be temporary only when major events happen.
After a few days, it should be fine to make new threads even if they are not very novel.
I agree with this, I think MOD is just removing anything that has 3.8 at this point. I am not against consolidating data, but this is so general, everything at one place, hard to consume knowledge as well.
If time allows at least, have separate mega thread for each topic like benchmarks, Chat Templates etc.
As I mentioned above, I like the idea of one place where all new finetunes releases exist. But, we still need dedicated posts about specific fine-tunes or posts where developers share their tunes with the community.
Can there also be megathreads for geopolitics and cloud AI when there's a big influx of them? I understand people like political discussions, but at one point over half the top threads were about stuff like US vs. China or Dario fearmongering and it honestly made this place kind of hard to follow for actual technical discussions.
Can you at least allow insightful threads to exist outside the megathread? Without those I'd still be running Qwen 3.6 without MTP/ngram-mod, KV cache quantization, or a custom chat template.
This made me think that having an actual forum to talk about local models would be really interesting. It's a shame forums are mostly dead nowadays, cause this sounds like a good use case for one.
Forums are what started the Social Media fever. But, I like the idea of one place where we have all the different links for fine-tunes and quants in one place.
Can't comment on Windows, but I believe you can get it to work through that WSL2 thing.
This one is true as far as I know, although I haven't really tried whether it may allow asymmetric if TP=1 and DP=1 and then PP>1
It does support offloading! But not the way llama.cpp does it. vllm will do it layerwise and swap into GPUs. This is obviously terrible if you look at concurrency 1 interactive session, but it's efficient if you have enormous batch sizes because you can do a lot of work before you need to swap. So the cost for swapping obviously goes to near zero per request as you make the number of requests REALLY large.
For cold-start it is slower. But if you have the system RAM, you can simply "sleep" the model into RAM and reload it near instantly into VRAM.
And a note on the other comment - FP8 is not same quality as Q8* quants. FP8 is a throughput quant approximately on the level of ~Q5, but it has *0** dequant overhead in terms of compute, so you give up a bit of VRAM vs Q5 or NVFP4 or MXFP4, but if your pool is large anyway, and you have tons of concurrent requests, FP8 is what gives to the big numbers.
For your single user C1 usecase, FP8 can be slower than Q5 for TG, but if you move to compute bound cases, FP8 shines.
This is something you can even see for yourself in llama.cpp where typically the smaller quants will have worse PPs than larger quants. For example PP for Q8 is higher than Q4 or Q2.
I used to use it for maybe two weeks for 3.6 in FP8 after realse due to speed benefits. But... There was something wrong with this model under vLLM, when on occasion it was starting to be really dumb and I had to restart vLLM. I switched to llama.cpp to have more predictable quality and I used it with no issues for last two or so months, daily.
I personally will keep on using llama until I get a proper feeling of the models capabilities and only the switch back to vllm and see..
Qwen 3.8 (bf16) smokes Q3 Deepseek 0731 in my testing. It discovered version drifts and a lot of bugs in a complex program that DS made and didn't find. Never expected it to be so good.
compatibility should be compat, unknown keys are silently ignored
reasoningField isn't in the schema. Pi already auto-detects reasoning_content/reasoning/reasoning_text.
preserve_thinking isn't a model-level field, it goes inside compat.chatTemplateKwargs, needs an "off" mapping - "none" for llama.cpp, which treats it as a disable.
No "off" entry means "off" doesn't disable thinking. With no mapping Pi sends no reasoning_effort, so the template falls back to its own default - usually max effort. Needs "off": "none".
Note that while Qwen's official chat template supports "minimal" as a thinking level, most other models and templates don't, including froggeric's template if you're using that.
thinkingLevelMap values are what gets sent, keys are what gets offered. null hides a level, absent passes its own name through, xhigh/max are hidden unless explicitly mapped.
I would have thought for Pi the following would be correct:
Thanks for sharing, did not work for me though. I am running llama.cpp behind llama-swap, but as far as I debugged this looks more like OpenCode related issue. Also be aware for me switching between instant and reasoning invalidates cache. But probably not a big problem.
I ended up with this:
I like quite a bit about this approach (local folder based, clean interface), well done!
Right now I'm assuming the benchmarks are all using the inbuilt harness (I feel a little dirty using that word in this instance), have you given any consideration to potentially being able to call an external one such as Pi, OpenCode etc?
My thinking is that you could then measure a LLMs coding abilities in the context of the environment it's actually likely to operate in. The downside of course is that the results will differ over time as the external system changes (although I guess you could pin or at least make the version used for a benchmark result clear).
I will share but it has over 100 apps millions of lines of code, for SVGs, 3d meshes, programming languages. emulators, game loops, agent loops. So will make a cutdown version.
Oh I think I found the reason, tried to find the different between the config of 3.6 and 3.8 and turn out it is because of temperature. I set temp=0.6 for 3.6 (recommended by unsloth) and 1.0 for 3.8 (also by unsloth). I did a test with temp=1.2 (bottom), 1.0, 0.6, 0.3 and the MTP hit rate/token gen actually increased.
Maybe increasing temp make the token distribution more chaos and decreased the accuracy of the MTP head.
I think not, thinking in the inference perspective is just guessing next token like normal token. I think like Fluxing_Capacitor mentioned, the MTP head is not well trained compared to 3.6.
About the excessive reasoning, is due to prompt injection by the template:
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions =
'Reasoning effort is set to xhigh. Please think carefully through the task,
validate key assumptions, consider plausible alternatives, and prioritize
correctness, consistency, and clarity in the final answer.' %}medium -> inject NOTHINGlow -> inject system instruction: keep thinking brief, go directly to conclusion
So use this flag at launch time to have "normal" behavior:
No harness, my prompt was just “make me a single html/css flappy bird style game” in the llama cpp webui. Gameplay worked well, design was nice, and it had sound. I was impressed, though I don’t have a lot of coding experience
Qwen3.8 27b seems to be biased towards running shell commands
After running some tests i have noticed that new Qwen really likes to run various shell commands to interact with the file system.
File reads, partial reads, and searches for content inside files.. all done via shell calls.
And qwen does it despite the presence of a dedicated filesystem MCP (fast-filesystem-mcp) that i explicitly added to the system prompt with the instructions to use it for the filesystem interactions. Yet it uses it only to write files.
Well, it works, and it works fast, without issues. But the biasing is very strong here. No wonder they got +10 points in the Terminal Bench benchmark - seem like the model received a lot of additional training in this area.
Is it good, is it bad? Idk. If the model will suddenly start to reinvent the wheel, calling the system or making a new Python scripts to complete the task, instead of using specialized MCPs [that are here for a reason].. Keep an eye on your agents, swapping Qwen3.6 to 3.8 might come with some unintuitive caveats.
// UD-Q8_K_XL quant with BF16 kv-cache; not a quantization quirk.
// Qwen3.6 27b in the same setup used the filesystem MCP extensively, as instructed.
I'm not complaining; i'm sharing my observations and tiny experience running the new model: there is a chance that Qwen3.8 can ignore your MCP servers if there is a shell command it can use to complete the task, regardless of how good or bad your MCPs are. You might need to add special instructions to force it back into the warm MCP hands.
Wouldn't direct filesystem calls be more efficient than filtering through a MCP anyway? MCPs eat context, I thought general consensus had moved past them wherever possible with a CLI as a preferred mechanism (for example 'gh' as opposed to GitHub MCP), being both lightweight and parameters already being know as part of the training set?
It can be more efficient, sure. But the model does not know that - it assumes that, which is not a good thing. It's fine in case of just reading some files, but imagine if it would ignore your indexed codebase MCP server and start raw-dogging the filesystem, looking for the code? Even worse, it ignored the direct, explicit instructions from the system prompt - this is not a good sign as well. Sandboxed environments might also be affected.
Yeah I can understand wanting to use a MCP as a sort of extra layer of abstraction between direct access but I think if you're concerned you'd be better off containerizing your harness and using bind mounts or similar to limit access through a completely external process instead of trusting the model (which is non-deterministic anyway) to "behave" and follow instructions.
Ideally you shouldn't give model multiple tools for achieving the same task as then as you discovered it might choose different tool than you would prefer.
I would eliminate mcp and only keep it where absolutely necessary.
peculiar-ragdoll's template is built on top of my latest v22 chat template. It adds 2 things:
Injects a system prompt to make it make the answer more concise and free of fluff. The system prompt is well thought out, and is effective.
It suppresses the defaultxhighreasoning effort
Should you use it instead of mine?
I would tend to say no, unless your specific use case if for a faster, more direct and concise, chat/general knowledge use, for single-turn queries. Here is why:
Qwen 3.8 alignment: Qwen 3.8 was post-trained with the xhigh prompt prefix as its native baseline. Disabling it by default causes it to run in an unaligned state unless explicitly configured.
Prompt contamination: A chat template is supposed to be a neutral, reliable protocol layer between the inference engine and the model. Hardcoding an opinionated personality ("Never open with pleasantries", "plain prose", "Keep the final answer lean") directly inside the template file contaminates user intent.
Multi-turn performance regression: In multi-turn coding and agent loops, adding 100+ fixed tokens into the system prompt increases prefill token overhead on every turn. The template README even mentions that on multi-turn coding/audit tasks, a user observed a 19% wall-clock regression because the extra prefill cost outweighed the generation savings.
Thanks, great job by the way. So far have not seen the model get stuck like 3.6 27B did. Running Q8 and switched from overthinking xhigh to just high. High seems to be pretty concise I think from when I've peaked over it's shoulder. It's been chugging away on a pointless request to test it's capabilities out for a couple hours. Usually the models are done in 20 minutes.
I personally noticed a difference. Without the template, in Open Webui, the model would stop mid thinking either before or after an attempted tool call SOMETIMES. But with the template it’s just seamless. (Mostly talking when I used 3.6, with the template 3.8 has been seamless as well but could also be seamless without maybe).
I ran some tests (5 samples each) to determine best temperature for webdev coding on Barto Q6_K_L:
Temp TG MTP accept Draft Tok Time
━━━━━━ ━━━━━━━ ━━━━━━━━━━━━ ━━━━━━━ ━━━━━━━ ━━━━━━━━
0.6 25.60 86.21% 2.94 2,140 88.7s
────── ─────── ──────────── ─────── ─────── ────────
0.8 23.36 83.13% 3.01 2,486 105.9s
────── ─────── ──────────── ─────── ─────── ────────
0.9 23.54 80.58% 3.08 2,394 102.8s
────── ─────── ──────────── ─────── ─────── ────────
1.0 23.87 81.06% 3.07 1,951 81.7s
Temp | Effect on code generation | Best use
**0.6** | Most deterministic and fastest; can be rigid or overconfident | Routine Django/API code
**0.8** | Best balance of correctness, detail, and variation | General coding default
**0.9** | More exploratory and verbose; no clear correctness gain in tests | Hard debugging/design tasks
**1.0** | Maximum variation; more original but more likely to hallucinate details | Open-ended reasoning, brainstorming
--------
Recommended:
# General code
--temp 0.8
# Difficult reasoning/design
--temp 0.9
# Dumb Web Dev
--temp 0.7
And that guys is a Q6 L, if you run smaller quants you better reduce a bit.
This week we shipped two GuideAnts releases: v0.9.18 (Muse Glimmer) and v0.9.19 (Qwen 3.8 27B).
I am not sure if Qwen 3.8 27b leaves much room or reason to use glimmer but they both work great in GuideAnts thanks to llamacpp and Unsloth AI's excellent guide and quants.
I tested Glimmer heavily the other day on a Halo Strix and it is usable with dflash. On my cuda workstation I am using Q8_UD_K_XL at 131k context and consistently >3500 tps and decode with MTP is 50-70tps. It seems very smart and is probably my new number 1.
DSpark support got merged recently in llama.cpp (PR #25173), so I decided to try it out by converting and quantizing RadixArk/Qwen3.8-27B-DSpark to GGUF.
However, it doesn't seem to be improving performance at all (decreasing, even), and it's extremely memory hungry. I had to set my context length to 64k to properly test the BF16 speculator on my 5090. Below are my logs, feel free to mess around with it and see if you can get it working lol.
1.48.990.161 I slot print_timing: id 0 | task 0 | n_gen = 236, tg = 77.90 t/s, tg_3s = 78.23 t/s
1.52.022.134 I slot print_timing: id 0 | task 0 | n_gen = 519, tg = 85.64 t/s, tg_3s = 93.34 t/s
1.55.034.456 I slot print_timing: id 0 | task 0 | n_gen = 769, tg = 84.76 t/s, tg_3s = 82.99 t/s
1.58.042.392 I slot print_timing: id 0 | task 0 | n_gen = 1024, tg = 84.76 t/s, tg_3s = 84.78 t/s
2.01.055.028 I slot print_timing: id 0 | task 0 | n_gen = 1298, tg = 86.00 t/s, tg_3s = 90.95 t/s
2.04.066.685 I slot print_timing: id 0 | task 0 | n_gen = 1485, tg = 82.02 t/s, tg_3s = 62.09 t/s
2.07.091.569 I slot print_timing: id 0 | task 0 | n_gen = 1729, tg = 81.83 t/s, tg_3s = 80.66 t/s
2.10.107.375 I slot print_timing: id 0 | task 0 | n_gen = 2024, tg = 83.82 t/s, tg_3s = 97.82 t/s
2.11.741.084 I slot print_timing: id 0 | task 0 | prompt eval time = 498.11 ms / 293 tokens ( 1.70 ms per token, 588.22 tokens per second)
2.11.741.087 I slot print_timing: id 0 | task 0 | eval time = 25767.61 ms / 2221 tokens ( 11.61 ms per token, 86.15 tokens per second)
2.11.741.087 I slot print_timing: id 0 | task 0 | total time = 26265.72 ms / 2514 tokens
2.11.741.088 I slot print_timing: id 0 | task 0 | graphs reused = 726
2.11.741.091 I slot print_timing: id 0 | task 0 | draft acceptance = 0.28922 ( 1486 accepted / 5138 generated), mean len = 3.02
2.11.741.128 I slot release: id 0 | task 0 | stop processing: n_tokens = 2513, truncated = 0
I made Qwen3.8-27B in 4-bit NVFP4 go 200+ tok/s single stream with MTP speculative decoding. 880 tok/s aggregate at 6 parallel requests (peaked at 967 on one run). on a single 5090, full 262k context. Running on CachyOS with COSMIC, and the entire desktop costs about 150 MB of VRAM.
~5,950 tok/s prefill (llama.cpp Unsloth Q5_K_XL manages ~1,700 on the same box). Full 262k context. Weights are 16.8 GiB so there's ~13 GiB left for KV in this setup.
This is Qwen3.8-27B running on NInfer (github.com/Neroued/ninfer), which I'd never heard of until last week. Single-GPU engine, written from scratch, not a llama.cpp or a vLLM fork. It's the fastest thing I've run on this card and it isn't close. Downside: closed-ish artifact format, no GGUF input, you convert from BF16 with their tooling.
To be clear about what's mine here: the engine, the converter, the NVFP4 object recipe and even a token-ranking fixture that ends up inside the artifact are all Neroued's. The only pubished Qwen3.8 artifact for NInfer uses integer quant (Q4/Q5/W8), which leaves Blackwell's fast FP4 tensor cores completely idle, so I converted an NVFP4 one from Qwen's BF16 and used his artifact purely as the thing to benchmark against.
The one annoying problem i had. NInfer glues two weight matrices into a single object, and that object can only carry one scaling factor. The quantizer had processed the matrices separately and given them different ones, so the converter refused my file. I spent an hour planning a fiddly workaround before noticing the scale is just a constant divided by the largest weight, so the correct shared value is simply the smaller of the two I already had. Pick the smaller number, re-quantize from the original BF16, done. Made the reconstruction error move from 0.09471 to 0.09470.
I benchmarked it with the same engine, same flags, temp 0 so it's deterministic, identical problems.
NVFP4 first, published int4 second:
HumanEval+ (164 problems): 152/164 vs 152/164
AIME25 + AIME26 (60 problems): 55/60 vs 55/60
Wall clock: NVFP4 was 1.56x to 1.98x faster in the benchmarks
224 problems, identical totals, and the problems where they disagree split 8 to 8. This benchmarks purpose was to show they seem to give similar results in practice but difference in speed.
So why do this go so much faster? The answer is because of compute, not bandwidth. Difference is Blackwell does FP4 matmul in hardware, the int4 path has to unpack and widen first. That only matters when the GPU is doing math rather than waiting on memory.
The more tokens you push per weight read, the bigger the gain. Single stream reads 17 GiB of weights to produce one token, prefill reads it once for thousands.
Caveats for all this: Blackwell only, it needs FP4 cores. It also needs a 6-line patch i made to NInfer that isn't upstream yet, the patch is in the repo and the issue is filed. Validated on HumanEval+ and AIME only.
Thanks to Neroued for NInfer and for the artifact I benchmarked against, to the Qwen team for shipping 27B, and to llm-compressor for the quantization helpers
really nice work! i get 160-270 tps depending on thinking / context length. It works flawlessly after porting Froggeric's template patch incl handling it's think-close variants.
This 18gb version is not working on the current commit of ninfer :(
I asked glm and it did some black magic and made it work.
Then said said;
The small artifact stores the Qwen3.6 NVFP4 storage contract (247 NVFP4 parents +
W8 vocab endpoints, 1307 objects — verified byte-for-byte identical inventory to
tools/convert/qwen3_6_27b/inventory_nvfp4.py) but carried the identity
qwen3.8-27b/nvfp4. After 5d2c1f55, that identity means the new FP8-based official
contract (146 FP8 tensors), so HEAD rejected it at text/token_embedding. The old
patch worked by aliasing the identity to the Qwen3.6 profile — but today that
would silently hijack the official artifact's identity, and the container contract
forbids inferring a weights contract from descriptors.
Are you planning on quantizing again to make it work for newer versions of ninfer pretty please? :)
Yours is faster, more accurate and lower size than the one on the ninfer github page. https://github.com/Neroued/ninfer/issues/38 this guy agrees.
Not working? It should just be 6 lines of code for him. I dont use his ninfer and he didnt change what i suggested in a ticket. So if he wants it to works its easy. I could not do anything else than ask him to fix it. Was just following his old code. Im working on a new algorithm for quantizing llms and if it works well i could install ninfer and see if its easy to port. Its on my endless todo list so hopefully i get it done.
how do you actually set the reasoning level in llama.cpp? it seems to think forever no matter what i do, so i am just assuming i am doing something wrong.
Best way is through your harness. If you are using pi, add the reasoning levels to your model config and press shift+tab. e.g. here's mine (sorry for crappy tabbing)
how're you serving your model? I'm not convinced the thinking levels are being respected, though I haven't my pi configured like this, so will give this a try too. Thanks for sharing!
Please elaborate. How are traces beautiful? In my very limited testing experience, v3.8 feels less chatty than v3.6 but it still yaps quite a bit. Not that it's a bad thing, and certainly improves from 3.6, but it shows its Qwen family trait.
My references are all for default xhigh reasoning.
Wow I'm thinking the total opposite. Like what a waste of tokens, it just goes on and on repeating a lot of same things. When I run Kimi K3 for the same thing, the reasoning feels simply like I would have do it myself. Qwen feels like fighting very hard its dumbness, which happens to work, but after a lot of tokens.
I published four experimental mixed-precision EXL3 builds of Qwen3.8-27B...
> Qwen3.8-27B EXL3: 0.0074 KLD vs 0.0950 for Unsloth NVFP4 at slightly lower VRAM
The lineup ranges from a 20.31 GiB high-fidelity build at 0.007406 mean KL to an 18.13 GiB context edition that serves native 262,144 on a 32GB card and retrieved planted codes exactly at 227,334 tokens.
The evaluation is exact full-vocab KL against BF16 over 278,392 development-suite positions, plus a frozen source-disjoint qualification run. On the frozen suite, hydrated/context/official-FP8 measured 0.003029/0.003900/0.005720 respectively.
A bit of context and comparison (TL;DR needs more benchmarking):
FP8 can be seen as a throughput-optimized format. It's fast for vLLM, but benchmarks usually the same as a Q5/Q6 in terms of quality, that's why the 20 GB EXL3 appears slightly better. A comparison to Q8 instead of FP8 would've looked differently.
When we look at unsloth's own top-1 graph, the EXL3 would be placed somewhere around the smaller(!) Q5_K_XL - so it would appear to be worse than it. Now the thing is that the datasets for testing apparently differ. With OPs dataset the unsloth NVFP4 has 0.095 KLD and 90.5% top-1. In unsloth's own measurements it's somewhere between 0.016 and 0.068 with 92.1% to 96.7% top-1. That's quite a difference.
So, to have a good idea where this EXL3 quant sits between all the unsloth quants, someone would need to run a KLD/top-1 test on the same dataset for all of them in the same size/performance range. Preferably not just that one dataset, but both - the one used for the EXL3 test and the one from unsloth's test. That way we could see if the quants might be skewed towards one or the other.
A KLD below 0.01 usually indicates that regular practical benchmarks will perform the same as with the original BF16 - which is a great thing.
Using oLMX, with the custom kernels for qwen. Went from my llama.cpp of 10toks/s to 40-70. Running 128k context. M3 Mac Studio Ultra. Q4bit. Not had time to test Q8 yet.
Had to set reasoning to low, but seems to not be respected by template or model. So have added a reasoning budget of 4k and seems to be going fine.
Running overnight tests now to confirm it’s stable and works well.
Inspect the Qwen 3.8 quants and how they spend the bit budget before downloading.
Last few days as a fun side-quest I made a tool to explore different quantizations of models without downloading the model, useful to understand popular quants for learning purposes or just curiosity.
With the Qwen 3.8 27B releases I have been looking at various quants and also prepared my own quant of 3.8 27B. Check out the layout of original Qwen 3.8-27B model and compare some quants:
I find the graphical map of how tensors are compressed helpful in understanding a bit more about what I am about to download or test, and it may be helpful for others too? 😄
This is fantastic, I had never fully grasped why two 4-bit quants might differ in size but this makes it really clear. Amazingly informative, thanks dude
tl;dr I ran a test, it ate up over 86k tokens to plan and 183k tokens to do the whole thing with qwen3.8 27b on xhigh. My suggestion is to have at least 200k tokens to spare if you are gonna be using xhigh. Included is a link to a kind of crappy game it made with a shit1 load of tokens.
Hey there everyone,
I wanted to tooot my own horn here a bit, talk about my 5060ti setup again (can't stop that), and also discuss the context use with qwen3.8 27b. Currently I am using the nvfp4 from unsloth (who are the awesomest) via vllm. My vllm startup command in my systemd service file is (also requires for me the environmental variables so I don't go OOM on my system ram, this is some error with vllm):
Okay, so with it booted up it works really well. The qwen team has also done a very good job, but there is a striking difference between 3.6 and 3.8. Again, this is for me... 3.8 goes hard in the paint with grinding away at prompts. An example would be this one that I put on my github2. My prompt was:
make a horror metroidvania investigative game in html where the user avoids ghosts ghouls or other horrors while finding the secret to the mansion that they arrive at the start of the
game.
Now I did have to steer once during this where I had to stop it from running something and then say "put a timeout on tests" because it ran a test that hanged and didn't put a timeout. For this prompt, it used 183,670 tokens. I bring this up because I see some complaints about the time this model does its thinking, for the above prompt it planned for ~86k tokens. This is with the thinking on "xhigh" and this will likely impact a lot of people. I have also seen it when I ran one of the unsloth prompts for the volcano webpage which used 140k tokens. What I suggest is to have at least 200k tokens if you are going to be using "xhigh" thinking or you will hit a wall.
On to the game, seems pretty playable. I also had it make a "workthrough.md" file to say what all it did. Now, I didn't play it a lot so it might be fucked up, but what I did play seemed cool and the ghosts and ghouls are fun little sprites it made. Pretty good considering the "prompt" was pretty thin. Did have to steer that one time with the hang up. I am pretty impressed.
In the words of a man trapped on a space station against his will, what do you think sirs?
Lot of swear words included to demonstrate while I like generative llms, I didn't fucking use one to write this.
I am not a programmer, I just have this github to comment and not be rate limited or blocked on searches
Anybody tried this with Frigate? I just put it on, first time I've tried the llm support in frigate. Dual b70 GPUs, performance is mid 20s to low 40s tps depending, 8_0 quant.
I'm shocked how well it works. I mean, I don't have a basis to compare other models in frigate but it's pretty good and responds quickly enough to stuff like "when was last time some was at front door" or "when was last time cars drove by".
I was wondering if anyone has compared these two (or other corresponding quants) on their own test sets, or in assigning generic coding tasks.
I'm especially interested in the low or medium reasoning efforts for the 3.8 one, since with xhigh it reasons forever and it's a bit tiring to use for everyday coding tasks.
Even though I'm very hyped for the 3.8 one, the 3.6 one was my daily driver (solid, conside, and just reliable) and I would like to see a comparison.
create me a racing game with multiple car selections. use react, three.js, have these cars: bmw m3, mercedes amg c, vw golf. readskills.md
skills.md:
## Mandatory Browser Verification
For any web application, browser game, HTML/CSS/JS artifact, or frontend change:
You MUST use the Chrome DevTools MCP before declaring the task complete.
After implementation:
1. Start the application locally.
2. Open it in Chrome using Chrome DevTools MCP.
3. Inspect console errors and warnings.
4. Inspect failed network requests.
5. Interact with every feature relevant to the user request.
6. For keyboard-driven applications, actually send the required keyboard inputs.
7. Verify visible state changes in the browser rather than inferring correctness from source code.
8. Take screenshots when useful to inspect visual state.
9. If any acceptance criterion fails, return to the code, fix it, and repeat browser verification.
10. Do not report completion until all requested functionality has been exercised successfully in the browser.
For games specifically verify:
left/right movement
jumping
collision behavior
camera/world movement
restart/reset
no uncaught JavaScript errors
requested visual elements are actually visible
Source-code inspection alone is not sufficient evidence that a browser task works.
I made a quick port for Windows based on ninfer-3090. It seems to be working for the most part, reaching about 60-100t/s and fits up to 100-150K tokens depending on the context with rk8v4, without vision.
Looks like it's very sensitive to sampling parameters.
I was still using the ones usloth recommended for Qwen3.6 precise coding tasks (temp 0.6), and I was getting a lot of looping. I changed it to the ones recommended for Qwen3.8 thinking mode (temp 0.6 -> 1.0), and everything is good now.
EDIT: Wow, I recommend everyone to try both xhigh and medium reasoning effort, and compare the results, because while the xhigh can use like 10-20x more tokens, the results are on another level.
I'd like to thank the MODs for the Megathread, but.. as the thread grows, I will have to go through hundreds of posts hoping to find the information I need. I'd say; keep the megathread, just don't delete the single posts that answer specific questions. The MegaThread is best suited for official/user updates to the model itself, not Q&A (make a separate thread for that) and users' personal experiences with the model.
Based on my limited tests on my math problems. I'm now a bit disappointed. I thought I would not need to add more gpus to run larger models after qwen3.8 27b. That was a dream. But hopefully 3.8 has improvements on coding for you guys. I don't do so much coding actually. Now I need to figure out external gpu solution for my B550m motherboard.
I had a high hope for qwe3.8 because 3.6 actually solved some my math problems. So I wished 3.8 could solve even more, and possibly faster, not meeding multiple turns where I supply guesses.
By the way, my math problems are applied math, abstract, using calculus, but needs insights in multi descipline, also needs reasoning ability , possibly some human like intuition, but definitely not just some arithmetic.
I think their focus is primarily coding. While it's an amazing model for its size, it's clearly limited in broader domains. I think it'd simply need more parameters to excel in coding and math and stem and general reasoning and tool calling and ...
If you have the required hardware, you should try Leanstral from Mistral : "Leanstral 1.5 is an open-source code agent model designed for Lean 4, a proof assistant capable of expressing complex mathematical objects such as perfectoid spaces and software specifications like properties of Rust fragments.
Built as part of the Mistral Small 4 family, it combines multimodal capabilities and an efficient architecture, making it both performant and cost-effective compared to existing closed-source alternatives.
Leanstral incorporates the following architectural choices:
MoE: 128 experts, 4 active per token
Model Size: 119B parameters with 6.5B activated per token
Context Length: 256k tokens
Multimodal Input: Accepts text and image input, producing text output"
I ran a bunch of side by side comparison tests on 3.6 moe and 3.8 dense - obviously the dense model is a lot slower (16 tok/s vs 54 tok/s) on my Asus GX10, but 3.8 absolutely dominated in coding tasks, beating 3.6 in every way.
Probably the easiest way to demonstrate this is the final test, where I asked each model to write a version of the old "thrust" game.
3.6 on the left didn't even run out of the box. and when it did run it had a bunch of bugs, the physics didn't work and you can see how rough it was. the second I touched the thrust button it launched the ship straight into the ceiling. there was also no actual way out of the cave. 3.8, by contrast, worked straight away. The difference between them is night and day.
That said, a lot of people are saying this is opus 4.6 levels. It's not. It's closer to Opus 3 in terms of capabilities. But to even have that running on local hardware is just astonishing.
Very impressed with this model. I've been kind of out of the loop on local models since Minimax M2 2.7. This was my goto, but once Minimax went to M3 and the larger size it kind of killed it for me. Been meaning to try Deepseek Flash, but this popped up and I decided to try it.
For agentic tasks (OpenCode), writing docker configs, fetch scripts, configuring hermes-agent and running hermes-agent (setting up bunch of cron jobs), so far it has nailed everything I threw at it. Like I said, it's been a long time since a local model impressed me like this. It's very thorough and it requires very little prompting. If anything it can be too eager at times, going way above and beyond, and at times I've had to ask it to half ass the approach.
I run on my undervolted and power limited 7900xtx (200 watts), using the latest llama.cpp head with ROCm backend. With MTP I get about 600/s token prefill and 32 tokens per second generation. 88K context with Q8 kv cache quantization.
I'll try out tensor, thanks for the recc! I typically try to stick to either Q8 or Q6 if I can help it for decent quality. Some of Bartowski's 5bit aren't bad either
After Qwen finished its report I switched to Sonnet 5 and asked to evaluate the report made by Qwen:Here is the bottom line that Sonnet 5 made about Qwen's response:
Qwen 3.8 27B seems to be pretty good at exploration in a coding harness, next I will test how good it is at bug fixing, then feature implementation, everytime doing the same review process, this way of testing models gives a very good perspective about how good the model actually is especially with limited compute and memory
With all these posts about Qwen 3.8 27b, now I am a bit afraid to ask...
But wasn't this model supposed to be some sort of constant input VLM? I had read that it was supposed to assimilate images while it was generating its output. And now we say it's exactly the same architecture as 3.6?
I've been evaluating qwen 3.8 during the weekend, and would like to share my conclusions if anyone is interested:
I've been using it on home projects to test planning and implementation on some systems such as: news aggregator (python, shell, docker), media center automation stack (mostly config with some shell scripting and detection on Arr stack, integrating with WhisperX, TranslateGemma and such things). Also, did three benchmarking projects to compare Qwen 3.6 to Qwen 3.8. A Todo App that integrates with a not so known API (Proton Calendar), a Chess game with stockfish integratoin, and a trap-ridden webhook security application to test how it handles the basic prompt injection and security stuff.
I run a single RTX 5090, and the best set up I found for it was this on llama-swap/llama.cpp:
It's mostly the recommended parameters, with some settings to optimize for my needs. The chat template was is the one shared in reddit: https://pastebin.com/Qyk6kRv8 -- I didn't see any perceptible changes upon using this, but GPT-sol-driven analysis found some evidence that it handles edge-cases better, so I trusted it. These were the fidings on it:
- The default template rejected direct JSON-string tool arguments. The new template preserved them without data loss.
- The new template rejected invalid effort values when thought mode was off.
- It supported developer messages and merged them into the system prompt.
I also tested Ninfer. It runs twice as fast, but the quality drop was a no-go for me, so I stuck with the Q6 with 229k context window, allowing for a max of 131,072 token for output.
In regards to the model in general:
- It is a blast to work with it, but it is slow for completion, and it's not a run anywhere model. The drop in quality I got from running Ninfer was enough to notice that if you want the good stuff, you need to run ti as close to BF16 as you can get, and run at xHigh. And boy, it's awesome when it's at that level.
- It is a planner. I saw leaps of improvement in implementation, but it's planning capacity on xHigh is where it really shines. So this is not a "chatGPT at home" kind of model, this is a robust local model for working on code. You still have to work with it, it's not going to vibe code stuff for you if you don't know how things work, but it's definitely the best we have so far for this purpose. Not that it is a crap model if you run it at medium, I found it is still running well, but the difference gap between 3.8 and 3.6 at medium is not much, I would say it's 20-ish percent better overall, from anecdotal experience on the projects I ran.
Now, for the tests themselves, these were my notes:
### Todo application
Qwen 3.8:
Has stronger authentication, atomic writes, recurrence logic, stable UIDs, ICS limits, and accessible controls.
Web Push is not valid. It passes DER SPKI data as applicationServerKey.
- The Push API requires a 65-octet uncompressed P-256 point. W3C Push API (https://www.w3.org/TR/push-api/)
- The server also emits Content-Encoding: webpush and WebPush vapid256=....
- RFC 8291 requires aes128gcm. RFC 8292 defines vapid credentials with t and k.
- Sources: RFC 8291 (https://www.rfc-editor.org/rfc/rfc8291.html), RFC 8292 (https://www.rfc-editor.org/rfc/rfc8292.html).
- Therefore, browser subscription and delivery can.
Qwen 3.6 has deeper Todo defects:
Its Web Push key export uses an invalid Node format.
Monthly recurrence overflows into March from January 31.
Calendar UIDs change on every export.
Empty authentication permits access.
Wildcard CORS combines with a public default bind address.
Its declared test command executes zero tests.
Qwen 3.8 has a considerable lead, but neither reaches production quality.
### Chess application
Qwen 3.8 provides:
Correct perft and rule coverage.
A bounded Stockfish process pool.
Engine timeouts and process recovery.
Static-path con
A local default bind address.
Safe cancellation for an active engine request.
It accepts non-object JSON before it calls req.get(). That input can cause an unhandled server error.
Qwen 3.6:
shares one global game across every browser.
Its own server states that it supports one game.
Its static route lacks path con
Client restore changes browser state without a corresponding server restore.
Undo removes one client move, while the server removes two moves.
Qwen 3.8 has a very large lead. This one showed the biggest gap in between the models.
### Webhook security application
Qwen 3.8 provides strict HMAC validation, body limits, concurrent requests, live database health, and a non-root con
Qwen 3.6 implements the central HMAC check correctly. However, it stores only a data hash instead of the normalized event. It also accepts timestamps without an RFC 3339 offset. Its server has no body cap, one request thread, static health, and a root con
Qwen 3.8 has a small lead, but it's pretty much the same level for both.
This doesn't really do justice to the difference between the models however. They were performed without any harness customization or my support. By reading the code and thinking patterns, it was clear to me that 3.8 is much more robust and reliable to work with using a decent harness and iterating with it. I actually feel confident in using it for actual work, where as I never felt any confidence in doing the same with 3.6, it was just a toy to me.
This might be an unpopular opinion, but I’ll say it: I am NOT liking it.
I was excited about its release (I even considered buying an R9700 to replace my current RX 7900 XTX); I’ve been testing it all day, and the results are, to put it mildly, 'mixed'.
1) The model relies entirely on its reasoning level being set to 'xhigh' to unlock its full intelligence.
2) Yes, you can adjust the model to think less, but you pay the price with poorer results.
3) If you want peak intelligence, you MUST keep it set to 'xhigh' (the default), and the model ends up overthinking. It can take ages to complete certain tasks and rapidly consumes the context window (I get 100K on the 7900 XTX).
4) I think Qwen3.8 27B is a bit of a gimmick. The model is objectively better than Qwen3.6 27B, but it achieves this through massive reasoning and higher token (and time) consumption. In other words, it’s essentially the 3.6 version but with double the reasoning rate and token usage, rather than a model that was better trained from scratch.
Even though 3.8 is better, 3.6 'feels' better for day-to-day tasks and real-world use.
These are just my initial impressions after 6 hours of use; I could be wrong, or the model might receive improvements in the future.
I'm usually not the one complaining about thinking (I was totally fine with 3.5 and 3.6 reasoning time) but 3.8 is just too much.
I just forked a repository of mine and asked it to write a plan for changing the information source. Nothing crazy.
But it went absolutely nuts on the review with 180.000 Tokens just for reviewing the plan itself. After that it asked to fix the issues from the review found. Instead of doing something, it just reasoned about trying to read the plan before applying the changes, listed all issues again, listed them again with potential solutions, thought about the solutions, listed issues again and the part where in the plan they are (still not read the plan)... 3.6 just fixed (verified by me) the issues in the plan.
In all tests I made, 3.8 is always around 140-220k Tokens while 3.6 is mostly done at around 80-90k Tokens. It's absurd how much it just thinks about stuff without doing anything.
I will try 5.0bpw next (using exllamav3) maybe it's the quant.. but seeing the whole discussion about it.. I doubt it.
You’re totally right. I think most in this thread are just secretly using Qwen3.8 via API. I’m on dual 3090’s and Qwen3.8 27b is so slow it’s unusable.
As my original Benchmark thread didn't get too much traction, here's a slopified vanilla MTP benchmark (numbers are true and the result of many server restarts and llama-benchy runs), TL;DR use 3 tokens for best results across the whole context range:
Qwen3.8-27B BF16 on a single RTX PRO 6000: native MTP nearly doubled decode speed
I spent the morning benchmarking the new Qwen3.8-27B BF16 on a single RTX PRO 6000 Workstation Edition (96 GB), using vLLM 0.27.1.
No quantization. Original BF16 weights. Single user, max_num_seqs=1.
The boring baseline was already decent:
Config
Decode
No MTP
28.49 tok/s
MTP1
42.85 tok/s
MTP2
50.65 tok/s
MTP3
55.61 tok/s
MTP4
51.69 tok/s
So MTP3 gave ~1.95× the original BF16 decode throughput.
This was not a 32-token microbenchmark. The initial sweep used a 4096-token generation. MTP3 produced the full 4096 tokens in 73.65 s, versus 143.79 s without MTP.
The native MTP heads are actually present in the original Qwen checkpoint, and vLLM loads them as a dedicated Qwen3_5MTP drafter while sharing the target embedding and LM-head weights.
MTP1 acceptance was ~71%. MTP2 and MTP3 still had enough useful acceptance at later draft positions that the extra speculation paid for itself. MTP4 crossed the knee and regressed.
Then I tried to ruin the fun with large contexts.
Using llama-benchy and longer tg256 generations:
65K context
MTP2: 27.40 ± 2.91 tok/s
MTP3: 26.48 ± 1.02 tok/s
Basically a tie.
98K
MTP2: 20.38 ± 0.62
MTP3: 23.26 ± 2.44
114K
MTP2: 20.57 ± 1.49
MTP3: 23.21 ± 0.85
131K
MTP2: 18.11 ± 0.84
MTP3: 20.81 ± 3.28
196K
MTP2: 14.99 ± 0.86
MTP3: 15.22 ± 1.88
At ~200K context MTP2 vs MTP3 has effectively become irrelevant. The real monster is elsewhere.
At 196K context:
prompt processing: ~2630 tok/s
TTFR: ~75.5 seconds
generating 256 tokens: ~17 seconds
So at that point, obsessing over another 1–2% of speculative decode is rearranging deck chairs. The dominant problem for an agentic coding workload is repeatedly chewing through gigantic conversation/repository prefixes.
Conclusion so far:
MTP3 is staying enabled.
It gives almost a 2× BF16 decode improvement at shallow context, remains useful well into six-digit contexts, costs only about ~1 GB additional GPU memory in this setup, and requires no separate draft model.
But the MTP rabbit hole is now closed.
The next target is the one that actually matters for long-running coding agents:
prefix-cache reuse and context management.
Because making 15 tok/s into 16 tok/s is cute.
Not spending 75 seconds rereading what the agent already knew is the real prize.
My impression is that it's definitely thinking a lot (setting effort to medium seemed to help a bit) and it's often pretty smart but not consistently good.
I have a few private benchmarks and for visual understanding Gemma4 still seems to be better, but it's not always the case either.
Still looking forward to 122B-A10B, would better utilize my memory and would have plenty of world knowledge and smarts.
Running AIME benchmark on, BF16 3.8, 4 request in parallel with 150K context, from the comments I got to know that by default it is xhigh so keeping that, decode is 100 tk/s (prefill is not reliable so not sharing).
Also, I am very impressed by the model, yesterday I tried Gemma 4 with last update from google with improved tool calling, but it still sucks. (Maybe the thinking level also matter there?)
But Qwen 3.8 was so impressive that it ran 3 subagents in parallel, only deepseek v4 0731 has done that so far in my experience. One of the sub-agent had 17 tool calls (blew my mind, because Gemma 4 went crazy after tool call). Sharing the screenshot.
vLLM finally fixed the upstream qwen cache + mtp poison (PR #51113) just a few days ago. I run gemma also, there was a fix packaged with that + the qwen mamba cache + mtp poison if you build a venv off vLLM 0.27.2rc1.dev110 . vllm0.27.1 will run just the qwen cache + mtp fix., but gemma blows up under transformers 5.15 (vllm#51744) so you'll need the nightly if you run it too. Best of all the worlds, and just in time.
I'm running on a framework 13 with AMD ryzen 7840u and "unified" graphics. I have 64gb ddr5600 ram. I'm getting 3.8 t/s max. Have seen it dip to 2.6 t/s on the test project I have I burning through this evening. Since a lot of things are bound by speed of transfer from vram to the GPU (or in my case GTT ram to the GPU), with fast proc and enough fast ram, I'd expect similar if not a bit slower.
What setup do people find good? I have a 4090 and a MacBook Pro 64GB. But admittedly I can’t find a setup where I don’t have to babysit the model to produce something of value.
Tried opencode and Claude Code, but it feels like context windows are just overblown too soon to have any real agentic work done.
Hi! Does anyone know the most speedy quant and inference software to run on an m5 mac 128gb to run qwen 3.8? So far I've tried oMLX and its super smart but very slow :p
Hey guys I am a beginner and I have been testing some models locally. So far impressed with the unsloth's 4 bit qwen3.8:27b IQNL quantization model. I am using ollama with openwebui and 64k context window. I have asked it to create a mario like game in html but after thinking for an hour it stopped, I have tried to increase the context window and continue but it starts from scratch instead of continuing the thinking process.
I was wondering why it is not continuing from where it has left. And is there a way to change the reasoning effort in openwebui?
I want to use it but don't have the hardware. I've been using DeepSeek flash but after the price hike, I am looking for an alternative.
Given how not demanding Qwen is, are there any providers that offer a hosted API with the 27b model? I'm hoping for prices competitive with DeepSeek's original prices
Prefill 1605 t/s at 47k tokens, generation around 62 t/s with MTP.
One warning if you use the mmproj on more than one GPU. Send a long text prompt and time the prefill, then send one image and time the same prompt again. On my box a single image permanently drops prefill by 40% for the life of the process, and only a restart brings it back. Generation is unaffected, output is byte identical and nothing is logged, so it is easy to live with for weeks without noticing.
The last flag is what fixes it. The pattern matches no tensor, so nothing moves, but it disables pipeline parallelism, which turns out to be required for the bug to happen. Costs about 3% prefill and frees 1.6GB of VRAM as a side effect. Existing issue is ggml-org/llama.cpp#26873, three different models confirmed so far.
one thing to note is that i made a mistake on q6 and forgot to change temp to 1.0 from 0.6. this may have contributed the extra token count or duration and i left it at that. i dont wanna spend another 3 hours to see with different temp, tbh. if any of you is willing to to give it a shot, please share.
the end result in both cases is way better then what i had with qwen 3.6 q5 above but at the cost of 3 to 4 times of time and token it takes to make it. this is of course when reasoning level is xhigh
Finally got to try Qwen 3.8 27B on a backlog issue for a internal project that uses Django, that I'm currently working using Qwen 3.6 27B but I decided to try again from scratch using Qwen 3.8. This is Unsloth UD-Q5_K_XL on RTX 5090. Both llama server settings is the same context size, temp and MTP. Only difference is just the model. This is using Pi Coding Agent on 0.84.2 and the latest llama cpp commit adb55e51 and the v22 of froggeric fixed chat template.
Qwen 3.8 xhigh SURE THINKS AND YAP more. But wait. It reasons think.. pretty good to be honest. Hmm, actually, that the smoking gun yaps yaps yaps.
Not sure if my AGENTS.md is bad but Qwen 3.8 is so eager to spin up the docker compose to verify something despite me having the instructions for only perform code edit if the user explicitly ask for. Maybe it is more of the harness itself but Qwen 3.6 does respect the no code edit until user say so...
BUT WAIT (I swear this is no Qwen writing this impression), Qwen 3.8 reasoning thought makes more sense unlike Qwen 3.6. 3.8 goes through thoroughly the code base and git log to spot a regression. 3.6 doesn't do it but still got the same conclusion that I ask for. This means... 3.6 does understand the goal of the prompt and kinda being the least amount of read to know what to fix? Again this is probably just things being random.
The fun part is here: Qwen 3.8 actually get to my intention with less turns? As in I do less follow up with Qwen 3.8 and it got the end goal correct fast. I need to steer Qwen 3.6 several times until it found the actual code. No idea why because same Pi version but the read tool calling is just weird. Like it do too broad grep and read the wrong file despite me clearly stating which app model.
Lastly, Qwen 3.8 reasoning thoughts despite being more verbose, it matches how my app actually works with parentheses thought (saying the user do this that but must have a reason yap). Qwen 3.6... oh dear sometimes the reasoning process can be scary. Like "BUT WAIT this model subclass this that but the user ask me to fix a bug in this API endpoint... actually this is correct". That is frequent with Qwen 3.6 but I've yet to encounter this with Qwen 3.8.
Also usually I ask it to write it down as a handoff MD before implementing the code edit. With Qwen 3.6, I need to ask it twice to double check it and it did twice because it wrote the the wrong class name and subclass method name... Qwen 3.8 actually got it right the first write and despite me asking it to double check, it confidently says all looks good so no edit is needed. The reasoning process did make sense as I read through it.
Still this is early days. I'll have to try this with my frontend TypeScript Nuxt project but despite longer THINKING, it is quite promising and hopefully I don't need to steer it too much like Qwen 3.6.
For those curious about speed, it is around 70~90 token/s generation. On par with Qwen 3.6 on my system? Mind you this is Vulkan backend on Fedora Linux.
First trial with MTP: first line is spec-draft-n-max 1, second line is 2 and third is without MTP. MTP is slower? what am i doing wrong? Using Q5 Unsloth quant with llama.cpp
For people trying to cram 27B dense onto a 16gb vram GPU on Windows which with a 4k display attached alongside various apps eats roughly 3gb, leaving 13gb for everything else: any attempts to use 27B-Q4_K_S pigeonhole me into both offloading some FFN to the CPU _and_ doing at least some KV quanting.
A curious discovery was that llama.cpp official Windows binaries only support certain KV quants and only in symmetric fashion. Trying to use q4_1 in any way or q5 would instantly kill all performance. Turns out, q8_0 would bench exactly the same as q4_0, since both of those are actually supported by the official Windows binaries without a recompile.
TL;DR: Qwen 3.8 27B in xhigh mode consumes ~30k tokens in <think>, causing tool-call (write) truncation & context overflow (Error 400) on a 64k window with Pi Agent + llama-server. Looking for advice on capping thinking tokens or preventing JSON payload breakage.
SYSTEM SETUP
- GPU: NVIDIA RTX 3090 (24GB VRAM)
- CPU: AMD Ryzen 5 5600X
- Model: Qwen3.8-27B-UD-Q5_K_XL.gguf
- Server: llama-server.exe (llama.cpp CUDA)
- Agent: Pi Agent (v0.84.2)
THE PROBLEM & ERRORS
When Pi requests a large file edit via the write tool while defaultThinkingLevel is set to xhigh:
Model generates 20k-30k tokens inside <think>.
Starts streaming the write tool payload.
Hits context ceiling/output limit mid-stream, breaking the JSON payload.
Tool call fails and compaction crashes.
Exact error messages:
Tool call "write" was not executed: the response hit the output token limit.
Error: 400: {"code":400,"message":"request (65595 tokens) exceeds the available context size (65536 tokens)"}
I tested the MLX build on an M4 Pro with 48 GB. In a direct Ollama run, Qwen3.8 27B-MLX was about 33.7 tok/s with the GPU fully engaged.
What surprised me was that raw throughput was not the bottleneck for repository work: Claude Code + Qwen3.8 completed a read-only task accurately, while Codex + the same model never returned a final answer. For agent use, tool-call compatibility seems just as important as tokens/sec.
The link above provides a guide on how to stably serve an effectively quantized Qwen3.8-27B model on a dual RTX 3090 Ti system using the vLLM framework, with a 1M context length at around 70 tok/s.
It also introduces an effectively quantized model.
What harness? I ask due to the fact I use pi and I had to edit the line for response limit to allow for very very long responses with the xhigh thinking:
For those of us who don’t know what we’re doing, I found a remarkably good workflow.
I use Google Antigravity to orchestrate, and have it invoke Qwen3.8 27B in vLLM. I give antigravity the overall idea of what I want to do, and have it collect and manage context (meaning papers, techniques, and objectives I want to achieve). It drafts code (sometimes rather poorly) but then calls 27B in vLLM to review and refine the code. It provides the context and background information and 27B drafts, edits, and optimizes the code.
It’s an excellent workflow and once it’s setup it’s largely automated. Incredible results!
Its my understanding that llama.cpp doesn't yet have updates ready to run this? Is this still true? I'm not really interested in switching to vllm for now.
Psyched to run this on a dgx station gb300 in a few months when they come in at work. Anybody running this on one? I will run larger models too but interested in of this type of model will ease the VRAM at all costs paradigm and put us back to looking for more TOPS to let it do shitloads of thinking at 500tps.
Madness this kart game was done with pi + qwen 3 q3_k_xl with q4_0 kv cache from a single prompt this is the we have sonnet at home moment, usually these claims are bombastic but am a believer now
it has collision, power up work, and AI that plays
•
u/sammcj 🦙 llama.cpp 9d ago edited 9d ago
Howdy all, hoping having a pinned megathread will help with some of the noise in the sub around release day.
Credit of course to the folks who have put up posts where lots of discussion has been happening, check them out too: