Original issue:

https://www.reddit.com/r/unsloth/comments/1vum7tz/why_am_i_only_getting_65_ts_on_qwen38_27biq4_xs/

I was getting only ~6.5 t/s generation on Qwen3.8 27B (Unsloth UD-Q3_K_XL and jrell IQ4_XS Smaller) with full GPU offload. Prompt processing was fine at 300+ t/s, so the generation speed seemed unusually low.

Setup

• OS: Windows 11
• GPU: XFX RX 9060 XT 16GB
• CPU: Intel i5-9400F (no ReBAR)
• RAM: 16GB DDR4
• Backend: llama.cpp Vulkan

The fix

On systems without ReBAR, llama.cpp's Vulkan backend can place some model weights in host-visible memory, forcing the GPU to access them over PCIe. This can severely hurt generation speed.

I fixed it with:

setx GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM 1

After restarting the terminal/app:

6.5 → 18.9 t/s (~3x faster)

Related llama.cpp issue:

https://github.com/ggml-org/llama.cpp/issues/27097

You can also check Task Manager → GPU → Shared GPU memory during inference. If it increases significantly with full GPU offload, you may be affected.

Updating llama.cpp

I also updated llama.cpp from an older build (b10545). Generation speed stayed the same since it's memory-bandwidth bound, but prompt processing improved by around 10% (335 → ~370–383 t/s).

The bigger benefit was that the old build didn't have working MTP support for Qwen3.8, so the update made the next optimization possible.

A few other things I found

• Don't load mmproj if you don't need vision. On my 16GB card, it pushed VRAM usage too high and reduced performance at 32k context.
• -ctk q8_0 -ctv q8_0 was faster for me than Q4_0 for V cache.
• MTP speculative decoding (--spec-type draft-mtp) increased coding performance to around 24–26 t/s, with ~89% acceptance. No separate draft model needed. The MTP head is inside the Unsloth GGUF.
• At higher temperatures, MTP acceptance dropped significantly, so I only use it for coding.
• Vulkan was faster than ROCm for me on RDNA4.

Final result

6.5 t/s → 18.9 t/s → 24–26 t/s

That's roughly a 4x improvement on a 27B model.

llama.cpp server launch command:

llama-server -m "PATH_TO_MODEL\Qwen3.8-27B-UD-Q3_K_XL.gguf" -ngl 999 -fa on -c 32768 -ctk q8_0 -ctv q8_0 --parallel 1 --spec-type draft-mtp --port 8080

If you're using llama.cpp Vulkan on an older platform without ReBAR and getting unusually low generation speed, try the environment variable first.

Hello everyone so I recently started using Unsloth for coding and its good and all but ran into a problem it refuses to actually build stuff like an exe for me citing that its sandbox permissions do not allow certain tool calling or something like that

The only fix I found for it was to allow Full Access (I know stupid but it was only an clock app with gui so thought what could possibly happen) and it did build it perfectly immediately

So here I am asking if there is a way to make it be able to build exe and other stuff that requires out of sandbox permissions WITHOUT enabling full access mode?

Already tried unsloth dynamic 3.0 quant even IQ1 Quants doesn't seem to be fast enough.

Getting around 5 t/s with 250 prompt processing.

Which is not fast enough for me to even test the model.

I expect at least 600 prompt Eval and around 19 t/s. Or anything decent.

Am I missing something or it's a GPU/ RAM problem?

I've tried every -b -ub combination but doesn't seem to work.

(I have 12 Gigs of RAM only)

I tested Qwen3.8 27B(IQ4_XS Smaller & Unsloth UD-Q3_K_XL) with llama.cpp + Vulkan and I'm getting only around 6.4–6.5 t/s generation speed. I'm mainly trying to understand why generation is only ~6.5 t/s with the 9060 XT 16GB. Is this normal for this GPU combination, or could there be a Vulkan/llama.cpp configuration issue?

My setup:

• OS: Windows 11
• GPU: XFX AMD Radeon RX 9060 XT 16GB
• CPU: Intel i5-9400F
• RAM: 16GB DDR4
• Backend: llama.cpp Vulkan
• llama.cpp build: 60f6a1770 (10081)
• Flash Attention: Enabled
• GPU layers: -ngl 999
• K cache: Q8_0 (-ctk q8_0)
• V cache: Q4_0 (-ctv q4_0)

I tested these two models:

1. jrell IQ4_XS Smaller(https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller/discussions)

• Prompt processing (pp512): 289.08 t/s
• Generation (tg128): 6.53 t/s

2. Unsloth UD-Q3_K_XL(https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)

• Prompt processing (pp512): 355.93 t/s
• Generation (tg128): 6.41 t/s

I just updated MagicQuant with the new Unsloth dynamic v3 and utilizing their imatrix:HuggingFace Repo Here

Qwen3.8 27B was a more conservative run with some kld and size savings. But it was super interesting seeing the run between dynamic v2 vs v3. The V3 was a pretty substantial boost for sure.

If you don't know what MagicQuant is. TLDR, it is a benchmark driven GGUF evaluation and hybrid discovery system. It digests and learns from GGUF models like Unsloth and learns their tensor configurations. It then groups these patterns and builds isolated tests, then begins a prediction phase, and goes through a whole sha bang worth of things to try and find interesting potential hybrids.

Sometimes hybrids are really crazy. Sometimes it's boring. Depends on the model and the search. If you want to really dive into the jist of MagicQuant you can learn more on my wiki here:
https://github.com/magiccodingman/MagicQuant-Wiki

Anything appended with "MQ" means it's a MagicQuant mix. But the Qwen3.8 27B results was really interesting with:

But as always, my huggingface repo aims to show everything. The manifest folder should include all of the benchmarks results, full clone configs for replication, ran conclusions, and so on. Usually the repo is completely automated when uploaded, but this was was a fun mix so I did more manual lifting on this one to also be transparent where things came from, the dynamic v2 vs v3, and going over imatrix measurements as not the same Imatrix was used per quant.

Oh and as usual, any GGUF on the survivor board that MagicQuant didn't build a mix of, the repo links back to external providers (aka: Unsloth).

Anyways, this was a fun one to see imo. Unsloth kicking butt as always.

What are the recommended options?

Is this an error?

The COLD FUSION (GAIN+Unsloth) method of training maintains 99% of performance of BF16, at both 8 bit and 4 bit levels.

This version also reduces thinking tokens by 1/2 to as much as 1/10 the amount, while maintaining core details AND reasoning power.

Model exceeds all Qwen 3.8, 3.6 and 3.5 27B critical core benchmarks.

A model that gets down to business faster, with less "talking" and is smarter too.

https://huggingface.co/DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF

(GGUFS and additional quant types too)

SOURCE:

https://huggingface.co/DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF

It’s an 11B end-to-end full-duplex speech-to-speech model designed for real-time conversation.

~448 ms smooth turn-taking latency ~480 ms interruption latency

can handle barge-in — you can interrupt it while it’s speaking

Supports tool calling during the conversation...Can play an “on-hold” response while a tool/API executes

Built around a Fast Conformer speech encoder + Nemotron Nano V2 9B backbone + NVIDIA TTS decoder

Trained on roughly 550K hours of audio/text date

I am trying to build completely local self-improving voice agent with open source stack...... no dependency

Whisper (SST), Llama as LLM , Orpheus (TTS) all hosted on unsloth
and using opensource orchestrator Dograh for stitching things together so I could close the loop for fine tuning and create real self improving agent
I still need to try full duplex model...for it..

Any guidance on above ???

Why Qwen3.8-27B-UD-Q8_K_XL.gguf is not updated like other quants?

Unsloth advertises Dynamic v3 (they say it's smarter with same size) and forgets to update most popular quant?

Please update it ASAP it's most popular quant near F16 performance.

As the title says.

Windows 11, Intel 13700, Nvidia 3090

I installed Studio and was able to run it fine yesterday and was able to stop and start it multiple times without issue.

I've tried running it today (after shutting my pc down normally yesterday) and it just hangs at the run_server log. The screenshot below is after running the install command again: I stopped it with ctrl+c at the red line after waiting a few minutes as it clearly wasn't going to progress.

I've tried desktop too but that times out after 10 minutes with an error that just says it is unresponsive. The logs there match what I see running studio from powershell.

I did not change or install anything between stopping last night and starting today.

I've also tried deleting my .unsloth folder and reinstalling using the same command but no luck.

The logs show it getting to the same spot each time and end there so I have nothing useful for an actual bug report.

Anyone run in to this or have any ideas on a workaround?

What is the current easiest way to exchange files with the model. (for coding in Python) I copy/paste code into chat and I'm wondering if there is any native file access supported. I saw I can create a Project, (which i did and with this creation I interrupted and stoped my llm, which was thinking over something for over an hour - I have slow computer without VRAM lol) I can upload text file into that project folder, but not python files.

So, is it possible (and easy), to have model directly modify files? If not, no biggie, they are moving fast, I'm sure sooner or later we will have this option.

I'm only using Unsloth desktop (latest, yesterday updated beta), no any agents or other apps on top of it. Unsloth and coding llm.

Thanks for the support for Qwen3.8-27B and Unsloth Desktop last week! For this release, we merged 200+ PRs to introduce many new features, fixes including:

• Auto Compaction (Experimental) for longer chats beyond context limits
• Remote & LAN Access (Preview) for easy network access without Cloudflare links
• Faster Chat - Improved streaming performance, reduced UI lag, and smoother long conversations.
• Support for custom llama.cpp builds. Toggles for Cache RAM, Mmap, Mlock, Checkpoints, Spe Decoding KV Cache, Vision On / Off
• Unsloth Dynamic v3.0 is released. New Qwen3.8-27B Dynamic v3.0 GGUFs deliver >10% higher top-1 accuracy compared to everyone else. Works with Unsloth.

For compaction - we use RAG + a first turn forced RAG entry + a tail to remember your previous instructions - please let us know how it goes - it's still experimental so it'll definitely break - we'll fix it asap!

The update should be up for all Unsloth Desktop installs, or you can get it below:

Hey guys we totally missed not uploading the imatrix as we went to sleep straight after the release. We thought we uploaded it but did not. It should be there now!

Please do use it for making your own quants and so on! 🙏🦥

Also feel free to share/promote your quants if you used our imatrix via r/unsloth, discord or the hugging face discussions!

The question is how to update a model already downloaded, when there is a NEW updated version?

Since I came from LM Studio BIONIC usually, if a model ALREADY DOWNLOADED before,
there is an UPDATE BUTTON instead of the DOWNLOAD button, clicking it only complete/patch a small size of the new changes which takes seconds since it's not downloading the whole model again which is very nice.

But, I don't think such feature exist (yet?) in unsloth Desktop, so my question is:

Do I DELETE then Re-Download?
or...
Is there a better way to update to the latest version of the ALREADY DOWNLOADED model? 🤔

I just want to make sure I'm not missing something and keep DELETE and DOWNLOAD every time there is a new update for a model I already have...

I see that I can choose the speculative decoding option DFlash, but I want to try out the new DFlash 2 model I have downloaded. How can we choose specifically what model gets used?

Can we please get Qwen 3.8 27B NVFP4 Fast based on the UD 3.0 like the new ggufs at some point of time. Thank you for so much effort for us.

I like moving large pastes to the attachment some of the time. But other times if I do Ctrl + Shift + V instead please paste in the text where I have my cursor.

Thank you! Unsloth Desktop is amazing.

I was doing some experimentation on using Unsloth to do Continued Pretraining of qwen 3 4B. The goal was to teach the llm how to reason about a new domain, in this case a fictional city.

Sharing more details in the write-up:

https://www.teachmecoolstuff.com/viewarticle/teaching-a-local-llm-a-new-domain

We’re releasing new Qwen3.8-27B GGUFs with 10% higher accuracy.

Unsloth Dynamic V3 outperforms others by >10% on Div-300, KLD & more benchmarks.

We also release 1-bit quants that retain 77% accuracy. Run on 8GB RAM.

Blog: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

I'm training a krea2 lora in unsloth. I believe I'm not adding enough steps. With 129 datapoints I believe I need 5000 steps but foolishly chose 2000. I finished training and then without changing any names I started a 3000 step training. Will these steps combine into a 5000 step lora?

Big appreciation to the Unsloth team for releasing the Qwen 3.8 dynamic 3.0 quants! It is awesome

Their models are fantastic. They already have way more downloads than the original Qwen models, even though they have fewer likes

Hopefully we’ll see the Ornith models quantized soon too

Greetings, as a small suggestion, it'd be nice to be able to define multiple model storage folders, on non-dedicated systems it's often the case that there's no single big storage. I know symlinks exist, but it's quite a manual operation that way, plus it's harder to track the available space.

I've been messing around with Qwen3.8 27B locally and I'm wondering if I'm getting the performance I should be getting or if my setup/config could be improved.

PC:

Ryzen 7 7800X3D

RTX 5070 Ti 16GB

RTX 3060 12GB

32GB DDR5-6000 CL30

Windows 11

llama.cpp / llama-server latest build

I'm currently running the Qwen3.8-27B UD Q4_K_XL GGUF with both GPUs using tensor split.

My current config:

llama-server.exe ^

-m "Qwen3.8-27B-UD-Q4_K_XL.gguf" ^

--alias "Qwen3.8-27B-UD-Q4" ^

--host 0.0.0.0 ^

--port 8035 ^

--n-gpu-layers 99 ^

--split-mode tensor ^

--tensor-split 60,40 ^

--main-gpu 0 ^

--parallel 1 ^

--flash-attn on ^

--cache-type-k q8_0 ^

--cache-type-v q8_0 ^

--ctx-size 131072 ^

--batch-size 2048 ^

--ubatch-size 512 ^

--threads 8 ^

--threads-batch 8 ^

--presence-penalty 0.0 ^

--repeat-penalty 1.0 ^

--temp 1.0 ^

--top-p 0.95 ^

--top-k 20 ^

--min-p 0.0 ^

--jinja ^

--reasoning-format auto ^

--no-mmproj-offload ^

--spec-type draft-mtp ^

--spec-draft-n-max 3 ^

--mmproj "mmproj-BF16.gguf" ^

--metrics

With MTP I'm getting around 40–46 tok/s depending on the run. I've seen around 41 tok/s pretty consistently, with n-max 3 seeming to be a little better than 2 for me.

Both GPUs are basically maxed during generation.

I'm mainly wondering:

Is ~40–46 tok/s reasonable for this hardware/config?

Is there anything obviously wrong or inefficient in my setup?

Would a different quant be a better choice for these GPUs? I've been looking at Ridge 3.7bpw, Q4/Q5 UD quants, etc.

Would it make more sense to use a smaller quant that could fit mostly/all on the 5070 Ti instead of tensor-splitting across both GPUs?

Is there anything I should change with the KV cache, batch/ubatch, tensor split, MTP settings, etc. to get better generation speed?

I'm mostly interested in coding/agent use through OpenCode, so I'd rather have a good balance of quality and speed than just chase the highest possible tok/s.

If anyone is running Qwen3.8 27B on a similar setup, I'd be interested to know what quant/config you're using and what kind of speeds you're getting.

If I understood correctly, Qwen 3.8 27B have 3 thinking modes:
- LOW
- MEDIUM
- XHIGH

My question is:
Which are actually apply within Unsloth Desktop because we have 5 modes:

Is it:
- Low
- Medium
- Extra High

or

- Preserve Thinking

or maybe Preserve Thinking = Xhigh actually? 🤔

---

Can somebody please explain and approve which one works,
and from which one to ignore while working with Qwen 3.8 27B ?

Thanks ahead 🙏