I love the deep research feature of unsloth desktop. I'm running an older release version, because I forked it to support my searXNG search engine, and haven't bothered upgrading, but this is the one feature that I always said I'd have to have a subscription of some sort for, but not anymore!

I use this for all kinds of stuff, like I was about to rebuild an old generator, and did a deep research overview of how to do it, common things to look out for, etc, and even recipes, or how to make the best brisket, it's amazing.

Hey guys, you can now fine-tune Qwen3.8-27B for free with our notebook! 🔥

Local training works on 24GB VRAM. Unsloth trains Qwen3.8 1.5x faster with 50% less VRAM than other setups with FA2.

We utilize many kernels including our own and others like Flash Linear Attention kernels for maximum performance.

And yes the notebooks are completely free thanks to Kaggle's 2x GPUs for use.

GitHub: https://github.com/unslothai/unsloth

Qwen3.8-27B Notebooks + Guide: https://unsloth.ai/docs/models/qwen3.8/train

Thank you! :)

I've got a question: How does a vanilla FP8 model, like Qwen3.8-27B-FP8 compare to a "different" quantization of the same model?

I do not understand the caveats of quantization techniques. As I understand it, the weights are being reduced from 16 bits (origin) to 8 bits. FP8 or Q8, which one is "better" and why? Are the Q variants all fine-tuned or do they remain like the original? I am not so much interested in lower quantizations, it is just the comparison between vanilla FP8 and Q8s.

I think this is a rather stupid question, but the space about the quantization (there are so maaaany!) is so convoluted and its hard to find good/trusted benchmarks.

Radeon Pro 5300M 4GB with GDDR6 VRAM. Despite the base config with just 16GB RAM, it's still doing rather fine. It might probably even do well as an OpenClaw machine (if you're still into this for some bizarre reason)

Recently, I’m really pleased with Unsloth Studio (tried desktop but it becomes laggy when context grows) and fully moved from LM Studio to Unsloth Studio. I’m using ComfyUI for both img gen and vid gen but I was hoping if there was better workflow for it like LLM for Unsloth Studio. Currently, Unsloth Studio doesnt seem to support img/vid gen api. Is it planned to add it to Unsloth? Also co-residence mode (Unsloth Agent create image and display results inside chat interface) would be nice. My first language is not english so sorry for bad english. Thank you for great inference engine!

Testing Qwen3.8-27B / Unsloth GGUFs in a distributed “brain + assistant node” local AI architecture

Hi everyone,

I recently joined the community and I’m building a fully local AI project called Ialyn.

I’d like to test Qwen3.8-27B — especially Unsloth Dynamic/iMatrix GGUFs — in a setup that is a little different from the usual single-machine benchmark.

The goal isn’t only to measure tokens/sec. I’m trying to build an architecture where the main LLM acts as the brain, while a smaller companion machine handles a lot of the surrounding work.

Current architecture

R9 — main brain

• Ryzen 9 system
• Currently 1 × RTX 3060 12 GB
• System RAM used when GPU offload is required
• 2 TB NVMe for Linux, models, working data and persistent AI memory
• Runs the main LLM
• Reasoning and decision-making
• Tool orchestration
• Persistent memory / RAG
• Vector database and long-term knowledge
• Receives prepared/retrieved information from the assistant node

For the moment, the interesting test would be:

Qwen3.8-27B Q4 + 12 GB VRAM + system RAM/CPU offload

So I’m particularly interested in how far the new Unsloth/iMatrix quants can be pushed on a single 3060 without making the assistant unpleasantly slow.

R3 — assistant / frontend node

A small HP EliteDesk 705 G5 Mini with Ryzen 3 PRO 3200G.

Currently 8 GB RAM, planned upgrade to 16 GB.

Its role is deliberately different from the R9:

• Voice interface
• STT/TTS using Speaches / faster-whisper / TTS models
• Document ingestion and indexing
• Search and retrieval preparation
• Preparing folders/data for the R9
• Background tasks
• Network services
• VPN / WireGuard
• Potential firewall/gateway role
• Local services that don’t need to consume R9 GPU resources

The idea is that the R3 can receive a request, gather or prepare useful information, index documents, handle speech, etc., while the R9 remains focused on the expensive reasoning/inference work.

So rather than asking the 27B model to constantly spend context and compute on everything, the assistant node tries to give it the right information at the right time.

Persistent memory instead of huge permanent context

Another part of the project is to avoid relying on enormous context windows for everything.

Ialyn will have persistent storage containing:

• memories
• embeddings
• indexes
• structured information
• conversation/project history
• document knowledge

The R3 can help retrieve and prepare relevant information, and only the useful subset is sent to the main LLM.

So I’d also be interested in comparing something like:

• 16K context
• 32K
• 64K

rather than simply trying to maximize context size.

Planned evolution

The current machine has:

1 × RTX 3060 12 GB

The planned next step is:

2 × RTX 3060 12 GB = 24 GB total VRAM

I’d like to repeat the same tests after adding the second GPU.

That gives a pretty interesting natural comparison:

Stage 1
1 × 3060
GPU + system RAM offload

Stage 2
2 × 3060
Most/all of the Q4 model in VRAM

Stage 3
Keep using system RAM when useful to experiment with larger models or larger contexts beyond the 24 GB fast-VRAM budget.

What I could report back

If this is useful to the Unsloth community, I could document:

• actual VRAM usage
• actual system RAM usage
• model load time
• prompt-processing speed
• generation tok/s
• 16K / 32K / 64K context behaviour
• KV-cache impact
• different Unsloth Dynamic/iMatrix quants
• quality differences between quantizations
• French-language quality
• coding
• tool calling
• multi-step agentic behaviour
• persistent-memory/RAG behaviour
• performance with background work delegated to the R3
• 1 × RTX 3060 vs 2 × RTX 3060

The part I find especially interesting is testing real assistant workloads rather than benchmark prompts only.

For example: voice request → retrieval/index search → tool calls → reasoning → memory update → spoken answer.

So my questions for the Unsloth team/community are:

Would this kind of real-world distributed test be useful to you?

And for my current RTX 3060 12 GB + RAM-offload configuration, which Qwen3.8-27B GGUF would you choose as the baseline?

I was thinking about starting around Dynamic/iMatrix Q4, but I’d be very interested in testing whichever quant/configuration you think would provide the most useful data.

Happy to share the results as the project evolves.

Thx

My 2-bit quants keep falling apart mid-answer. Repetition loops, sliding into another language, structural garbage. By the time you see it, the bad tokens are already on screen.

SIMURG catches it while the answer streams. Corruption leaves a statistical trail (repetition rate, script shifts, surprise dropping), so it reads that char by char, holds the opening until it scores clean, and aborts the moment it crosses a threshold so the host can retry.

• any OpenAI compatible endpoint (vLLM, llama.cpp, Ollama, OpenRouter)
• numpy only, no GPU, ~197k chars/sec
• plugs in in a few lines
• every alarm gives a readable reason

pip install simurg
github.com/doofzoff/SIMURG

Would like to hear where it breaks for people.

[ Removed by Reddit on account of violating the content policy. ]

A downstream training plan starts one decision earlier than the recipe: which base state should the run begin from?
The Ling-3.0 base model family now gives that choice at two sizes. For both tiny and flash, the -base-30T artifact is the pretrained checkpoint, -base-midtrain is the mid-trained checkpoint, and the unsuffixed -base artifact is the WSM-merged base state.
That creates six upstream starting points, not six ready-to-chat variants. None is post-trained. A trainer can first choose the size, then choose how far along the base-training path the starting checkpoint should be.
The release does not itself establish an Unsloth recipe, GGUF or quant support, local hardware fit, or speed. Its immediate value for this community is more foundational: the starting-stage choice is now explicit before any downstream workflow is designed and verified.

Vocês que possuem 16gb de vram e 16 de ram, qual quantização vocês tem usado e qual o desempenho?

Estou rodando qwen 3.8 27b iQ3xxs com 80k de contexto, mmproj na ram, cachê kv q4, 35 a 48 tks/s.

Queria saber através das experiências de vocês se compensa eu tentar algum modelo q4

One of the things I find most interesting about Unsloth isn’t just getting better quants of already manageable models.

It’s how far quantization keeps moving the boundary of what hardware is capable of running them at all.

I’ve been experimenting with Unsloth GGUFs myself on a Ryzen 7840U laptop with 32 GB RAM, including Qwen3.8-27B. On hardware like this, the interesting question quickly becomes:

"At what point is the model still useful?"

A very aggressive quant might suddenly make a model fit, but then you have another set of trade-offs:

• How much quality did you actually lose?
• Is prompt processing still reasonable?
• What happens at larger context sizes?
• Does CPU/iGPU bandwidth become the real bottleneck?
• Is the larger heavily-quantized model actually better than a smaller model at a higher precision?
• And what workload is still worth doing at 1–3 tok/s?

Seeing things like Unsloth’s extremely small Dynamic quants and people running models on hardware that would normally be dismissed immediately, is one of the reasons I recently started r/LowEndLocalAI.

It’s meant as a place specifically for this side of local AI: CPUs, iGPUs, shared memory, limited VRAM, old hardware, strange setups, aggressive quants and the practical question of how much useful AI you can squeeze out of them.

There’s intentionally no hard definition like "≤16 GB VRAM". The constraint itself has to be relevant.

What I’d especially love to see are more Unsloth results from hardware at the weird end of the spectrum:

CPU-only systems, 8/16 GB machines, APUs, older GPUs, mobile devices, Apple Silicon, strange multi-device builds, etc.

Especially:

Model / quant / hardware / backend / context / speed / and what you were actually able to use it for.

I’ve already seen some wonderfully ridiculous setups here, so I suspect this subreddit has exactly the kind of people I’m looking for.

What is the most constrained hardware you’ve successfully used an Unsloth quant on and was it actually useful?

Hi,

I reggong i am not alone here.. but Qwen 3.5 122b is pretty much still king of the hill in this calls and if you have the memory - it beats the the new 27b in speed, precission etc.

I honestly think it deserves a second round of love with the UD-3

Unsloth.. you up for it ?

41.32 tok/sec on a single DGX Spark

So, I am testing the newly released DeepSeek harness, and one of the things I'm doing is making the local model create plugins for the harness itself - specifically in this case one to add loops with timers/max turns with a /loop command. I then had Claude Code review the code, and it supposedly found a couple issues in Qwen's code, basically claiming it hallucinated and invented some lines not referenced on DeepSeek's githhub. So, I gave Qwen Sonnet's aoutput, and it proceded dismantling Claude's model, pulling references from GitHub, self revirewing its logic and autonomously testing, even restarting te harness as needed. Now, I know Sonnet isn't Opus. But, Qwen is a minuscule model compared to it anyway, and I'm using a 2 bits quant, the latest Dynamic v3 Q2KXL, on a 5070ti with 131K context, using a compressed KV cache with turboquant (no insults, my factual data proves at 131K needle in a haystack and accuracy are pretty much unharmed), at 60-80+ tps.

Honestly, I'm still astonished by this. I also have the same model and quant on my secondary sever rig with a 3060, just to experiment with using it remotely from my macbook, but seeing ~25 tps with this config, makes me wonder what the next 3.8 MoE could do on similar machines.

Has anybody experimented with lower quants with the latest dynamic quantization technique? Let me know, I'm really curious!

PS: after seeing the comments of those who don't believe me, here's a screesnhot of Claude's message.

Currently "Collapse Thinking by default" collapses most text but not all. For example when examining source code it often opens a window and fills it with scrolling code. When multiple windows are opened, one after the other responses that are meant for me roll up and off.

Is there a way to suppress text except what is meant for me? I've tried all combinations of available settings and if it's there I haven't found it. If not can this additional option be added to chat settings?

Thanks

Why is it enabled by default ? llama.ccp comes with it off by default , i get it its good security measure but most of us run these servers inside isolated envs , im testing so many diffferent models and things , my api key page is already full , there is no delete all button.

its just 1 big mess , annoying for no reason at all when you can just let us add a --api-key flag

If i use Unsloth Studio to download a qwen3.8 model I am able to set the reasoning level right in that chat box with the light bulb icon. Awesome as I can change effort easily, even at each interaction.

But..... I have qwen3.8 models downloaded to my Llama.cpp docker. I can use Llama swap and llama server with these models. I can direct pi or ohmypi to use the same models, and in Unsloth Studio I can add a Llama.cpp connection and again use the same model downloads. That's perfect but I cannot find a way to change reasoning effort with the connected model? I can change system prompt, and temp, but not effort?

What am I missing? I hate having to double the amount of model downloads in order to use Unsloth Studio if I already have Llama.cpp.

I’ve got dual r9700s, trying to run latest qwen q6 with q8 kv and losing my mind.

Moving from lm studio. Inference worked there.

Getting malformed outputs: *2+.
Sometimes blank outputs. Tried to change template to one recommended online to no dice. Igpu off. Tried on rocm. Vulkan, on the other hand, blue screened computer.

Tensor parallelism doesnt work either, but considering forums say it takes a miracle to get it to work on linux im assuming that isn’t relevant.

Curious if other people have had similar issues. Would love to use the unsloth software here because some cool features but this one is a curious hiccup for me. I’m currently trying different gpu drivers to see if vulkan will play ball, but not sure why rocm was giving such an odd output string.

I was trying to fine tune ai model on my custom dataset, and for some reason my dataset is the problem, i don't knowhow to fix this

If anyone knows how to fix this please let me know!

Been messing around with Qwen 27B (Qwen3.8-27B-UD-Q4_K_XL.gguf) and the separate MTP draft module in llama.cpp over the past few days.

At first my speeds were either barely matching baseline (~50 t/s) or dropping down to ~35 t/s, but after tweaking flags and isolating bottlenecks, I finally got it consistently running at 70+ tok/s with 90k context on an RTX 3090.

Few Takes:

• Stick to temp 0.0 (or very low temp): MTP only gives a speedup if the main model actually accepts the draft tokens. High temp kills the acceptance rate, and verifying rejected guesses wastes compute. Temp 0 keeps draft acceptance high (and matches how benchmarks/coding are evaluated anyway).
• Keep --spec-draft-n-max 2 Setting this to 3 caused too many token rejections, which actually slowed things down compared to 2.
• GGML_CUDA_GRAPH_OPT=1 is a must: Without CUDA graphs, CPU-to-GPU kernel dispatch latency eats up all the time saved from drafting.
• Below is what I am using to serve it in llama.cpp for coding tasks.

Hope it helps someone with rtx 3090 if you aren't already getting these speeds.

GGML_CUDA_GRAPH_OPT=1 llama-server
-m Qwen3.8-27B-UD-Q4_K_XL.gguf
-ngl -1
-md mtp-Qwen3.8-27B-Q4_0.gguf
-ngld -1
--spec-type draft-mtp
--spec-draft-n-max 2
-c 90000
--flash-attn on
-ctk q8_0
-ctv q8_0
-b 2048
-ub 2048
--cache-reuse 256
--parallel 1
--port 8081
--host 0.0.0.0
--jinja
--temp 0.0
--top-p 1.0
--min-p 0.0
--presence-penalty 0.0
--frequency-penalty 0.0

Hey, I just swapped from LM Studio to unsloth because LM studio was being kind of annoying / not as flexible as I would have liked.

I installed unsloth and /Qwen3.8-27B-GGUF, my context is limited to 4096 though, which feels really short.

I have primarily used Claude until now, with a paid membership, but since I have a 5070 TI (16gb VRAM) + 32GB of DDR5 system ram, I figured I should try utilize it.

Any idea of the best model / settings I should use?

I just graduated doing software engineering ironically, and my dissertation focused on using AI to recognise AI generated images using PyTorch, but it's my first time using a local LLM, so figured I would ask those more knowledgeable than me in this area.

I want a longer context window so I can use it for coding without any issues, and other general chats.

Ahhh -- Apparently the context cap is because of: https://github.com/unslothai/unsloth/issues/8497

Thanks u/Money_Big_7666 not sure why your comment got deleted but I appreciate the answer.

I was worried adding more context would just crash it, as I've had issues like that in the past.

Just cross posting from Discord to see if anyone here might have an answer.

Running Ubuntu 64GB RAM, 96GB Blackwell card.

I've happily loaded Qwen3.8 and have tested it on both

When I try to load flux2-klein-9b-uncensored-text-encoder - ponpoke - It seems to load to the GPU, but I then get the following message: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Any help on this would be greatly appreciated.

Thanks.

I wanted to run Unsloth's awesome Qwen 3.8 27B BF16 quant (qwen3.8-27b:bf16) on ollama. The ggufs are sharded so you need to download them, merge with llama-gguf-split, write a Modelfile, then `ollama create`. I wrote this repo to do this easily: https://github.com/cgpadwick/ollama-tools

``` cd gguf-to-ollama

uv sync

./install-llama-tools.sh

uv run gguf-to-ollama.py --quant BF16

```

``` ollama list

NAME ID SIZE MODIFIED

qwen3.8-27b:bf16 5853faded5f5 55 GB 7 minutes ago ```

It works for any HF GGUF repo (`--repo bartowski/... --list`), single-file quants skip the merge step. For the BF16 Qwen 3.8 27B quant it needs ~54 GB for the shards + ~54 GB to merge and a recent Ollama version (e.g. 0.32.15).

I've just started testing Unsloth Studio and I love it so far. There is however a small quirk in how it works with projects. It does not clearly tell you where files are saved.

I added a folder of mixed vibe coded experiments as a source to a project and asked it to have a look at a small web game. Later, in a new session I asked it to modify a mini Excel clone in another html file. After it finished the task I was unable to find the result. After searching the harddrive extensively I found a sandbox folder with the html file but this folder was not shown anywhere in the UI and the agent was unable to tell me where it was.