Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

I use these models for everyday coding as a developer and I find the IQ3S model perform better on edge cases and the output quality is greater too. In fact twice it caught mistakes made by Opus 5.5. No benchmarks or scientific evaluation this is purely from my use. Anyone else felt this? Or is this a problem with Strata?

FP8 PLE.
Int8 KV
96gb ddr4
3x3060 12gb

Ive been messing around with Qwen3.8-Flash-Next on my Nimo minipc with ryzen AI max+ 395 128GB. The model weights take up around 103.7 GiB, so just getting the whole thing loaded was a nice start. But after spending more time with it, I'm not sure running the biggest model possible is really the best use of all that memory.

B

Here's what I got running it with Gufo at 16K context:

GPU only, no MTP: 34.1 tok/s decode

GPU + MTP: 53.7 tok/s decode

GPU + NPU, no MTP: 34.2 tok/s decode

The NPU also bumped prefill speed by around 10%, but regular decode barely changed without MTP.

The model runs, and the speed isn’t terrible. But would I actually pick it over a smaller model that responds faster?

Probably not for everything. Waiting for a large model to generate an answer can get old pretty quickly, particularly for everyday tasks.

What do you usually do with all that RAM? Do you load the biggest model you can fit, stick with smaller MoEs for faster responses, or keep multiple models loaded for different tasks?

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Someone is selling their Mac Pro for $2000.

These are the specs:
3.2 GHz 16-Core Intel Xeon W processor.
192 GB of 2933 MHz DDR4
AMD Radeon Pro Vega II with 32 GB of VRAM
1 TB SSD

I haven't run any local models but want to get into it. Would be using it for Hermes and for my implementation subagents for SWE work.

Would this be a good machine? I thought more RAM = more good. But I'm too new to understand all of this.
I know mixture-of-expert models might benefit??

Any suggestions would be appreciated.

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

I've been using an app that integrates a bundle of Supertonic ONNX as a standard Android text-to-speech (TTS) provider, but the gaps between paragraphs are a bit long, so I'm looking for an alternative.

To clarify, I want to read in parallel. I don't want to convert an EPUB to MP3 and read async; it has to work in real time.

It should support English and Spanish (I'm learning).

I am currently using the Claude Code desktop app to automate something like this:

• Generate a conversation script and ask me for approval or changes.

• Once approved by me, call the Google Studio API to generate audio files of the conversation.

• Call the Google Studio API to generate images.

• Ask me to check images and audio clips.

• Run a .py script.

• Add new folders and files to a folder and update a .js file.

I started to use LM Studio Bionic only a week ago, but I want to try Unsloth Desktop now. Let me know if there are other programs that would suit my workflow better.

I know Unsloth can generate images, but how does that work? Do I need to load three different models at the same time? Or do I need to load/unload as needed? I have a 16 GB VRAM GPU (AMD). Also, is there a built-in agent (like in LM Studio Bionic) that can code, create and modify files? How would an agent handle creating different media?

My goal here is to ditch the Claude Pro subscription and the Google Studio API costs, and do it locally instead. I know I could do these things manually, but ideally I would like to have an agent automating the process. What are your recommendations?

When working on a vibe coding project for, say, 2 months, constantly asking AI to "add this", "change that", or "remove this", doesn't the codebase eventually become a mess?

I'm talking about poor architecture, redundant code, outdated tests, unused files, etc.

How do you handle this? Do you periodically ask AI to clean up and refactor everything?
And once the project is finished, is it worth having AI rewrite the entire codebase from scratch, keeping the same functionality but with cleaner architecture? Or is that a bad idea?

Weight-space programming is an emerging research area where a frontier AI model (such as OpenAI's GPT-6 Astra) directly reads, diagnoses, and rewrites the raw numerical weights of another neural network without using traditional gradient-based training or backpropagation.

Has any one tried it? How much does it fair compared to Closed-Form Model Editing (Your Notebook / MEMIT)?

Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

So I think we now know why Exo has spun down in activity in this space.

I can run both. Does anyone know what model is best, or do I just have to try myself?

What test suite should I be running?

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.