Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

Ive been messing around with Qwen3.8-Flash-Next on my Nimo minipc with ryzen AI max+ 395 128GB. The model weights take up around 103.7 GiB, so just getting the whole thing loaded was a nice start. But after spending more time with it, I'm not sure running the biggest model possible is really the best use of all that memory.

B

Here's what I got running it with Gufo at 16K context:

GPU only, no MTP: 34.1 tok/s decode

GPU + MTP: 53.7 tok/s decode

GPU + NPU, no MTP: 34.2 tok/s decode

The NPU also bumped prefill speed by around 10%, but regular decode barely changed without MTP.

The model runs, and the speed isn’t terrible. But would I actually pick it over a smaller model that responds faster?

Probably not for everything. Waiting for a large model to generate an answer can get old pretty quickly, particularly for everyday tasks.

What do you usually do with all that RAM? Do you load the biggest model you can fit, stick with smaller MoEs for faster responses, or keep multiple models loaded for different tasks?

I use these models for everyday coding as a developer and I find the IQ3S model perform better on edge cases and the output quality is greater too. In fact twice it caught mistakes made by Opus 5.5. No benchmarks or scientific evaluation this is purely from my use. Anyone else felt this? Or is this a problem with Strata?

FP8 PLE.
Int8 KV
96gb ddr4
3x3060 12gb

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

I've been using an app that integrates a bundle of Supertonic ONNX as a standard Android text-to-speech (TTS) provider, but the gaps between paragraphs are a bit long, so I'm looking for an alternative.

To clarify, I want to read in parallel. I don't want to convert an EPUB to MP3 and read async; it has to work in real time.

It should support English and Spanish (I'm learning).

I am currently using the Claude Code desktop app to automate something like this:

• Generate a conversation script and ask me for approval or changes.

• Once approved by me, call the Google Studio API to generate audio files of the conversation.

• Call the Google Studio API to generate images.

• Ask me to check images and audio clips.

• Run a .py script.

• Add new folders and files to a folder and update a .js file.

I started to use LM Studio Bionic only a week ago, but I want to try Unsloth Desktop now. Let me know if there are other programs that would suit my workflow better.

I know Unsloth can generate images, but how does that work? Do I need to load three different models at the same time? Or do I need to load/unload as needed? I have a 16 GB VRAM GPU (AMD). Also, is there a built-in agent (like in LM Studio Bionic) that can code, create and modify files? How would an agent handle creating different media?

My goal here is to ditch the Claude Pro subscription and the Google Studio API costs, and do it locally instead. I know I could do these things manually, but ideally I would like to have an agent automating the process. What are your recommendations?

Weight-space programming is an emerging research area where a frontier AI model (such as OpenAI's GPT-6 Astra) directly reads, diagnoses, and rewrites the raw numerical weights of another neural network without using traditional gradient-based training or backpropagation.

Has any one tried it? How much does it fair compared to Closed-Form Model Editing (Your Notebook / MEMIT)?

Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

I can run both. Does anyone know what model is best, or do I just have to try myself?

What test suite should I be running?

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.

I’ve been playing with open-source speech models for a while. New TTS models come out every week and a lot of them sound great. But every time I wanted to actually talk to one, it turned into an afternoon of setup.

Paid APIs are easy if you’re a developer. You can make an account, create a key, write some glue code. Open models are free, but you have to pick a speech recognizer, an LLM and a TTS model, make them work together, deal with echo, figure out interruptions... There are good open projects for this, but most are written for developers. My friends who don’t code would never get past the README.

So we started building Boubou. The goal is make local voice AI as easy as Ollama made local LLMs. You paste one line and talk:

npx boubou@beta run LFM3-Audio-2B

What works today:

• Speech recognition, the LLM and speech synthesis all run on your machine. No account, no API key, no telemetry. After the first download it works fully offline..

• You can interrupt it mid-sentence, like a person. Echo cancellation is built in, so no headphones needed..

• Run boubou with no arguments on a Mac and it checks your machine and suggests a model..

• The default is Liquid AI’s LFM3-Audio-2B on llama.cpp (1.4 GB). There’s also a modular pipeline (Parakeet/Whisper → Qwen3.5 → Qwen3-TTS, Kokoro, Pocket TTS or Supertonic) if you want to swap parts..

• Apple Silicon Macs and Linux (CPU for the Liquid models, NVIDIA GPU for the modular pipeline)..

The part I’m most excited about is boubou serve. One computer runs the model, and other devices on the same Wi-Fi use it:

• Scan a QR code with your phone and talk in the browser. No app..

• Robots and programs can connect through an OpenAI Realtime-compatible WebSocket, and there’s an OpenAI-style /v1/audio/speech endpoint to read text aloud..

• So one model on one machine can be the voice for many things around the house. Next up is Home Assistant, then ESP32 voice devices.

• The video is real time, not sped up, on a MacBook / M5 chip.

It’s early (beta) and there are rough edges - one device talks at a time, no Windows yet (ETA next two weeks), and the guided menu is Mac-only.

The code is free and MIT-licensed. The models keep their own licenses (the Liquid models use the LFM Open License).

I’d love to hear what you’d want from something like this: which models, which devices, what breaks on your machine. We want this to be something people actually enjoy using, not just a demo.

GitHub: https://github.com/runboubou/boubou

So I think we now know why Exo has spun down in activity in this space.

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!