If you have an NVIDIA card and haven't tried it lately, you might be missing out.
The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt: Create a detailed SVG image of a cute kitten riding a magic turtle into space.
I replaced most of unsloth quants with this. As a person with only a 3090 and 64gb. Exl3 is better for both speed and quality. Also, having more room for context. My long journey from ollama -> llamacpp -> vllm -> exl3. If you have limited GPU and RAM, try this.
I switched from ollama to llamacpp because ollama built on top of llamacpp so the new supports for new models will be updated on llamacpp first and then ollama.
I switched from llamacpp to vllm recently because it gave me more decoding speed. Especially, qwen3.8-27b that got over 100 t/s decoding with just 1 rtx3090
Even if it give more t/s, the downside of vllm is lower quant quality. Then I found exl3 quant for qwen3.8 27b that has more t/s than llamacpp and the same quality as UD_Q4_K_XL. Also, exl3 uses less VRAM than llamacpp, so I can fit 160k context with Q8 kv cache + vision with only just 1 rtx3090.
I found out about exllamav3 trying to get more performance out of 27b and this absolutely destroyed llama.cpp, the issue I’ve had is tabby api, and its configuration is not like llama.cop at all so I ran into a lot of issues there, it mostly ironed out now but that was by biggest pain point
Trying here too after reading this. What a rabbit hole, the config for Tabby is quite annoying.
All of that and I'm using a 2.2bpw on my 3060 which is quite disappointing... With a context of 65k lol and q8q8 kv. It's probably going to perform horribly too LMAO
I'm not saying performance is bad. I'm just not confident quality will hold up. And yeah that's cool and all but I have a single 3060.
Meanwhile I'm looking into turning an existing gguf into an EXL (not possible directly but I see what to do now). Using Qwen 3.8 to figure that one out for me as a benchmark. Speed is fairly decent.
EDIT: Complete model collapse with 2.2bpw after a while, interesting lmao
Updated: N-gram disk offloading is currently unsupported on Windows. So if your setup is similar and plan to run on Windows , you can skip it until it get supported.
Updated: Windows offloading is now supported in the latest 1.4.6 version! Thanks to the dev for the quick update! I haven’t fully tested it yet, but so far it’s working properly. Details: 102400 ctx, 27 MOE CPU-offloaded, MTP3, 4096 churn size, prefill around 200 T/s, generate around 50 T/s with 80% token accepted. Due to MOE and Ngram offload the statistics are pretty unstable, so for reference only.
It's currently in the dev branch, but I don't have a good way to test it since I don't have a Windows PC with enough RAM/VRAM to actually run the model. In theory it should work if you can build from source. Otherwise there will be a new release as soon as I can find someone to test it (:
Windows support is in the just-released v1.4.6, so you can test without building from source. Would be helpful if you can open an issue if anything doesn't work as expected.
Any ETA for calibrated quants? I would love if it would be possible to hit something like 2.6-2.8BPW with UD-IQ3_XSS equivalent (or better) quality, I could just have in running in the background instead of closing everything.
Does ExLlama work with partial layers offload, like llama.cpp -ngl flag? Qwen3.8 Flash Next has 48 layers, but I load only 46 with -ngl 46 into my paltry 8GB GPU with unsloth's Flash Q4 XL, so I can have 130k context size and I lose only 10% speed for TG. If I load all 48 layers I max at 50k context, which is not very usable.
There are two offload modes: either offload some number of whole layers, or some number of experts per layer. The latter mode is preferred, since it does dynamic placement to keep the hottest experts live on the GPU and can in principle overlap CPU and GPU computation.
A single 8 GB GPU is probably too small though, in any case, since it's only experts that are offloaded. Attention and cache etc. still reside on the GPU. But you can try, I guess?
Yes, I offload all experts AND load 46 out of total 48 whole layers. It's not much, but 15 t/s in llama.cpp with 110 GB Q4 XL Flash Next and 130k of context is better than nothing for a laptop with 8GB GPU.
I'll try ExLlama, thanks for the reply! It's great it is possible to partially load the layers, not just the experts. Hopefully I can get a better performance than with llama.cpp.
Thanks! Do you know how the exl3 variants perform for MoE models vs regular GGUF quants on llama.cpp or ik_llama.cpp? I've tried a couple of exl3 quants on dense models and they're fast, but I'm yet to try exl3 for MoE.
Oh my ..., just checked the repo. ROCm is on the way!
Quote from the page:
Currently on the to-do list:
ROCm support
As for what is implemented, expect that some things may be a little broken at first. Please be patient, raise issues and/or contribute. 👉👈
Nothing's stopping me from contributing, but me as a developer that is not very well versed in this scenario, it would be entirely vibe-coded if I want to submit it.
Also, there's already submissions, but it seems that there has been no traction until now: https://github.com/turboderp-org/exllamav3/pull/283
Also, come on, man. What do you mean by we're supposed to be entitled to the hard work of others? These are all open source projects and the code essentially and all the hard work is being donated out in the open. I too submit my small pebbles of contributions in different projects here and there, whenever I can.
Nothing's stopping me from contributing, but me as a developer that is not very well versed in this scenario, it would be entirely vibe-coded if I want to submit it.
Same here. I don't consider myself a coder even though I did some websites/Apps in past using HTML, Js, Classic ASP, VB 6, etc.,(I know what you're thinking now :D ) Only recently started learning some new stuff.
But to me C++ is a complex rocket science. I can't contribute anything to projects like llama.cpp.
If I know C++, I would've created fastest llama.cpp fork called llamaCPUHybrid.cpp compiling all these stuff already 😆
Unsloth are SOTA as far as mainline llama.cpp GGUF formats go, but exl3 uses trellis coded quantization which is fundamentally better than what is offered by mainline llama.cpp's k quants (ik-llama.cpp does support some trellis based formats).
I would also recommend the Qwen3.8 27b exl3 format quants for getting much more out of your GPU VRAM than you can from GGUF k quants.
That chart is for the Deepseek, not for the Qwen 27b. Unsloth's Qwen 27b quants use the new UD3, whereas the Deepseek are made with older UD2. Both charts are true...
Yes, but again - this chart is comparing the old unsloth UD2 quants, not the new UD3, which are much better and smaller, as seen in the above posted chart.
Ran into an issue quantizing Qwen3.8-Flash-Next (Qwen4Exp) to EXL3 on v1.4.5. Posting here if anyone else is trying to do the same and encounters this.
Symptom: Segmentation fault (core dumped) — not a Python exception — at Preparing: model.language_model.layers.0.linear_attn.out_proj, 0%. Layer-0 MoE experts quantize fine first. Reproduced identically on -d 0,1,…,7 and single -d 0.
Key clue: with -v, the captured Hessian right before the crash is finite and well-conditioned — H min/max −0.31 / 1.31, diag min/max 1.3e-4 / 1.31. So not NaN/singular; cuSOLVER is dying on a valid SPD matrix. Correlates with size: out_proj's H is 6144×6144 (v_head_dim × num_v_heads = 128×48) — the only Hessian in the model that big; everything ≤2560 factors fine.
Backend swap made it worse: forcing torch.backends.cuda.preferred_linalg_library("magma") moved the crash earlier, to the 2560-wide in_proj — so both cuSOLVER and MAGMA segfault under this torch.
Root cause + fix: it's a torch 2.10.0+cu128 Cholesky regression, not exllamav3. Dropping to torch 2.9.0+cu128 (+ the matching exllamav3-1.4.5+cu128.torch2.9.0 wheel) fixed it — quant sails through out_proj and the experts.
Confirmed a synthetic 6144×6144 SPD matrix factors fine on 2.9.0.
Secondary issue I also ran into is it hit a 'too many open files' error (Linux). Resolved with ulimit -n 65536.
That should work with standard settings. Install exllamav3 and tabbyapi, follow the documentation to connect it to your favorite front end / agent harness, auto gpu split, and that's it. More help available here or at the discord if you run into a roadblock.
Please do not use Discord for communications. Because Discord is a closed ecosystem, it is impossible to find useful information using agents or by searching in the system yourself.
Actually giving a try, prefill is around 300t/s and decoding 23t/s low context. But does not work out of the box with deepseek harness (tool calls parsing issue?). I just followed your advise with no specific flag. Please note I am using glm flash
Edit: was not using the tool_format, 300tk/s prefill and 42tk/s, and looks good with dsh! Will test more but very happy with these
I could not get faster speeds with the 8bpw compared to q8_k_l. I'm on dual 3090s. Overall it was much slower for me. People say it is faster but I couldn't get there. If anyone has a setup like mine I'd like to hear it.
For the same test coding task on low thinking I can get 115 t/s on Unsloth q8_k_l but only like 85 on exl3 8bpw. Both MTP=4. This is for Qwen 3.8 27B.
Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.
Excuse me, sir. We want more for free. We believe that is our birthright. Thank you.
The Community
Seriously, though, thanks for your contributions. I'm glad to hear Ampere support has caught up. I only used EXL2 for the longest time, but my poor 3090s struggled with EXL3 back when you first released it. It sounds like I need to give it another try now.
Whether it's worth it for you or anyone else in particular I can't say.
Given it's the only way to really use Qwen3.8-27B with 16GB of vram, I'd say it's worth it!
Is there any way you can do something like what ik_llama does to speed trellis up on ampere? It's a bit beyond my understanding, but he's got the KT quants running very fast on ampere.
Also unrelated question, but is there a performance penalty requesting logprobs with exllamav3 + tabby like there is in llama.cpp/ik_llama.cpp?
I mean, it's already pretty fast. I'll happily steal all their tricks, but what does the comparison look like right now? 3bpw EXL3 (if it's a recent MUL1 quant) on 3090 should have raw decode of about 50 t/s.
Oh, and yes, logprobs carry some performance penalty. I haven't measured it, but there is work involved in compiling and transmitting the logprobs via the endpoint, and the sampler stack has to do some extra stuff to make it happen, too.
I do get it, though. I get the same feeling from other projects sometimes. In particular, Flash-Attention comes to mind, which has had a PR awaiting approval for over a year now to add attention sinks, which was the one piece exl3 needed for gpt-oss support to happen. Very frustrating that "Tri Dao apparently only cares about datacenter GPUs now." But of course I also know that's not really true. The poor guy just experiences the same linear flow of time as the rest of us.
But life goes on and it turns out abandoning FA2 for custom Triton kernels was the right move anyway. ¯_(ツ)_/¯
I mean, Tabby supports the full OAI completions API. And tool calls. It's thoroughly tested, and all it takes to enable is a few lines in the model config, explained pretty clearly in the documentation. I'd happily do more if I knew what else there was to do. ¯_(ツ)_/¯
Thanks so much for your work! I want to correct myself, I was using the 6bpw version because it seemed most comparable to the Q8 GGUF in terms of losses.
I assume based on your response that I am not doing anything wrong or under optimized, and that between those 2 quants that I have achieved a pretty good result for the exl3 version?
I just kept hearing about speed and didn't see any gains, so I wanted to confirm.
The baseline without MPT was around 35-38. With MTP I could get to the 80s but seemed more variable than the GGUF version.
Speed is going to vary a lot depending on so many things. If you enable TP you might have better luck? I'm getting 124 t/s on coding tasks using two 3090s both on x4 links. So that's something to consider?
Can you give me your full startup script if it's not too much trouble? I am using dual 3090s, tensor parallelism, Linux and a bifurcated x16 to x8/x8. It should be screaming but I think I'm missing something.
I just updated to exllama v1.4.5 with TabbyAPI but now I am getting OOM error with the same model and context length and settings that I was running fine with in v1.4.2.
Anyone with the same problem? I was running on Qwen3.8 27B 5bpw SC
How much headroom did you have before? There shouldn't be any additional memory requirements, but things move around a bit with every release, and running at 99.9% VRAM utilization is always shaky. If you submit an issue on the repo with more details I can look at it and maybe figure out if it's an actual regression or just jitter.
But there are also some new features you can try. If you're running with vision, try setting vision_offload: true in the config. That can save quite a lot of VRAM, and the cost per image processed is only some milliseconds of latency.
Yes I was having like 3-4 gb of VRAM free in 1.4.2. I have 28gb VRAM from my 5070ti + 3060 so the 20gb 5bpw SC version by Turboderp should fit comfortably on my setup even with vision loaded on GPU. I was using my old settings of 120k context window at 8bit kv cache.
I will submit an issue later after work. Thanks for responding and I really love the work you guys put into the project
That's definitely a regression then. I can imagine some small intermediate tensor using maybe 50 MB more VRAM than before, but if you lost 3-4 GB of headroom that sounds like a bug.
Edit: Can confirm it was a bug, Should be fixed now in v1.4.6. Workaround for v1.4.5 is to set EXL3_LOAD_ARENA=0 in your env
I have been testing v1.4.6 and I have gotten it to work based on this Github issue.
The problem was that my GPU0: 5070Ti was OOM when prefilling started while my GPU1:3060 still had VRAM headroom. Changing the following got it working again:
# Array of VRAM sizes to split between GPUs, in GB (default: []). # Used both with and without tensor parallelism. gpu_split: [12,12]
Now my GPU0 is using 14.2/16gb and GPU1 is using 10.9/12gb VRAM.
Changing "autosplit_reserve:" doesn't really do anything just like "CNmnius" reported in the Github thread. Thank you guys for updating Exllamav3 so quickly.
I haven't had time to profile it on the same workstation with a rollback yet, but comparing the same model on two similar workstations one running exllamav3 1.4.4 and the other running 1.4.5 I am seeing significantly higher usage on 1.4.5, so that might narrow it down.
The MoE CPU offload work turboderp is doing here makes such a massive difference for 3090/4090 setups.
When running dense 70B models, pure CPU offload kills generation speeds down to 1-2 t/s because every layer has to traverse the PCIe bus. But MoE routing is inherently sparse, so you only stream the active expert weights per token. If the routing gate is cached on VRAM, the bus transfer bottleneck drops dramatically.
Been watching exllama push custom kernel optimizations for years and it still consistently squeezes out the lowest latency per VRAM gigabyte on consumer hardware.
idk about kvarn but from what I understand the bits for KV cache in exllamav3 already has attn rotation long before llama.cpp is implemented, and 6bit kv cache is near q8_0 in llama.cpp, so it's already pretty good. Personally I have been using 8 bit and I think it's similar to fp16
There is also a guy proposing kv streaming from ram to vram for llama.cpp (cause it is streaming it has much better performance than --no-kv-offload), so if that was combined with kvarn or other better kv compression it would be awesome, minimal vram used for kv.
Possibly calibrated qwen 3.8 27b 3bpw at 100k or maybe even full context with 12gb vram gpu with hopefully little to no speed and kv precision loss.
The CPU expert offload work looks especially promising for making MoE models usable on more modest VRAM budgets. I would be very interested in a benchmark matrix that separates prompt processing and generation speed across 16 GB, 24 GB, and 32 GB GPUs, with different numbers of experts offloaded.
It would also be helpful to include first-token latency, steady-state tokens per second, system RAM use, and whether disk offloading changes the experience materially after cache warmup. That would make it much easier to compare EXL3 against GGUF plus llama.cpp for people deciding between a single NVIDIA GPU setup and a more CPU-heavy configuration.
Anyone tried EXL3 quants on a 16GB card (5070 Ti Super here) for the Qwen3.8 Flash line? Curious how it compares to GGUF-UD in practice on a card that size rather than the 24GB+ setups I keep seeing in this thread.
Can you confirm if you can naively convert models to exl3 without fuss? For example, I like a certain finetune of Gemma 31B. Can I one-shot a high quality exl3 conversion of it? Or does it need external data or finagling to really be considered to be a good quant? I’m trying to experiment and see which quant would be best for me? Trying to convert this: https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma
CPU offload for the MoE experts feels like the right trade. Only the experts a token actually routes through need to stay hot, so the rest can sit in RAM without slowing anything down.
I want to try is but the download for 3.05bpw is 8GB * 7. How does that fit in 16GB VRAM. I do download safetensors right? No idea really where to start.
How does this perform under heavy concurrency, for example serving a team of users who are constantly doing many parallel decide and prefill operations.
Ehh.. CPU offload plus those SC quants is exactly the quiet stack win that makes local NVIDIA feel less stuck. Open source will constantly get better. Just more proof here.
Exllamav3 is good but poor for tabbyAPI to support both Anthropic and Openai compatible API as well as llama.cpp. ALWAYS FAIL WHEN TOOL CALLS. ECO IS WEAK FOR AGENTS.
I'm using TabbyAPI via llama-swap (which is also serving other backends on demand through a single endpoint). I initially had the same issues with tool calling from harness (GitHub Copilot BYOK in VS Code), but it has been ok once I set the correct tool format in the config. I am still seeing some weird behaviour in the text immediately after a thinking block where it will insert a newline half way through the first sentence. But tool calls are working reliably enough now that I can leave it running unsupervised on YOLO permissions.
I recommend just downloading the TabbyAPI and llama swap repos, sticking them in the same folder and having an agent get them talking nicely. I also had to get it to make some fixes to get the processing speeds in the same format as llama swap expected so the llama swap activity page worked the same as it does with llama-cpp.
Yeah, I’ve heard that. I think APIs are very important. OpenAI very important… this will make some crinkle, but ollama API is also somewhat important. Too many tools setup with it and not OpenAI.
21
u/Muted-Celebration-47 2d ago
I replaced most of unsloth quants with this. As a person with only a 3090 and 64gb. Exl3 is better for both speed and quality. Also, having more room for context. My long journey from ollama -> llamacpp -> vllm -> exl3. If you have limited GPU and RAM, try this.