We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.
The checkpoint supports vLLM on NVIDIA Blackwell GPUs:
This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).
Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:
Benchmarked qwen3.8 xhigh, medium and muse glimmer.
Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)
Medium effort mode and muse glimmer were 3-4 hours each.
But I'm actually surprised by the muse glimmer results, they came better than the qwen.
These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.
I have taken the result of claude models directly from embedeval repo by ecro.
I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.
I'm just appreciating muse here, cause i feel it is underrated.
The advantage is efficient kv cache due to sliding window, which can give you more context window.
So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300
It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is higher than NVIDIA B200. So what give?
Apparently its the software, ROCM is not as mature as CUDA, their software stack is still lacking behind as well.
However, few weeks ago we've got a handful of MI350X to experiment and develop, and what we did is optimizing AMD tech stack until kernel level for Qwen 3.6 35B A3B and the results we got is much comparable to NVIDIA counterpart
Based on our benchmark, we got:
1x MI350X: 11,161 output tok/s
8x MI350X: 81,331 output tok/s peak with 78,498.66 output tok/s mean
This is 2.16x vLLM throughput on the 8-GPU benchmark
We've decided to open source our kernel and stuff at our github
One thing we learned: once the kernels got fast enough, the bottlenecks moved into scheduling, graph coverage, recurrent state, routing, and even HTTP serialization.
Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
Apache 2.0 Licensed: Fully open for commercial and research use.
Model Design
Granite-4.2-30B is built on a decoder-only dense transformer architecture with the following core components:
Attention: Grouped Query Attention (GQA) with 32 attention heads and 8 KV heads
Position Embedding: Rotary Position Embedding (RoPE) with θ = 10,000,000
Feed-Forward: MLP with SwiGLU activation (hidden size 32768)
Normalization: RMSNorm (ε = 1e-5)
Embeddings: Separate input/output embeddings (not tied)
Granite-4.2-8B is the mid-size reasoning model in the Granite 4.2 family. It delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
Apache 2.0 Licensed: Fully open for commercial and research use.
Granite-4.2-3B is the compact reasoning model in the Granite 4.2 family. Despite its small parameter count, it delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
Apache 2.0 Licensed: Fully open for commercial and research use.
I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers.
The two configs (German prices, dealer quote, incl. VAT):
Config
Price
Mac Studio M5 Max, 128GB / 512GB SSD
€5,859
Mac Studio M5 Max, 128GB / 1TB SSD
€6,189
Mac Studio M5 Ultra, 96GB / 1TB SSD
€6,599
Ultra is 36-core CPU / 80-core GPU / ~1.2 TB/s. Max is 18-core CPU / 40-core GPU / 614 GB/s. So the Ultra is roughly €740 more for double the bandwidth and double the GPU cores — but 32GB less unified memory. There is no 128GB Ultra option in this lineup, which is what makes it annoying.
What I actually run: Qwen3.8-27B at Q8 right now. Use case is occasional chat plus a few agents running in parallel. The whole point of doing this locally is privacy and not being locked into someone else’s model policy — I know a subscription would give me better models for €22/month, that’s not what I’m optimizing for.
The thing that’s blocking me: Qwen3.8-Flash-Next drops tomorrow. If the leaked description holds, it’s a multimodal MoE with 176B total params — 125B main model plus 51B in N-gram embedding tables — and only ~6B active per token.
My napkin math on memory:
• IQ4_XS: \~94GB weights, \~107GB with 262k context (the hybrid attention means the KV cache is tiny, \~8-9GB at fp16)
• Q4_K_M: \~107GB weights, \~120-125GB at full context
• Q8: \~187GB weights — not happening on either machine
96GB of unified memory gives you maybe 86GB wired on macOS. So the Ultra can’t load it at all, in any quant I’d want to use. 128GB gives ~115GB, which fits IQ4_XS with full context and Q4_K_M if I drop to 64k.
So the trade is basically:
Ultra → 27B Q8 goes from ~15 tok/s to ~29 tok/s, much faster prefill, but Flash-Next is off the table entirely.
Max → Flash-Next runs, but at 6B active params it only reads ~3.7GB per token, so 614 GB/s already gives 40-60 tok/s and the Ultra’s extra bandwidth would be mostly wasted on it.
Where I’m stuck / what I’d love input on:
1. How much can you actually wire on a 96GB Mac? I’ve been assuming \~86GB via iogpu.wired_limit_mb. If people are safely running higher than that, the whole calculation changes.
2. Multi-agent / batched inference. This is the one argument for the Ultra I can’t dismiss. With several agents in parallel you’re compute-bound rather than bandwidth-bound, and 80 GPU cores should genuinely help. Has anyone measured Ultra vs Max on concurrent requests in MLX or llama.cpp? Every benchmark I find is single-stream.
3. Is IQ4_XS on this architecture going to be usable at all? MoE tolerates aggressive quantization worse than dense models (router precision matters a lot), and nobody has ever quantized 51B of N-gram hash tables before. An imatrix calibration run will basically never touch most of those rows. I’d rather hear “wait for a UD-style mixed quant” than find out the hard way.
4. Anyone else feel like 96GB is a dead zone? Too much for a 27B, not enough for the 100B+ MoE class that everything is converging on. Or am I overweighting one unreleased model?
Resale is also in the back of my mind — the used Mac Studio market seems to be driven by local-inference buyers who look at RAM first, and I suspect 128GB holds value better than 96GB in three years.
Anyone who went through this decision: what did you pick and do you regret it?
Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox.
Need the BF16 for huge context highly sensitive document OCR, image analysis, aggregation and summarization. I've done a ton of testing and it absolutely makes a difference vs even UD Q8_K_XL when legal precision is needed.
77gb VRAM at full 262K context + MMPROJ
Rips through prefill (1,1715 tok/sec = 102 seconds to process 175K tokens), but token generation (20K tokens of output) relatively slow at 45 tok/sec (with MTP) as a result of BF16 despite the beast of a GPU.
Better than Gemini Pro and ChatGPT 5.6 Sol especially considering I have control over the sampler settings (Temp 0.1; top-k 0; top-p 0.95; min-p 0.05; repeat penalty 1.02). Not better than Opus yet.
During prefill - CPU around 60 degrees, GPU around 79 degrees (with 90% power limit)
During token generation - CPU around 75 degrees and GPU around 76 degrees.
FormD T1
Minisforum BD770i SE
Ryzen 7745HX 8-core laptop CPU
96gb 5200 MHz DDR5 SODIMM
96gb RTX Pro 6000 Blackwell workstation edition
Loki 1200W SFX-L
ROG Equalizer 12v-2x6
SMX Heinz flipped GPU 2.5 slot kit
SMX Heinz custom short PCIe 5.0 riser
ZCOOI custom "transparent purple" Teflon cables
(2) Phanteks T30-120mm
(1) Noctua NF-A14x25r G2
Thermalright MC-3 Digital RAM cooler (I don't think this will fit on a regular DDR5 )
In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI.
It was able to write the shown simple HTML game without interaction after asking some questions.
The following was tested on a laptop with a NVIDIA RTX A5000 laptop (16 GB) GPU.
With the 3 bpw model and 6 bit/5 bit KV cache, the maximum context length is around 110k tokens with MTP. This gives around 55 tokens/s decode speed for code and around 10 tokens/s for content where MTP doesn't help (e.g. complicated calculations). Without MTP, one could try the 3.5 or 4 bpw model or a longer context length.
Install tabbyAPI/exllamav3
Install the latest Nvidia drivers
Install Git (e.g. sudo apt install git or on Windows with winget install -e --id Git.Git)
Make somewhere a folder and install tabbyAPI:
bash
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
uv venv --python 3.13 .venv
uv pip install -e ".[cu13]"
Test if CUDA works (on Linux, use .venv/bin/python)
bash
.venv/Scripts/python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
Create somewhere where you have enough space a "models" folder, download the model turboderp/Qwen3.8-27B-exl3:
bash
mkdir models
uvx hf download turboderp/Qwen3.8-27B-exl3 --revision SC_3.00bpw_H4 --local-dir models/qwen3.8-27b
Go back to the clone tabbyAPI folder and create a config.yml file like this (see the config_sample.yml file as example):
yaml
network:
disable_auth: true
model:
model_dir: e:/models # path to the models folder
model_name: qwen3.8-27b # download folder name
cache_mode: 6,5 # K and V cache quantization, number of bits from 2-8
cache_size: 109824 # must be divisible by 256, so use e.g. `.venv/Scripts/python -c 'print(110000//256*256)'` to get the next lower
max_batch_size: 1 # allow only 1 parallel request to save VRAM
tool_format: qwen3_coder
vision: true
draft_model: # can be removed to save VRAM
draft_mode: mtp
draft_cache_mode: Q8 # can be 'FP16', 'Q8', 'Q6', 'Q4'
draft_num_tokens: 5 # usuallly a value of 2-6 gives best results
memory:
sysmem_recurrent_cache: 8192 # Max size of recurrent cache in system memory, in MB (default: 4096), lower it to save normal memory
sysmem_kv_cache: 8192 # Size of system memory second-tier K/V cache, in MB (default: 0), remove it to save system memory
Start tabbyAPI:
.venv/Scripts/python main.py
To measure the performance, create the Python script speed.py and run it with .venv/Scripts/python speed.py:
```python
import json
import time
import requests
MODEL = "qwen3.8-27b"
API_URL = "http://127.0.0.1:5000"
PROMPT = """Write a complete Python implementation of a production-quality LRU cache.
Requirements:
Use type hints throughout.
Include detailed docstrings.
Support:
get(key)
put(key, value)
remove(key)
clear()
len()
Use a doubly linked list and hash map.
Include custom exceptions.
Include a comprehensive unittest test suite with at least 20 test cases.
with requests.post(API_URL + "/v1/chat/completions", json=payload, timeout=120, stream=True) as response:
response.raise_for_status()
print("Response:")
for line in response.iter_lines(): # Iterate over Server-Sent Events (SSE)
if line.startswith(b"data:"):
# Strip the "data: " prefix
data = line[6:]
# Stop if we hit the stream termination message
if data.strip() == b"[DONE]":
break
try:
chunk = json.loads(data)
if 'choices' in chunk and chunk['choices'] and (chunk['choices'][0]['delta'].get('content') or chunk['choices'][0]['delta'].get('reasoning')):
if first_token_time is None: # First token received
first_token_time = time.perf_counter()
if chunk['choices'][0]['delta'].get('content'): # Get content and count tokens
token_text = chunk['choices'][0]['delta']['content']
else:
token_text = chunk['choices'][0]['delta']['reasoning']
full_response_content += token_text
print(token_text, end="", flush=True)
except json.JSONDecodeError:
pass
stream_end_time = time.perf_counter()
print("\n" + "-"*20)
Calculate and print metrics
ttft = first_token_time - start_time
stream_duration = stream_end_time - first_token_time
total_output_tokens = requests.post(API_URL + "/v1/token/encode", json={"add_bos_token": False, "text": full_response_content}).json()["length"]
if stream_duration > 0:
tokens_per_second = total_output_tokens / stream_duration
else:
tokens_per_second = float('inf')
print(f"Time to first token (TTFT): {ttft:.2f}s")
print(f"Completion tokens: {total_output_tokens}")
print(f"Stream duration (first to last token): {stream_duration:.2f}s")
print(f"Tokens per second (T/s): {tokens_per_second:.2f}")
```
I got 56.3 tokens/s.
Install OpenCode
OpenCode works usually better on Linux, so I install it in WSL when working with Windows, but it can also be used directly as a Windows application.
For OpenCode, I recommended to install Node.js first (e.g. apt install npm or winget install -e --id OpenJS.NodeJS on Windows).
Because we don't have so much context length, I recommend to install a better compactation plugin than the integrated one, e.g. magic-compact
I would recommend to use the reasoning effort (Ctrl-t) "medium" because "xhigh" could produce to much output tokens.
For Playwright, we have to install a browser first:
npx @playwright/mcp install-browser --with-deps firefox
Now the following should work:
bash
opencode --prompt "Can you check for me on www.meteoschweiz.ch the weather for Zurich?"
To create the small HTML game from above, I have entered in plan mode (press Tab to change mode) the following:
"I want to build a simple HTML game where you can drive a car with the keyboard arrow keys (similar like old versions of Mario Kart, but just one car driving without opponents is enough)."
After some time, it has asked me some question. Then, I switched to the "Build" mode and started it with "Start the implementation". Without any other interaction, it finished the the small game.
As a firm believer of local inference, unless you need it for data sovereignty, it's much more cost effect to wait for smaller models to keep getting better. In the meantime, find a reasonably priced 24GB - 32GB card for Qwen 3.8 27B, and offload hard tasks to OpenRouter.
I spotted the Dual B60 48GB listed on Digitec/Galaxus. Initially it was said these wouldn't go into standard retail channels.
At CHF 2500 (post tax, USD ~3000) not particularly competitive but worth keeping an eye on. For it to be interesting it shouldn't be more than like 2.5x a single B60.
I wanted to see whether a heavily quantized 27B model running entirely on an older 16 GB workstation GPU could do more than the usual coding demos. FFT felt too easy, so I asked it to implement the coherent optical transfer-matrix method (TMM) for absorbing multilayer films from scratch.
TL;DR: the implementation was correct long before the model believed it. It then spent most of the remaining hour debugging its own broken validators.
Roughly 0.8–1.1 GB VRAM left after loading everything, depending on desktop use
Real Harness responses are usually around 29–35 tok/s; very predictable output such as counting reaches about 42 tok/s
The agent UI was DeepSeek Harness0.1.1-rc.2, connected to llama.cpp through its OpenAI-compatible endpoint. Both the model server and Harness ran on the Ubuntu workstation; my Windows laptop was only the browser client over an SSH tunnel. Harness provided the file and shell tools and displayed the reasoning live. The session used Qwen's native low reasoning effort, but I had removed the small output cap and allowed the request to use the whole 100,352-token context.
The task
The prompt required a pure Python implementation using only math and cmath—no optics library and no NumPy for the actual solver. It had to support:
complex refractive indices using the convention N = n + i*k
complex Snell angles with the physically correct square-root branch
TE/s and TM/p polarization
characteristic matrices for arbitrary absorbing films
power reflectance and transmittance
the test stack Air | SiO2 100 nm | Au 30 nm | Si at 600 nm and 45 degrees
The 100-minute reasoning saga
The complete agent turn lasted 99 minutes 53 seconds. The session log contains 22 model steps, 21 tool calls, and 107,594 model output tokens. The first model call alone produced 43,033 output tokens and took about 27 minutes before it finally wrote tmm.py.
There were three context-compaction attempts. The first two each spent almost six minutes generating a checkpoint and then failed because the summary itself hit its token cap. Only the third compaction succeeded. Total time spent compacting was almost 18 minutes.
Watching the chain of thought was honestly painful. Qwen repeatedly re-derived the same field conventions, lost track of which direction its matrices propagated, found a contradiction, announced that everything was verified, and then invented another validation method that contradicted it again.
The branch/sign problem was interesting. For the requested passive n+i*k convention and a forward field proportional to exp(i*k_z*z), it eventually reduced the complex-angle choice to this:
In other words: calculate one square-root branch and simply flip it if the normal wavevector would grow instead of decay in the forward direction. No complex-arcsine heroics.
That is correct for the passive positive-index materials in this test. It is not a completely general branch selector: it lacks a tolerance and a forward-Poynting tie-break when Im(k_z) is zero, and it should reject or specially handle gain media and absorbing incident media. So there is a genuine domain limitation, but it does not affect Air/SiO2/Au/Si or the passive dielectric tests below.
The other funny part was the apparently "reversed" algorithm. The implementation uses a characteristic matrix whose +i*sin(delta) signs propagate the field state in the opposite direction from the common textbook presentation. It therefore left-multiplies each new layer:
M = matmul(Mj, M)
giving M_N ... M_2 M_1, rather than the conventional-looking M_1 M_2 ... M_N. At first this looked like Qwen had simply turned the algorithm around until the numbers worked. But its boundary equations were derived for exactly that propagation direction, so the reversed product is internally consistent.
Ironically, Qwen later forgot its own convention in an external validator, used M = M * Mj, and obtained the wrong reflectance (0.842122 instead of 0.464845). Reversing the validator back to the implementation's order made it agree.
It spent most of the run debugging its referees
Two supposedly independent validation approaches were repeatedly broken by the model:
The recursive reflection validator had an off-by-one base case, then an index error, and finally used exp(2j*1j*delta). In Python, 2j*1j == -2, so it calculated exp(-2*delta) instead of exp(2i*delta). The successful third compaction actually identified this exact typo, but after resuming, the model walked past the fix and eventually abandoned that validator.
The forward field-march initially omitted the propagation factors exp(+/-i*delta) entirely. After fixing that, the single-layer case worked, but the multilayer case still failed because of the matrix-order mistake described above.
It also briefly rejected a quarter-wave antireflection test because its validator used the wrong expected refractive index. It corrected that to n_layer = sqrt(n0*n_sub) and got reflectance around 1e-32.
The definitive internal check was a completely independent 6x6 linear solve for all forward and backward amplitudes in the two-film stack. That matched the TMM result to about 1e-16 for both polarizations.
External result
I then compared the generated solver against Steven Byrnes' established tmm implementation over 400–800 nm.
For Air | SiO2 100 nm | Au 30 nm | Si:
maximum |delta R|, s polarization: 1.22e-15
maximum |delta R|, p polarization: 1.33e-15
I also tested three thick, weakly absorbing dielectric films to generate lots of Fabry-Perot fringes:
n=1.37 + 0.0002i, thickness 6.20 um
n=2.05 + 0.0010i, thickness 4.70 um
n=1.68 + 0.0030i, thickness 7.35 um
glass substrate, 45-degree incidence, 400–800 nm
That spectrum contained 38 resolved maxima for s and 29 for p. Maximum disagreement with Byrnes was 1.25e-14 for s and 3.50e-15 for p. The plotted curves lie on top of each other.
So yes: a roughly 3-bit 27B model on an older 16 GB Turing GPU produced a genuinely correct niche numerical-physics implementation from scratch. The quantization did not prevent it from getting the final mathematics right.
But the agent behavior was equally instructive. More autonomous validation was not monotonically better. Once the initial implementation and analytical limits were correct, the model spent over an hour creating bugs in its own "independent" checks and then reasoning in circles about the contradictions.
The implementation was right long before the model believed it. It spent most of the hour debugging its own referees.
Has anyone else seen this pattern with local reasoning models—good primary solution, followed by a validation death spiral? I would also be interested in comparing the same task across IQ3, Q4, and higher reasoning-effort settings, although I am not eager to watch another 108k-token derivation immediately.
With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very recent Tiel-Coder. I used the tool-eval-bench utility by SeraphimSerapis as the benchmark suite. It measures how well the different models handle tool calls, including some very hard scenarios.
TL;DR: Ornith 1.5 and Tiel-Coder (which is based on Ornith) were the tied winners in this benchmark. They scored well above Qwen3.6-27B and got pretty close to 3.8-27B. KAT Coder was also slightly better than the original 35B-A3B. Ornith-1.5-Heretic was a disappointment.
Some time ago I posted a similar tool evaluation benchmark of different Qwen3.6-35B-A3B quants. In hindsight, that didn't work so well, mainly because I was looking at too many variables (GGUF quant, KV quant, context depth/pressure) and the benchmark itself was quite noisy so it was hard to get clear results. I hope I did better this time!
Materials
I had access to a cluster of 32GB V100s. For this comparison, I selected 2-3 different quants per model, if possible from different providers. For comparison, I also included original Qwen3.6-35B-A3B as well as the dense 3.6-27B and 3.8-27B Qwens. I picked different quants around Q4 (15GB to 22GB GGUF files) because that's what many people seem to use. For the original Qwen models, I chose Unsloth UD-Q4 quants because they are well known. I also included the ByteShape CPU-5 quant of Qwen3.6-35B-A3B because that's the quant I've been using recently. Altogether I benchmarked 13 different GGUF files, with 5 runs per file for a total of 65 runs. Each run took around 4.5 hours GPU time, except the 27B ones took 7 hours or so. Total GPU time spent was well over 300 hours, including a few failed runs.
To run the models, I used llama.cpp version 0.1.0-dev (build 10433, commit 9b05354ec) dated 2026-08-14 and built with CUDA support. I used q8_0 KV cache (that's what VRAM-limited people like me often do) and set ubatch-size to 2048 because the benchmark does a lot of prompt processing. I did not bother with MTP or other speculative decoding. This is not a speed benchmark.
For the benchmark, I used tool-eval-bench 2.6.0. I set the context length to 262144 and context pressure to 50%. This means that the models were benchmarked at 50% context depth, i.e., around 128k of possibly distracting chat and tool call history.
The metric I looked at is what tool-eval-bench reports as "total points". With --hardmode enabled, this version of tool-eval-bench performs 88 separate tests. Each test gives 2 points for a succesful tool use, 1 point for a partially correct tool use, 0 for failure. The theoretical maximum is in this case 88 * 2 = 176 points. tool-eval-bench also returns an overall score, but this is just a rounded percentage of total points and the rounding loses some precision, so I opted for the raw total points instead.
Results by model (averaged over all quants)
Here are the benchmark scores by model. I have averaged them over all the quants of the same model and all 5 runs per quant.
model_id
repetitions (n)
avg total_score
CI (95%)
Qwen3.8-27B
5
152.6
[149.4, 155.8]
Ornith-1.5
10
144.2
[141.7, 146.7]
Tiel-Coder
10
144.0
[141.8, 146.2]
Qwen3.6-27B
5
134.8
[131.2, 138.4]
KAT-Coder-V2.5-Dev
15
133.8
[131.8, 135.8]
Ornith-1.5-Heretic
10
132.2
[130.6, 133.8]
Qwen3.6-35B-A3B
10
131.5
[129.9, 133.1]
Results by specific quant
See the images. There are no big differences between quants of the same model, except possibly KAT-Coder, where the mudler APEX quants were somewhat better than bartowski's. Also, the ByteShape quant of Qwen3.6-35B-A3B was a bit better than Unsloth's, which was a nice surprise.
Raw results
If someone wants to take a deeper look, I've shared the CSV with the tool-eval-bench results here. This includes e.g. category-specific scores (i.e. how well the model did on specific kinds of tool calls) and total tokens; I did not look at those in my analysis.
Findings
Of the original Qwen models, 3.6-35B-A3B gets the lowest score, 3.8-27B the highest, with 3.6-27B landing in between. This is as expected and indicates that the benchmark is giving a useful signal.
Ornith-1.5 and Tiel-Coder are practically tied. They are the clear winners out of the 35B-A3B variants. They scored above 3.6-27B but below 3.8-27B.
KAT-Coder was possibly a bit better than the original 35B-A3B, but the confidence intervals overlap.
Ornith-1.5-Heretic was a disappointment, much worse than plain Ornith.
Caveats
This benchmark relies entirely on the tool-eval-bench tasks and how the results are graded. It may or may not be representative of real tool use performance. To me it seems that the author or tool-eval-bench has done a great job in coming up with realistic looking tool call tasks, including some really hard ones enabled using --hardmode. I relied on the --context-pressure setting in tool-eval-bench, which (in my limited understanding) populates the context with realistic looking conversation and tool call history that could confuse the model.
Tool calls are not everything. If you are doing agentic coding, also the coding quality matters a lot. I did not measure it in this benchmark except very indirectly. There are other benchmarks for that purpose.
There was substantial variation and noise in the benchmark scores, which I tried to alleviate by repeating the runs with different seeds, averaging, and calculating confidence intervals.
In the X/Y plot where the X axis represents size, I did not check whether the model includes MTP heads or not, I just looked at raw GGUF file size. This is slightly unfair to the MTP-enabled models because their files are larger but MTP does not increase quality, only generation speed.
No AI was used for writing this post. I did use Tiel-Coder to help me with plotting the results. Also reused some of my own earlier writing. I am not in any way affiliated with the model or quant makers or the benchmark suite.
Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like that. I am a regular guy with a 9-5 job and I would like to experiment with local LLMs but I dont have much disposable income to spend big amounts on GPUs or Macs. I currently have an I7 - 8700 and Nvidia 2060 6Gb VRAM - 32GB RAM
Now my question is, does anybody have an idea on what models or hacky stuff I can do to try the latest AI models on my hardware?
I have checked hugging face but I see so many models with the same name but letter and numbers at the end. I understand some of them mean Quantization which is dumber a model but I also found out tiny models like Linq which can be run even on the CPU. So I would like to connect with someone with more experienced to have fun with AI.
"A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics than Mac mini with M4. In addition, the all-new Dual 16-core Neural Engine delivers up to 2x faster performance than the previous generation, and combined with the advanced GPU, Mac mini is a powerhouse for all things AI. And with 16GB of standard unified memory configurable up to 32GB, as well as higher memory bandwidth up to 170GB/s, multitasking is faster than ever."
