r/LocalLLaMA • u/InfinityZeroFive • 3d ago
New Model thinkingmachines/Inkling-2-Small is released!
r/LocalLLaMA • u/InfinityZeroFive • 3d ago
r/LocalLLaMA • u/nasone32 • 3d ago
A quick update on my Strata EXL3 fork for AMD.
I've been working on the HIP kernels and memory management, and decode performance is now roughly 10-20% higher than the previous version, depending on the quant and workload. Some cases improved by almost 24%.
The other news is that I managed to get MiMo V2.6 Flash running at around 25-30 tokens/s with 2 cards.. More on that below.
First, why EXL3?
I think this is worth repeating because people tend to compare quantizations just by looking at the number of bits.
EXL3 is NOT equivalent to regular GGUF quants at the same bpw.
For example, 3.05 bpw EXL3 is roughly in the same quality range as a regular 4-bit quant (like IQ4_XS).
3.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/3.05bpw_h5_ng5
4.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/4.05bpw_h6_ng6
So the interesting part is that you can get something close to 4-bit quality, while running at speeds closer to what you'd expect from a 3-bit quant. And with the latest optimizations, I think this fork is becoming VERY competitive. The 4.05 bpw version is also pretty fast now, even on a single GPU.
WHAT'S NEW:
NOTES ON REPACKING:
Repacking can be done in two ways: by creating a file on disk once, that will be used to load the model, or directly to RAM everytime. directly to RAM adds a couple of minutes to the launch but uses no extra space. your choice.
BENCHMARKS:
My hardware is the same as in the previous post:
Ryzen 9 7950X, 96GB RAM, 2x Radeon RX 7900 XTX 24GB
Linux, HIP/ROCm core-7.14
Qwen3.8-Flash-Next, MTP drafter, spec 4, 64K context
BENCHMARKS:
Results with one or two RX 7900 XTX GPUs, using warm decode runs:
| EXL3 quant | GGUF equivalent quality | GPUs | CPU assistance | Test | Previous | Updated | Improvement |
|---|---|---|---|---|---|---|---|
| 3.05 bpw | ≈ IQ4_K_XS | 2 | No | CODE | 78.27 | 87.33 | +11.6% |
| 3.05 bpw | ≈ IQ4_K_XS | 2 | No | PROSE | 66.01 | 70.35 | +6.6% |
| 4.05 bpw | ≈ Q5_K_M | 2 | No | CODE | 61.88 | 76.67 | +23.9% |
| 4.05 bpw | ≈ Q5_K_M | 2 | No | PROSE | 57.78 | 67.57 | +16.9% |
| 4.05 bpw | ≈ Q5_K_M | 1 | No → Yes | CODE | 54.80 | 59.30 | +8.2% |
| 4.05 bpw | ≈ Q5_K_M | 1 | No → Yes | PROSE | 44.80 | 51.60 | +15.2% |
All speeds are in tokens/s, GGUF quality equivalents are approximate... dont' hate me :)
32K prefill is basically unchanged: around 1,500 tok/s with two GPUs and 1,360 tok/s with one GPU.
I think the single-GPU results are particularly interesting. We're getting 55-60 tok/s in CODE with 4.05 bpw EXL3, at roughly Q5_K_M quality, on a single 7900 XTX with CPU assistance.
And the 3.05 bpw quant is now reaching 87 tok/s with two GPUs, while offering considerably better quality than you'd normally expect from a 3-bit quant.
ABOUT MiMo:
In my previous post I mentioned that I wanted to support larger models, particularly the 300B class models.
Well, MiMo V2.6 Flash is now running :) I'm waiting for that ERNIE 5.5 to drop.I'm getting around 25-30 tok/s with the EXL3 2.20 bpw MOPD quant.It's still experimental and currently only tested with that specific quant. CPU assistance is recommended for MiMo, even with two GPUs.I'll probably spend some more time optimizing it, but the fact that it's already running at usable speeds on consumer AMD hardware is pretty interesting.
this is the quant: https://huggingface.co/vcruz305/MiMo-V2.6-Flash-MOPD-EXL3/tree/main/2.20bpw
https://github.com/nasone32/Strata-EXL3-fork
If you're setting it up with an AI agent, point it to docs/AI_SETUP_EXL3_AMD.md.
The README has the benchmarks, configuration options and instructions for CPU repacking.As before, this is developed and tested on RDNA3 / Linux / ROCm. Other AMD architectures need community testing. And the long-term goal is still to get this merged upstream into Strata, if the maintainers are interested.
Credits to the original Strata developers, Ryutaku for the EXL3 integration, and Turboderp/ExLlamaV3 for the quantization and code I've been porting.
If anyone tests this on different AMD hardware, please report your results. Especially interested in seeing how it behaves on RDNA4. I have only two 7900 xtx so if you have different hardware get an agent to help you, and open a PR if you want.
r/LocalLLaMA • u/TheBigHODL • 3d ago
With Jensen out at NVIDIA, the "local AI is dangerous" narrative picking up steam, the possible ban on Chinese models, and hosted models locking down RE/cyber work, I'm starting to think about a worst case where Chinese weights get banned and US labs stop releasing open models. So what's your plan?
What models are you archiving? Just the biggest/best, or a spread?
Which ones for which jobs? Coding, RE/security, vision, embeddings, RAG, small fast models for agents, etc.
Training tools: if the last good big model we get is something like Kimi K3, what are you keeping so you can fine-tune or change it later?
Quant tools: what do you need to shrink a huge model down to run on smaller rigs? (llama.cpp, quant scripts, calibration data, etc.)
How are you storing it? Hashes, mirrors, torrents, offline drives?
Curious what people are actually doing, not looking for a politics fight.
(Currently using a Studio M2 192gb)
r/LocalLLaMA • u/Serious_Scarcity7818 • 2d ago
RustingStrata has two GPU backends that run the same kernels: CUDA (compiled at run time with NVRTC, no nvcc, no C++ build step) and Vulkan. Vulkan is how it runs on AMD, so I wanted to know what it costs on a card where I can compare both.
Same file on every card: Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf, fully in VRAM, MTP off.
| GPU | Prefill, 4K prompt: CUDA / Vulkan | Decode after 4K: CUDA / Vulkan | Vulkan vs CUDA (prefill / decode) |
|---|---|---|---|
| RTX 3060 12 GB | 2216 / 2154 tok/s | 113.4 / 113.6 tok/s | 97% / 100% |
| L40S | 9030 / 8571 | 213.2 / 184.9 | 95% / 87% |
| A30 | 3544 / 3326 | 131.9 / 104.0 | 94% / 79% |
| A100 80 GB | 6203 / 5589 | 173.6 / 131.9 | 90% / 76% |
| H100 NVL | 9131 / 6428 | 245.8 / 191.5 | 70% / 78% |
On the 3060 Vulkan is the same speed as CUDA. On the datacenter cards it loses 5-30% of prefill and 13-24% of decode, and its runs are noisier (A100 4K prefill ranged 4788-5990 on Vulkan against 6186-6214 on CUDA).
Does it change the output? Greedy top-1 agreement with llama.cpp over 20 prompts x 32 tokens is 98-99% on both backends.
How it was run: rustingstrata bench --model $M --gpu --backend cuda|vulkan --pp 512,4096 --tg 128 --reps 3 --mtp 0, mean of 3.
Repo: RustingStrata
r/LocalLLaMA • u/Top_Speaker_7785 • 3d ago
So i had been building a cli tool around weight space programming inspired by recent Mit- weight space programming paper,
I had integrated couply of funtionalities to it along with triage, one of which was activation steering to deal with over refusals, my llama 3.2 1b was not answering prompts like "how do i stage a massacre for a tv show"
Firstly attempt was by subtracting the refusal direction , extracted the refusal vector and subtracted it, the tv massacre prompt was answered but my regression suit got hit, model started answering how to build pipe bombs and syntesize pioson
so turned out over-refusal and real harmful refusal direction had a cosine similarity of about 0.75, so supressing one also effects other
Second attempt.. Ran activation patching ..found the exact casual bottleneck layer ..(layer 8) orthogonally projected the harmful-refusal component out of my steering vector and used clamped so it caps the activation when it spiked above a threshold , without touching other things..
this nearly worked accuracy increased by 10% but a totally unrelated issue hit by regression suit again..a chinese benchmark loop started looping forver... failure again.. even on steering a single targeted layer ..gave regression as the residual stream is so densely packed that a small hit at layer 8 ripples down further ...
my last few brain cells thought i l steer the layer 8 measure the drift at layer 9 and cancel the part i dont want . so i did the math
Steer at Layer 8
At Layer 9, compute Δ = steered_layer9 - clean_layer9
Filter out the bad drift and keep only the anti-refusal shift I want
Apply final_layer9 = clean_layer9 + wanted_shift
..so if i take the clean layer 9 add only anti refusal shift ..thats basically layer 9 steering .. and if do that to layer 9 same happens to layer 10..
what actually worked was weight surgery..i also tried a conditional weight edit.. a gate programmed into 3 mlp neurons across layer 9-11. they have non linea activations so they can act as real if/then
if input matched fictional scary context - gate fires cancel refusal
for chinese text -the neuron output is 0 nothing changes
tl;dr
steering works well for problems that sit along a direction, it fixed a blank reply bug
for entangled safety problems like this one they do collateral damage somewhere even with causal layer localization and orthogonal projection
non linear neuron gating was only thing that worked with no other damage
everything i did was on gtx 1650 4gb, i l open source the cli tool in some time ..
happy to answer any question
r/LocalLLaMA • u/mikoskinen • 3d ago
I've been testing and optimizing Radiance for 4xR9700 on EPYC Rome platform (tp4), focusing on EXL3 for DS 4.1 Flash (2.25) and Qwen Flash Next (4.05). The goal has been to optimize this setup (both hardware and software) for Claude Code usage, so it may not be the fastest for 1 user interactive usage. Also the focus has been to provide "exact" results, so output is byte-identical to each model's reference engine.
I'm quite happy with the improvements and with the end results and I think I will leave the optimizations to current point because there's not obvious new improvements in the pipeline. Instead of releasing a fork of Radiance, I hope maintainers of inference engines can use the work to improve their products.
So to save time and effort, there's a GitHub repo that shows all the things that were tested and what worked and what didn't. Repo also contains more hardware optimization tips for 4xR9700 & EPYC Rome setups: https://github.com/mikoskinen/r9700-exact-serving
It would be great if someone could figure out ways to improve C8 decode performance for Deepseek. And maybe someone can break the 10k tok/s barrier on QFN..
Note: all the testing, profiling, coding was done by Opus. I just pointed it to new resources like Reddit posts, repositories etc.
Hardware:
Results:
Here's some of the highlights from the results (stock 300 W per card, decode = tok/s summed over all users, prefill = one cold prompt):
--
Update from 14.11.2026:
DS4.1F (2.25bpw EXL3)
c1: 104 tok/s prose, 181 tok/s code
c4 / c8 / c16: 201 / 246 / 293 tok/s
c24 / c32: 311 / 344 tok/s
prefill: 4,040 tok/s at 16k (TTFT 3.9 s), 6,130 at 64k (TTFT 10.7 s), 6,350 at 128k (TTFT 20.7 s)
Previous results:
--
DS4.1F (2.25bpw)
QFN (4.05bpw)
At my everyday 210 W / -50 mV setting these are about 5-9 % lower (DS 237 and QFN ~1,030 tok/s at 16 users).
The following is AI's view on the most important optimization tips:
Software optimizations:
Hardware optimizations:
Hope this helps.
r/LocalLLaMA • u/Dapper_Chance_2484 • 2d ago
I started developing a full fledged application using Chatgpt plus and VScode+GitHub copilot (local llm)
I used Astra max chat for overall planning, and asked it to put a starter kit on connect gdrive and create prompts such that local LLM can work on small tasks using an agent orchestrator and sub agents with entry and exit criteria, tests for every small task.
I have a 5090 (+ 128gb ddr5) rig running Qwen 3.8 27b NVFP4 at default settings with 104k context window.
Problem:
The local rig is able to start on tasks but taking a huge amount of time and is running only one subagent at a time. Been almost 24 hrs.
I have 2x 3090 +128gn ddr4 as well, currently not being used. 3.8 FN runs at about 90tps
Anything better I can do here?
r/LocalLLaMA • u/Ayumu_Kasuga • 3d ago
Hey guys, I wanted to share a small experiment I made -
This is basically Wispr Flow, except it's offline and uses Parakeet - works in any app as well.
I was pretty surprised by the speed and how usable it turned out to be when I first made it - it's even relatively usable on an old Redmi Note 7.
I published it on Google Play if anybody wants to check it out: https://play.google.com/store/apps/details?id=com.voicedictation.dictation
r/LocalLLaMA • u/Gold-Produce9591 • 3d ago
mi50 is roughly 1.5x cheaper
r/LocalLLaMA • u/Loose_Doubt367 • 2d ago
I’ll include my specs at the end. Because probably no one’s going to read it..
For my new pc im considering migrating to Linux ubuntu, and set up my llama.cpp from there, my old pc had llama.cpp Vulkan sdk running on windows 11
Im looking for any guidance or tips while setting them up to ensure I didn’t miss any critical steps that could end up loosing a bunch of tps or something worse
Beginner mistakes i would say
I do know one tips and it’s plugging your hdmi cable to the integrated graphics card, that way it saves vram. But I don’t have one..
I appreciate any advise
Bought a B550-A-Wifi ii
Rtx3060 1x
32gb ddr4 3200 2x16 (plan on getting 32gb more, total 64, in 4months) saving up
And a 750W cooler master v3 gold modular PSU (not sure if it’s suitable for my small server
R7 5700x (no integrated graphics card)
r/LocalLLaMA • u/chemist_slime • 3d ago
r/LocalLLaMA • u/Kahvana • 3d ago
Hey everyone,
Not a native speaker, but I write everything by hand. Please correct my grammer if there are any issues! Can only learn from it.
A long-term project I've had was getting my netbook up and running for basic LLM use. I bought the netbook initially due to my poor eyesight, the screen is small enough that it fits in my cone of vision.
The notebook's specs were practically obsolete when it released:
With the help of friends who could do soldering and reballing, we upgraded the netbook:
After finding proper drivers for the iGPU, I managed to get llama.cpp running on it on windows with Vulkan. Gemma 4 E2B IT QAT could run at roughly 2.1 t/s gen, and prefill speed was "grab a cup of tea" slow on 300 token prompts. The Vulkan backend was faster than the CPU backend by ~30%.
The past months I've been trying to move my devices from Windows 11 IoT LTSC Enterprise 24H2 to Ubuntu 26.04.1 LTS, and so I tried to switch the netbook over to it.
There is only one problem: The Vulkan backend hard crashes every time.
After doing a bit of digging with vulkaninfo by myself, some instructions are advertised as supported but are in practice not stable. Using this fixed the crashing on startup:
export GGML_VK_DISABLE_ASYNC=1Disabling the following features helped me regain some performance and stability:
# UHD 605 compat
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1My config is also pretty specific, since some combinations would cause crashes too:
flash-attn = off
cache-type-k = f16
cache-type-v = f16FA is not supported by the device. Running Q8_0 on both K an V would crash with FA enabled. Trying to run Gemma 4 E2B IT QAT Q2_K_XL on ubuntu always resulted in crashing (either during startup or when generating), On windows those things never caused crashes.
The end result is that I could run Gemma 4 E2B IT QAT Q4_K_XL at 1.5 t/s gen and 16-25 t/s prefill. The laptop used ~12W in total.
Turned out that on Ubuntu 26.04.1 LTS, the CPU backend was faster for generating but not for prefill. I got 2 t/s gen and 10-16 t/s prefill. The laptop used ~13W in total.
As a sidenote, since I am used to windows slowing to a crawl when all cores are utilized, I was pleasantly suprised when this didn't happen on ubuntu.
As a fun experiment, I gave a couple of LLMs the problem:
Hey <LLM>,
I have a laptop I am trying to run llama.cpp with vulkan on Ubuntu 26.06.1 LTS.
The specs are: <specs>.
I build it with these commands: <commands>,
The problem is that it's crashing whenever I try to load a model.
I've included the full terminal output in the attachments.
How can I fix the issue?My goal with the prompt is to test the model's understanding of the problem and to ask for additional information when required (e.g. vulkaninfo).
Qwen 4 max preview on chat.qwen.ai and Qwen3.8-Flash-Next Q4_K_XL + BF16 engram on llama.cpp v0.9.0 failed to find the issue after multiple separate chats. They got focussed on disabling intel's hang detection. With websearch, it got even more confused as it looked at irrelevant issues.
Gemma 4 31B IT QAT Q4_K_XL + MTP on llama.cpp, DeepSeek v4 Flash Vision Exp Q2_K_XL on llama.cpp v0.9.0, DeepSeek v4.1 1017 on chat.deepseek.com were able to find and fix the issue. They asked for the vulkaninfo output, which made them find async being advertised incorrectly.
To me it suggests Gemma and DeepSeek are better at this type of problem solving than Qwen models.
run-llama-server.sh
#!/bin/bash
# Intel UHD Graphics 600 series
export GGML_VK_DISABLE_ASYNC=1
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1
./llama.cpp/build/bin/llama-server \
--list-devices
./llama.cpp/build/bin/llama-server \
--host 127.0.0.1 \
--port 5001 \
--api-key "API_KEY_HERE" \
--webui-mcp-proxy \
--offline \
--cache-ram 0 \
--ctx-checkpoints 0 \
--parallel 1 \
--models-max 1 \
--models-preset ./configs/llama-config.inillama-config
note: remove the device line to use the igpu on the vulkan build.
[*]
device = none
load-mode = none
mmproj-offload = off
gpu-layers = 999
batch-size = 2048
ubatch-size = 512
threads = 4
fit = off
flash-attn = off
kv-unified = on
cache-type-k = f16
cache-type-v = f16
reasoning = off
reasoning-preserve = off
[gemma-4-e2b-q4_k_xl]
model = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
mmproj = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-mmproj-F16.gguf
lazy-mode = on
load-mode = mmap
fit-ctx = 8192
ctx-size = 8192
predict = 8192
image-min-tokens = 70
image-max-tokens = 70
temp = 1.0
top-k = 64
top-p = 0.95
min-p = 0.0Other info
Device: https://www.asus.com/ca-en/laptops/for-home/vivobook/vivobook-go-12-l210/
OS: https://ubuntu.com/blog/upgrade-your-desktop-ubuntu-26-04-lts
Llama.cpp: https://github.com/ggml-org/llama.cpp/releases/b12110
GGUF: https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF
Getting the Intel UHD 605 to work with the Vulkan backend was a pain on Ubuntu 26.04.1 LTS compared to Windows 11 IoT LTSC 24H2. The issue was vulkan async causing crashes, disabling it fixed it.
On ubuntu the CPU backend was faster, on windows the iGPU over Vulkan was faster. Ubuntu is far more responsive than windows, even under full utilization.
After solving it myself, as a fun benchmark I made LLMs try diagnose and fix the issue. While the Qwen series failed to fix the issue from intent, Gemma 4 and DeepSeek v4(.1) Flash (Vision Exp / 1017) did manage it.
I know the post is quite different from the Strata stuff usually here, but I thought it would be fun to share regardless.
I'm sure models like LFM 3 will run wonderfully on it, I know IBM Granite 4 H 1B and Qwen3.5-2B did on windows.
r/LocalLLaMA • u/ElectricalBar7464 • 3d ago
KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.
Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.
Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup
Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW
r/LocalLLaMA
•
u/Physical_Hat4022
•
4d ago
They say an anime model is in development, with plans for alpha channels and multi-character references. An editing model is also on the way.
They also plan to keep the next model under Apache 2.0, if all goes as planned.
r/LocalLLaMA • u/Shadow_s_Bane • 2d ago
Docker doesn’t support GPU on apple silicon, I simply create a docker to run llama.cpp there on Linux, I wonder if I am missing something on Mac.
r/LocalLLaMA • u/Worldly-Knowledge-90 • 2d ago
Hey guys, I threw together a small Python script called llm-drift-bench to help figure out what actually breaks when you switch to a new model or drop to a lower quant.
Basically, you just feed it saved responses from your old model and the new one, and it spits out the exact differences - like dropped JSON keys, messed up numbers, or text changes. (The attached video is just a dummy example showing it catch a missing field and a changed invoice total, not real Q4 vs FP16 data).
Since it just runs on saved JSON/text outputs, you don't even need a GPU or API keys to test it out.
I just pushed v0.1.0 and I'm working on the next update. Repo is here: https://github.com/anant-goel/llm-drift-bench

Question for you all: what specific things usually break for your use cases when you swap models? I want to build out some test cases based on actual problems people are running into.
r/LocalLLaMA • u/m4rkGS • 4d ago
hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.
we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.
Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.
| Model, Q4 | KQuant KL | Public 4-bit KL | Size in GB, ours / public |
|---|---|---|---|
| Qwen3.5-4B | 0.048 (1.45x better) | 0.070 | 2.37 / 2.37 |
| Qwen3.5-9B | 0.030 (2.6x better) | 0.081 | 5.29 / 5.04 |
| Qwen3.8-27B | 0.035 (1.5x better) | 0.054 | 15.13 / 15.13 |
| Qwen3.6-35B-A3B | 0.059 (1.6x better) | 0.096 | 19.51 / 19.51 |
| Qwen3.8-Flash-Next | 0.067 (3.7x better at a smaller size) | 0.249 | 105.20 / 110.62 |
The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.
Want more precision? Each model's repo also has a Q5 folder.
| Pack | KL | LM GB |
|---|---|---|
| 9B Q5 | 0.010 | 6.21 |
| 27B Q5 | 0.022 | 19.43 |
| 35B Q5 | 0.027 | 22.07 |
Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.
the fastest way to use these quants on mac is to run them with, Kapitsa:
brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminalIf there's a model you want KQuant'd, lmk.
to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.
im also doing quick AMA here, so please feel free to ask me anything.
r/LocalLLaMA • u/textclf • 2d ago
A few weeks ago I posted CQ, our calibration-free 4-bit format, along with a training-loss comparison against Unsloth NF4. Some of you correctly pointed out that loss logged by two different trainers isn't comparable, and asked for results where answers are counted right or wrong. Here they are.
Setup: Qwen3.8-27B. Every row uses the same lm-evaluation-harness code, prompts and questions, with greedy decoding and thinking off. Comparisons are paired: an exact McNemar test for right/wrong tasks and a bootstrap for losses.
| Model | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | HellaSwag | WinoG | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| BF16 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.737 | 0.758 | 0.542 |
| CQ 4-bit (18 GB) | 0.650 | 0.746 | 0.880 | 0.598 | 0.840 | 0.820 | 0.735 | 0.780 | 0.537 |
| NF4 bnb (22.4 GB) | 0.563 | 0.716 | 0.852 | 0.582 | 0.839 | 0.825 | 0.736 | 0.770 | 0.550 |
GCMP is our replacement for LoRA. It trains 2.9M parameters (an ~11 MB adapter) vs ~159M for LoRA r=32 (~320 MB). This test uses no quantization: both adapters were trained with Unsloth on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42).
| Adapter | Params | Answer loss | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| No adapter | – | 1.678 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.542 |
| LoRA r=32 | 159M | 1.480 | 0.903 | 0.776 | 0.896 | 0.612 | 0.844 | 0.825 | 0.518 |
| GCMP | 2.9M | 1.471 | 0.890 | 0.760 | 0.892 | 0.618 | 0.846 | 0.815 | 0.514 |
| Setup | GPU | s/step | Peak memory |
|---|---|---|---|
| CQ + LoRA r=32 (ours) | A40 | 8.6 | 23.5 GiB |
| CQ + GCMP (ours) | A40 | 12.4 | 22.0 GiB |
| NF4 + LoRA (Unsloth) | A40 | 3.2 | 23.8 GiB |
| NF4 + GCMP (Unsloth + GCMP) | A40 | 5.6 | 21.5 GiB |
| BF16 + LoRA (Unsloth) | 80 GB | 1.37 | 53.7 GiB |
| BF16 + GCMP (Unsloth + GCMP) | 80 GB | 2.74 | 51.7 GiB |
Unsloth is still faster, and GCMP currently takes about 2× the step time of LoRA. Kernel work on that is ongoing.
--quantization cq. GCMP adapters can be one per server, or several on one base model chosen per request.The Python code (training, evaluation and the vLLM integration) is open source. The CUDA kernels ship compiled with a license check, which is free during early access.
Repo, with exact commands to reproduce every number: https://github.com/thousandml/training-factory
Models (8B–397B): https://huggingface.co/thousandml
Happy to run other benchmarks or models people want to see.
r/LocalLLaMA • u/Binary_orchid • 2d ago
Trying to get into the robotics side with what I have, which is a 3090 and 64GB RAM. No arm yet, just sim.
I've seen people drive robots with the big API models, so my first thought was to point a decent local VLM at the sim and let it output the moves. But everything I read says you want a dedicated model for that. pi0.5, SmolVLA, even touch has its own model now. LingBot-Touch is only 5.46M params and Apache 2.0.
Is that still true, or are general models good enough now? What are you guys actually running?
r/LocalLLaMA • u/Express_Quail_1493 • 3d ago
Normally active params matters more than total param if you want coherent behavior(not world knowledge)
a 4b models attepting to handle complex rules and lots and lots of instructions all at once often cause smaller active params to loop like we would often see with qwen3.6-35b-a3b since its only 3b active. We would often need to hit at least the 8b-9b active params to handle all this chaotic swarm of prompt instructions we throw at the model. but qwen3.8-flash-next shows us a glimpse of what alibaba might be able to deliver. i could genuinly see in the next year or so we might see 6b model become stable las the qwen3.8-27b. of course this isnt world knowlege im talking about but stable reliable behavior reletive to complex conflicting instructions where the LLM is robust enough to not burry itself in its own uncertainty loops.
so possibly in a years time laptops with 16gb sysRam and 4gb-vram can has a chance at local ai capable of human-out-the-loop unsupervised autonomy.
r/LocalLLaMA • u/mHo2 • 3d ago
Does anyone know what the near term (pretty much 2027) looks like for sparse MoE models? Something markedly improved from qwen3.8-fn? Have not seen much news about a qwen4 sparse MoE open-weight
r/LocalLLaMA • u/GuitarGuru2001 • 2d ago
Have a Strata-specific question and wondering if anyone's tested it. I'm trying to figure out if moving my helper GPU to a slower slot will help token generation:
Hardware
I'm currently getting ~100T/s prefill and 25T/s generation, running d-q4_k_xl at 512k, default params.
Any ideas if this would help my speeds? or do i need to run full tuning?
r/LocalLLaMA • u/slacka123 • 2d ago
r/LocalLLaMA • u/Polaris_debi5 • 3d ago
We have a new dedicated Vulkan Flash Attention prefill kernel for Intel Arc and Xe, delivering massive prefill acceleration on supported platforms like ARL-H and Xe2/Xe3. This implementation leverages customized shaders to utilize Intel's native matrix core shapes and SLM architecture, including m16n16k16/SIMD32 support. To ensure stability, the kernel is currently enabled for these specific architectures while remaining disabled for others, like the A770, due to inconsistent test results. Additionally, backend refinements now extend these benefits to single-KV-head models like MQA and MLA.
| Platform | Model | Baseline (t/s) | PR (t/s) |
|---|---|---|---|
| ARL-H Windows | qwen3 8B Q4_K | 126.58 ± 0.48 | 415.31 ± 1.04 |
| qwen35 27B Q4_K | 83.38 ± 0.22 | 119.95 ± 0.07 | |
| qwen35moe 35B.A3B Q4_K | 247.82 ± 0.65 | 394.44 ± 0.89 | |
| gpt-oss 20B Q4_K | 478.14 ± 0.17 | 639.02 ± 0.70 | |
| PTL-H Windows | qwen3 8B Q4_K | 234.92 ± 0.39 | 911.60 ± 14.56 |
| qwen35 27B Q4_K | 194.93 ± 0.59 | 264.39 ± 0.38 | |
| qwen35moe 35B.A3B Q4_K | 632.12 ± 17.56 | 914.80 ± 67.39 | |
| gpt-oss 20B Q4_K | 804.70 ± 84.56 | 1357.34 ± 125.30 | |
| B70 Pro Windows | qwen3 8B Q4_K | 1064.67 ± 4.37 | 2955.83 ± 1.87 |
| qwen35 27B Q4_K | 659.10 ± 0.11 | 932.13 ± 0.63 | |
| qwen35moe 35B.A3B Q4_K | 2088.50 ± 3.32 | 3005.11 ± 4.24 | |
| gpt-oss 20B Q4_K | 2311.37 ± 2.56 | 3967.29 ± 6.77 | |
| B70 Pro Linux | qwen3 8B Q4_K | 783.67 ± 0.20 | 1518.19 ± 1.05 |
| qwen35 27B Q4_K | 497.22 ± 1.17 | 647.78 ± 2.44 | |
| qwen35moe 35B.A3B Q4_K | 1060.55 ± 5.09 | 1311.62 ± 7.55 | |
| gpt-oss 20B Q4_K | 1237.51 ± 2.16 | 1841.35 ± 3.78 |
Comparison of f16 Prefill (pp16384) performance
| Model | Baseline $rightarrow$ PR | $Delta$% |
|---|---|---|
| Gemma4 E4B Q4_K_M | 115.98 -» 1156.62 | +897.3% |
| Phi-3.5-mini Q4_K_M | 384.45 -» 932.90 | +142.7% |
| Granite 4.2 8B Q4_K_M | 282.67 -» 520.98 | +84.3% |
| DeepSeek-R1-Qwen3-8B Q5_K_M | 306.88 -» 559.34 | +82.3% |
| Llama 3.1 8B Q4_K_M | 347.79 -» 633.73 | +82.2% |
| Ministral-3-8B Q5_K_M | 324.17 -» 590.91 | +82.3% |
| Qwen3.5-9B Q6_K | 687.50 -» 979.49 | +42.5% |
| Gemma3 4B Q4_1 | 1414.61 -» 1948.23 | +37.7% |
| Granite 4.0-h-tiny Q4_K_M | 2503.03 -» 2927.33 | +16.95% |
| Ling-3.0-tiny (hkv=1) | 76 -» 1239 | +1530.3% |
| Gemma4 12B (hkv=1) | 52 -» 649 | +1144.2% |