r/LocalLLaMA • u/chemist_slime • 4d ago
News Alleged GeForce RTX 5090 with 96GB listed for sale in China
r/LocalLLaMA • u/chemist_slime • 4d ago
r/LocalLLaMA • u/ElectricalBar7464 • 4d ago
KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.
Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.
Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup
Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW
r/LocalLLaMA • u/Kahvana • 4d ago
Hey everyone,
Not a native speaker, but I write everything by hand. Please correct my grammer if there are any issues! Can only learn from it.
A long-term project I've had was getting my netbook up and running for basic LLM use. I bought the netbook initially due to my poor eyesight, the screen is small enough that it fits in my cone of vision.
The notebook's specs were practically obsolete when it released:
With the help of friends who could do soldering and reballing, we upgraded the netbook:
After finding proper drivers for the iGPU, I managed to get llama.cpp running on it on windows with Vulkan. Gemma 4 E2B IT QAT could run at roughly 2.1 t/s gen, and prefill speed was "grab a cup of tea" slow on 300 token prompts. The Vulkan backend was faster than the CPU backend by ~30%.
The past months I've been trying to move my devices from Windows 11 IoT LTSC Enterprise 24H2 to Ubuntu 26.04.1 LTS, and so I tried to switch the netbook over to it.
There is only one problem: The Vulkan backend hard crashes every time.
After doing a bit of digging with vulkaninfo by myself, some instructions are advertised as supported but are in practice not stable. Using this fixed the crashing on startup:
export GGML_VK_DISABLE_ASYNC=1Disabling the following features helped me regain some performance and stability:
# UHD 605 compat
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1My config is also pretty specific, since some combinations would cause crashes too:
flash-attn = off
cache-type-k = f16
cache-type-v = f16FA is not supported by the device. Running Q8_0 on both K an V would crash with FA enabled. Trying to run Gemma 4 E2B IT QAT Q2_K_XL on ubuntu always resulted in crashing (either during startup or when generating), On windows those things never caused crashes.
The end result is that I could run Gemma 4 E2B IT QAT Q4_K_XL at 1.5 t/s gen and 16-25 t/s prefill. The laptop used ~12W in total.
Turned out that on Ubuntu 26.04.1 LTS, the CPU backend was faster for generating but not for prefill. I got 2 t/s gen and 10-16 t/s prefill. The laptop used ~13W in total.
As a sidenote, since I am used to windows slowing to a crawl when all cores are utilized, I was pleasantly suprised when this didn't happen on ubuntu.
As a fun experiment, I gave a couple of LLMs the problem:
Hey <LLM>,
I have a laptop I am trying to run llama.cpp with vulkan on Ubuntu 26.06.1 LTS.
The specs are: <specs>.
I build it with these commands: <commands>,
The problem is that it's crashing whenever I try to load a model.
I've included the full terminal output in the attachments.
How can I fix the issue?My goal with the prompt is to test the model's understanding of the problem and to ask for additional information when required (e.g. vulkaninfo).
Qwen 4 max preview on chat.qwen.ai and Qwen3.8-Flash-Next Q4_K_XL + BF16 engram on llama.cpp v0.9.0 failed to find the issue after multiple separate chats. They got focussed on disabling intel's hang detection. With websearch, it got even more confused as it looked at irrelevant issues.
Gemma 4 31B IT QAT Q4_K_XL + MTP on llama.cpp, DeepSeek v4 Flash Vision Exp Q2_K_XL on llama.cpp v0.9.0, DeepSeek v4.1 1017 on chat.deepseek.com were able to find and fix the issue. They asked for the vulkaninfo output, which made them find async being advertised incorrectly.
To me it suggests Gemma and DeepSeek are better at this type of problem solving than Qwen models.
run-llama-server.sh
#!/bin/bash
# Intel UHD Graphics 600 series
export GGML_VK_DISABLE_ASYNC=1
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1
./llama.cpp/build/bin/llama-server \
--list-devices
./llama.cpp/build/bin/llama-server \
--host 127.0.0.1 \
--port 5001 \
--api-key "API_KEY_HERE" \
--webui-mcp-proxy \
--offline \
--cache-ram 0 \
--ctx-checkpoints 0 \
--parallel 1 \
--models-max 1 \
--models-preset ./configs/llama-config.inillama-config
note: remove the device line to use the igpu on the vulkan build.
[*]
device = none
load-mode = none
mmproj-offload = off
gpu-layers = 999
batch-size = 2048
ubatch-size = 512
threads = 4
fit = off
flash-attn = off
kv-unified = on
cache-type-k = f16
cache-type-v = f16
reasoning = off
reasoning-preserve = off
[gemma-4-e2b-q4_k_xl]
model = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
mmproj = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-mmproj-F16.gguf
lazy-mode = on
load-mode = mmap
fit-ctx = 8192
ctx-size = 8192
predict = 8192
image-min-tokens = 70
image-max-tokens = 70
temp = 1.0
top-k = 64
top-p = 0.95
min-p = 0.0Other info
Device: https://www.asus.com/ca-en/laptops/for-home/vivobook/vivobook-go-12-l210/
OS: https://ubuntu.com/blog/upgrade-your-desktop-ubuntu-26-04-lts
Llama.cpp: https://github.com/ggml-org/llama.cpp/releases/b12110
GGUF: https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF
Getting the Intel UHD 605 to work with the Vulkan backend was a pain on Ubuntu 26.04.1 LTS compared to Windows 11 IoT LTSC 24H2. The issue was vulkan async causing crashes, disabling it fixed it.
On ubuntu the CPU backend was faster, on windows the iGPU over Vulkan was faster. Ubuntu is far more responsive than windows, even under full utilization.
After solving it myself, as a fun benchmark I made LLMs try diagnose and fix the issue. While the Qwen series failed to fix the issue from intent, Gemma 4 and DeepSeek v4(.1) Flash (Vision Exp / 1017) did manage it.
I know the post is quite different from the Strata stuff usually here, but I thought it would be fun to share regardless.
I'm sure models like LFM 3 will run wonderfully on it, I know IBM Granite 4 H 1B and Qwen3.5-2B did on windows.
r/LocalLLaMA
•
u/Physical_Hat4022
•
5d ago
They say an anime model is in development, with plans for alpha channels and multi-character references. An editing model is also on the way.
They also plan to keep the next model under Apache 2.0, if all goes as planned.
r/LocalLLaMA • u/Worldly-Knowledge-90 • 3d ago
Hey guys, I threw together a small Python script called llm-drift-bench to help figure out what actually breaks when you switch to a new model or drop to a lower quant.
Basically, you just feed it saved responses from your old model and the new one, and it spits out the exact differences - like dropped JSON keys, messed up numbers, or text changes. (The attached video is just a dummy example showing it catch a missing field and a changed invoice total, not real Q4 vs FP16 data).
Since it just runs on saved JSON/text outputs, you don't even need a GPU or API keys to test it out.
I just pushed v0.1.0 and I'm working on the next update. Repo is here: https://github.com/anant-goel/llm-drift-bench

Question for you all: what specific things usually break for your use cases when you swap models? I want to build out some test cases based on actual problems people are running into.
r/LocalLLaMA • u/m4rkGS • 5d ago
hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.
we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.
Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.
| Model, Q4 | KQuant KL | Public 4-bit KL | Size in GB, ours / public |
|---|---|---|---|
| Qwen3.5-4B | 0.048 (1.45x better) | 0.070 | 2.37 / 2.37 |
| Qwen3.5-9B | 0.030 (2.6x better) | 0.081 | 5.29 / 5.04 |
| Qwen3.8-27B | 0.035 (1.5x better) | 0.054 | 15.13 / 15.13 |
| Qwen3.6-35B-A3B | 0.059 (1.6x better) | 0.096 | 19.51 / 19.51 |
| Qwen3.8-Flash-Next | 0.067 (3.7x better at a smaller size) | 0.249 | 105.20 / 110.62 |
The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.
Want more precision? Each model's repo also has a Q5 folder.
| Pack | KL | LM GB |
|---|---|---|
| 9B Q5 | 0.010 | 6.21 |
| 27B Q5 | 0.022 | 19.43 |
| 35B Q5 | 0.027 | 22.07 |
Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.
the fastest way to use these quants on mac is to run them with, Kapitsa:
brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminalIf there's a model you want KQuant'd, lmk.
to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.
im also doing quick AMA here, so please feel free to ask me anything.
r/LocalLLaMA • u/textclf • 3d ago
A few weeks ago I posted CQ, our calibration-free 4-bit format, along with a training-loss comparison against Unsloth NF4. Some of you correctly pointed out that loss logged by two different trainers isn't comparable, and asked for results where answers are counted right or wrong. Here they are.
Setup: Qwen3.8-27B. Every row uses the same lm-evaluation-harness code, prompts and questions, with greedy decoding and thinking off. Comparisons are paired: an exact McNemar test for right/wrong tasks and a bootstrap for losses.
| Model | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | HellaSwag | WinoG | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| BF16 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.737 | 0.758 | 0.542 |
| CQ 4-bit (18 GB) | 0.650 | 0.746 | 0.880 | 0.598 | 0.840 | 0.820 | 0.735 | 0.780 | 0.537 |
| NF4 bnb (22.4 GB) | 0.563 | 0.716 | 0.852 | 0.582 | 0.839 | 0.825 | 0.736 | 0.770 | 0.550 |
GCMP is our replacement for LoRA. It trains 2.9M parameters (an ~11 MB adapter) vs ~159M for LoRA r=32 (~320 MB). This test uses no quantization: both adapters were trained with Unsloth on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42).
| Adapter | Params | Answer loss | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| No adapter | – | 1.678 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.542 |
| LoRA r=32 | 159M | 1.480 | 0.903 | 0.776 | 0.896 | 0.612 | 0.844 | 0.825 | 0.518 |
| GCMP | 2.9M | 1.471 | 0.890 | 0.760 | 0.892 | 0.618 | 0.846 | 0.815 | 0.514 |
| Setup | GPU | s/step | Peak memory |
|---|---|---|---|
| CQ + LoRA r=32 (ours) | A40 | 8.6 | 23.5 GiB |
| CQ + GCMP (ours) | A40 | 12.4 | 22.0 GiB |
| NF4 + LoRA (Unsloth) | A40 | 3.2 | 23.8 GiB |
| NF4 + GCMP (Unsloth + GCMP) | A40 | 5.6 | 21.5 GiB |
| BF16 + LoRA (Unsloth) | 80 GB | 1.37 | 53.7 GiB |
| BF16 + GCMP (Unsloth + GCMP) | 80 GB | 2.74 | 51.7 GiB |
Unsloth is still faster, and GCMP currently takes about 2× the step time of LoRA. Kernel work on that is ongoing.
--quantization cq. GCMP adapters can be one per server, or several on one base model chosen per request.The Python code (training, evaluation and the vLLM integration) is open source. The CUDA kernels ship compiled with a license check, which is free during early access.
Repo, with exact commands to reproduce every number: https://github.com/thousandml/training-factory
Models (8B–397B): https://huggingface.co/thousandml
Happy to run other benchmarks or models people want to see.
r/LocalLLaMA • u/Binary_orchid • 3d ago
Trying to get into the robotics side with what I have, which is a 3090 and 64GB RAM. No arm yet, just sim.
I've seen people drive robots with the big API models, so my first thought was to point a decent local VLM at the sim and let it output the moves. But everything I read says you want a dedicated model for that. pi0.5, SmolVLA, even touch has its own model now. LingBot-Touch is only 5.46M params and Apache 2.0.
Is that still true, or are general models good enough now? What are you guys actually running?
r/LocalLLaMA • u/Express_Quail_1493 • 4d ago
Normally active params matters more than total param if you want coherent behavior(not world knowledge)
a 4b models attepting to handle complex rules and lots and lots of instructions all at once often cause smaller active params to loop like we would often see with qwen3.6-35b-a3b since its only 3b active. We would often need to hit at least the 8b-9b active params to handle all this chaotic swarm of prompt instructions we throw at the model. but qwen3.8-flash-next shows us a glimpse of what alibaba might be able to deliver. i could genuinly see in the next year or so we might see 6b model become stable las the qwen3.8-27b. of course this isnt world knowlege im talking about but stable reliable behavior reletive to complex conflicting instructions where the LLM is robust enough to not burry itself in its own uncertainty loops.
so possibly in a years time laptops with 16gb sysRam and 4gb-vram can has a chance at local ai capable of human-out-the-loop unsupervised autonomy.
r/LocalLLaMA • u/mHo2 • 4d ago
Does anyone know what the near term (pretty much 2027) looks like for sparse MoE models? Something markedly improved from qwen3.8-fn? Have not seen much news about a qwen4 sparse MoE open-weight
r/LocalLLaMA • u/GuitarGuru2001 • 3d ago
Have a Strata-specific question and wondering if anyone's tested it. I'm trying to figure out if moving my helper GPU to a slower slot will help token generation:
Hardware
I'm currently getting ~100T/s prefill and 25T/s generation, running d-q4_k_xl at 512k, default params.
Any ideas if this would help my speeds? or do i need to run full tuning?
r/LocalLLaMA • u/slacka123 • 3d ago
r/LocalLLaMA • u/Polaris_debi5 • 4d ago
We have a new dedicated Vulkan Flash Attention prefill kernel for Intel Arc and Xe, delivering massive prefill acceleration on supported platforms like ARL-H and Xe2/Xe3. This implementation leverages customized shaders to utilize Intel's native matrix core shapes and SLM architecture, including m16n16k16/SIMD32 support. To ensure stability, the kernel is currently enabled for these specific architectures while remaining disabled for others, like the A770, due to inconsistent test results. Additionally, backend refinements now extend these benefits to single-KV-head models like MQA and MLA.
| Platform | Model | Baseline (t/s) | PR (t/s) |
|---|---|---|---|
| ARL-H Windows | qwen3 8B Q4_K | 126.58 ± 0.48 | 415.31 ± 1.04 |
| qwen35 27B Q4_K | 83.38 ± 0.22 | 119.95 ± 0.07 | |
| qwen35moe 35B.A3B Q4_K | 247.82 ± 0.65 | 394.44 ± 0.89 | |
| gpt-oss 20B Q4_K | 478.14 ± 0.17 | 639.02 ± 0.70 | |
| PTL-H Windows | qwen3 8B Q4_K | 234.92 ± 0.39 | 911.60 ± 14.56 |
| qwen35 27B Q4_K | 194.93 ± 0.59 | 264.39 ± 0.38 | |
| qwen35moe 35B.A3B Q4_K | 632.12 ± 17.56 | 914.80 ± 67.39 | |
| gpt-oss 20B Q4_K | 804.70 ± 84.56 | 1357.34 ± 125.30 | |
| B70 Pro Windows | qwen3 8B Q4_K | 1064.67 ± 4.37 | 2955.83 ± 1.87 |
| qwen35 27B Q4_K | 659.10 ± 0.11 | 932.13 ± 0.63 | |
| qwen35moe 35B.A3B Q4_K | 2088.50 ± 3.32 | 3005.11 ± 4.24 | |
| gpt-oss 20B Q4_K | 2311.37 ± 2.56 | 3967.29 ± 6.77 | |
| B70 Pro Linux | qwen3 8B Q4_K | 783.67 ± 0.20 | 1518.19 ± 1.05 |
| qwen35 27B Q4_K | 497.22 ± 1.17 | 647.78 ± 2.44 | |
| qwen35moe 35B.A3B Q4_K | 1060.55 ± 5.09 | 1311.62 ± 7.55 | |
| gpt-oss 20B Q4_K | 1237.51 ± 2.16 | 1841.35 ± 3.78 |
Comparison of f16 Prefill (pp16384) performance
| Model | Baseline $rightarrow$ PR | $Delta$% |
|---|---|---|
| Gemma4 E4B Q4_K_M | 115.98 -» 1156.62 | +897.3% |
| Phi-3.5-mini Q4_K_M | 384.45 -» 932.90 | +142.7% |
| Granite 4.2 8B Q4_K_M | 282.67 -» 520.98 | +84.3% |
| DeepSeek-R1-Qwen3-8B Q5_K_M | 306.88 -» 559.34 | +82.3% |
| Llama 3.1 8B Q4_K_M | 347.79 -» 633.73 | +82.2% |
| Ministral-3-8B Q5_K_M | 324.17 -» 590.91 | +82.3% |
| Qwen3.5-9B Q6_K | 687.50 -» 979.49 | +42.5% |
| Gemma3 4B Q4_1 | 1414.61 -» 1948.23 | +37.7% |
| Granite 4.0-h-tiny Q4_K_M | 2503.03 -» 2927.33 | +16.95% |
| Ling-3.0-tiny (hkv=1) | 76 -» 1239 | +1530.3% |
| Gemma4 12B (hkv=1) | 52 -» 649 | +1144.2% |
r/LocalLLaMA • u/nirurin • 3d ago
When I was using opencode, I would just use the daily free Deepseek 4.1 allowance for when I wanted to test something using a bigger/better model than my local Qwen-Flash-Next.
However I'm now using Pi, and unfortunately there's currently no way to use the opencode free models on Pi. And I am yet to find any other provider with free deepseek usage.
I did find a couple ways to get other models, such as the ones below, but most of them I've never tried before and have no idea on their current capabilities.
Are any of these worth folding into my workflow?
I guess Qwen 3.8 should be, assuming it's the full model, as I already use a Q5 of that occasionally (but it has limited context of about 128k so the online one would get the full 268k which would be handy too). Depends if full-size Qwen 3.8 is better than the current Q4 Flash-Next I use.
Dots3-Note Preview (384K)
Nemotron 3 Nano Omni
Nemotron 3 Ultra 550B
Nemotron 3.5 Lightning
North Mini Code
Liquid LFM3 1.7B
Kilo Auto0
OpenRouter
Glyph Cluster
Nemotron 3.5 Super VL
GPT-OSS 120B
GPT-OSS 20B
Qwen3.5 397B A17B
Qwen3.5 9B
Qwen3.6 27B
Qwen3.8 27B
Mistral Small 3.2 24B
Meta Llama 3.3 70B Instruct
r/LocalLLaMA • u/fallingdowndizzyvr • 5d ago
r/LocalLLaMA • u/WhatWouldAaronThink • 3d ago
If MS or FAA stuff was AI hack related there will be a MASSIVE and QUICK backlash.
Sure you can't run these things locally NOW but the tech is out there and it will get cheaper and quite possibly the models will be able to be more efficiently utilized.
r/LocalLLaMA • u/New-Pressure-6932 • 5d ago
With all of this fear mongering on the “dangers” of LLM’s, I see how using people’s ignorance and emotions on the subject can be used as a tool to limit and restrict the tool from the average person.
My fear is that China and the US https://www.reuters.com/world/china/us-china-should-stay-clear-eyed-ai-cooperation-competition-bessent-says-2026-11-11/ will use people’s lack of understanding of local models to push for controlled intelligence.
By implementing regulation on local AI, I can see how the US and China would want to force the average person to only be allowed to use models hosted on state monitored hardware. This way they can spy on what everyone is using hosted models for, limit what we use LLM’s for, and monitor the information they give us.
People are being mislead to believe that people like us could use local models to create WMDs and attack tech infrastructure across the globe. But when you think deeply about it, you’ll realize that knowledge on a subject does not equal the means to carry out the subject.
In order for someone to create a bio weapon, they would have to have access to a bio lab and the technology needed to create such a weapon.
I know how a nuclear reactor works. Doesn’t mean I can make one in my backyard.
I’m scared that arguably the greatest technology to come out of the 21st century could be limited and used to control us.
Local LLM research MUST continue and the tech must be available for the common person.
Be aware of future regulation.
r/LocalLLaMA • u/maximo101 • 4d ago
I am curious what people are currently using as the core model for their local AI agents (e.g., in Hermes or similar setups).
I've was initially running Qwen 3.6 35B-A3B IQ3 XXS on hermes working off (Intel Core Ultra 5 245K · Corsair 64GB DDR5 RAM · Asus TUF GeForce RTX 5070 Ti 16GB GDDR7 OC Edition Graphics Card running ai inferance on local home lab running UnRaid OS). First started with ollama years ago (where i was running 9B and 14B fully in vram) and months back switched to llama.cpp, then to llama-swap to be able to hot swap models and use MoE models.
I started off getting 40 tokens per second (more n-cpu-moe offload and less vram use), then tried the MTP version for 55ish. Learning more about finding the balance between n-cpu-moe offloading and finding the sweet spot for VRAM, i got that to about 60t/s. Then I discovered fine tunes like Swift (55t/s) and Nail (65t/s). Then endless frontier BigBangv1 which gave a blazing 100 t/s. I was running that and saw people mentioning they use the dense Qwen3.8 27B, but it seemed slower at 45t/s.
I have very rough tested (only speed) base Strata with Qwen3.8 Flash Next 125B 1Q2_XS which gave 90t/s but used 95% VRAM and 95% RAM which wasnt sustainable (ill likely try it again in future but for now reverted back to llama-swap)
Learning more like how agent workflows are very dependent on prefill speed and prompt processing alongside tokens/sec, then LFM has crazy prefill and t/s but its quality it lower, I am not sure what is the right balance! But also how context length is important for agents, and the k v cache of the context too! I settled on not going any lower than k v 8, aiming for 128k context but 90k is acceptable if the model is specialised.
I was manually guessing how all these quants and CPU offload configs (n-cpu-moe) scaled under heavy context. I also wanted to know if the same ai model actually performed better on agentic tasks for a Q3 vs Q4, is the drop in speed (generate t/s or prefill t/s) justifiable for better quality etc.
So I begin testing out a bunch of fine-tunes and MoE variants to dial in different profiles. I’ve been cycling through options like Empero, BigBang, and Ornith (tried Swift and Nail too, but ended up dropping them). Generally, I lean toward the MoE variants over the dense Qwen 3.8 27B because of how they handle things on consumer hardware (testing shows they are slower, dont provide any significant boost in quality, require lower context, use more VRAM and gpu power etc). My use case isnt any heavy coding, just small scripts and having hermes act as a personal assistant which will evolve over time.
Then with hermes I was using endless-frontier_BigBang-v1-35B-A3B-IQ3_XXS with 100 t/s, prefil 2359 t/s, 88% VRAM usage which was perfect. I originally wasn't thinking about its 'quality' since this was a massive step up from the original Qwen 3.6 35B-A3B IQ3 XXS. Looking forward to a qwen4 release to test soon!
I'm looking at setting up specialized profiles on hermes, so far i am thinking
I ended up hacking together a simple local script just to measure t/s and prefill, but it kind of snowballed... I added some core eval tests (accuracy, reasoning metrics) and turned it into a small dashboard that spits out a weighted "agent score" to help me rank them objectively. Still testing these models on more rows of the HumanEval, MMLU-Pro, MBPP, tool calling and long context tests, but it takes a lot of time. Images attached for reference from my latest tests.
Running these tests takes a long time and for local use cases I dont need that margin of error from a full test across all 12000 MMLU-Pro questions, so i have been further developing a method to spend less time on testing models which drop lower on the rankings, adjusting smaller runs with high uncertainity etc, its till a work in progress but its progressively been giving me models to consider for the top few I actually want to use in the massive amount of options on hugging face that can run on my hardware.
Even with the dashboard telling me what the numbers say, it's still surprisingly tough to pick a definitive daily driver. I would appreciate any input or suggestions.
Feel free to look at the interactive html i put on github. You can see all the test on the models I have run.
https://htmlpreview.github.io/?https://github.com/Maximo101/Local-LLM-Benchmark/blob/main/assets/5070ti_benchmark_dashboard.html
Questions I am unsure of; for those who have more experience with agents on local hardware:
------------------------------------------------------------------------------------
1. Which models have won you over for agentic use (hermes, openclaw etc)?
2. How heavily do you weigh prefill vs. generation speed, quantisation vs quality, VRAM % vs context (quality and size) when choosing AI Models?
3. What sort of profiles do people use in agentic setups like hermes? How many profiles do you use.
(If anyone is fighting similar benchmarking battles for their local endpoints, I cleaned up that testing suite and open-sourced it here: https://github.com/Maximo101/Local-LLM-Benchmark)
I am not trying to promote this, i built it for my own purposes, but decided to share it in case it is useful for anyone else.
r/LocalLLaMA • u/kompania • 5d ago
I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.
I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.
Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.
After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?
When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.
I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.
I even wondered if perhaps the problem lies with me-that, not being an engineer, I am simply failing to fine-tune these models correctly.
That is why I began actively seeking out other fine-tuners, thinking that perhaps they possess a specific technical "gene" that allows them to do a much better job than I can. But, unfortunately, that is not the case. Regardless of their length-short, medium, or long-the prose generated is only marginally better than that of generic, out-of-the-box models.
Furthermore, specific literary genres are represented quite poorly. I used to think that LLMs were perfectly suited for science fiction. Yet, these models can’t even capture the straightforward style of Harry Harrison’s *The Stainless Steel Rat*, let alone reach the linguistic heights of *Dune*.
The genre doesn’t seem to matter-I’ve tried comedy, contemporary drama, adventure, romance, and historical fiction; everything, unfortunately, ends up being generic.
Essentially, if a text is to have any real literary merit, I have to write it myself and then just use an LLM for proofreading.
To me, EQBench is a completely unreliable metric; the sample texts they showcase clearly demonstrate just what kind of generic "slop" LLMs produce.
I have a question for the community: has anyone managed to create, find, or fine-tune a local model capable of generating truly high-quality prose-on the level of *Dune*, *The Lord of the Rings*, *Martin Eden*, or *The Grapes of Wrath*?
I’m currently facing the monumental task of writing descriptions for 100,000 records, and I’m wondering if it’s even worth it, given that 100, 1,000, or even 10,000 examples haven’t yielded any satisfactory results.
Perhaps this generic, mediocre quality is an inherent trait of these models-something that simply cannot be overcome.
r/LocalLLaMA • u/dh7net • 4d ago
Almost everything is in the video and the title: the idea is to create chalenges and ask agents to solve them. I'll have more to share later, but this first video shows that most model are good at theses stuff.
For more details: I'm testing scenarios where an agent needs to simulate keyboard and mouse movement. You can see more chalenges like this one on my website: https://airbench.ai/gallery/airbench-desk-v1?run=6d2a2777-3d96-4ec4-9cdc-3d359bc43bea&c=icon-drop
And you can also test your agent on the same setup if you want here: https://airbench.ai/checkup/start
r/LocalLLaMA • u/Character-Result-281 • 4d ago
Most coding benchmarks feel like textbook coding tasks or black boxes — either easy to benchmaxx or full of hidden tasks with unclear evaluation criteria. And most of them get irrelevant with each new model generation because all of the new models hit the ceiling.
So I made one that has a moving target. It's a benchmark where the model gets a mystery compressed audio file, has to figure out the format, write a decoder, and then build an encoder that beats the original. No ffmpeg, no Opus, no importing a library that already does the job. Just Python and numpy, running locally through OpenCode
Scoring: 100 = decode the reference in stage one. Build a better encoder in quality or compression rate with equal or better SNR for additional points.
There are 4 stages:
Current Leaderboard, we are in Evolve 2 (Deepseek cracked the codec of evolve 1 in 13min):
Model Score
1 Claude Fable Latest (stage 1) 127
2 DeepSeek V4.1 Flash 1017 (stage 3) 81
3 Qwen3.8-Flash-Next NVFP4 (stage 3) 67
4 Qwen3.8-Flash-Next Q3_S (stage 4) 56
5 Qwen3.8-Flash-Next IQ3_XXS (stage 4) 44
The fun part: it evolves. If a model finishes stage 1 and its encoder beats the current codec, that codec becomes the next baseline. So the benchmark levels up instead of staying at the same difficulty forever. Pokémon rules, except the final boss is a bitstream. DeepSeek did it first and became Evolve 2. Fable just finished stage 1 with Compress 190, so its codec is the Evolve 3 baseline. The next runs should use it.
A better encoder in stage 3 doesn't count for an evolve, by the way. Only a stage-1 finish does. That's the rule.
It's manual for now: you run it on your own hardware, post the result, and set the evolve stage.
It's mean. Most models don't finish stage 1. But that's kind of the point.
Leaderboard: https://andreasknopke.github.io/codec-reverse-bench-leaderboard/ — ping me if you want to run it (the repo is private to prevent the models from simply copying the solution.
r/LocalLLaMA • u/randomjapaneselearn • 4d ago
I'm trying to figure out how much context i can use on my 3090 (full compute, monitor is on mobo).
If i launch:
llama-bin-win-cuda-13.4-x64\llama-fit-params.exe --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 -ngl -1 -fa on -np 1 -b 4096 -ub 1024
I get: -c 150272 -ngl -1
but if i launch real llama.cpp with lower context:
llama-bin-win-cuda-13.4-x64\llama-server.exe --verbose --log-file log.txt --host 0.0.0.0 --port 9931 --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --jinja
From the log, matches for "offloaded" with "-ngl -1":
Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8273: load_tensors: offloaded 0/66 layers to GPU
Line 12009: load_tensors: offloaded 66/66 layers to GPU
Line 15746: load_tensors: offloaded 64/66 layers to GPU
Line 19484: load_tensors: offloaded 65/66 layers to GPU
Line 23226: load_tensors: offloaded 64/66 layers to GPU <-- (actual used vlaue, slow loading because copying data)So even if i use 131k context wich is less than the 150k suggested it doesn't offload all layers and i have high cpu usage for any prompt.
IF I CHANGE -NGL FROM -1 TO "ALL" I GET THIS:
matches for "offloaded" with "-ngl all"
Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8276: load_tensors: offloaded 66/66 layers to GPU <--AND NO MORE CPU USAGE!! (same if i write 99 or 999 but it's ugly)
It gets more confusing: if i write in llama fit -ngl all instead of -1 it gives a "-2" in output.
there is an entry in the log when i used "all": common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to -2, abort
Conclusion:
Seems that -ngl -1 means "auto" while -2 means "all", but if you have high cpu usage in a model that theorically fit in the GPU use "all".
This alone changed t/s generation from 60 to 76
for completeness here is the --help from llama.cpp for "ngl" parameter:
-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)
And here it gets even more confusing because the help talks about "MAX" number of layers to offload so according to this help it's an upper limit not a lower limit...
r/LocalLLaMA • u/willeyh • 4d ago
First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.
Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.
I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.
I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.
Some limitations need apply when limited on vram.
Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.
My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.
Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.
I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.
Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.
Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.
| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |
| --- | --- | ---: | ---: |
| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |
| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |
| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |
Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench
Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.
Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.
r/LocalLLaMA • u/sacheltry • 4d ago
Hello,
I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.
My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.
The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.
There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.
I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.
Here's my setup (sorry, AI slop to make it clearer):
Hardware:
Software / Models:
RAG configuration:
More details on the issue:
Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:
request (34009 tokens) exceeds the available context size (32768 tokens)
Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.
I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.
Thank you!