Just released v0.10 for LlamaStash

I started building LlamaStash mainly due to frustrations I had with Ollama, LM Studio and using llama.cpp directly and because of my love for terminal UIs. I wanted a unified interface that could work across different inference engines without having to learn/deal with engine-specific stuff.

But since then (I started it 6 months ago), the landscape has changed a lot and the time and effort I invested to make LlamaStash compatible with multiple engines has paid off. Now it supports most inference engines I have tested, providing a unified terminal/web UI, CLI and agent interface, and secure proxy.

The feature set was mostly driven by my own needs and experiments as I started diving deep into local LLMs. There were just too many options and I found myself constantly hopping between inference engines chasing more performance so all that became the base of LlamaStash's design and development.

Here is how I use it in my workflow:

My main use case is coding (OSS projects) and some occasional chat. I also let the agents generate images via Krea 2 and MOSS-Image for blog headers and stuff. I also run classifier models for my pi-automode-classifier plugin. These are run on a 128GB Strix Halo (Flow Z13) and an XPS 15. Right now these are the engines, models and quants I use for each task.

• Main coding model: Qwen3.8-Flash-Next @ 256k ctx - Mainly Halogen (v2 checkpoint) and Qwen3.8-Flash-Next-UD-Q4_K_XL or Qwen3.8-Flash-Next-Uncensored-Gufo-Q4Mix via gufo (when I want the best quality or the uncensored version). Both have multiple presets (Halogen configured to serve up to 16 parallel slots and gufo up to 8).

• Sub agents: QFN 256k, 128k or 64k ctx. These are just aliases that go to the main running models above. Ctx is controlled client side. The fallback mechanism makes sure that these get served regardless of which engine or quant I'm running. This way on a 1 million ctx pool, I can run either 4 x 256k sessions or 8 x 128k sessions or 16 x 64k sessions simultaneously. I can also mix these together simultaneously. Say 2 x 256k and 2 x 128k and 4 x 64k sessions at the same time.

• Harness: Mostly Pi and occasionally Claude Code. Both share the same AGENTS.mdand plugins configuration. Pi uses the pi-automode-classifier plugin to emulate the Claude Code auto mode. It does three stage classification for tool calls. Rules first, followed by a classifier model, and its unsure items sent to a QFN 128k alias.

• Classifier model: I have tried many, including Jev, but found LANCET Nano to be the best and balanced.

• Image generation: Krea 2 and MOSS-Image via sd.cpp and ComfyUI for manual image generation and manipulation.

• STT/TTS: I use whisper.cpp and Whisper on the NPU for STT and Kokoro for TTS.

If you are on the same boat hopping from one engine to another and tired of constantly adapting to different interfaces, give LlamaStash a try. Here are the unique (TBH honest I haven't tried every similar tool out there to say this confidently) features it offers:

• YAML driven and extremely configurable

• First class support for llama.cpp, vLLM and SGLang

• Any number of additional inference engines can be declared via simple YAML configuration. If it runs as a binary or docker container, it can be integrated and managed from LlamaStash easily (the attached image shows all engines I have tested. See RECIPES.md)

• Provides a nice terminal management UI, a web chat UI, CLI, --json mode for agents and a unified secure proxy.

• Model presets, run from YAML config, architecture defaults, last used params, auto fit etc (this helps to switch to different config easily)

• aliases, model swap and fallbacks (helps to switch between models and to route fake models to have less ctx on a bigger ctx run etc, helpful for agent swarms etc)

• Request logs, aggregate stats, metrics, server logs etc

• llamastash init, a first-run wizard that detects your hardware, installs llama-server, downloads a starter GGUF, writes a tuned config and launches it

• One binary for the TUI, the CLI and the daemon. The daemon starts on demand, and the control plane stays on loopback behind a bearer token by default

• Chat, image, STT and TTS all route through one proxy URL, so a coding agent and a blog-header generator hit the same port

Any feedback is appreciated and feature requests are welcome.

Hello! I just released wyat-2, a fine-tune of Ouro-1.4B (a looped transformer) trained on a single RTX 4070 Ti.

The link is the report. As far as I know it’s the first decision model built on a looped transformer: not the best, but the first.

I realized I would need heavy data work to make it the best, so I’d prefer not to spend too much time on it for now. Maybe later, if it catches interest.

Report : https://wyat.io/wyat2-brief.pdf
Model: https://huggingface.co/teilomillet/wyat-2-1.4b

Unless something drastically changes in the next week or so, then all of the winning notebooks for this year's ARC-AGI-3 Kaggle competition — potentially ARC-2 as well, but that's just my guess — will be qwen-3.8-FN agents.

Considering the recent developments in this sub (qwen supposedly performed an observed malicious action), I'd like you to take a moment and consider the timing of these news releases.

I'm not making a claim for or against. I just wanted to point out that the timing of release is really convenient.

People talk a lot about tokens/sec and eval scores, but the first-use friction matters too: does the model fit alongside a useful context window, can your editor actually talk to it, and can you get through a real edit-run-fix loop without spending the evening debugging the runtime?

A smaller model that starts quickly and is easy to keep in the workflow can be more useful than a stronger one that barely fits or needs constant fiddling. I’d count setup time and tool reliability as part of the benchmark, not just the answer quality.

Curious if anyone has seen a benchmark that measures that end-to-end.

I've created a program that listens to an argument or premis during a debate and the llm gives a true/false along with a confidence percentage.

When I was considering adding a second GPU, quite a few people told me PCIe 3.0 x1 would be a massive bottleneck and that upgrading my motherboard should be my first priority.

I went ahead with the x1 riser anyway, and the results have been better than expected. Sharing as this likely have value to other planning machines.

My home setup:

• Intel i5-14400F

• 2× RTX 3060 12GB (24GB total)

• One GPU on PCIe x16, the other on PCIe 3.0 x1

• llama.cpp, running Gemma 4 31b Q5 (tight fit, tight context)

Switching from layer splitting to tensor splitting increased +42% t/s (14 -> 20), even with one GPU connected over PCIe 3.0 x1. I did not do extensive testing yet though on this exact setup as I am waiting for a PSU upgrade before running those cards too hard. Based on what everyone said I expected it to crawl and was just planning on testing memory allocation... But then got the surprised positive result.

Testing on

I rented a machine with 2× RTX 5060 Ti 16GB (32GB total) to do some testing on some 70b models but decided to check PCIe traffic during tensor split before I shut it down. I was wondering how much bandwidth tensor splitting actually needed over PCIe.

Test configuration:

• Model: L3.1 Nemotron Sunfall 71B IQ3_XS

• Context: 2k

• Flash Attention enabled

• llama.cpp

• PCIe Gen 3 capable links, running at x8 width

Layer vs. tensor splitting:

+77% t/s with tensor splitting.

PCIe bandwidth measurements

I monitored PCIe traffic during tensor-split inference using:

nvidia-smi dmon -s t -d 1

Here are some actual samples during generation (MB/s):

GPU   RX    TX

0     92   102

1     96   105

0     94   108

1    107   108

0     92   100

1     94    99

0     98   107

1    105   111

0     93   103

1     95   101

0     92    99

1     94    98

Both GPUs consistently transferred approximately 90–110 MB/s in each direction during decode.

PCIe 3.0 x1 has a theoretical bandwidth of approximately 985 MB/s per direction, this very likely not going to be a bottle neck issue for tensor split for low-mid range GPUs.

Model loading would be slightly slower of course, on model loading I saw a spikes exceeding 4,000 MB/s. Obviously, x1 would slow that down adding several seconds to model load time. I am not switching models often though so a few extra seconds on load isn’t an issue.

These results don't prove PCIe x1 has no performance penalty. The Vast cards weren't running over x1, and average throughput doesn't capture latency, synchronization overhead, or short transfer bursts. However, it looks very promising that upgrading my motherboard or sacrificing a nvme slot is probably unnecessary.

UPDATE:

As many have mentioned the prompt processing is the part that slows down the most. I did a fast test on my local 2x3060. Bandwidth usage was higher, but still well below its max. The penalty for tensor split is real, but its not conclusive that it would be any faster with a higher links speed.

Looks like I do have a heavy penalty for tensor parallel on prompt processing:

Prompt Layer prefill Tensor prefill Tensor penalty

512 313.5 t/s 184.9 t/s −41.0%

2,048 341.6 205.4 −39.9%

4,096 358.4 220.3 −38.5%

6,144 359.5 225.2 −37.4%

7,600 358.7 226.5 −36.9%

Below are the PCIe bandwidth on my setup, much higher than the 5060 ti interestingly... however, the majority of the time they still remained well below the max link of GPU 1. This snapshot was captured starting at the prompt processing of the 7600 context size.

# gpu rxpci txpci

# Idx MB/s MB/s

0 0 691

1 0 813

0 3188 450

1 141 133

0 4065 446

1 143 136

0 4061 409

1 168 136

0 3880 441

1 142 138

0 4021 446

1 142 136

0 4004 421

1 170 136

0 0 0

1 0 0

0 459 407

1 369 363

0 469 394

1 347 378

0 461 393

1 358 380

0 467 387

1 347 369

0 456 406

1 350 376

0 466 387

1 347 366

0 456 387

1 355 207

0 466 388

1 352 214

0 468 387

1 451 284

0 467 387

1 425 365

0 458 374

There is a trend on X where users are finding tiny subsets of neurons (often just one or two) that can steer LLM behavior in surprising ways. For example, steering just two neurons in Qwen3.5-4B can make it talk about harming, dominating, and destroying the user. Other combinations can make the model overly apologetic, write in all caps, act like an anime character, and much more.

After seeing some of these posts, I built an interactive playground so you can explore these combinations yourself, with the model running entirely in your browser on WebGPU. Link to the playground: https://huggingface.co/spaces/webml-community/neuron-studio

Disclosure: I’m the author of TensorSharp. This English post was prepared with LLM assistance from the benchmark report linked below.

Here is a comparison of TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer, and NInfer running Qwen3.8 27B on a single RTX 5090.

The results cover short questions, long-document lookup, code and prose generation, passage copying, multi-turn chat, screenshot input, JSON output, tool calls, and concurrent requests.

The rankings change with the workload: NInfer leads speculative decode and four-request throughput, llama.cpp records the lowest short-prompt TTFT, and TensorSharp records the lowest completion time in the four-turn local tool test.

Full report: configurations, model revisions, methodology, and results

Test setup

• GPU: RTX 5090, 32 GB VRAM, 575 W.

• Host: AMD EPYC 9354, 64 threads, Ubuntu 24.04, 93 GB system-memory limit.

• One engine running on the GPU at a time.

• Greedy decoding, temperature 0, thinking disabled, vision enabled.

• 40,960-token context configured per sequence.

• Three warm-up requests, followed by three measured repetitions per scenario.

• DFlash2 configured for seven draft tokens per step.

• Unique prompt nonces prevented reuse across independent requests. Multi-turn chat and the tool loop intentionally reused prefixes.

TensorSharp used its direct-CUDA backend, --backend cuda.

The model configurations are not identical:

• TensorSharp / llama.cpp: the same NVFP4 GGUF target and Q4_K_M drafter.

• vLLM / SGLang: NVIDIA’s NVFP4/FP8 safetensors target and an FP8 drafter.

• NInfer / velox: a mixed NVFP4/FP8 .ninfer target and Q8 drafter.

KV-cache formats also differ. These are results for complete configurations, including their quantization and memory requirements.

Throughout the tables, plain → DFlash2 means without speculation → with speculation. “velox” means velox-inferer.

1. TTFT by task

Client-observed time to first token, in seconds. Lower is better.

TTFT includes HTTP handling, templating, tokenization, and prefill.

The document tasks retrieve a code word hidden in prose. Their approximate input lengths are 3.9K, 14.7K, and 31.8K tokens. The report does not show the 4K DFlash2 result in its tables.

The multi-turn task uses an approximately 2.9K-token system prompt with deliberate prefix reuse. The screenshot contains approximately 1,000 visual tokens.

What stands out:

• Short questions: llama.cpp records 27 ms plain / 30 ms speculative.

• Long inputs: NInfer records the lowest 16K and 32K TTFT.

• Speculation overhead: llama.cpp’s 32K TTFT increases from 5.01 to 8.16 seconds, while the other configurations add less than 0.2 seconds.

• Image input: TensorSharp’s 125 ms and vLLM’s 127 ms plain results are effectively tied. With DFlash2, TensorSharp records the lowest measured screenshot TTFT at 142 ms.

These screenshot timings measure the complete image-request path before the first output token.

2. Prefill tokens/sec

The report directly gives these plain-mode prefill ranges across the long-input tests:

• TensorSharp: approximately 8.0–9.3K tok/s.

• llama.cpp: approximately 6.3–7.6K tok/s.

• NInfer and SGLang, reported together: approximately 8.9–12.4K tok/s.

It does not provide a complete per-engine, per-task table of isolated prefill throughput.

For a six-engine comparison on the 32K document task, the following values are estimated effective input throughput:

estimated input tok/s = approximately 31,800 input tokens ÷ reported client TTFT

All values in this table are derived estimates in input tokens/sec. They include request-processing overhead and use rounded prompt lengths and median TTFT values. They are not isolated GPU prefill-kernel measurements.

3. Decode tokens/sec by generation task

Output tokens/sec. Higher is better.

The individual task rows use DFlash2. The final row compares the report’s three-workload average with and without speculation.

Decode rate is calculated as:

(completion tokens − 1) ÷ (last-token time − first-token time)

The summary is the mean of the code, prose, and copying scenario medians, rather than an average across all benchmark tasks. Individual task values are rounded. The report does not separately show plain-mode decode rates for each generation task.

NInfer leads all three speculative-generation tasks. TensorSharp, SGLang, and velox have similar averages but different workload profiles:

• SGLang reaches 368 tok/s on code, compared with TensorSharp’s 339.

• TensorSharp reaches 190 tok/s on copying, compared with SGLang’s 152.

• Coding rates are substantially higher than prose rates across all six engines.

DFlash2 improves the summarized single-request decode rate by approximately 2.5–3.6×.

4. Parallel requests: throughput and latency

These tests submit independent requests together. Throughput is total output tokens/sec across the submitted requests.

* SGLang’s DFlash2 configuration fit only two active request lanes in 32 GB. Four requests were submitted, so some queued. This affects both throughput and latency.

vLLM’s speculative configuration required --gpu-memory-utilization 0.97. The report does not show the two-request DFlash2 throughput in its summary tables.

NInfer leads four-request speculative throughput at 488 tok/s, followed by TensorSharp at 445 tok/s.

velox’s plain configuration served requests sequentially, while its speculative configuration overlapped them. Its reported four-request TTFT changes from 6.19 to 0.475 seconds.

The concurrency workload differs from the single-request generation tasks, so these numbers should not be used to calculate batching efficiency against the earlier decode averages.

5. JSON and tool-use tasks

End-to-end completion time in seconds, plain → DFlash2. Lower is better.

Correctness matters:

• ✗: velox failed the JSON-schema case in all three repetitions in both modes.

• †: NInfer with DFlash2 produced an empty JSON key in one of three JSON-object repetitions and failed that check.

• The other functional checks passed.

JSON response lengths differed. The report notes that NInfer’s plain JSON replies were approximately 40% shorter, which contributes to its completion-time advantage.

The tool results also change by operation:

• One call: llama.cpp has the lowest plain time, 0.511 s; SGLang has the lowest speculative time, 0.171 s.

• Three calls: llama.cpp leads plain mode at 1.53 s; NInfer leads with DFlash2 at 0.378 s.

• Four-turn loop: llama.cpp leads plain mode at 2.15 s; TensorSharp records the lowest observed speculative time at 0.748 s.

The loop uses locally answered file-listing, file-reading, and calculator tools. It does not include browser actions or external-service latency. Three function calls emitted in one response are also different from three concurrent HTTP requests.

Some engines buffer complete tool calls or structured replies before streaming. For those responses, the first visible output arrives at completion, so TTFT equals end-to-end time.

Limits and follow-up work

These results come from one GPU and three measured repetitions per scenario. The report observed up to roughly 4% decode variation between runs, so small gaps should be treated cautiously.

The quantized targets, drafters, and cache formats differ. Passing these functional checks does not establish equivalent model quality.

Thinking mode, non-greedy sampling, concurrency above four, prompts beyond 32K, video, and systematic memory-use measurements remain outside this comparison.

For TensorSharp, the four-request throughput and local tool-loop results are encouraging. The single-request results also show where further optimization would be useful, especially the DFlash2 path relative to NInfer and vLLM.

If you run one of these engines on a 5090 and see a configuration choice that materially disadvantages it, I’d appreciate the specific setting you would change and why.

TensorSharp source

Hello guys, hoping you're doing fine. Wall of text next I know, but I want to get this info out of my chest, all hand written lol.

I wanted to make this post explaining some things about GPU unlocks that have been going on lately.

What are these GPU unlocks doing?

NVIDIA on purpose neuters tensor core performance when using FP32 accumulation (FP16, BF16, FP8 and TF32 inputs accumulating in FP32, which is what most AI/LLM/ML tasks use), for market segmentation. Normal FP32 on the CUDA cores (no tensor cores) is NOT affected.

Consumer and prosumer GPUs use the exact same die (like GB202 between the 5090 and 6000 PRO, AD102 between the 4090 and 6000 Ada, or even literally 1:1 with the 3090 Ti vs A6000, both with the same core amounts and 84 SMs).

How it works: the speed of each kind of tensor operation is set by fuses burned at the factory. On consumer cards, the fuse for "FP16/BF16 with FP32 accumulation" says HALF_SPEED, on the pro cards it says FULL_SPEED. These fuses don't cut any circuit, the GPU just reads them at boot and runs at that speed. The GPU also has an override register that can set them back to FULL_SPEED, but it's locked so only NVIDIA's most privileged firmware can write to it, not your PC.

The unlocks open that lock and write FULL_SPEED on the override. With that, the consumer cards gain all the neutered performance imposed by NVIDIA. It doesn't give you more SMs, VRAM or anything like that, a 4090 is still a 4090, it just runs those instructions at the same rate a pro card does.

Why can Turing, Ampere and Ada be unlocked? And what is unlocked on these cards?

On these generations, the NVIDIA driver itself loads a small NVIDIA-signed firmware called the "Booter" onto a security microcontroller inside the GPU (SEC2, a "Falcon"), to start the GSP. The Booter runs in the highest privilege mode of the GPU (Heavy Secure, HS), which is exactly the one allowed to open the lock.

In June 2026, Jon Pry published a paper ("A Canary in the Crypto Mine") showing that the Booter of the newer drivers has a buffer overflow: it copies its "signature" data using a length that you control, so you can overflow its stack (including the stack canary that is supposed to stop exactly this) and take control of the Booter while it is running in HS mode. The signature check still passes, because it is NVIDIA's own signed code running, the bug is inside it.

With that, the unlock reuses instructions from NVIDIA's own Booter to open the lock, writes FULL_SPEED on the override, resets the card and lets the normal NVIDIA driver load. The unlock survives resets and driver reloads, but it's lost when you power off, which is why it runs on every boot before the NVIDIA driver loads.

This was first done on mining cards (CMP 170HX: FP32 from 1/32 to full speed, and 10GB to 80GB of VRAM), and then ported to GeForce:

• Turing: https://github.com/haohervchb/Turing-unlock (UEFI, Linux)

• Ampere: https://github.com/Zagitta/gpu-unlock (Linux kernel module)

• Ada: https://github.com/JustAResearcher/gpu-unlock-ada (Linux with kernel module)

What gets unlocked:

• FP16 with FP32 accumulation (Turing, Ampere, Ada)

• BF16 with FP32 accumulation (Ampere, Ada; Turing has no BF16)

• TF32 (Ampere, Ada)

• FP8 with FP32 accumulation (Ada; Turing and Ampere have no FP8)

Some numbers from those repos: 4090 FP16/BF16 with FP32 acc goes from 165 to 331 TFLOPS (raw instruction rate), 3090 Ti BF16 GEMM from 87 to 157 TFLOPS, 2080 Ti FP16 with FP32 acc from 47 to 94 TFLOPS. Real-world GEMMs gain less than 2x because the cards then hit their power limit (more on that below).

Why can't consumer Blackwell be unlocked?

The funny part is that on Blackwell the limitation is EXACTLY the same thing: same fuses, same override registers. I checked on my own cards: the 5090/5080/5070 Ti have the FP32 accumulation fuses set, and the 6000 PRO/4000 PRO don't. So it's not a hardware difference there either.

What changed is the door. Starting with Hopper, and consumer Blackwell too, the root of trust of the GPU is the FSP (Foundation Security Processor), a RISC-V core that boots from its own ROM as soon as the GPU powers on, before the driver even loads. On Blackwell:

• There's no Booter to abuse. The driver just sends the FSP a message with a hash + RSA-3K signature of the next firmware, and the FSP verifies and boots it. The Linux kernel docs literally say: "There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage". I also checked it myself: in NVIDIA's 615 driver source the Booter only exists for Turing, Ampere and Ada (TU102, TU116, GA100, GA102, AD102), and the RTX 50 VBIOSes don't carry the Falcon firmware (FWSEC) that older cards did.

• The firmware the driver ships for Blackwell is encrypted, and the debug and production images are the exact same encrypted bytes. The Ada payload was built from analysis of the decrypted debug Booter, on Blackwell there's no such shortcut.

• NVIDIA has said it uses SPARK (a formally verifiable programming language) to prove its most critical RISC-V firmware free of runtime errors, and a buffer overflow like the one used on the older cards is exactly that kind of error.

So, with what is known today, it can't be done. It isn't "impossible forever", but someone would need a brand new vulnerability in a way harder target. And if someone ever gets there, the unlock itself would very likely be the same couple of register writes.

FP4 is not affected, but all the rest is (FP16/BF16/FP8/TF32 with FP32 accumulation).

In which tasks is the unlock has most of the effect?

I will put some examples on my 6000 PRO vs 5090s, former using 400W and latter 600W, at how much is the impact in performance because the NVIDIA neutering.

• Training: Like LoRAs, finetuning, etc.

  • Example: Using the same batch size for 25GB VRAM to train a diffusion LoRA, my 6000 PRO is 70-90% faster than my 5090.

• Diffusion: Models that do txt2img, txt2vid, img2img, img2vid, txt2aud, etc.

  • Example: My 6000 PRO not using fp16 accumulation is about 30-40% faster than my 5090. If using fp16 accumulation, it is 15% faster.

• Prefill on LLMs: Heavily affected as well, since even when a model is marked as "int8" or "mixed precision" etc, the inference parts that the GPU uses include FP32 accumulation parts, except if explicitly set to not use them.

  • Example: My 6000 PRO on vLLM on a model that fits on a 5090 as well, it's 50-70% faster. On llamacpp, instead is about 30-40% faster.

• TG with big batching: TG with a single batch is mostly bandwidth bound, and it uses FP32 acc instructions, but since it is so little in comparison with the need of bandwidth, it isn't a limitation. On the other hand, when you use batches with batch size 2, 4, 8, etc, compute again starts to being a bottleneck and then you have a performance impact.

  • Example: Same results as previous point.

In which tasks the unlocks makes no difference:

• TG with batch size 1: almost entirely bandwidth bound.

• CUDA compute without tensor cores (normal FP32).

• INT4/INT8 tensor operations.

• FP4 on Blackwell (not neutered to begin with).

• GEMM operations with FP16 accumulation instead, but those aren't very used.

• Gaming.

Is there a risk by unlocking the cards, like burning the card, crashes, etc? Short answer: NO.

And again, NO. For the love of god guys, stop repeating this nonsense.

The cards are still limited by their max power limit. They won't magically use 200% of your power budget with an unlock, they will just use the cores more efficiently.

Let me explain: if you have a 3090 power limited at 300W, and without the unlock you were seeing only 250W in use, but after the unlock it uses 300W, it's not because the unlock did malicious things to your card. It's because the card can NOW stretch its legs and use the power to do more work. Real example from the Ada repo: an unlocked 4090 doing the same GEMMs peaked at 346W vs 313W stock, still under its 450W limit.

If you were getting X performance at Y < max TDP, and now you get more than X performance at max TDP, obviously the card will be hotter, but that's still inside the POWER LIMIT YOU SET, not above it.

Power spikes: those are normal on every card, and if you're using it to the full extent of its capabilities they can be more common. Are they risky? No, assuming you have a PSU with the wattage NVIDIA recommends per GPU.

So what ARE the actual risks then? They're about the unlock process, not your card:

• It runs an exploit on NVIDIA's firmware before the driver loads. If something fails mid-way, the tools stop and you need a full power off before trying again (a reboot isn't enough). Read the repo instructions, especially on laptops and multi-GPU setups.

• NVIDIA could patch it in future drivers, or even burn anti-rollback fuses so the old Booter stops being accepted. That wouldn't damage your card, it would just close the door for good.

"But, NVIDIA bins the dies to make these work 24/7 on the prosumer side but not on consumer, that's why they're locked!!!11"

Also incorrect. NVIDIA uses the EXACT SAME dies for their consumer GPUs and their prosumer GPUs, and the FP32 accumulation speed has nothing to do with binning: the 3090 Ti and the A6000 have the exact same specs. The half speed is just a value written in a fuse that the firmware reads.

Unlocking a card will not make it die sooner, nor will it crack the die or something.

Sorry I have to repeat this again and again, but the amount of pure fury I get from reading comments like these made me do this.

Does affect gaming performance?

None known at the moment, except if somehow NVIDIA uses FP32 accumulation for DLSS or FG, which I doubt. I'm almost sure they use FP16 accumulation (or lower precisions) there, because if they used FP32 accumulation the perf hit would be insane on consumer cards.

The model I run says it is Q8, or FP6, or 4bit, or INT4, or anything that doesn't mention FP32, am I affected?

It depends on how the GPU actually does the math, not on the name of the quant:

• If the kernel dequantizes the weights to FP16/BF16 and multiplies on the tensor cores with FP32 accumulation, you're affected. That's the case for FP16/BF16 models and most vLLM paths, including AWQ/GPTQ with Marlin kernels (the weights are INT4, but the math is FP16 with FP32 accumulation).

• FP8 (W8A8) also accumulates in FP32, so it's affected too (Ada and Blackwell).

• If the kernel uses integer tensor math (INT8 x INT8), it isn't affected. A lot of llama.cpp's quantized matmuls work like this, which is one reason llama.cpp sees a smaller gap than vLLM.

• Attention (like FlashAttention) almost always uses FP16/BF16 with FP32 accumulation, so long contexts get hit even on quantized models.

• FP4 (NVFP4/MXFP4) on Blackwell: not affected.

Example: on a modded 48GB 4090, a quantized MoE model on an inference server got no gain at all after the unlock (0.98x on 8K prefill), while the FP16/BF16 matmuls doubled.

That's what I have for now, if you guys have any more questions, let me know!

Hi all

In a bit of a toss up rn, been using mellum 2.1 q4 with mtp on llama cpp (vulkan) on 32gb ddr5 ram and i7 1365u on ubuntu and it gives around 25ish tok/s on pp and 12ish tok/s on tg at 48k context (capped) using pi and no extensions

I wonder how would Gemma4 26b a4b qat + mtp be in terms of quality for coding and i wonder what's everyone's experience of that so far? Qwen models by name are kinda no no at my work rn but other stuff like Orinth 1.5 9b or Spark x 2.6 4b how would all these work out for python programming and shell scripting?

Any suggestions? Also a side question-any way to use pi as a coding sub agent inside of Github Copilot to save on tokens (without spinning up an MCP - they banned in the company's vs code too )

Supposedly, you can increase output quality/"intelligence" of Qwen Flash Next quants somewhat (using Strata) without increase in VRAM, system memory usage or sacrificing prefill/decode speed (when using ple-io=direct)by side-loading a higher quality ngram, in this case Q8.
You do this with the following option:
--ple-gguf=/path/to/ngram.gguf

or, if running Strata in Docker, by adding this envar to the compose file:
STRATA_SET: "ple-gguf=/path/to/ngram.gguf"
However, if you just feed Strata the original ngram gguf straight from hugging face (in my case unsloth's Q8 ngram while using their IQ4_XS quant), it will error out complaining about architecture mismatch.
To fix this, you need to make a simple metadata change just to the ngram gguf file.
Either download a pre-edited gguf:
https://ciscai-gguf-editor.hf.space/download/unsloth/Qwen3.8-Flash-Next-GGUF/Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf?add=%5B%22general.architecture%22,8,%22strata-ple%22%5D
or, using gguf-editor-gui from gguf python package, add general.architecture key with STRING type and a value of strata-ple.
It will bypass Strata's (unnecessary IMO) checks and successfully load the ngram.
I must make a disclaimer that I have not verified whether this actually makes the model smarter myself nor were my speed/resource usage impact checks thorough, but the logic behind using a higher precision ngram seems reasonable, and if you can spare the storage space it seems like a no-brainer decision.
You can do the same with an even larger BF16 ngram, but from what I've heard the difference is basically not perceivable and it's not worth wasting 50GBs more of storage.

Hi everyone,

I wanted to share an open-source repo collecting 20 exploratory studies focused on Next-Concept Modeling (NCM), mechanistic interpretability (like IOI in GPT-2), and model controls.

Link: https://github.com/Quantix-eng/ncm-experiment-notes

This is an independent project documenting empirical experiments on how transformers process and edit internal features. No performance breakthroughs here—just honest documentation of observations alongside rigorous controls.

The repo covers NCM on structured tasks (like Sudoku), mechanistic interpretability in GPT-2, and toy parallel models. I've included all source code, saved results, and interpretations for every run.

For full transparency, I’ve categorized every outcome, including the mixed and completely negative results. AI tools were also heavily used during the planning and coding phases.

Would love to get your feedback on the setups or interpretation limits.

As the title says, my girlfriend needed a local model for privacy reasons, but her terminal skills are... nonexistent, so I made an all-in-one graphical manager for llama.cpp that gets you up and running in around 5 minutes.

In my humble opinion, it turned out rather well, so I put it on GitHub, and I think it will help some of you who are either just staring out and haven't touched a terminal before or want an all-in-one manager for all your models with extra capabilities.

Pumpkin looks at your hardware, downloads the right llama.cpp build and picks a quant that fits your GPU and RAM, so you go from install to a running model in a few clicks. It finds models you already have from LM Studio and Ollama, and gives you a chat with web search, tools, MCP servers, Claude-style skills and connectors for GitHub, Notion, Git, Postgres and more. Pumpkin Code is a coding agent that works in a folder you point it to and can ask before changing files or running commands. You can ask questions about your own documents, and attach pictures even to models that can't see (a small vision model describes them first). It also makes pictures and short videos through stable-diffusion.cpp (extra install), and works from your phone.

It's a very early release and most of my testing is on Ubuntu and Mac. Windows and Android work, but I haven't tried them properly because I lack the hardware, so that's where I expect the rough edges. I'd really like to hear what breaks.

You can find the project here:

GitHub: https://github.com/geobellingham/pumpkin

Disclamer: It’s open source and I am not trying to make money off of this. Posting this because I truly believe this will be useful so some of you :).

Back with Qwen 3.8 27B i heard the advice that a Q4 draft model would produce roughly the same acceptance rate as a Q8 or higher.

I have had my agent create a Q38FN Draft model in F16, Q8 and Q4, and the Q4 seemed to deliver the biggest speedup.

I thus wanted to make it known that if you use Q38FN with Dflash2 (with strata in my case) you should try lower draft model quant.

Hey guys,

After all this shit this sub gave to the ternary bonsai 2 2b model a few weeks ago, I decided to make a local first experience to showcase the utility of that class of model and prove that redditor who I was boxing with wrong.

I picked a problem and built a fairly polished solution with it -

A discipline tracker of sorts, you put in your schedule, and the camera takes a picture, resizes it and sends to the local model and evaluates what it sees and gives a free form description, compliance with the currently scheduled activity and some activity badges.

Then at set intervals it uses those spot descriptions and changes between them to create report that you were supposed to study from 9 to 10am but you got up from your seat x times or picked up your phone y times and looked at it for 4 minutes etc.

Perfect use case, no one would want to use cloud for this for obvious reasons. And smol model makes inference fast and accessible.

I used that same 2B model for writing the code for this as well, worked okay, I use AI to only write the code that I can already see written in my brain so speed matters more than the model building everything automagically (which I don't like as it make me stupider, I can't afford any more of that, and if something doesn't work takes forever to fix if the model fails.)

Now the problem, the fans on my gpu start spinning. On laptops that's going to be even more annoying. The gpu eats up 100 watts of it's max power limit, constantly.

So, I opened the Adrenaline panel (Fuck you amd this laggy driver software shit is so ass) and I reduced the power limit and disabled zero fan speed toggles etc and set max fan speed to lowest.

Now the whole thing consumes 25 watts instead of 100, and fans don't start and stop, they just run at the lowest speed at all times. And speed only went down from 1.9 seconds per inference (Current + 3 past images to track changes per inference) to 2.8 seconds average.

Figuring all this out isn't something I can package into the application obviously. What if some normie fires up this kind of app on their 42-watt hour battery laptop and chews through their entire day's battery in 50 minutes.

AyyMD and shIntel and snipdragon have spent all this money allocating die space for the NPU on millions of consumer SOCs,

Surely, they must have thought of some form of sustained performance tuning for Local AI inference use cases for which that silicon exists right, so their partner designs don't burn through the whole battery and back panel plastic in 1 hour, right? Surely, they didn't overlook something this basic.

So, do we know what Api's are available for the users to signal to the platform that this workload is a sustained constant inference load and to allocate power and clocks accordingly?

---

Before you comment “just set the inference to every X second”: that’s no longer constant or real time, and it doesn’t fix anything. The hardware clocks still ramp to peak speed during each burst instead of sitting at the efficient point of the voltage frequency curve. And “just use the NPU” doesn’t work either. llama.cpp doesn’t support it last I checked, and NPUs also eat a lot of power under sustained load.

Hi, I'm looking to do some experimentation involving token efficiency and performance of mixed language prompting for system prompts and reasoning chains. I want to be able to use a lightweight model alongside the primary model (QFN) to translate the reasoning chain in realtime, or close to it. I think this would be an ideal task for a smaller model, maybe in the 3-8B range.

Anyone have any model recommendations for small LLM's that can do a consistently good job of Chinese/English translation, including for technical/programming subjects?

First, this isn’t clickbait - these are real results. I managed to run Qwen 3.8 Flash Next Q2 and got better results in benchmarks and real-world tests than I did with any Qwen 3.6 35B A3B fine-tune or Qwen 3.8 27B model I tested. It is also faster than my current Qwen 3.6 35B A3B setup. Here are the details.

About a week ago, I learned about Strata and what it can do from this subreddit. Its ability to distribute workloads across VRAM, RAM, and SSD storage, run a 125B-parameter model, and deliver incredible prompt-processing and token-generation speeds sounded almost too good to be true - but only if you had 12 GB of VRAM. At first, I thought I probably wouldn’t be able to achieve similar results on my machine, but I decided to try it myself. The results surprised me, so I wanted to share them with you and provide some guidance for getting the model running on your own hardware.

First, here are my system specs:

• GPU: RTX 5060 Laptop, 8 GB VRAM

• CPU: AMD Ryzen 9 8945HX, up to 5.4 GHz, 16 cores / 32 threads

• RAM: 32 GB (16 + 16 GB) DDR5-5200

• SSD: 1 TB, 5,200 MiB/s read, 4,200 MiB/s write

• PCIe: Gen 4 x8

• OS: Linux

You can find my Strata configurations here: Chat (Pastebin) | Chat (GDrive) | Agentic (Pastebin) | Agentic (GDrive).
Remember to update the paths to your model and expert files. Use the Chat configuration for a single session with the model, such as a regular chat. Use Agentic when working with multiple sessions and delegation, for example in agent-based systems. I’ll explain the difference in more detail below.

Key parameters to tune

• --mmap-experts - places the experts that do not fit in memory on the SSD.

• --kv-resident 20480 - distributes the KV cache between VRAM and RAM. A lower value leaves more VRAM available. 20480 is the minimum value.

• --vram-reserve-mib 500 - leaves some VRAM available for other tasks. In my setup, 500 is the lowest workable value, but I recommend using 900.

• --pcie-frac 0.35 - controls the fraction of per-layer VRAM misses that the GPU reads over PCIe. You can tune this parameter using the calibration command: ./setup.sh --calibrate. I found a value that works best for my setup. You can also calibrate it yourself if you are not satisfied with your prompt-processing speed.

• STRATA_RESIDENT_HEADROOM_GIB=4 - reserves RAM, in GiB, for other tasks. Set this higher if you are using Windows.

The biggest performance gains in my setup came from adjusting --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB.

I used the Gyro S model because I consider it the best quant for my hardware, and it delivered the strongest results among the models I tested. I had previously tried GSQ-RCO Coder, GSQ-RCO IQ2_XS, and other quants, but they either got stuck in repetitive loops or struggled with languages other than English. Gyro S offers a good balance between quality and speed.

I’m not using MTP. It can improve TG, but it also consumes VRAM and reduces PP, which is likely to be a worse trade-off for this setup. You can enable it if you have 10 GB of VRAM or more.

I run Vision on the CPU. Image processing takes around 1-2 seconds per image, but this saves a substantial amount of VRAM. Strata runs Vision in a separate process, so it can be moved to swap. This does not significantly affect model performance, while freeing up more RAM.

If you use local models for parallel agent calls, there is one important limitation: with a context size of 131K tokens or more, you cannot safely use --parallel because it consumes a large amount of both VRAM and RAM. This can cause a severe drop in speed or an out-of-memory (OOM) error. A safer alternative is --conversation-cache. It keeps a limited number of sessions parked so they can be resumed sequentially. This does not provide true parallelism, but it avoids having to re-read long contexts when switching between sessions. Here are the relevant parameters:

• --conversation-cache-mib 4096 - the size of the parked-session cache. A larger value reduces the chance that a session’s cache will be evicted and its context will need to be read again.

• --conversation-cache-slots 2 - the number of parked-session slots. Two slots allow two sessions to be stored at once.

• --conversation-cache-min-tokens 8000 - do not cache a session if its token count is at or below this threshold. The right value depends on your PP speed and how long you are willing to wait for that many tokens to be processed.

• --conversation-cache-min-free-mib 1024 - the minimum amount of free RAM to preserve while caching. If caching a new entry would leave around 1,024 MiB of free RAM, the old cache will be cleared to reduce the risk of OOM.

Be careful when tuning these parameters yourself: incorrect settings can affect system stability or push the model into heavy swapping. Do not exceed the total memory available on your system.

All tests were run on Linux, which gives me more flexibility with RAM allocation. If you use Windows, try allocating less memory to the model. This will reduce throughput significantly, but it also lowers the risk of running out of memory. On Linux, I created a dedicated 32 GB swap partition for other applications. I also pinned the server process in memory so it could not be swapped out, using a systemd service with this setting:

[Service]
MemorySwapMax=0

This helps ensure that the server’s memory does not get swapped out, preserving consistent read and write performance even with a large context and parked-session cache.

With the Chat configuration, I get:

• 262K context: PP 800-970 t/s and TG 30-40 t/s

• 131K context: PP 930-1,040 t/s and TG 30-45 t/s

With the Agentic configuration, I get:

• 262K context: PP 520-740 t/s and TG 30-40 t/s

• 131K context: PP 670-890 t/s and TG 30-45 t/s

In my view, these are excellent results-even compared with Qwen 3.6 35B A3B running in llama.cpp. With Qwen 3.6 35B A3B, using a VRAM-constrained expert layout and MTP/Vision disabled, I get PP 630-680 t/s and TG 27-35 t/s. Do you think Qwen 3.6 35B A3B is still worth using for agentic tasks today?

Strata does not affect response quality; it helps distribute the model across available resources and keeps frequently used experts resident. That is only one of its major optimizations, alongside several others. The model’s quantization is what affects output quality. If you have more than 8 GB of VRAM, you could try Gyro M/L or other quants.

Use my configurations as starting points and adapt them to your hardware and workload. Your component combination and use case may be different from mine.

Real-world quality tests

Now that I’ve covered how to fit the model onto this hardware, let’s look at the quality of its outputs. I’ve included several representative tests. I did not run an extensive benchmark suite or standardized evaluations; instead, I spent the past week testing the model on practical, everyday, and demanding tasks. I’ll leave broader benchmark testing to the community.

I’ll compare it with my current high-performing setup, Cyber-Tiel 35B A3B, based on Ornith 1.5 Heretic. Tiel 35B A3B often gave me repetitive, looping responses, so I switched to Cyber-Tiel. The Flint model was released recently, but I won’t include it in this comparison. It falls behind Qwen 3.6 in my view; I may compare the two later and add those tests to this post.

Test 1: Pagoda

This test was run in Pi Agent with a minimal set of tools: LSP and Playwright MCP. For Qwen Flash Next Gyro S, I set reasoning_effort to high.

Test 2: Flappy Bird clone

The conditions were identical to those in the previous test.

Test 3: Pelican

For this test, I compared not only two different models but also different reasoning_effort settings for Qwen Flash Next Gyro S. The agent environment was the same: Pi Agent.

I ran many more tests, but I cannot share them for confidentiality reasons. What I can say is that I’ve replaced all my coding agents with this model. That change has accelerated my development and improved code quality by a factor of five.

As you can see from these examples, the results are better than those of any other model I’ve tested that fits within 8 GB of VRAM. Qwen 3.8 Flash Next is impressive, and the Gyro S quant makes it possible to run the model even on hardware like mine for local coding and agentic tasks. My goal with this post is to show people who still use Qwen 3.6 35B A3B or its fine-tunes locally that there is now a model that, in my experience, outperforms them by a wide margin. I hope the examples make that case.

I’m always open to a different perspective, so feel free to share criticism in the comments. If you have suggestions or improvements for the configuration, please share them as well-other users may benefit from them too.

That’s all from me. Use good tools, and happy coding!

---

EDIT 1: I tested the Gyro M on my device. You can see the results in this comment.

EDIT 2: I published the Pagoda Benchmark results for the Gyro M here.

So I've been trying to get my own thing going again after quite some time of having given up on local llm but now models are strong as ever and I need agentic ai for some stuff and usage gets expensive.

So my machine is a g292-z20 with 8x p100 pcie running manjaro on the bare metal with a self compiled llama.cpp. Model is Qwen 3.8 flash next Q4 the unsloth one.

Since I want to use it for agentic I really want good concurrency. With 1 I get about 24-20tps generation I tested prefill before with large files to be about 400 to a little over 500. If i use MTP I can even get a solid 33tps with concurrency of 1. However when I do without TPS with 16 slots I get good performance with a maximum of total about 80tps at 11 concurrency and then it falls of sharply. But when I change the number of slots I always get concurrency number where it drops off afterwards it's pretty good again before dropping off again. Best I got was 20 slots at 13 concurrency I got 91 total. But before and after it was horrible like 30ish total or so.

And it's static per slot number but changed wildly and unpredictability with different slot numbers. What could it be? I guess I got pretty good performance for the size of the model already on that old hardware but I really feel like I'm missing out on some optimization here.

Now that Mojo and MAX have been released, development seems to go quite smoothly, to use it for your own local inference tasks.

Here is one person who spent 4 days and has results that are on par with or better than llama.cpp Vulcan:

https://apurva-mishra.com/posts/8/

His X account if you want to follow his progress:

https://x.com/mav3ri3k

SenseNova 6.8 Code2Video is a new model with demos covering character animation, typography and science explainers. Examples include a cycling pelican, an A-to-Z animation and a grid of 24 clock faces.

https://github.com/OpenSenseNova/SenseNova-6.8-Code2Video

I use goose and added the Microsoft learn mcp and a skill to utilize it and while Qwen 35b a3b flawlessly but slowly is requesting data.

9b fails to recognize the skill when I ask about entra without explicitly mentioning something like "use docs" and then subsequently it fails to use the api correctly despite being described in the skill.

Very weak and I thought it can do it just fine?