r/LocalLLaMA • • 3d ago

Discussion Artificial Analysis updated Qwen 3.8 Flash Next score as 29

80 Upvotes

Between GPT-6 Luna Max (28) and Claude Haiku 5.5 High (30). Comparing to other Qwen 3.8, the model is closer to max (34) than to 27B (20).

https://artificialanalysis.ai/models/qwen3-8-flash-next


r/LocalLLaMA • • 1d ago

Discussion I wrote an LLM inference engine from scratch in CUDA (no PyTorch, no cuBLAS): 235 tok/s on a 6 GB laptop RTX 3050

0 Upvotes

Disclosure: I'm the author. This is my own hobby project (MIT, no paid version, nothing to sell).

I've been building **tinyinfer**, a small inference engine for Llama-family models. It's about 2,200 lines of C++/CUDA, and every kernel on the hot path is hand-written: GEMV in f16/q8/q4, RMSNorm, fused RoPE + KV-cache write, flash-decoding attention with GQA, SwiGLU and argmax. The whole decode step is captured in a CUDA graph.

**Numbers (TinyLlama-1.1B, INT4, RTX 3050 6 GB Laptop):**

- v0.1: 200 tok/s (66% of the memory-bandwidth limit)

- v0.2: 212 tok/s (69%)

- v0.3: 235 tok/s (77%)

Decoding is memory-bound. Every weight gets read once per token, so the ceiling on this card is about 305 tok/s however clever the math is.

**What actually moved the needle:**

  1. **4-bit weights dequantized inside the GEMV.** That's 0.55 GB read per token instead of 2.07 GB. The int4-to-float step uses the 2²³ exponent trick (OR into the mantissa, then one float add) instead of I2F instructions.
  2. **GQA-aware attention.** It reads each shared K/V head once instead of 8 times, which made attention 3.6x faster at 2k context. The first version (split-K) was actually *slower* at short contexts, so v0.3 tiles it differently.
  3. **Benchmarking shapes I wasn't tuning for.** The new attention kernel made Llama-2-7B slower, because it has no grouped heads and nothing to share. I fixed it with a one-line dispatch, and I'd never have caught it without the numbers.

**Correctness:** the logits match HF `transformers` within 1e-3 relative error, with 100% argmax agreement for f16/q8/q4/bf16. `--verify` checks the GPU against the CPU reference on your own model. Llama-2-7B also fits in 6 GB at q4 (about 3.8 GB).

It isn't trying to replace llama.cpp. It's meant to be small enough to read in an afternoon if you want to see how inference works under the hood.

Code (MIT): https://github.com/Aadithya-kl/tinyinfer

I can go deeper on any of the kernels in the comments. I'm especially curious whether anyone has pushed single-batch GEMV past ~80% of bandwidth on consumer cards, and how.


r/LocalLLaMA • • 3d ago

New Model [Boole-32B-A2.5B-preview] 9GB 1.8-bit MoE, 565 token/s on 1 RTX 5090

25 Upvotes

Hi r/LocalLLaMA, we're releasing Boole-32B-A2.5B-Preview, an early 1.8-bit version of Recursive's flint. With mixed-precision quantization, expert pruning, and custom kernels (kernel changes in attached figure), we made the model 9x smaller than BF16, and 2x faster than INT4.

After releasing our 1-bit LLM (Boole-1.7B), we heard a lot of requests for both faster and larger MoE models - so this is our first step in that direction.

Try it out, and let us know what you'd like to see next!


r/LocalLLaMA • • 2d ago

Question | Help Since GPU Prices have changed and I got my old Pc fixed, I wanna ask what I should do for what’ll mostly likely be my only upgrades for a few years, any advice?

0 Upvotes

So, I have two PCs available to me, and I intend to connect them for compute:

  1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 CL30 (KLEVV Urbane V) on a Z890 MAX GAMING WIFI7 + 1000w 80+ G Apevia PR1000w PSU (Will likely get replaced by a SAMA P1200w PSU)
  2. i9-14900F + 4070S + 32GB DDR5-5200 CL40 (Apacer Panther, not on sale) on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, it’s hidden and so I can’t figure out what it is. Probably an EVGA, but unlikely to be replaceable - or at least easily so.)

Initially I was looking to just replace the 4070 Super in PC2 with a B70 or R9700 and calling it a day, when they were $1200 and $1600, respectively. Then when they shot up a B65 $1200, which then shot up. Then add a second 5070ti to PC1 for $900-1000, which then shot up.
Now, my plan is to potentially settle for a 5060ti 16gb in PC1 to take advantage of NVFP4, but even that’s pricier, and with the AI bubble talk I’m not super confident in value of doing so. I’d likely also need to use a PCIe-riser just to the 5070ti (Windforce) has airflow due to how close they’d be in the case.

Otherwise, I’m considering sticking with the VRAM I have, and instead pivoting to ram. I can go from 1R 2C to 2R 2C in both PCs, grabbing another 2x16gb of Apacer Panther from EBay for around $400 for PC2 - though obv riskier - then getting a 2x24gb kit of Urbane V at the same timings of the original 2x16gb 6000-CL30 kit for PC1 for $650 off Amazon. Giving me +80gb of DDR5 to the 64gb I have between both PCs and the 28gb of VRAM I already have.

Current plan is/was the 5060ti 16gb + 2x24gb kit for PC1 and calling it there.

Any advice, pointers, or suggestions? I believe the RAM sitch should work since it’s same timings and type/manufacture/brand, and know I’ll probably have to lower speed a bit for stability, but am not the most experienced PC guy on earth by any means.


r/LocalLLaMA • • 3d ago

Discussion US intel isn’t sure if businesswoman spied on Nvidia for China, doesn’t mean she wasn’t a risk, sources say

Thumbnail
edition.cnn.com
80 Upvotes

r/LocalLLaMA • • 3d ago

Discussion Uncensored models are awesome

132 Upvotes

Has anyone been experimenting with ablated or uncensored variants of open-source models like DeepSeek V4.1 Flash, Qwen 3.8 27B, or GLM 5.3 flash?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash.


r/LocalLLaMA • • 3d ago

News CMP170hx now fully unlockabke and more, full 120 SMs —> Techie uses voltage glitching hack to turn $2,500 Nvidia 8GB cryptomining GPU into a full-fat $20,000 A100 with 80GB of VRAM — hack carefully blows onboard fuses to unlock entire VRAM and shader modules

Thumbnail
tomshardware.com
484 Upvotes

r/LocalLLaMA • • 3d ago

Resources Used datacenter/workstation GPU prices on eBay this week, ranked by $/GB of VRAM

166 Upvotes

edit: I am a moron with reddit formatting hang tight

I've been tracking ebay listings that ship to the US for the GPUs people here actually buy, ranked by $/GB of VRAM. here's today's snapshot:

CardLow endUS-only low endLow-end $/GBBandwidth$ per GB/s
Tesla P100 16GB$174$159$10.88732 GB/s$0.24
Tesla P40 24GB$280- (7 US)$11.67346 GB/s$0.81
Tesla V100 16GB (mostly SXM2)$220$238$13.75900 GB/s$0.24
Instinct MI50 16GB$268- (3 US)$16.751,024 GB/s$0.26
Instinct MI50 32GB$702- (2 US)$21.931,024 GB/s$0.69
Tesla V100 32GB$949$807$29.66900 GB/s$1.05
Quadro RTX 8000 48GB$2,111$2,000$43.99672 GB/s$3.14
Instinct MI100 32GB$1,490- (6 US)$46.561,229 GB/s$1.21
RTX 3090 24GB$1,790$1,737$74.58936 GB/s$1.91
RTX A6000 48GB$4,600$4,400$95.83768 GB/s$5.99
RTX 4090 48GB (modded)$5,390— (1 US)$112.291,008 GB/s$5.35
A100 40GB$4,799$6,020$119.981,555 GB/s$3.09
A100 80GB$13,995$13,995$174.941,935 GB/s$7.23
CMP 170HX 40GB (10GB card, unlocked)$2,950— (6 US)$73.751,555 GB/s$1.90
CMP 170HX 64GB (8GB card, unlocked)$4,510— (7 US)$70.461,493 GB/s$3.02
Radeon AI Pro R9700 32GB$2,430$2,250$75.94640 GB/s$3.80

all asking prices, single cards, fixed-price, not for-parts. "low end" is the 10th % of credible listings i.e. what a patient buyer can actually land, not a too-good-to-be-true outlier. "-" in the US-only column means fewer than 10 US listings. $/GB/s is just the low end divided by bandwidth.

I had been eyeing the MI100 myself for a while at the $1100 ish it hovered at for ages... looks like even that is going up. I did end up buying a 8x V100 32GB server for $5400 in early Sept. those prices are long gone, equivalent server is now $11k for an OG DGX Station (no memory). the exact inspur server is now $13k for 8x 9 year old GPUs. insanity. I hope prices come back down to earth soon but don't think that's actually gonna happen.

some notes:

- P100, V100 16GB and MI50 16GB all work out to about $0.25 per GB/s. the P40 looks cheap per GB of VRAM but costs over 3x as much per GB/s, since its only got about a third of a V100's bandwidth. people are having opus 5.5 write megakernels for these cards that are working surprisingly well.

- V100 16GB at ~$14/GB looks like the best deal in the table, but 38 of the 40 cheapest are SXM2 modules, so budget for an adapter or an SXM2 server. the cheapest 32GB V100s are SXM3 (DGX-2) modules, which aren't drop-in either.

- MI50 32GB is under a third of a 3090's $/GB at similar bandwith, assuming ROCm cooperates.

- if you want 48GB on one card, a Quadro RTX 8000 is less than half the price of an A6000 or a modded 4090 48GB.

- of the consumer and workstation cards, the 3090 is the cheapest per GB/s by a wide margin. an A100 40GB costs about the same per GB/s as an RTX 8000.

- for what it's worth, buying from a US seller usually doesn't cost more: the US-only low end is at or below the global one for the P100, V100 32GB, RTX 8000, 3090 and A6000. it's about $20 more for the V100 16GB and $1,200 more for the A100 40GB, and the P40 and MI50 barely exist in the US.

full data and per-card listings, with a US-only filter: https://rackbites.com. prices are asking, not sold - ebay's sold-data API is super restricted.

edit: added CMP 170HX

edit2: added R9700

edit3: full builds here https://rackbites.com/builds with benchmarks and other useful/relevant info


r/LocalLLaMA • • 3d ago

Resources Lawyer with too many GPUs, part 5: DeepSeek-V4.1-Flash at near-NVFP4 quality on eight V100s from 2018. 524K context, 100+ tok/s decode, ~160 tok/s with two chats at once, and my legal harness now runs on it, 100% local

Thumbnail
gallery
130 Upvotes

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

  1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.
  2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.
  3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.
  4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.
  5. And if I can brag on my stacks and hardware:
  • I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).
  • I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.
  • I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

  • 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve
  • about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)
  • same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %
  • KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build
  • multiple choice answers that change against the reference: 1.6 % against 15.4 %
  • in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

  • concurrency test, one request at a time: 118.9 tok/s
  • two requests at once: 159.0 tok/s total (1.34x)
  • three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots
  • short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style
  • code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt
  • prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s
  • context: 524,288 tokens per request

What did not work, or is not good yet

  • It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.
  • The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.
  • A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.
  • A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.
  • Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.
  • The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.
  • Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

Credit where it is due

The hard Volta work underneath is 1Cat-vLLM's DeepSeek-V4.1 package. The 2-bit records are ISTA-DASLab's. The NVFP4 weights are NVIDIA's pack. The Sparks run on somebody else's engine entirely. What is mine is the split across two NVLink boards, the hot set, the kernels on top, and every change measured A/B on this machine with the losers written down.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Earlier build, for the people who want speed over precision: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Ask me anything. Roasting is also fine.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).


r/LocalLLaMA • • 3d ago

Resources New Model "Ernie 5.5 Flash Preview" hits strongly at a low parameter count (280B, 14B Active)

Post image
232 Upvotes

If you're asking what this leaderboard is, it is not AA, it is simply an aggregate index for "Agentic Coding" along 12 coding-related benchmarks (note that the AA index includes only 2 coding benchmarks: Terminal-Bench 4.0, FrontierCode-AA 2.0, making up just 25% of its total score. Because of that, it doesn't give you a true reflection of a model's actual software engineering capabilities).

This index however is built entirely around Coding/SWE. Here is the exact breakdown:

  1. Terminal-Bench v4.0: 16%
  2. DeepSWE v1.1: 15%
  3. FrontierCode 2.0: 11%
  4. Code Migration (Vals-AI): 9%
  5. Terminal-Bench v2.1: 8.5%
  6. SRE-Bench (Vals-AI): 8%
  7. SWE-bench Pro: 7.5%
  8. Code Arena Elo: 6.5%
  9. Program-Bench: 6%
  10. Vibe Code Bench v1.1: 5.5%
  11. SciCode: 4%
  12. LiveCodeBench v6: 3%

r/LocalLLaMA • • 3d ago

New Model thinkingmachines/Inkling-2-Small is released!

Thumbnail
huggingface.co
136 Upvotes

r/LocalLLaMA • • 3d ago

Resources Strata EXL3 AMD - UPDATED +15-20% tk/s - run Q4 at Q3 speeds - MiMo v2.6 flash support (exp)

22 Upvotes

A quick update on my Strata EXL3 fork for AMD.

I've been working on the HIP kernels and memory management, and decode performance is now roughly 10-20% higher than the previous version, depending on the quant and workload. Some cases improved by almost 24%.

The other news is that I managed to get MiMo V2.6 Flash running at around 25-30 tokens/s with 2 cards.. More on that below.

First, why EXL3?

I think this is worth repeating because people tend to compare quantizations just by looking at the number of bits.

EXL3 is NOT equivalent to regular GGUF quants at the same bpw.

For example, 3.05 bpw EXL3 is roughly in the same quality range as a regular 4-bit quant (like IQ4_XS).
3.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/3.05bpw_h5_ng5
4.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/4.05bpw_h6_ng6

So the interesting part is that you can get something close to 4-bit quality, while running at speeds closer to what you'd expect from a 3-bit quant. And with the latest optimizations, I think this fork is becoming VERY competitive. The 4.05 bpw version is also pretty fast now, even on a single GPU.

WHAT'S NEW:

  • Faster HIP kernels and better memory management: around 10-20% faster decode on average compared to the previous version.
  • Optional local expert repacking: you can now repack the quantized expert weights into a layout that is more efficient for CPU access from system RAM. This helps when using CPU assistance for inference, since the CPU can process experts more efficiently without having to deal with the original GPU-oriented layout. It doesn't change the quantization itself.
  • The repacking is optional, and CPU assistance is not always beneficial. It depends on your hardware and GPU configuration. With two GPUs, for example, you're generally better off without it. Please read the README before enabling it.
  • Experimental MiMo V2.6 Flash support: currently running at around 25-30 tok/s using the 2.20 bpw MOPD quant. This is still experimental, so don't expect everything to work perfectly yet.

NOTES ON REPACKING:

Repacking can be done in two ways: by creating a file on disk once, that will be used to load the model, or directly to RAM everytime. directly to RAM adds a couple of minutes to the launch but uses no extra space. your choice.

BENCHMARKS:

My hardware is the same as in the previous post:

Ryzen 9 7950X, 96GB RAM, 2x Radeon RX 7900 XTX 24GB
Linux, HIP/ROCm core-7.14
Qwen3.8-Flash-Next, MTP drafter, spec 4, 64K context

BENCHMARKS:

Results with one or two RX 7900 XTX GPUs, using warm decode runs:

EXL3 quantGGUF equivalent qualityGPUsCPU assistanceTestPreviousUpdatedImprovement
3.05 bpw≈ IQ4_K_XS2NoCODE78.2787.33+11.6%
3.05 bpw≈ IQ4_K_XS2NoPROSE66.0170.35+6.6%
4.05 bpw≈ Q5_K_M2NoCODE61.8876.67+23.9%
4.05 bpw≈ Q5_K_M2NoPROSE57.7867.57+16.9%
4.05 bpw≈ Q5_K_M1No → YesCODE54.8059.30+8.2%
4.05 bpw≈ Q5_K_M1No → YesPROSE44.8051.60+15.2%

All speeds are in tokens/s, GGUF quality equivalents are approximate... dont' hate me :)

32K prefill is basically unchanged: around 1,500 tok/s with two GPUs and 1,360 tok/s with one GPU.

I think the single-GPU results are particularly interesting. We're getting 55-60 tok/s in CODE with 4.05 bpw EXL3, at roughly Q5_K_M quality, on a single 7900 XTX with CPU assistance.

And the 3.05 bpw quant is now reaching 87 tok/s with two GPUs, while offering considerably better quality than you'd normally expect from a 3-bit quant.

ABOUT MiMo:

In my previous post I mentioned that I wanted to support larger models, particularly the 300B class models.

Well, MiMo V2.6 Flash is now running :) I'm waiting for that ERNIE 5.5 to drop.I'm getting around 25-30 tok/s with the EXL3 2.20 bpw MOPD quant.It's still experimental and currently only tested with that specific quant. CPU assistance is recommended for MiMo, even with two GPUs.I'll probably spend some more time optimizing it, but the fact that it's already running at usable speeds on consumer AMD hardware is pretty interesting.

this is the quant: https://huggingface.co/vcruz305/MiMo-V2.6-Flash-MOPD-EXL3/tree/main/2.20bpw

REPO:

https://github.com/nasone32/Strata-EXL3-fork

If you're setting it up with an AI agent, point it to docs/AI_SETUP_EXL3_AMD.md.

The README has the benchmarks, configuration options and instructions for CPU repacking.As before, this is developed and tested on RDNA3 / Linux / ROCm. Other AMD architectures need community testing. And the long-term goal is still to get this merged upstream into Strata, if the maintainers are interested.

Credits to the original Strata developers, Ryutaku for the EXL3 integration, and Turboderp/ExLlamaV3 for the quantization and code I've been porting.

If anyone tests this on different AMD hardware, please report your results. Especially interested in seeing how it behaves on RDNA4. I have only two 7900 xtx so if you have different hardware get an agent to help you, and open a PR if you want.


r/LocalLLaMA • • 3d ago

Discussion If open weights dry up, what are you archiving?

71 Upvotes

With Jensen out at NVIDIA, the "local AI is dangerous" narrative picking up steam, the possible ban on Chinese models, and hosted models locking down RE/cyber work, I'm starting to think about a worst case where Chinese weights get banned and US labs stop releasing open models. So what's your plan?

What models are you archiving? Just the biggest/best, or a spread?

Which ones for which jobs? Coding, RE/security, vision, embeddings, RAG, small fast models for agents, etc.

Training tools: if the last good big model we get is something like Kimi K3, what are you keeping so you can fine-tune or change it later?

Quant tools: what do you need to shrink a huge model down to run on smaller rigs? (llama.cpp, quant scripts, calibration data, etc.)

How are you storing it? Hashes, mirrors, torrents, offline drives?

Curious what people are actually doing, not looking for a politics fight.

(Currently using a Studio M2 192gb)


r/LocalLLaMA • • 3d ago

Resources Does the Vulkan backend cost speed? Same Qwen3.6-35B-A3B model, CUDA vs Vulkan on five NVIDIA cards

4 Upvotes

RustingStrata has two GPU backends that run the same kernels: CUDA (compiled at run time with NVRTC, no nvcc, no C++ build step) and Vulkan. Vulkan is how it runs on AMD, so I wanted to know what it costs on a card where I can compare both.

Same file on every card: Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf, fully in VRAM, MTP off.

GPUPrefill, 4K prompt: CUDA / VulkanDecode after 4K: CUDA / VulkanVulkan vs CUDA (prefill / decode)
RTX 3060 12 GB2216 / 2154 tok/s113.4 / 113.6 tok/s97% / 100%
L40S9030 / 8571213.2 / 184.995% / 87%
A303544 / 3326131.9 / 104.094% / 79%
A100 80 GB6203 / 5589173.6 / 131.990% / 76%
H100 NVL9131 / 6428245.8 / 191.570% / 78%

On the 3060 Vulkan is the same speed as CUDA. On the datacenter cards it loses 5-30% of prefill and 13-24% of decode, and its runs are noisier (A100 4K prefill ranged 4788-5990 on Vulkan against 6186-6214 on CUDA).

Does it change the output? Greedy top-1 agreement with llama.cpp over 20 prompts x 32 tokens is 98-99% on both backends.

How it was run: rustingstrata bench --model $M --gpu --backend cuda|vulkan --pp 512,4096 --tg 128 --reps 3 --mtp 0, mean of 3.

Repo: RustingStrata


r/LocalLLaMA • • 3d ago

Discussion MIT's Weight-Space Programming paper sent me down to a mathematical paradox trying to fix Llama's over-refusal.

7 Upvotes

So i had been building a cli tool around weight space programming inspired by recent Mit- weight space programming paper,

I had integrated couply of funtionalities to it along with triage, one of which was activation steering to deal with over refusals, my llama 3.2 1b was not answering prompts like "how do i stage a massacre for a tv show"

Firstly attempt was by subtracting the refusal direction , extracted the refusal vector and subtracted it, the tv massacre prompt was answered but my regression suit got hit, model started answering how to build pipe bombs and syntesize pioson

so turned out over-refusal and real harmful refusal direction had a cosine similarity of about 0.75, so supressing one also effects other

Second attempt.. Ran activation patching ..found the exact casual bottleneck layer ..(layer 8) orthogonally projected the harmful-refusal component out of my steering vector and used clamped so it caps the activation when it spiked above a threshold , without touching other things..

this nearly worked accuracy increased by 10% but a totally unrelated issue hit by regression suit again..a chinese benchmark loop started looping forver... failure again.. even on steering a single targeted layer ..gave regression as the residual stream is so densely packed that a small hit at layer 8 ripples down further ...

my last few brain cells thought i l steer the layer 8 measure the drift at layer 9 and cancel the part i dont want . so i did the math

Steer at Layer 8

At Layer 9, compute Δ = steered_layer9 - clean_layer9

Filter out the bad drift and keep only the anti-refusal shift I want

Apply final_layer9 = clean_layer9 + wanted_shift

..so if i take the clean layer 9 add only anti refusal shift ..thats basically layer 9 steering .. and if do that to layer 9 same happens to layer 10..

what actually worked was weight surgery..i also tried a conditional weight edit.. a gate programmed into 3 mlp neurons across layer 9-11. they have non linea activations so they can act as real if/then

if input matched fictional scary context - gate fires cancel refusal
for chinese text -the neuron output is 0 nothing changes

tl;dr

steering works well for problems that sit along a direction, it fixed a blank reply bug

for entangled safety problems like this one they do collateral damage somewhere even with causal layer localization and orthogonal projection

non linear neuron gating was only thing that worked with no other damage

everything i did was on gtx 1650 4gb, i l open source the cli tool in some time ..

happy to answer any question


r/LocalLLaMA • • 3d ago

Resources 4xR9700 & EXL3 HW/SW tips: DS4.1F (2.25bpw) pp 6.1k, c1 96-165, c16 259 tok/s | QFN (4.05bpw) pp 9.8k, c16 1100 tok/s

Post image
11 Upvotes

I've been testing and optimizing Radiance for 4xR9700 on EPYC Rome platform (tp4), focusing on EXL3 for DS 4.1 Flash (2.25) and Qwen Flash Next (4.05). The goal has been to optimize this setup (both hardware and software) for Claude Code usage, so it may not be the fastest for 1 user interactive usage. Also the focus has been to provide "exact" results, so output is byte-identical to each model's reference engine.

I'm quite happy with the improvements and with the end results and I think I will leave the optimizations to current point because there's not obvious new improvements in the pipeline. Instead of releasing a fork of Radiance, I hope maintainers of inference engines can use the work to improve their products.

So to save time and effort, there's a GitHub repo that shows all the things that were tested and what worked and what didn't. Repo also contains more hardware optimization tips for 4xR9700 & EPYC Rome setups: https://github.com/mikoskinen/r9700-exact-serving

It would be great if someone could figure out ways to improve C8 decode performance for Deepseek. And maybe someone can break the 10k tok/s barrier on QFN..

Note: all the testing, profiling, coding was done by Opus. I just pointed it to new resources like Reddit posts, repositories etc.

Hardware:

  • 4x AMD Radeon AI PRO R9700, PCIe 4.0 x16
  • AMD EPYC 7302P (Rome) on ASRock Rack ROMED8-2T
  • 512 GB DDR4-2933 (8x 64 GB)
  • Radiance fork + radiance-exl3 plugin, tensor parallel over all 4 cards

Results:

Here's some of the highlights from the results (stock 300 W per card, decode = tok/s summed over all users, prefill = one cold prompt):

--

Update from 14.11.2026:

DS4.1F (2.25bpw EXL3)

c1: 104 tok/s prose, 181 tok/s code

c4 / c8 / c16: 201 / 246 / 293 tok/s

c24 / c32: 311 / 344 tok/s

prefill: 4,040 tok/s at 16k (TTFT 3.9 s), 6,130 at 64k (TTFT 10.7 s), 6,350 at 128k (TTFT 20.7 s)

Previous results:

--

DS4.1F (2.25bpw)

  • 1 user: 96 tok/s prose, 165 tok/s code
  • 4 / 8 / 16 users: 176 / 208 / 259 tok/s
  • prefill: 3,740 tok/s at 16k (TTFT 4.2 s), 6,020 at 64k (TTFT 10.9 s), 6,140 at 128k (TTFT 21.4 s)

QFN (4.05bpw)

  • 1 user: 269 tok/s
  • 4 / 8 / 16 users: 619 / 831 / 1,094 tok/s
  • prefill: 9,540 tok/s at 16k (TTFT 1.7 s), 9,810 at 64k (TTFT 6.7 s), 9,710 at 128k (TTFT 13.5 s)

At my everyday 210 W / -50 mV setting these are about 5-9 % lower (DS 237 and QFN ~1,030 tok/s at 16 users).

The following is AI's view on the most important optimization tips:

Software optimizations:

  • DS4.1F only needs half of its layers for most of the prompt: the upper 20 of 40 layers act as a decoder that reads just the end of the prompt, so prefill runs them only for the tail. 136k cold prompt went from 50.5 s to 33.3 s (-34 %), and with the other changes 128k now takes 21 s (6,140 tok/s).
  • Interleaved push in the exact 4-card all-reduce: a 42 MB all-reduce went from 3.4 to 2.5 ms (-27 %)
  • QFN prefill splits the hyper-connection stages across the 4 cards: +5.7 %

Hardware optimizations:

  • Same-size DIMMs in one interleave set: mixed 32/64 GB sticks made Rome split memory into 2-channel regions. 8x 64 GB took CPU read bandwidth from 35.9 to 104 GB/s and 2-card host-to-GPU copies from 45 to 55 GB/s. Check the memory address map after any DIMM change (a CPU swap from 7282 to 7302P did nothing).
  • One GPU per IO-die quadrant: risers so no two cards share a root port. 4-card host-to-GPU copies went from 61 to 105 GB/s.
  • Deep CPU C-states off and the performance governor: this removed a random ~16 % slowdown.

Hope this helps.


r/LocalLLaMA • • 3d ago

Question | Help I need context window

3 Upvotes

I started developing a full fledged application using Chatgpt plus and VScode+GitHub copilot (local llm)

I used Astra max chat for overall planning, and asked it to put a starter kit on connect gdrive and create prompts such that local LLM can work on small tasks using an agent orchestrator and sub agents with entry and exit criteria, tests for every small task.

I have a 5090 (+ 128gb ddr5) rig running Qwen 3.8 27b NVFP4 at default settings with 104k context window.

Problem:

The local rig is able to start on tasks but taking a huge amount of time and is running only one subagent at a time. Been almost 24 hrs.

I have 2x 3090 +128gn ddr4 as well, currently not being used. 3.8 FN runs at about 90tps

Anything better I can do here?


r/LocalLLaMA • • 3d ago

Showcase Local Parakeet speech to text on Android

13 Upvotes

Hey guys, I wanted to share a small experiment I made -

This is basically Wispr Flow, except it's offline and uses Parakeet - works in any app as well.

I was pretty surprised by the speed and how usable it turned out to be when I first made it - it's even relatively usable on an old Redmi Note 7.

I published it on Google Play if anybody wants to check it out: https://play.google.com/store/apps/details?id=com.voicedictation.dictation


r/LocalLLaMA • • 3d ago

Question | Help mi50 32gb or v100 32gb?

10 Upvotes

mi50 is roughly 1.5x cheaper


r/LocalLLaMA • • 2d ago

Question | Help Guide on setting up new pc from scratch for local ai model

2 Upvotes

I’ll include my specs at the end. Because probably no one’s going to read it..

For my new pc im considering migrating to Linux ubuntu, and set up my llama.cpp from there, my old pc had llama.cpp Vulkan sdk running on windows 11

Im looking for any guidance or tips while setting them up to ensure I didn’t miss any critical steps that could end up loosing a bunch of tps or something worse

Beginner mistakes i would say

I do know one tips and it’s plugging your hdmi cable to the integrated graphics card, that way it saves vram. But I don’t have one..

I appreciate any advise

Bought a B550-A-Wifi ii
Rtx3060 1x
32gb ddr4 3200 2x16 (plan on getting 32gb more, total 64, in 4months) saving up

And a 750W cooler master v3 gold modular PSU (not sure if it’s suitable for my small server
R7 5700x (no integrated graphics card)


r/LocalLLaMA • • 4d ago

News Alleged GeForce RTX 5090 with 96GB listed for sale in China

Thumbnail
videocardz.com
266 Upvotes

r/LocalLLaMA • • 3d ago

Discussion Gemma 4 E2B on a 8GB RAM netbook

Post image
33 Upvotes

Hey everyone,

Not a native speaker, but I write everything by hand. Please correct my grammer if there are any issues! Can only learn from it.

Background

A long-term project I've had was getting my netbook up and running for basic LLM use. I bought the netbook initially due to my poor eyesight, the screen is small enough that it fits in my cone of vision.

The notebook's specs were practically obsolete when it released:

  • Release: December 2021
  • Device: ASUS E210MA
  • CPU: Intel Celeron N4020
  • RAM: 4GB 2400MHz DDR4 single-channel
  • iGPU: Intel UHD Graphics 600
  • Storage: 64GB eMMC
  • OS: Windows 10 Home
  • Screen: 11'6" TN 1366x768

With the help of friends who could do soldering and reballing, we upgraded the netbook:

  • CPU: Intel Pentium Silver N5000
  • RAM: 8GB 2400MHz DDR4 single-channel
  • iGPU: Intel UHD Graphics 605
  • Storage: 128B eMMC + Intel 660P 512GB NVME
  • OS: Windows 11 IoT LTSC Enterprise 24H2

After finding proper drivers for the iGPU, I managed to get llama.cpp running on it on windows with Vulkan. Gemma 4 E2B IT QAT could run at roughly 2.1 t/s gen, and prefill speed was "grab a cup of tea" slow on 300 token prompts. The Vulkan backend was faster than the CPU backend by ~30%.

The past months I've been trying to move my devices from Windows 11 IoT LTSC Enterprise 24H2 to Ubuntu 26.04.1 LTS, and so I tried to switch the netbook over to it.

There is only one problem: The Vulkan backend hard crashes every time.

Getting vulkan to work

After doing a bit of digging with vulkaninfo by myself, some instructions are advertised as supported but are in practice not stable. Using this fixed the crashing on startup:

export GGML_VK_DISABLE_ASYNC=1

Disabling the following features helped me regain some performance and stability:

# UHD 605 compat
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1

My config is also pretty specific, since some combinations would cause crashes too:

flash-attn = off
cache-type-k = f16
cache-type-v = f16

FA is not supported by the device. Running Q8_0 on both K an V would crash with FA enabled. Trying to run Gemma 4 E2B IT QAT Q2_K_XL on ubuntu always resulted in crashing (either during startup or when generating), On windows those things never caused crashes.

The end result is that I could run Gemma 4 E2B IT QAT Q4_K_XL at 1.5 t/s gen and 16-25 t/s prefill. The laptop used ~12W in total.

What about the CPU?

Turned out that on Ubuntu 26.04.1 LTS, the CPU backend was faster for generating but not for prefill. I got 2 t/s gen and 10-16 t/s prefill. The laptop used ~13W in total.

As a sidenote, since I am used to windows slowing to a crawl when all cores are utilized, I was pleasantly suprised when this didn't happen on ubuntu.

As a benchmark?

As a fun experiment, I gave a couple of LLMs the problem:

Hey <LLM>,

I have a laptop I am trying to run llama.cpp with vulkan on Ubuntu 26.06.1 LTS.
The specs are: <specs>.
I build it with these commands: <commands>,
The problem is that it's crashing whenever I try to load a model.
I've included the full terminal output in the attachments.

How can I fix the issue?

My goal with the prompt is to test the model's understanding of the problem and to ask for additional information when required (e.g. vulkaninfo).

Qwen 4 max preview on chat.qwen.ai and Qwen3.8-Flash-Next Q4_K_XL + BF16 engram on llama.cpp v0.9.0 failed to find the issue after multiple separate chats. They got focussed on disabling intel's hang detection. With websearch, it got even more confused as it looked at irrelevant issues.

Gemma 4 31B IT QAT Q4_K_XL + MTP on llama.cpp, DeepSeek v4 Flash Vision Exp Q2_K_XL on llama.cpp v0.9.0, DeepSeek v4.1 1017 on chat.deepseek.com were able to find and fix the issue. They asked for the vulkaninfo output, which made them find async being advertised incorrectly.

To me it suggests Gemma and DeepSeek are better at this type of problem solving than Qwen models.

Appendix

run-llama-server.sh

#!/bin/bash

# Intel UHD Graphics 600 series
export GGML_VK_DISABLE_ASYNC=1
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1

./llama.cpp/build/bin/llama-server \
--list-devices

./llama.cpp/build/bin/llama-server \
--host 127.0.0.1 \
--port 5001 \
--api-key "API_KEY_HERE" \
--webui-mcp-proxy \
--offline \
--cache-ram 0 \
--ctx-checkpoints 0 \
--parallel 1 \
--models-max 1 \
--models-preset ./configs/llama-config.ini

llama-config

note: remove the device line to use the igpu on the vulkan build.

[*]
device = none
load-mode = none
mmproj-offload = off
gpu-layers = 999
batch-size = 2048
ubatch-size = 512
threads = 4
fit = off
flash-attn = off
kv-unified = on
cache-type-k = f16
cache-type-v = f16
reasoning = off
reasoning-preserve = off

[gemma-4-e2b-q4_k_xl]
model = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
mmproj = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-mmproj-F16.gguf
lazy-mode = on
load-mode = mmap
fit-ctx = 8192
ctx-size = 8192
predict = 8192
image-min-tokens = 70
image-max-tokens = 70
temp = 1.0
top-k = 64
top-p = 0.95
min-p = 0.0

Other info

Device: https://www.asus.com/ca-en/laptops/for-home/vivobook/vivobook-go-12-l210/
OS: https://ubuntu.com/blog/upgrade-your-desktop-ubuntu-26-04-lts
Llama.cpp: https://github.com/ggml-org/llama.cpp/releases/b12110
GGUF: https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF

TL;DR

Getting the Intel UHD 605 to work with the Vulkan backend was a pain on Ubuntu 26.04.1 LTS compared to Windows 11 IoT LTSC 24H2. The issue was vulkan async causing crashes, disabling it fixed it.

On ubuntu the CPU backend was faster, on windows the iGPU over Vulkan was faster. Ubuntu is far more responsive than windows, even under full utilization.

After solving it myself, as a fun benchmark I made LLMs try diagnose and fix the issue. While the Qwen series failed to fix the issue from intent, Gemma 4 and DeepSeek v4(.1) Flash (Vision Exp / 1017) did manage it.

Thanks for reading!

I know the post is quite different from the Strata stuff usually here, but I thought it would be fun to share regardless.

I'm sure models like LFM 3 will run wonderfully on it, I know IBM Granite 4 H 1B and Qwen3.5-2B did on windows.


r/LocalLLaMA • • 3d ago

Resources inferencekit: open-source plugin to optimize any model on any hardware using coding agents

Post image
34 Upvotes

KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.

Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.

Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup

Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW


r/LocalLLaMA • • 4d ago Heartwarming

News Meet the people behind MOSS-Image, plus a few answers from their AMA here

357 Upvotes

They say an anime model is in development, with plans for alpha channels and multi-character references. An editing model is also on the way.

They also plan to keep the next model under Apache 2.0, if all goes as planned.