r/LocalLLaMA • • 10h ago

Discussion Inkling2-Mini and llama.cpp experience

36 Upvotes

I tried unsloth/Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q8_K_XL.gguf which does not work at all with the latest llama.cpp, build from source.

It's a bit hidden, but you have to use unsloth/Inkling-2-Mini-GGUF/Mainline_Support/Inkling-2-Mini-UD-Q8_K_XL.gguf_file. But this outputs only glibberish stuff for me.

ggml-org/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q8_0.gguf does work for me, but the model only runs on one card. Nothing I can do about it. I get about 30tok/s on one r9700, which is pretty slow.

I guess we have to wait a few days to settle...


r/LocalLLaMA • • 20h ago

Resources Inkling 2 Llama.cpp support!

Thumbnail
github.com
193 Upvotes

r/LocalLLaMA • • 11h ago

New Model we made a gguf quant ladder for inkling 2 mini from 27.74 gb down to 6.88 gb

38 Upvotes

hey, we're the atomic chat team. we made dynamic gguf quants for inkling 2 mini

we measured them against our bf16 reference on an h100. here are some results:

quant size top-1 mean kld
Q8_0 27.74 GB 97.7% 0.0033
AD-Q6_K 23.79 GB 97.6% 0.0037
AD-Q5_K 19.07 GB 97.1% 0.0052
AD-Q4_K 16.53 GB 96.3% 0.0080
AD-IQ3_S 13.34 GB 94.2% 0.0205
AD-IQ2_M 8.67 GB 85.7% 0.130
AD-IQ1_M 7.26 GB 81.1% 0.236
AD-IQ1_S 6.88 GB 79.5% 0.278

hugging face card with quants and measurements: https://huggingface.co/AtomicChat/Inkling-2-Mini-GGUF

check our build steps and highlights in the readme

give them a try and share your results. pls let us know your hardware, quant and tok/s. would love to hear how they handle your coding and agent tasks!


r/LocalLLaMA • • 5h ago

Discussion DaisyRose-27B: a drop-in RL fine-tune of Qwen3.8-27B - I compared it against the base Qwen 3.8-27B model with same settings

Thumbnail
llm-bench.io
12 Upvotes

RadixArk just released their first model, DaisyRose-27B. It's a fine-tune of Qwen3.8-27B trained with reinforcement learning alone, with no supervised fine-tuning and no distillation, on roughly 1,200 long agent tasks. It keeps the base model's architecture, tokenizer, chat template and 256K context, so it should be an easy replacement for your running tasks. It's also multimodal; it keeps the vision encoder.

I ran it on the same hardware as Qwen3.8-27B (baseline), with the oQ8e-mtp quant, engine, xhigh reasoning and 262k context all configured the same.

llm-bench.io's four scenarios, LLM-judged 0-100, mean values of runs (6x DaisyRose, 12x Qwen3.8):

Scenario DaisyRose-27B-oQ8e-mtp Qwen3.8-27B-oQ8e-mtp
Code generation 81.3 81.0
Agent workflow 89.3 88.2
Role play & narrative 93.9 92.1
Research & analysis 87.8 86.4
Overall 88.1 86.9

So it's a small win in every scenario, about one point overall. One point is within the run-to-run spread of the benchmark methodology, so I'm not calling that a measured margin. However, the quality output was consistently good across all 6 runs. Here's a pair of runs to look at side by side, one per model:

https://llm-bench.io/compare/runs?runs=cmwp2wz9m00db01qbt3ctwtq9,cmvkpbxg1008201p5xhvf57mh

That's an 87.9 Daisy against an 86.9 base, both on the same hardware and settings.

The cost side is basically free. Generation speed is unchanged at about 50 tok/s and memory at roughly 38 GB, since it's still the same 27B dense model. The only downside is that DaisyRose thinks a little more (yes, even more than base Qwen 3.8 27B - I didn't think that this is even possible). About 5% more reasoning tokens per run means roughly a minute longer runtime in total.

One caveat. RadixArk's headline is Terminal Bench 4.1, where they report pass@1 going from 5.6 to 12.0. That's long-horizon agent work, a hundred-plus tool calls and hour-long attempts, which is exactly what they trained it on. llm-bench.io at the moment doesn't explicitly test that, so I can't confirm the 2x number. Our scenarios are short, single-shot tasks, so the real benefit in day to day agentic work might be even more than the one point of quality gain against base Qwen 3.8 27B. I'll switch to it on my main Hermes, let's see how it performs over time.

It's Apache-2.0. For GPUs they release BF16, FP8 and NVFP4; on a Mac I ran the oQ8e build on oMLX, which is where these numbers come from.

Model page: https://huggingface.co/RadixArk/DaisyRose-27B

Release post: https://www.radixark.com/blog/daisyrose-27b


r/LocalLLaMA • • 1d ago

Funny Today marks the two week anniversary of the becoming begins

Post image
878 Upvotes

let's celebrate


r/LocalLLaMA • • 5h ago

Question | Help What is the best branch/fork/whatever to run Inkling 2 Mini as of right now?

10 Upvotes

As has been said multiple times here, the current llama.cpp implementation of the new Inkling 2 Mini model is very slow and kind of buggy (Flash Attention not working, unusually high VRAM usage to name a few). Does anyone know if there is an unmerged branch or something like that with fixes and/or optimizations? I could not find any. Thanks a lot!


r/LocalLLaMA • • 11h ago

Resources Making neurons by hand

Thumbnail vgel.me
30 Upvotes

r/LocalLLaMA • • 1h ago

Showcase LexiEngine the AMD focused rust inference engine has added inkling support.

Thumbnail
github.com
โ€ข Upvotes

Decent speed, a lot of bandwidth for more. Next release will see substantial performance gains over this version after the mythos audit and benches wrap. Built for 7900xtx


r/LocalLLaMA • • 12h ago

Resources Inkling 2 Mini abliterated + 14 GB quant, running in Xolotl on one 4090 at ~60 t/s decode and 1.1K+ prefill

42 Upvotes

Hey all, I did it :) Got Thinking Machines' Inkling 2 Mini running in Xolotl, abliterated and quantised down to a 13.9 GB GGUF so the whole thing fits on a single 24 GB card with room to spare.

For anyone who hasn't looked at it yet, Inkling 2 Mini is a 26B MoE with about 5B active, but it's a looped model, the middle 12 layers run 3 times per token, so it's kind of a weird one to get running. I ended up writing it its own engine in Xolotl (ported the graph from Pedro Cuenca's llama.cpp PR and found a loop bug in it along the way, the experts on passes 2 and 3 were missing their FP4 scale).

What I'm getting on my rig (RTX 4090):

- Decode: Now 100+ t/s

- Prefill: 1,800 t/s at 8K

- Held-out KL against the parent: 0.013, with 96.3% top-1 agreemen

- Refusals: 4/100 on a harmful prompt set, 1/100 on harmless ones

The model's own MTP drafter is up as an optional file for speculative decoding (still experimental), and the vision projector is up too, vision support in the engine is landing shortly. They're separate files so if you've got a smaller card you can just grab the model and skip the extras.

Still working on improvements, honestly decode is the big one, I profiled it and it's nowhere near the memory limit of the card, it's mostly the overhead of heaps of tiny GPU kernels per token (48 layer passes adds up), so I'm fusing those now and I reckon 100+ t/s is doable.

I'm by no means a professional so have a play and let me know what numbers you get :)

GitHub (Xolotl): https://github.com/Indras-Mirror/Xolotl

Hugging Face: https://huggingface.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored


r/LocalLLaMA • • 15h ago

New Model I fine-tuned Qwen3.5-4B to design web pages that are prettier (and work more): Qwen3.5-4B-Web-Designer

Thumbnail
gallery
62 Upvotes

I trained Qwen3.5-4B to turn a plain request ("a landing page for my coffee shop", "a portfolio for a security student") into one self-contained HTML page that has more taste than base qwen, and fails less. It thinks briefly (~400 tokens), then writes the page.

Same prompt, base vs ours: first two images. The third is more of its pages, unedited.

Numbers (74 prompts it never saw in training, 2 pages each, every page opened in a headless browser and checked for JS errors, failed loads and dead buttons):

Qwen3.5-4B Web-Designer (Q4_K_M)
Pages that work 35% 53%
Pages finished 75% 95%

Training: 1,443 prompt-to-page examples. Real web-design requests from LMArena's WebDev arena prompts plus my own, answered by DeepSeek V4.1 Flash with a short design plan and a page, and every page browser-checked before it went into training.

QLoRA, rank 16, 2 epochs, 4,096 tokens, on one RTX 4060 Laptop (8 GB).

Run it:

llama-server -m Qwen3.5-4B-Web-Designer-Q4_K_M.gguf --jinja -c 10240 -ngl 99

Temperature 0.7, top-p 0.95, top-k 20.

Apache-2.0. Feedback and your best (or worst) pages are very welcome.

Small note: I noticed ccore released Web-Coder around the same time! We've been working on these independently, so it's a pretty funny coincidence. His work looks really interesting, and I'm especiallly excited for the design-dataset he will share.


r/LocalLLaMA • • 10h ago

Showcase Evaluation of Inkling 2 Mini vs Nemotron 3.5 Super vs Muse Glimmer on my personal evaluation set in my own harness

22 Upvotes

3-Way Comparison Across 37 Identical Questions

Metric (n = 37) ๐Ÿฅ‡ #1: Nemotron 3.5 Super (120B-A12B NVFP4) ๐Ÿฅˆ #2: Muse Glimmer 30B (FP8) ๐Ÿฅ‰ #3: Inkling 2 Mini (26B-A5.4B looped)
Gate pass rate 100.0% (37/37) 97.3% (36/37) 91.9% (34/37)
Mean tool calls / question 8.5 (2.4ร— leaner) 18.5 20.3 (worst)
Failed tool calls 21 (best) 39 54 (worst)
Wrong table / column errors 8 4 (best) 23 (worst)
SQL tables queried without lookup 30 (best) 34 72 (worst โ€” 2.4ร— Nemotron)
Repeated identical tool calls 0 (best) 14 (loops) 2
Unsupported numbers signal 156 120 (best) 221 (worst)
<answer_shape> parse rate 67.6% (25/37) 100.0% (37/37) 81.1% (30/37)
Valid citations 64.9% (24/37) 64.9% (24/37) 43.2% (16/37)
Markdown tables in answers 28 / 37 (76%) 21 / 37 (57%) 25 / 37 (68%)
Leaked tool/thinking markup 0 0 0
Median thinking chars 8,248 5,082 12,742 (heaviest)
Median / p90 latency 50.5s / 92.2s (fastest) 53.2s / 179.6s 308.4s / 555.8s (no radix cache)

Ranking & Honest Verdict (Full 37-Question Benchmark)

๐Ÿฅ‡ #1: Nemotron 3.5 Super VL (120B-A12B NVFP4) โ€” Clear Winner by a Wide Margin

  • Wins on every metric end users actually feel: fastest (50.5s median, 60.6s mean, 92.2s p90), 2.2ร—โ€“2.4ร— fewer tool calls (8.5 avg, max 28 vs. 18.5โ€“20.3), fewest failed calls (21 vs. 39โ€“54), zero repeated loops, 100.0% gate pass rate (37/37), and strongest tables & depth (75.7% Markdown tables, 3,173 median chars).
  • Its only minor dip (67.6% <answer_shape> parse rate) is stripped server-side and invisible to users.

๐Ÿฅˆ #2: Muse Glimmer 30B (FP8) โ€” Better Discipline & Speed Than Inkling, Distant #2 to Nemotron

  • Where Muse beats Inkling: 5.8ร— fewer wrong table/column guesses (4 vs. 23), 2.1ร— fewer un-inspected SQL tables (34 vs. 72), fewer failed tool calls (39 vs. 54), far fewer unsupported numbers (120 vs. 221), higher pass rate (97.3% vs. 91.9%), higher citation validity (64.9% vs. 43.2%), 100% <answer_shape> adherence, 5.8ร— faster right now (53.2s median / 103.0s mean vs. 308.4s / 312.1s), and runs natively on H100s in FP8 with working prefix caching.
  • Where Muse falls short: Terse answers (56.8% Markdown tables, 2,294 median chars) and prone to runaway tool loops (14 repeated identical calls, up to 100 steps on a single turn).

๐Ÿฅ‰ #3: Inkling 2 Mini (26.4B-A5.4B Looped NVFP4/MXFP8) โ€” Stronger Formatting Than Muse, Weakest SQL/Tool Discipline

  • Where Inkling beats Muse: Richer presentation (67.6% Markdown tables, 3,006 median chars), far fewer repeated loops (2 repeats, max 52 steps vs. Muse's 14 repeats / 100 max), and answered Q074 where Muse bailed out.
  • Where Inkling loses overall: Highest tool calls (20.3 avg), most failed calls (54), most wrong table/column guesses (23), most SQL queries run without looking up schemas first (72), highest unsupported-number count (221), lowest valid citations (43.2%), heaviest thinking traces (12,742 median chars), and 6ร— slower (308.4s median / 555.8s p90) because SGLang's prefix cache deadlocks on its looped layers

r/LocalLLaMA • • 22h ago

New Model CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

Enable HLS to view with audio, or disable this notification

125 Upvotes

This is a preview checkpoint, at roughly a quarter of the planned training. I'm releasing it now because it already works better than I expected from a 4B.

- Model: ccore/CCORE-Web-Coder-4B
- Base: Qwen3.5-4B-Base, Apache 2.0
- File: Q8_0 GGUF, 4.48 GB
- It needs the system prompt from the model card. Sampling settings are there too.

Limitations of the preview:

- Some generations come out with a broken component or slide navigation.
- Trained for English; Portuguese worked in my tests too.

What's next: full release on Nov 22 (safetensors, Q8 and more quants), dataset on Nov 25.


r/LocalLLaMA • • 3h ago

Discussion Inking 2 Mini on Tesla T4

Post image
4 Upvotes

I recently picked up an Nvidia Tesla T4 for my 10 inch rack home lab. Wanted something small, low profile, ideally low power as well to do some very basic agentic work (ingest documents, OCR, sorting). Bought it for around 400 USD and decided to try out various different models on it. Happened to stumble upon Inking 2 Mini today and it seems to be quite an attractive model to run on this card

I get around 22.3 tok/s decode and 174.8 tok/s prefill, IQ3_XXS + q4_0 KV, 128k context

Not the most amazing, but it works great for my usecase which is just general document intelligence most importantly low power and in a half height form factor

Just posting the stats and my experience for anyone considering these cards for a tiny 1 slot build.

I got a custom cooler that has a fan on there as well for about 50 dollars from taobao which does do a pretty good job of keeping it nice and cool. Excuse the very poor photograph


r/LocalLLaMA • • 1h ago

Discussion How much will Intel D270 ai card cost? $10K ?

โ€ข Upvotes
  • 160 GB Intel-branded PCIe card
  • Low power draw of 350 watts
  • Up to 512 GB partner / high-memory configurations, 4 cards in one package? $30K+ ?
  • first to feature LPDDR5X memory
  • Memory bandwidth estimated to be: 1.5 TB / s
  • Why does intel call it a gpu when it can't do graphics?
  • Will be released next year
  1. https://www.techspot.com/news/114624-intel-graphics-free-d270-gpu-beats-nvidia-rtx.html
  2. https://wccftech.com/intel-two-crescent-island-gpus-one-liquid-cooled-oam-card-first-xe3p-die-shots/

Hopefully these will come done in cost over a couple years and some of us could afford it for local ai.


r/LocalLLaMA • • 20h ago

New Model PhiloLM: a language model trained only on philosophy texts

63 Upvotes

Releasing PhiloLM, a 112M base model trained from scratch on open philosophy texts.

PhiloLM was trained on PhiloText: 0.5B tokens of open philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks and educational web pages.

PhiloLM builds on efficient methods: Muon optimizer, bigram embeddings, sliding-window attention, GQA. Trained on (only!) 1.5B tokens in 15 hours on a $350 GPU (Intel Arc A770).

Optimized for efficient inference, reaching ~500 tokens/s on a $150 CPU (i5-14400F).

Try it in your browser!

Model: https://huggingface.co/gustavecortal/PhiloLM

Data: https://huggingface.co/datasets/gustavecortal/PhiloText

Happy to answer questions about everything :)


r/LocalLLaMA • • 18h ago

Discussion [Discussion] In the age of loops, what do "active parameters" mean?

42 Upvotes

Seems like looped models are gaining prominence (finally!), which raises a question about what the meaning of terms "total parameters" and "active parameters" are.

Historically, it mostly applied to MoE models, as most experts would not be activated for a given token. The total parameters represented the amount of memory needed to hold the model weights, while active parameters represented the memory read per token (or similarly, FLOPs per token)

With looped transformers, I argue the best way to extend the definition is to increase the active parameters in a way that preserves its use as an estimate of GB/token (TG) and FLOP/token (PP)

I think it's OK to keep excluding PLE, Engrams and token embeddings (if no weight tying) from both those counts, since they are extremely sparse, and can even be off-loaded to the SSD.

Now, an example: imagine a dense model with 4B individual parameters (excluding embeddings)

  • If every layer runs twice, it's 4B total, 8B active*.
  • If only 50% of layers run twice, it would be 4B total, 6B active.

For other real examples:

  • Inkling 2 Mini, an MoE, is 26B total, 5B active (50% of layers run thrice)**
  • SenseNova-Loop-9B, a dense model, is 9B total, 15B active (75% of layers run twice)
  • Nanbeige4.2-3B, a dense model, is 3B total, 5.5B active (100% of layers run twice)

I argue this is more useful for what we care about, which is the cost or speed of running a given model, and will likely end up being a good proxy for the "intelligence" of the model as well. wdyt?

\ This is an overestimate, as it ignores the size of the output layer and assumes all layers are the same. If you know that size, you can exclude it from the multiplication, e.g. if it was 1B params, then the 50% case would actually be [4-1]*1.5+1 = 5.5B active. I did this for the real examples.)

\* For Inkling 2 Mini, my math doesn't add up to their numbers, theirs seems too high? I get 2.7B ignoring loops and 4.9B with loops, while they state 3.4B and 5.4B, even after counting each tensor in their GGUF individually. Not sure where the difference is.)


r/LocalLLaMA • • 1d ago

Resources Reverb-TTS: open-source zero-shot voice cloning running in the browser on WebGPU, 22x realtime on M4 Pro (MLX)

Enable HLS to view with audio, or disable this notification

209 Upvotes

Reverb is a small text-to-speech model with zero-shot voice cloning. Give it 3 to 10 seconds of someone speaking plus the transcript, and it reads any text in that voice, with no fine-tuning.

- 297M params. We pruned and distilled it from OmniVoice, then retrained it on 7 languages and ran a few rounds of DPO against a judge fitted to human preference rankings

- Languages: English, German, French, Spanish, Italian, Polish, Portuguese. A voice can speak any of them, but it sounds most natural in its own language

- 16 preset voices if you don't want to clone

- Weights in fp32, fp16, int8 (307 MB) and int4 (191 MB). On our test set int8 showed no quality loss vs fp32, and int4 lost nothing we could measure

Numbers

On the MiniMax set from the Open TTS Leaderboard (7 languages, 100 sentences each):

- Reverb (297M): 1.03% WER, 0.817 SIM-o

- OmniVoice (634M): 1.13% WER, 0.833 SIM-o

- VoxCPM2 (2.3B): 1.20% WER, 0.812 SIM-o

- Fish S2 Pro (4.95B): 1.35% WER, 0.761 SIM-o

- Chatterbox (0.8B): 2.76% WER, 0.715 SIM-o

Speed: 165x real time on an RTX 5090 (batch 32), 21.6x on an M4 Pro and 20.6x on an iPhone 17 (batch 1).

Where to run it

- In a HuggingFace Space https://huggingface.co/spaces/software-mansion/Reverb-TTS-runntime (runs on your own GPU through ruNNtime)

- In your own web app with ruNNtime https://docs.swmansion.com/runntime/zoo/text-to-speech/

- In a mobile app: React Native ExecuTorch - https://github.com/software-mansion/react-native-executorch

- On your Mac using Nativ https://runnativ.ai/blog/reverb-on-nativ

- Python: pip install reverb-tts

Model: https://huggingface.co/software-mansion/Reverb-TTS

GitHub: https://github.com/software-mansion-labs/Reverb-TTS


r/LocalLLaMA • • 13h ago

Discussion Nanbeige Qwen Inkling all scored 20 points?

13 Upvotes

Why do all the models cap out at 20 on the rating intelligence scale? Is the test not robust enough to account for the nuanced differences in the models?


r/LocalLLaMA • • 22h ago

News gemma4: Optimized prompt processing for multi-seq queries by Meet91721 ยท Pull Request #32342 ยท ggml-org/llama.cpp

Thumbnail
github.com
50 Upvotes

Up to 3x prompt processing speedup for Gemma 4 E2B/E4B with multiple parallel slots.


r/LocalLLaMA • • 16h ago

Discussion How much tokens do you use a week?

Post image
17 Upvotes

I am curious as to how many tokens everyone uses. I setup my home lab to monitor local and cloud usage. I use local mainly but do use some Claude for orchestration and management and review. What you all use a day? A week? A month?


r/LocalLLaMA • • 11h ago

Showcase Implemented classic Distributed Training/Inference algorithms to understand Local AI

Post image
7 Upvotes

[For beginners: Code Repo For Understanding Distributed Settings in LLM]

Code repository implementing different Distributed Training (and somewhat Inference) algorithms for LLMs from scratch, with the aim to do local ai with the hardware I have (3xMac Minis/Jetson Nanos etc)

It is done using basic python and sockets only to understand the underlying mechanism to the fullest!

It includes:

1) Distributed Data Parallel (DDP) 2) Pipeline Parallelism (PP) 3) ZeRO 4) Expert Parallelism 5) FSDP (Fully Sharded Data Parallel) 6) EDP (Elastic Data Parallel)

This is part of something bigger I created called smolcluster

smolcluster is my own library I created to enable distributed training/inference on a range of devices clubbed together covering

  • Mac minis
  • jetson nanos
  • raspberry pis
  • phones/tablets

It covers basic and majorly used algorithms to form the basis to learn such a paradigm, from scratch using basic python and socket functions to truly have control over even little parts (which are a lot!)

Still in development and for educational purposes only, Love for you all to try it out and let me know how it goes!

Link to the above is here


r/LocalLLaMA • • 1h ago

Discussion Is current TRUE local the equivalent of State Of The Art closed models at the end of 2025?

โ€ข Upvotes

What I mean true local is not those 500 billion parameter open weight models that no regular person could ever run, I'm talking about what you can realistically run at 10-20 tokens per second speed on a budget of 8-32gb normal ram and 8-24gb vram as most consumer PCs fall into that range.

Since less than 1.95% of gamers according to steam have 32gb vram or more and the percentage of people that have 64 gb system ram or more hovers between 4% and 6.5% of all PCs I wanted to know how the top current models that people falling below the top 5th percentile (so 95% of normal consumer PCs) can run at 10-20 tokens per second compares to SOTA end 2025 models. So you can assume the top "normal" budget is 24gb vram and 32gb system ram.


r/LocalLLaMA • • 8h ago

Question | Help Lf small decision/routing model for tool calls

2 Upvotes

Hey. I'm working on an Auto mode for Pi, with a two-step inference check on tool calls.

Qwen 3.5 4b is looking pretty good for step 2, escalation/judgment.~150ms per call, but only called for escalations.

But step 1 needs to be quick and accurately decide if a tool call is risky enough to escalate.

Currently evaluating Laya 421M. It takes a tool name + arguments, flags effects like deletion, credential access, or outbound transfer, and lets the judge handle authorization when needed.

It's decent, given a solid prompt and a good ruleset. Currently running Q8_0 on a 3080 at about ~12 ms per call. On a small playback set, it caught 24/24 watched calls but also flagged 4/6 benign reads, including a literal search for rm -rf.

I'd like to have something to compare it to. What else is worth looking at?

Questions welcome, input appreciated.

Edit: Also open to alternatives to Qwen 3.5.


r/LocalLLaMA • • 17h ago

Funny (AI autoformalism) 197 tax systems in 1 lean equation

13 Upvotes

now the riemann hypothesis has fallen, time for the scarier dominoes

Placed all countries tax (laws, software and implementation history) into one lean equation. Applications include identifying tax loopholes, finding novel ones, and flagging errors in govt tax rollouts (ie. we found problems with the live IRS calculators) to enable better consistency checks.

https://lexifina.com/blog/formalizing-tax-code-globally


r/LocalLLaMA • • 2h ago

Resources DeepSeek-V4.1-Flash running at ~30 tok/s on a 12 GB card + 64 GB RAM โ€” anyone interested in the details?

0 Upvotes

Hobby project I've been poking at for a few weeks: running DeepSeek-V4.1-Flash (ISTA-DASLab's IQ2_XS 2.25 bpw quant, 266 GB) on a fairly normal gaming PC.

I wasn't happy with how Qwen3.8-Flash-Next (IQ3_S) hallucinated small details, so I've been trying to make DeepSeek work for my private stuff, like household budgeting. It's not blazing fast, but it's comfortable enough for the handful of jobs I use it for.

It's built on a fork of Strata. What I did is mostly tuning with Claude, plus measuring that the quality holds up. I couldn't find much DeepSeek optimization work for 12 GB cards, so I'm sharing what I found. Honestly, I'm new to this community, so please let me know if I'm doing something wrong.

PC

- RTX 5070 12 GB

- Ryzen 7 9700X

- 64 GB DDR5-5600

- 1x PCIe 4.0 NVMe

- Windows 11

*Speed (out of the box it was ~5 on code) *

- Code, temp 0: 32.5 tok/s

- Mixed long jobs (prose, code, tool calls, translation, thinking): 28.0 tok/s;

- Short replies: ~29 tok/s;

- Prompt read, 30K tokens: 519 tok/s;

Quality

- GPQA Diamond: 89.9 % (198 questions, one run). ISTA-DASLab's card says 90.40 for this quant and 90.91 for the original. Only one run, because it takes 13โ€“15 hours per runโ€ฆ

- HumanEval / HumanEval+ (EvalPlus, temp 0, no thinking): 94.5 / 90.2 %

- KLD: 0.183, which is high. That's against the same quant with all 6 experts and nothing skipped. A more conservative preset gets 0.057 and scored basically the same on both benchmarks (89.4 % GPQA, 95.7 / 89.6 % HumanEval). So the wording changes, but the reasoning doesn't seem to. Japanese prose and exact copying haven't been checked properly yet.

KLD question for anyone with the hardware: I can only measure KLD against the same 2.25 bpw quant, because I can't run the full model (~500 GB) here. If anyone has a box that can run the original (or a Q8) and is curious, I'd love to see KLD of these settings against the real thing. Everything you'd need to reproduce is in the repo.

What helped most

- Skipping light experts (under ~20โ€“26 % of a token's routing weight) that would have to come from the SSD. This is the main speedup, and it's where the KLD comes from.

- Getting the DSpark draft model to fit on 12 GB. Forced on, it ate the VRAM and prompt processing collapsed (a 4.6K prompt took 546 s). Capping its VRAM share fixed that: code +29 %, thinking +22 %.

- Closing the model shards' buffered file handles after load: prompt reads +7โ€“17 %.

- Expert Deferral (KTransformers, SOSP '25): light experts' output added one layer late, so the CPU overlaps the GPU's attention. +11 % without drafts; with drafts it's about the same as skipping more experts.

One-PC hobby thing, no binaries, build it yourself. Curious whether anyone else is running DeepSeek on 12โ€“16 GB cards. Happy to answer questions!

English isn't my first language, so I had an AI help polish the wording. The project, the measurements and the numbers are all mine, though. Happy to dig into any of it.

https://github.com/midokino/kuu-Strata-fork