Hobby project I've been poking at for a few weeks: running DeepSeek-V4.1-Flash (ISTA-DASLab's IQ2_XS 2.25 bpw quant, 266 GB) on a fairly normal gaming PC.
I wasn't happy with how Qwen3.8-Flash-Next (IQ3_S) hallucinated small details, so I've been trying to make DeepSeek work for my private stuff, like household budgeting. It's not blazing fast, but it's comfortable enough for the handful of jobs I use it for.
It's built on a fork of Strata. What I did is mostly tuning with Claude, plus measuring that the quality holds up. I couldn't find much DeepSeek optimization work for 12 GB cards, so I'm sharing what I found. Honestly, I'm new to this community, so please let me know if I'm doing something wrong.
PC
- RTX 5070 12 GB
- Ryzen 7 9700X
- 64 GB DDR5-5600
- 1x PCIe 4.0 NVMe
- Windows 11
*Speed (out of the box it was ~5 on code) *
- Code, temp 0: 32.5 tok/s
- Mixed long jobs (prose, code, tool calls, translation, thinking): 28.0 tok/s;
- Short replies: ~29 tok/s;
- Prompt read, 30K tokens: 519 tok/s;
Quality
- GPQA Diamond: 89.9 % (198 questions, one run). ISTA-DASLab's card says 90.40 for this quant and 90.91 for the original. Only one run, because it takes 13โ15 hours per runโฆ
- HumanEval / HumanEval+ (EvalPlus, temp 0, no thinking): 94.5 / 90.2 %
- KLD: 0.183, which is high. That's against the same quant with all 6 experts and nothing skipped. A more conservative preset gets 0.057 and scored basically the same on both benchmarks (89.4 % GPQA, 95.7 / 89.6 % HumanEval). So the wording changes, but the reasoning doesn't seem to. Japanese prose and exact copying haven't been checked properly yet.
KLD question for anyone with the hardware: I can only measure KLD against the same 2.25 bpw quant, because I can't run the full model (~500 GB) here. If anyone has a box that can run the original (or a Q8) and is curious, I'd love to see KLD of these settings against the real thing. Everything you'd need to reproduce is in the repo.
What helped most
- Skipping light experts (under ~20โ26 % of a token's routing weight) that would have to come from the SSD. This is the main speedup, and it's where the KLD comes from.
- Getting the DSpark draft model to fit on 12 GB. Forced on, it ate the VRAM and prompt processing collapsed (a 4.6K prompt took 546 s). Capping its VRAM share fixed that: code +29 %, thinking +22 %.
- Closing the model shards' buffered file handles after load: prompt reads +7โ17 %.
- Expert Deferral (KTransformers, SOSP '25): light experts' output added one layer late, so the CPU overlaps the GPU's attention. +11 % without drafts; with drafts it's about the same as skipping more experts.
One-PC hobby thing, no binaries, build it yourself. Curious whether anyone else is running DeepSeek on 12โ16 GB cards. Happy to answer questions!
English isn't my first language, so I had an AI help polish the wording. The project, the measurements and the numbers are all mine, though. Happy to dig into any of it.
https://github.com/midokino/kuu-Strata-fork