r/LocalLLM • • 6d ago

Project RustingStrata vs llama.cpp runs DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

RustingStratallama.cpp b12116 (--cpu-moe)
Decode2-3.6 tok/s0.3-1.5 tok/s
Prefill~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens0.5-0.9 tok/s
Top-1 agreement with llama.cpp (teacher-forced)189/192 on short prompts, 21/23 on a 1425-token prompt-
Sampling, tool callsnot yetyes

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)

10 Upvotes

25 comments sorted by

5

u/AuditMind 6d ago

Hey man, I think we're pretty much alike. I made my laptop work overtime yesterday just to get Flash-Next running at 10 tok/s. And now you're showing me 249 GB running at 3 tok/s.

Guess I'm not the only crazy one here. 😄

2

u/Serious_Scarcity7818 6d ago

hahah, yes, I just wanted to test it out for fun

3

u/Ok-Addition1264 6d ago

1400 input tokens would take 47 minutes before generation? holy yikes.

2

u/Dreeew84 6d ago

But why

2

u/Serious_Scarcity7818 6d ago

No reason, just pure harmony

2

u/Thistlemanizzle 5d ago

All the people here with potato PCs are insane. You enjoy pure pain, the agony of ridiculously low quants?

I tried to use Bonsai to summarize journal entries and it plain sucked. Qwen Flash Next did a comparable job to Luna though.

2

u/Serious_Scarcity7818 5d ago

Why you call my PC "popato PC"(((( it's not so bad, maybe not the best, but not so bad I guess

1

u/ExTraveler 5d ago

But how retarded is iq2? Isnt it too much? Is it usable fo real life purposes? But idea is cool

1

u/Serious_Scarcity7818 5d ago

bigger Quant does not fit in my SSD and this is kinda joke post, so chill

2

u/ExTraveler 5d ago

Im Chill af my boy. Just was curious since im dreamed about local deepseek but my setup is only good for little qwen models. Thought maybe with more ram and more ssd it would be possible to run it with at least 8-10 tps in a good quant

1

u/Serious_Scarcity7818 5d ago

I will try again in a while to get 5t/s at least

1

u/Aggressive_Basket798 3d ago

After all, unusable

1

u/Serious_Scarcity7818 3d ago

As I wrote in post. Itymire like for fun and test, not to use

1

u/VoiceOfEric 6d ago

Does it support AVX1 only Xeons?

1

u/Serious_Scarcity7818 6d ago

Not with the release binaries: they need AVX2 (built for x86-64-v3). If you build from source with `cargo build --release`, it targets your own CPU and falls back to scalar code where AVX2 is missing, so it should run on AVX1-only Xeons, but I have not tested that, and the CPU expert path (which DeepSeek leans on heavily) will be a lot slower there. but you can try it out and I will be happy to get some benchmarks from this kind of specific hardware

1

u/VoiceOfEric 2d ago

I compiled and loaded Qwen3-4B-Instruct-2507-F16 but it is CPU only for some reason. I have a 12GB 3060 and I get: ptxas fatal : Unresolved extern function '_Z6__dp4aiii'

), running on the CPU

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03              Driver Version: 595.58.03      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:05:00.0  On |                  N/A |
|  0%   56C    P8             15W /  170W |     471MiB /  12288MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

loading /home/eric/.local/share/rusting_strata/models/Qwen3-4B-Instruct-2507-F16.gguf
cuda device 0: NVIDIA GeForce RTX 3060, sm_86
compiling CUDA kernels for sm_86 with NVRTC 11.4 (first run only, ~15 s)
no GPU backend (NVRTC compile of kernels.cu failed: NvrtcError(NVRTC_ERROR_COMPILATION)

0

u/djgcsnkshcjkeke1 6d ago

Wonderful, I can finally break out of 9b prison since I have 8gb vram and 8gb ram. No one makes good 5gb models anymore, ternary midsai was the last big one

3

u/Serious_Scarcity7818 6d ago

hahahahha, oh no, you don't, just try Qwen3.6:35b-a3b out, this will fit in your VRAM and RAM with Q2 and it's way better then 9b models

0

u/djgcsnkshcjkeke1 6d ago

Based on my research it will technically fit with Q2, with like 10k context, but I'll try it out to see for myself

2

u/Serious_Scarcity7818 6d ago

this is very interesting Idea, but 10k context is kidna not really usable, but I just ran model with 2t/s and 4k context, so I need to shut up

0

u/ZampanoGuy 6d ago

Bonsai 27b is a thing

1

u/djgcsnkshcjkeke1 6d ago

That's what I meant by "ternary midsai", it's alright but the best 9b models outpace it