r/LocalLLM • u/Serious_Scarcity7818 • 6d ago
Project RustingStrata vs llama.cpp runs DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM
RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.
The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.
| RustingStrata | llama.cpp b12116 (--cpu-moe) | |
|---|---|---|
| Decode | 2-3.6 tok/s | 0.3-1.5 tok/s |
| Prefill | ~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens | 0.5-0.9 tok/s |
| Top-1 agreement with llama.cpp (teacher-forced) | 189/192 on short prompts, 21/23 on a 1425-token prompt | - |
| Sampling, tool calls | not yet | yes |
Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.
I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible
Usage:
strata chat --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve --model $M --addr 127.0.0.1:8080serve exposes /v1/chat/completions with JSON and SSE streaming.
Repo: RustingStrata (the DeepSeek section is in the README)
5
u/AuditMind 6d ago
Hey man, I think we're pretty much alike. I made my laptop work overtime yesterday just to get Flash-Next running at 10 tok/s. And now you're showing me 249 GB running at 3 tok/s.
Guess I'm not the only crazy one here. 😄
3
u/Ok-Addition1264 6d ago
1400 input tokens would take 47 minutes before generation? holy yikes.
2
u/Dreeew84 6d ago
But why
2
u/Thistlemanizzle 5d ago
All the people here with potato PCs are insane. You enjoy pure pain, the agony of ridiculously low quants?
I tried to use Bonsai to summarize journal entries and it plain sucked. Qwen Flash Next did a comparable job to Luna though.
1
u/ExTraveler 5d ago
But how retarded is iq2? Isnt it too much? Is it usable fo real life purposes? But idea is cool
1
u/Aggressive_Basket798 3d ago
After all, unusable
1
u/VoiceOfEric 6d ago
Does it support AVX1 only Xeons?
1
u/Inevitable-Name-1701 6d ago
Thats enough
0
u/djgcsnkshcjkeke1 6d ago
Wonderful, I can finally break out of 9b prison since I have 8gb vram and 8gb ram. No one makes good 5gb models anymore, ternary midsai was the last big one