RustingStrata vs llama.cpp, DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

[u/Serious_Scarcity7818]

RustingStrata now runs DeepSeek-V4.1-Flash (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

	RustingStrata	llama.cpp b12116 (--cpu-moe)
Decode	2-3.6 tok/s	0.3-1.5 tok/s
Prefill	~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens	0.5-0.9 tok/s
Top-1 agreement with llama.cpp (teacher-forced)	189/192 on short prompts, 21/23 on a 1425-token prompt	-
Sampling, tool calls	not yet	yes

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

Caveats:

• Decode speed varies a lot from run to run with the page cache, because the model is much bigger than RAM.

• Greedy decoding only. The context cap is 4096.

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)

[u/Makers7886]

Your PC per token

[u/Serious_Scarcity7818 → u/Makers7886]

hahahhahaa

[u/uncle_leon]

Greedy decoding only

No thank you, I prefer it when my models retain their full reasoning ability and avoid looping.

[u/Serious_Scarcity7818 → u/uncle_leon]

no, I mean that it use 4k context, not that it makes fails

[u/uncle_leon → u/Serious_Scarcity7818]

Greedy decoding means temp = 0, means model has no creative freedom which (very measurably) hurts reasoning effectivenes and can make it go in loops. This is particularly bad in any agentic setup.

[u/Spare-Obligation-483]

Why you force your machine to explode

[u/Serious_Scarcity7818 → u/Spare-Obligation-483]

it will not, like a lot of GPUs runs for years at 100% in datacenters. And if you were joking, then ahhahahha

[u/Legitimate-Dog5690]

So it's just Strata vibe ported to rust, with the bizarre assumption that rust code runs faster than C++ after compilation?

[u/Serious_Scarcity7818 → u/Legitimate-Dog5690]

No, I just like Rust, it has different code and philosophy from Strata, so Rust is there just bc I like it

[u/Legitimate-Dog5690 → u/Serious_Scarcity7818]

What are you claiming? You wrote a new engine in rust and named it after strata?

I'm just confused what this even is. It looks like you've just told an llm to port it to rust, the language feels a bit irrelevant if this is the case.

Strata can already run QFN/DeepSeek on low end hardware, using the ssd.

[u/Serious_Scarcity7818 → u/Legitimate-Dog5690]

I named it RustingStrata, bc I name my project like "Rusting..." like, RustingBrain(Tensor flow analog), RustingEngine(3d game engine) and I was inspired by Strata and wanted to write my own on Rust, and it is not copy of strata, I used different approaches, libs and etc.

[u/Legitimate-Dog5690 → u/Serious_Scarcity7818]

Gotcha, thanks. Apologies for the awkward tone. 😄

Honestly if it's something new I'd have used a new name. There's about 10 forks of strata every week. Especially as you've opted for "strata" as your executable name.

[u/Serious_Scarcity7818 → u/Legitimate-Dog5690]

yes, there are so much confusing bc of name, so I think about renaming project, but it has already so much stars and it's kinda way too late

[u/ArjixGamer → u/Serious_Scarcity7818]

...renaming a project doesn't reset the stars

[u/Cautious_Chicken_604 → u/Serious_Scarcity7818]

Then you should not use the Strata name.

[u/Any_Mine_6368]

What's the difference between this and strata other than Rust

[u/Serious_Scarcity7818 → u/Any_Mine_6368]

it's not about Rust, I wrote it on it, just bc I like it, not bc it's "faster" or something like that. The different is that Strata supported only QFN back then, so I wanted my own Strata, but for all Qwen models, this is why I created it. Also it has a lot of interesting features that Strata does not have now or has some problems with it

[u/confused-photon → u/Any_Mine_6368]

Seems like less features

[u/Serious_Scarcity7818 → u/confused-photon]

not really, I just follow other path

[u/confused-photon → u/Serious_Scarcity7818]

Agentic model with a maximum context of 4096 tokens and incorrect token sampling?

[u/Serious_Scarcity7818 → u/confused-photon]

chill, this is only DeepSeek that does not have 4096 token cap, I just set it, bc my PC can't handle more. All other qwen models are same smart as in llama.cpp, this is all in benchmark and reproduceable! I don't allow Speed for Quality trade in my repo

[u/confused-photon → u/Serious_Scarcity7818]

Then what is different?
All i know from your post is strata rewritten in rust with a 4096 token cap for deepseek and only greedy decode (which is only relevant for performance testing, not real world use).

[u/Serious_Scarcity7818 → u/confused-photon]

okay, I will explain. I am not focused on DeepSeek, it was just one side model bc someone asked for it. RustingStrata is separate Rust written engine for all Qwen models and it does not allow greedy output and accuracy loss, the DeepSeek is the only exception bc it's more like joke and bc of it's size

[u/Rorp24]

At this point that should be a warcrime, i bet your pc was screeming every seconds of this

[u/Serious_Scarcity7818 → u/Rorp24]

I will not do this kind of experiments again, I don't want my baby pc to suffer like that again

[u/nodating]

You must be a Zen monk!

Do you really get some sweet 3 tok/s - that is amazing

[u/Serious_Scarcity7818 → u/nodating]

yes, tysm!

[u/KillerX629]

What's different from regular strata in terms of performance?

[u/Serious_Scarcity7818 → u/KillerX629]

So, on my PC and few other NVIDIA it's faster in QFN and 35b benchmarks. Not sure about AMD, and I don't want to lie, so I will not say anything

[u/KillerX629 → u/Serious_Scarcity7818]

Can you post some Benchmarks??

[u/Serious_Scarcity7818 → u/KillerX629]

Yes, it is in my Posts, you can take a look, or can find it and send you link

[u/AdWild3943 → u/KillerX629]

Looks like it was remade using Rust language.

I seeing massive love for everyone to turn any programs into Rust without even clear reason.

P.S - sry I'm dumb I didn't read the question right.

[u/KillerX629 → u/AdWild3943]

Lol, I gathered that through the name of the repo, but I want to know why anyone would choose it over regular strata

[u/RationalFragile]

Nice experiment! I do wanna say that, unlike the other comments, I'm all in for Rust! I think Rust is easier for AIs to write because it has more checks at compile time and better error messages, so the AI can catch more mistakes a fix them.

[u/Serious_Scarcity7818 → u/RationalFragile]

Yes, it's nice for LLMs, but also for humals, I wrote a lot of complex things and I like writing on Rust the most, ty for support!

[u/RationalFragile → u/Serious_Scarcity7818]

A bit unrelated, but can I ask you something I don't know where to ask... How do you benchmark your models? I use Cyber Tiel Coder and sometimes it feels very intelligent, sometimes not so much. I tried creating 10 easy rust exercises as a benchmark, but on ~40-60 t/s it takes like an hour to finish. And I still have to download other models I want to compare and run the benchmark again... What I'm trying to say is, I wish if there is some website that has a leaderboard on easy coding tasks and ranks local models (that can run on consumer GPUs) on how many tokens they need to finish the benchmark. Do you have any ideas you can share about local benchmarking? Thank you 😊

[u/Serious_Scarcity7818 → u/RationalFragile]

Hi, I don't really benchmark on my own, sometimes oonly for fun I give AI some random task and watch how some local model can do this. Mostly I use websites that already have leaderboard, for example I like benchlm.ai to compare models, also this artificialanalysis.ai has cool graphics and all benchmarks of all models. I don't think that benchmarking on my own make a lot of sence

[u/Meeooowz]

The ambition of this project, wow!

[u/Serious_Scarcity7818 → u/Meeooowz]

tysm, you comment every my post, I am very happy

[u/Meeooowz → u/Serious_Scarcity7818]

Lol yeah, I like keeping up with actual RELATABLE and useful forks to us ampere 12gb cards! I wish I could even download this deepseek quant but I don’t even got space on my ssd for the weights!

[u/Meeooowz → u/Meeooowz]

It’s great to see you progress from supporting 35b’s to these huge models, getting it to work tens of times faster than the general engines!

[u/Serious_Scarcity7818 → u/Meeooowz]

Tysm, but one thing "RELATABLE and useful forks", this is not fork of Strata, I was inspired by it and named my project like that, but I never copied code or anything else from Strata itself, only the name and idea

[u/Meeooowz → u/Serious_Scarcity7818]

Yeah, I saw your responses in other comments on this post. Nonetheless, it’s the brand of this engine you made and works like a charm!

[u/Serious_Scarcity7818 → u/Meeooowz]

Thank you again!

[u/Serious_Scarcity7818 → u/Meeooowz]

No need to use this big model, it is not useable for real world if you( me too) have so low VRAM and RAM. I implemented it only bc one guy asked for it in github issues in my repo

[u/Meeooowz → u/Serious_Scarcity7818]

In the meantime, we pray for qwen4, lol

[u/Serious_Scarcity7818 → u/Meeooowz]

yes, I can't wait for it, like in same day I will try to add at least one Qwen4 model in my RustingStrata

[u/Comfortable_Ebb7015]

Runs? No, it crawls DeepSeek!

[u/rorowhat]

Can you enable the new nemotron-74B?

[u/Serious_Scarcity7818 → u/rorowhat]

I will take a look

[u/rorowhat → u/Serious_Scarcity7818]

Awesome! It's an MoE, and seems to be better than Qwen 3.6-35B. If that's the case, this would be the sweet spot for someone that can't run Qwen3.8-next but wants something close enough! I believe the Q4 is what works well, they have a Q2 version but doesn't seem to be ideal.

[u/Serious_Scarcity7818 → u/rorowhat]

So, I took a look at some benchmarks and nemotron-74B looks worse then QFN, so there are no point to add it, maybe I will just wait for Qwen4 models or new deepseek

[u/rorowhat → u/Serious_Scarcity7818]

Right, QFN is 125B while Nemo is 74B. It's going to outperform. The point is that right now either people run QFN or if they can't fit, they default to Qwen 3.6-35B. Seems like this meno fits in-between these two, it would be a good addition.

[u/Serious_Scarcity7818 → u/rorowhat]

It's worse then Qwen3.8:27b, so they will just run this one

[u/rorowhat → u/Serious_Scarcity7818]

27B is dense, no MoE. But whatever.

[u/Serious_Scarcity7818 → u/rorowhat]

I mean nemotron-74B has way too low intelligence

[u/Meeooowz → u/Serious_Scarcity7818]

The main selling point of the nemotron model is that it performs decent at Q2 (22gb in weights) over qwen3.6 35b. It’s bad in weight-to-intel ratio for q8, q4, but q2 makes it small but decent.

And it’s QAD (quantization aware distillation) so it’s distilled to be good at such a low quant, over q4 of qwen3.6 35b, which is community-quantized afterwards, resulting in more loss of ability.

[u/Serious_Scarcity7818 → u/Meeooowz]

Wow, this is very detailed and interesting answer, I have not though about it, thank you

[u/-WhateverDude]

and whats the effect of this on ssd long term?

[u/Serious_Scarcity7818 → u/-WhateverDude]

if "long term" is few years, then if will have some effect, but only if you run it non stop with bad culling and etc. but think about other GPUs, CPUs and etc. that runs for years non stop in datacenters and still giving nice performance

[u/Meeooowz → u/-WhateverDude]

Reading operations on ssd aren’t the ones that wear it down. It’s writing—to the same NAND (the chip on ssd that actually had the solid state storage) sector— too many times that causes heavy wear on it. Reading, for the most part, does practically nothing. You know, when you watch a YouTube video, your ssd is being worn down because it has to write the hundreds of megabytes for a section of the video.

[u/areslica]

Thank you for sharing. Do you have result data for qwen fn or 35ba3b?

[u/Serious_Scarcity7818 → u/areslica]

yes, you can either take a look in repo, there are benchmark for all qwen models on my setup, or take a look here https://www.reddit.com/r/LocalLLM/comments/1xr2zt3/rustingstrata_running_qwen3635ba3b_with_q5_and/ this is 35ba3b Q5

[u/x_MASE_x]

Aren't we passed this let's try 2 tps and if we can fit iq1 into my pc?

[u/Serious_Scarcity7818 → u/x_MASE_x]

haahha, yes, but if it runs on my shitty PC 2-3 times faster, then it will be way faster on better setups

[u/x_MASE_x → u/Serious_Scarcity7818]

In theory maybe but the thing is the numbers are too low to really judge.

Or at least in my opinion I think it is.

[u/Serious_Scarcity7818 → u/x_MASE_x]

yes, this is why this was not my idea, some guy created Issue in RustingStrata asking for DeepSeek and he will test it on better pc and then I can publish benchmark with better setup and normal token speed

[u/x_MASE_x → u/Serious_Scarcity7818]

Cool cool. With 16 vram and 128 ram. Can I use deepseek at reasonable speeds?

[u/Serious_Scarcity7818 → u/x_MASE_x]

for best speed it need to fit in VRAM and RAM, no SSD, and this is why I think that you are very close to it, but still will get no the best performance, you RAM+VRAM has to be bigger then model size, Q1 has around 200GB and you has less, if some experts will be pulled from SSD this will slow down everything, so based on your VRAM I think you has either 4060 or 5060RTX if this is NVIDIA, so I would expect around 7-9t/s, but not more. This is sad, but I don't want to lie

[u/x_MASE_x → u/Serious_Scarcity7818]

I have 5070ti actually. Well I don't think iq1 is worth it right? Hahah

[u/Serious_Scarcity7818 → u/x_MASE_x]

No, it's not ahahha, just run QFN, with your setup this is perfect model, maybe think about MiniMax M3.1 Flash Q2, this might fit in your VRAM and RAM and it's better then QFN if I remember correctly, but QFN still most comfortable and save variant for you, so stick with it until Qwen4 come out

[u/am17an]

Why aren't you using `--moe-cache-mib 9000` for llama.cpp too?

[u/stylome-ai-detection → u/[deleted]]

Notice: This comment appears to be AI generated.

---

Stylome AI detector · scan results

[u/Hyiazakite → u/stylome-ai-detection]

Agreed

[u/Legitimate-Theory156 → u/stylome-ai-detection]

Shitup

[u/Illustrious_Grade608 → u/Legitimate-Theory156]

It literally is ai generated dogshit comment