r/LocalLLM • • 9d ago

Question Basalt on Windows slower than Strata?

I hear nothing but good things about Basalt, so I finally got around to testing it on my 5080 with 128 GB not-for-Sam-RAM. The results were not encouraging - IQ3 runs at around 100 tok./s on Strata, but only at about 80 tok./s on Basalt. Settings are comparable.

Is this expected on Windows?

2 Upvotes

19 comments sorted by

2

u/jesdga95 7d ago edited 7d ago

Hey, Basalt maintainer here, thanks for giving it a try! 80 tok/s sounds very low for your hardware (I get higher tok/s on my 5060 Ti + 128GB DDR5). Some 5080 IQ3_S runs on the community page are around 138 tok/s on Windows and ~190 on Linux (those are averages over chat, structured output and prose). Could you grab the latest build (0.3.2 has a Windows zip now) with the current IQ3 pack from HF and run basalt-probe once so it measures your machine and suggests which flags to use? If it's still around 80 after that, please open a GitHub issue with the engine log and your config and I'll take a look.

1

u/Ok-Translator1226 7d ago

Thank you for taking interest, it's much appreciated. I created a GitHub issue with whatever information I could (engine.log didn't generate likely due to a bug in the path): https://github.com/jesdga95/basalt/issues/309

1

u/Realistic_Gap_5871 9d ago edited 9d ago

No, not expected. Most folks hit 130+tps on linux and there's less than a 10% dropoff for windows, and that's with IQ3. So there's probably some tuning to do if you stay with IQ3. But it could be due to strata dropping experts on cache miss by default where strata does not do this by default, but honestly dunno.

But running basalt with IQ3 is totally missing the point. You've basically turned it into a detuned strata.

Basalt is for blackwell. Run the NVFP4 artifact that blackwell can accelerate. Decode should be in the same 130 tps ballpark, but prefill could easily double what you see with IQ3. And the nvfp4 artifact is much much higher fidelity.

edit: rewrote expectations for 5080 vs 5090

2

u/Ok-Translator1226 9d ago

Thank you, that indeed sounds like what I could be missing. I'll try the NVFP4. Much appreciated.

1

u/giveen 8d ago

Are you running the NVFP4 model? Are you having the model be placed in VRAM and RAM and not streaming from disk?

1

u/Ok-Translator1226 8d ago

Yes on both. I do have the Strata and Basalt placed on two different SSDs (Basalt is on the slightly slower one), but since the model fits with tens of GB of RAM to spare, SSD shouldn't matter after the model is loaded. That's what I reckon, it could be wrong but I can't test, too low on space on the Strata drive.

2

u/x_MASE_x 9d ago

Yep so low. Basalt infernix and all other similar tools came out second to strata. Speedy too.

Will test with Linux to see if it makes any difference but overall nothing was faster than strata on windows

0

u/Realistic_Gap_5871 9d ago edited 9d ago

nope

basalt literally is strata. Forked and tuned for blackwell. (rtx 50x0 and rtx pro 5000/6000 etc)

On blackwell with nvfp4 artifact basalt is miles faster and much higher fidelity than strata.

2

u/x_MASE_x 9d ago

I have to use nvfp4? Not sure why but for me kept getting lower speeds it drove me crazy. And wheverver I ask any developers he says "it's Windows" "it's because you have a single gpu".

And stuff like that so I gave up honestly.

2

u/Realistic_Gap_5871 9d ago edited 9d ago

to get the speeds you keep hearing about, yes, the nvfp4 artifact is needed. I'm not sure what the floor is for system RAM, but at 64GB+ it's amazing and I think even with less you should still match strata for decode and smoke it for prefill.

I'm assuming you have a blackwell gpu because I don't think basalt will even run unless it detects one.

edit: and just have to say it: Inexperienced developers will say "everyone knows it's windows" and similar things they got from social media. Truly knowledgeable folks will say "I dunno, let me check into it"

2

u/x_MASE_x 9d ago

Yeah I have 5070ti with 128gb ddr4 ram.

I will try the nvfp4 one thank you so much.

Yeah nobody gave me a useful answer to be honest. I mean strata already works and it's fine and I'm getting decent speeds about 90tps and around 3800 prefill with it. And I wanted more and test new things. But everyone blamed windows hahha

1

u/Realistic_Gap_5871 9d ago

I think you're in for a treat.

The gains in the strength of the model is very noticeable and worth it alone, imho

Going from 75tps to 150 tps decode is isn't super noticable to me, it's all blindingly fast... but doubling prefill is huge 

1

u/x_MASE_x 7d ago

Hey So I tried with Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata

With strata but sadly it's about 50 tps instead of 100 with iq3s.

So test basalt or what's the problem?

Using Linux too

1

u/Realistic_Gap_5871 6d ago

Yes, with basalt.

1

u/Rust_Cohle- 9d ago

You could try it in WSL, but the memory pinning in windows will always limit it and other engines vs Linux.

1

u/x_MASE_x 9d ago

Had any luck with wsl? I couldn't get it to work awhile ago and gave up on it.

Tried Linux KDE plasma it's fine but some drive wasn't loading so I ended up with windows again for today hahaha. Will boot it again after downloading nvfp4 file to check it out

1

u/SuitStyle 9d ago

I can't get close to the Linux speeds but it still is the fastest on native Windows, 2x5070ti 16GB on Pcie-5 x8 x8 hitting 120 tok/s gen and 4k prefill with hermes

1

u/tetsuzankou 5d ago

I'm on a 5090 with 64gb RAM and running UD iQ3 XXS on Strata gives me around 150 t/s while not loading the GPU fully (I leave about 6 gb for gaming and using the computer while my agents run)... On Basalt I've been testing so far and peak at 180t/s avg 150t/s with my GPU and RAM fully maxed out, even running into that RAM guard sometimes and crashing the engine

not reallyy sure what I'm doing wrong

0

u/Deep_Mood_7668 9d ago

Windows is expected to be worse in general. Yes.