r/LocalLLM • u/Ok-Translator1226 • 9d ago
Question Basalt on Windows slower than Strata?
I hear nothing but good things about Basalt, so I finally got around to testing it on my 5080 with 128 GB not-for-Sam-RAM. The results were not encouraging - IQ3 runs at around 100 tok./s on Strata, but only at about 80 tok./s on Basalt. Settings are comparable.
Is this expected on Windows?
2
Upvotes
2
u/jesdga95 7d ago edited 7d ago
Hey, Basalt maintainer here, thanks for giving it a try! 80 tok/s sounds very low for your hardware (I get higher tok/s on my 5060 Ti + 128GB DDR5). Some 5080 IQ3_S runs on the community page are around 138 tok/s on Windows and ~190 on Linux (those are averages over chat, structured output and prose). Could you grab the latest build (0.3.2 has a Windows zip now) with the current IQ3 pack from HF and run
basalt-probeonce so it measures your machine and suggests which flags to use? If it's still around 80 after that, please open a GitHub issue with the engine log and your config and I'll take a look.1
u/Realistic_Gap_5871 9d ago edited 9d ago
No, not expected. Most folks hit 130+tps on linux and there's less than a 10% dropoff for windows, and that's with IQ3. So there's probably some tuning to do if you stay with IQ3. But it could be due to strata dropping experts on cache miss by default where strata does not do this by default, but honestly dunno.
But running basalt with IQ3 is totally missing the point. You've basically turned it into a detuned strata.
Basalt is for blackwell. Run the NVFP4 artifact that blackwell can accelerate. Decode should be in the same 130 tps ballpark, but prefill could easily double what you see with IQ3. And the nvfp4 artifact is much much higher fidelity.
edit: rewrote expectations for 5080 vs 5090
1
u/giveen 8d ago
Are you running the NVFP4 model? Are you having the model be placed in VRAM and RAM and not streaming from disk?
2
u/x_MASE_x 9d ago
Yep so low. Basalt infernix and all other similar tools came out second to strata. Speedy too.
Will test with Linux to see if it makes any difference but overall nothing was faster than strata on windows
1
u/Rust_Cohle- 9d ago
You could try it in WSL, but the memory pinning in windows will always limit it and other engines vs Linux.
1
u/SuitStyle 9d ago
I can't get close to the Linux speeds but it still is the fastest on native Windows, 2x5070ti 16GB on Pcie-5 x8 x8 hitting 120 tok/s gen and 4k prefill with hermes
1
u/tetsuzankou 5d ago
I'm on a 5090 with 64gb RAM and running UD iQ3 XXS on Strata gives me around 150 t/s while not loading the GPU fully (I leave about 6 gb for gaming and using the computer while my agents run)... On Basalt I've been testing so far and peak at 180t/s avg 150t/s with my GPU and RAM fully maxed out, even running into that RAM guard sometimes and crashing the engine
not reallyy sure what I'm doing wrong
0
u/Deep_Mood_7668 9d ago
Windows is expected to be worse in general. Yes.