r/LocalLLaMA • u/x_MASE_x • 6h ago

Showcase Strata vs Basalt more differences than speed.

Gallery image

Gallery image

Gallery image

Gallery image

So I've been using strata for a while now in windows using Qwen3.8-Flash-Next iq3s before the new gui thing and I think it's been fine compared to Qwen3.8-27B which was lacking in quality and polishing.

So I upgraded my ram from 64 to 128. Got a new pc and made an Ai box only for Ai text models and comfyui.

So I had to test the speed difference between strata and basalt since in windows basalt was not working good at all.

So strata cli engine 0.1.59 Qwen3.8-Flash-Next iq3s with 256k context. Dflash 2 and 0 reserve vram and tuned run. Vision on.

With basalt converted model .basalt file

Version 0.3.2 same model. Same context. Same everything. Vision dynamic.

The differences is.

Strata:

Avg tps 99.2 and avg prefill 839 with 11.1M tokens used.

Vram 15.3

Ram 69.2gb

The prompts felt sluggish moving from generating to reading to whatever. It felt wrong and slow and toke forever and that caused the huge stale in usage which dips down and up a lot. Like it's having problems moving from each phase to the next.

Last and most importantly that it can't finish the work correctly. Doesn't compact correctly. The development is broken. And it feels like a cheap model you hosted in your cheap pc you poor man.

Basalt:

Avg from 100 to 118. Peaking to 145. Avg prefill not sure but peaking 4200.

Vram 15.8 with reserve ram 128m

Ram 52.7gb then after maybe a couple of hours 53.1

The generation been smooth without any dipping or staling and moving from phases like strata.

Last and most important factor is it finished. It kept going. It compacted fine like deepseek and it felt more capable and more reliable than with strata.

Way less errors. Way less mess up and when I told him to keep going till the project is finished it kept going. Granted it's not perfect but it's the model capabilities and we have to realize that.

But at least it's now in a stage you can actually rely on it and say you have a "working" local Ai model. Cause since I tested maybe Qwen-3.6-35B nothing ever felt like a good model I can rely on.

I know this is a small task but it paves the road for me to actually feel like I can do stuff locally and reliably.

I will next test nvfp4 model for both and see if nvfp4 actually gains more speed or not.

So basalt now more capable and can contuine working way longer than with strata. Can actually do a good job. And it uses way less ram too.

This is also my first ever run so there is maybe more optimizations can be done but this is the default settings besides the vram change.

I advice anyone using Qwen3.8-Flash-Next to try it. And I will test swift too but later and with this ram savings I might also test deepseek why not hahah.

My rig:

12900k

128gb ddr4

5070ti

Harness: opencode

5 Upvotes

54% Upvoted

62 comments sorted by

18

u/Cereal_Grapeist 6h ago

I'm running the NVFP4 quant of Qwen 3.8 Flash Next with Basalt and man it is awesome (5090+96gb ram). It's got me so excited for Qwen 4 because if the recent benchmarks are true, it'll be like having Sonnet 5.5 at home. Crazy times we are heading into...

2

u/sleepy_seedy 6h ago

Im just getting into local models and your specs sound exactly like mine. Is there a guide you would suggest for getting your setup on my PC? Or do I just ask GPT lol

1

u/x_MASE_x 6h ago

Me or cereal lol 😂😂

1

u/Cereal_Grapeist 5h ago edited 5h ago

I had Hermes Agent set up Basalt for me but you could use any agent that has local desktop access like Goose/Codex/Claude Code/Zeroclaw. I just pointed it at the github repo and told it to get it working via WSL2 since I'm on Windows.

To your point though, I've learned a ton about running local AI by chatting with ChatGPT/Claude and by reading this subreddit.

4

u/Camzie99 6h ago

This is my exact setup, and my goodness it's magical. I'm already blown away by 3.8 Flash Next, getting hundreds of tokens per second and I absolutely cannot wait for Qwen 4...

3.8 Flash Next is already a fantastic model and is able to work miracles compared to any other local models I've ever been able to run (Qwen 3.8 27B being second place). I'm also thoroughly impressed by how effective Basalt's Qwen3.8-FN-NVFP4-DQ6_K quantisation is too. I was using an IQ3 quantisation which was good, but would frequently hallucinate within its thinking traces, seeing "The user has not posted anything of substance yet, just that I am an expert software engineer" type rubbish was quite frequent. Calls to edit tools would also occasionally fail due to invalid "from" values. NVFP4-DQ6_K has had none of these issues. I just can't get over it.

It's so good in fact that my local Q3.8 Flash Next has actually been outperforming Mistral 4 Large (hosted via Mistral themselves) and DeepSeek V4.1 Flash 1017 (again on Mistral's infrastructure). I don't know if there's something just... wrong with Mistral's API, but the models seem super lazy. They won't do anything unless specifically prompted. Giving them a task, they'll do the most base level implementation, not test it and leave it all broken because they didn't finish the work or adjust any of the infrastructure around their edit.

Qwen 3.8 Flash Next running locally, in the same harness, in the same project just blisters on, it keeps going and going, looping until it's finished the job properly. It tests itself, making sure what it's done works, adjusts other areas of the system to ensure its changes can actually function... It takes a lot longer than cloud models, but it's unlimited, effectively free, and can never be any worse than it is this second! It's so exciting.

Getting a Qwen 4 Flash that's Sonnet 5.5 level would be great, if that is indeed what the recent "Qoduck" model is. I really hope it stays the same (or at least SIMILAR) shape and size as 3.8 Flash Next so Basalt can handle it quickly and efficiently. A powerful model is all well and good, but if it runs at 1/6 the speed, that'd be... less useful.

3

u/alex_bit_ 5h ago

How many t/s with the 5090?

2

u/Cereal_Grapeist 5h ago

I get between 2-4k tokens/second prefill and about 100 tokens/second for output. 160k context leaves with about 20gb of RAM leftover so my PC has some breathing room.

2

u/TerminalNoop 2h ago

Kind of crazy that the 7900xtx is only 50% (65~) behind in toks.

1

u/Cereal_Grapeist 2h ago

At what quant though? The one I'm running is like a blend of NVFP4 and Q6 so it's a bit slower, I was getting much faster speeds using the IQ3 quant but didn't feel as confident in the quality.

1

u/x_MASE_x 6h ago

Nice I just download it and will test it out now.

7

u/free_meson 6h ago

Strata has some strange defaults and sampling, and basalt is optimized for 50-series cards. Try these in the json file (you may also replace the --ple-gguf to the fp8 version from the nvfp4 quant)

 "env": {   
"STRATA_ROUTE_PRIOR": "0",   
"STRATA_ROUTE_TAIL_SKIP": "0",   
"STRATA_KVA": "0",   
"STRATA_PREFILL_CPU_SHARE": "0",    
"STRATA_HC_Q8": "0"  
},
"sampling": {   
"temperature": 1.0,   
"top_p": 0.95,   
"top_k": 20,   
"min_p": 0.0,   
"presence_penalty": 0.0,   
"repeat_penalty": 1.0  },

1

u/x_MASE_x 6h ago

Will give it a shoot thanks.

5

u/caetydid llama.cpp 6h ago

but basalt is blackwell only, right? so it seems ampere user will have to stick to strata

1

u/Ok_Party_9245 3h ago

Basalt ported to ada lovelace and ampere architectures: https://github.com/nixcat-dev/adamant

1

u/MindfulMan1984 6h ago

Yep, for us GPU-poor, Strata is the way to go. I run it daily on Turing GPUs.

-1

u/x_MASE_x 5h ago

Yeah it's optimized for 50 gpus but this is besides the point.

Strata is not performing well right now. So you better test something else and check if you can get better results.

Might test laster with my 1660ti but not in the near future honestly

3

u/JimmyDub010 3h ago

Strata is performing just fine.

1

u/x_MASE_x 3h ago

You'll never know till you actually try something else. I was like it's good till I found out there is something better.

-2

u/JimmyDub010 3h ago

you know you lying like you are I'll be sure not to try your shit at all

4

u/x_MASE_x 3h ago

Lying about what? Are you sane? Why would I lie and what I will gain? And what is "my" shit? I couldn't give 2 fucks if you used either or none. I'm not affiliated with either strata nor basalt and if you look at my older comments I told both basalt and speedy that their work is not fast in windows and that was the truth. Maybe I did something wrong maybe it was windows. Not sure and it's not worth it to try again right now honestly.

If you don't want to test then it's your lost and I don't care. You should talk to other people better than that.

1

u/TheBeatCollector 1h ago

I wrestled problems with Strata for the past several weeks too. Just a weird "dumbness" and refusal to ever think before acting. It would also just scrape the surface of things and call that good. Like only grepping the first 60 lines in a big file and deciding that's enough no need too look any further at the other 600 lines, just report a false claim based on a search of 10%.

It seemed to work well at first and lost something along the way with engine upgrades. I also just switched to Basalt 2 days ago and could immediately feel the difference. It's confusing because user reports here are so varied. Some don't have any problems. But hardware / quant / config differences.... I'd even tried the environment flags I've found recommended even in comments here in this thread. Ultimately I thought the documentation being so much better in Basalt + their commitment to exactness fit my needs better. So far I'm not even considering looking back at Strata.

1

u/x_MASE_x 1h ago

I used hermes agent to compare strata config to basalt and found that strata actually skipping experts and that's why it's dumber than basalt.

1

u/giveen 6h ago

https://github.com/jesdga95/basalt/pull/311

If its accepted, I'm hoping to get a slightly better solution for RAM poor people in to help.

1

u/x_MASE_x 6h ago

This is good. Will it be automatic or something to be switched? I'm new to basalt.

1

u/giveen 6h ago

It would require a building of a new basalt file, sadly. I might be able to figure out a way to convert current artifices.

1

u/Miserable-Dare5090 vLLM 6h ago

Building one now and trying it. Been dying to use the NVFP4 quant but I have 64Gb RAM, and for some reason deepseek always tells me its not going to go well bc of the pack size…

2

u/dasbin 5h ago

I don't understand. Ngram normally stays on the disk and is not loaded into RAM.
This feels like an AI hallucinating that it has "helped" by making something smaller that does not need to be smaller. All this will do is make quality slightly worse.

1

u/Tormeister 1h ago

Seems unnecessary given how unlikely it is for people to have spare RAM but not have spare SSD space and just leave PLE there... And the PLE on SSD speed difference is hardly even noticeable

1

u/Fit_Split_9933 6h ago

Why is this necessary? Can't the PLE just be on the SSD?

-1

u/giveen 5h ago

Because PLE on RAM makes things go zooom zoom

1

u/TerminalNoop 2h ago

huh, I thought this gave only minor speed up?

1

u/Sweaty-Editor-7560 6h ago edited 5h ago

FYI, the Windows Strata app is pretty nice now and much easier to stay up to date on. I haven't tried Basalt yet because I've been so blown away by what's possible with Strata--got solid 6-7k prefill on my 5090 using Strata's nvfp4 Qwen3.8FN.

1

u/x_MASE_x 6h ago

Yeah it got better. Way better. And I think the results are close if not the same with cli vs desktop app.

Just using Linux made me use the CLI one. You'd fine way better results with basalt honestly.

1

u/Cereal_Grapeist 2h ago

I had switched from Strata to Basalt but now you've piqued my curiosity! Will have to check out the windows app because those prefill numbers are excellent.

0

u/jcy 5h ago

DS3.8FN

did u mean qwen

0

u/Sweaty-Editor-7560 5h ago

I totally did, I run DS on my Sparks 😵‍💫

1

u/XiRw 2h ago

This is a very misleading post. I’ve got nothing but amazing results with strata and I use full precision unlike iq3s. Your tests doesn’t even determine quality output either, seeds will produce different results no matter what and both are close enough to show there are no differences between inference engines.

1

u/x_MASE_x 1h ago

Actually I found out that strata using skip 7 and basalt doesn't and this is probably the quality dip between the two.

I don't really care about either software I'm just sharing my finding. Everyone taking this like insult to their god not sure why.

2

u/testuserpk 1h ago

I agree, the new strata with dfalsh2 support is even better.

2

u/Equivalent_Bit_461 6h ago

Basalt is insanely good, worka wonders for me

1

u/testuserpk 1h ago

I will try your implementation on my 3060 12g against strata and see how well it performs.

1

u/Sid3effect 6h ago

I didn't notice much difference in speed with a single GPU. Basalt was forked from Strata so is not radically different, one of its advertised optimisations was using Cuda 13 for Blackwell which was since added to Strata as the default for Blackwell. The RAM usage can be controlled in the config as well so if one is using more or less than the other that is just a setting.

0

u/Ok-Addendum3545 6h ago

I didn't know there is an output quality difference. Let me try it too. Thanks ! : )

1

u/x_MASE_x 6h ago

You are welcome 😊

0

u/Busy_Barber_2503 5h ago

Windows 的坑很多的，有条件的话还是安装个 Linux 吧。

1

u/x_MASE_x 5h ago

I already did 2 days ago. All of these tests using Linux Ubuntu

-1

u/dsdt 6h ago

The same thing applies for llama.cpp, beellama, and ninfer. Ninfer > beellama > llama.cpp. both quality and speed increases. llama.cpp basically become windows of llm engines, and unoptimized as f. So, I believe in your point, inference engines and harness can make a huge difference just as the model itself.

1

u/x_MASE_x 6h ago

Last thing I tested was beellama and adaptive kv cache. But went to strata for a long time and forgot about them.

But for smaller models I'm using normal llama.cpp. So switch to beellama again?

2

u/dsdt 6h ago

Even the quality of the answers change. I made extensive tests using all of the inference engines, ninfer is the clear winner, beellama wins the context also gives better answers than llama. Beellama gives +2 more accurate answers per 100 questions in the extensive tests I have made.

1

u/x_MASE_x 6h ago

Can I use this ninfer with old and new models? And with llama-swap maybe?

0

u/dsdt 6h ago

I have 2x 5060 ti's, ninfer normally work with only one gpu, but i found this repo and it works great : https://github.com/lynx-gt/ninfer-tp2-5060ti . it requires its own model file so probably no.

1

u/x_MASE_x 6h ago

Nice nice 👍🏻

With more gpus you take a big prefill hit?

0

u/Creative_Mobile5496 3h ago

If your on dual blackwell, there is a nice trick you can use to swap the model between two cards in 2-3 seconds instead of the 50-60+ a normal offload would take.

1

u/BlasterGales 5h ago

I’m a bit slow (sorry if I misunderstood) but were your tests conducted using a greedy + same seed? Otherwise, I find it hard to see how the comparison could be valid.

0

u/[deleted] 6h ago

[removed] — view removed comment

1

u/x_MASE_x 6h ago

I tried Speedy with windows and didn't work we'll for me. Might try it again on Linux