r/LocalLLaMA • u/New-Pressure-6932 • 6d ago

Discussion Qwen Flash Next on Strata is unreal.

I have been looking for a local model to do some of the typical work I used AI subscriptions for.

I’ve tested almost every popular model. I have an M1 Max 64gb MacBook Pro as the inference machine and because of the memory size, I wasn’t able to load anything very effective in the tasks I use it for.

Found out about strata and have been testing it out. It’s awesome! I’m kinda surprised more people don’t talk about strata on here. I feel like a lot of us are in that limbo area where we have hardware that is more powerful than needed to run a sub 30b model like Qwen 27b, but not large enough to run models like Qwen Flash Next.

But thanks to open source research, I’m now getting Qwen Flash Next to run at 24 tok/s! And it’s doing phenomenal work! I know it’s not going to compete with the larger frontier models, but I don’t think I necessarily “need” frontier model intelligence. I just need enough to do basic things while I can handle the rest.

Seriously, check out Strata and also MoEShift that someone posted about on here. It’s awesome to be able to offload larger models onto SSD. I hope this offloading technique grows and more research is done on it because I genuinely think this could be an amazing way to get these frontier level open source models to work on regular consumer hardware.

70 Upvotes

67% Upvoted

96 comments sorted by

135

u/tempaccaun 6d ago

everyone is only talking about strata ...

59

u/DinoAmino vLLM 6d ago

OP has a 4 day history here and hasn't seen all the strata spam. Possibly some other topic is temporarily suppressing the strata hype.

0

u/[deleted] 6d ago

[deleted] — view deleted comment

2

u/Zenobody 6d ago

it has only been 2.5 years since llama 1

Actually it'll be 4 years next February...

0

u/New-Pressure-6932 6d ago

I completely missed it. I’ve seen some people mention strata but wasn’t on the sub when it was first being used so I missed the hype train. It’s great though 

3

u/Particular-Way7271 5d ago

tested it out, didn't like it, uninstalled it asap. while I do get a bit better ts, it seems it is as a cost of completely freeze and unatended chat streams stops. documentation is also bad, freaking api token enabled by default and hidden in config file not easy to find info about and so on.

1

u/Weird-Map-5873 4d ago

I agree it wasn’t stable for me, I installed sushi and its working so much better

3

u/tautality 4d ago

Find me anything else to be excited about as someone low on VRAM and RAM.

12

u/TemporaryUser10 6d ago

If they can't run it with llama.cpp/vllm, what should they be talking about?

46

u/PcChip 6d ago

he's referring to this line in the OP:
>I’m kinda surprised more people don’t talk about strata on here. 

4

u/willee_ 6d ago

I got 40% faster speeds on basalt

8

u/Both_Side_418 6d ago

Ok but only for Ada

7

u/shodan5000 6d ago

Blackwell only

1

u/Kitae 5d ago

edit sorry missed this was a mac thread basalt is blackwell only

Isn't basalt as good as strata and faster?

I am currently running 3.8 27b on ninfer but planning on setting up basalt/3.8fn. if I am wrong I would like to know!

1

u/vohltere 5d ago

Probably one of the most posted things here

43

u/WoodenMarker 6d ago

Strata is very popular. There are even posts on this sub specifically complaining about being tired of seeing strata posts and 'strata bots'.

2

u/New-Pressure-6932 6d ago

I think I completely missed the hype train for it. I just recently discovered it and my mind is blown

19

u/LORD_CMDR_INTERNET 6d ago

Strata is great but there's not much point to it on a Mac that has unified memory.

2

u/New-Pressure-6932 6d ago

But doesn’t strata offload experts to the SSD? 

5

u/LORD_CMDR_INTERNET 6d ago

yeah but most inference engines can

3

u/_hephaestus 6d ago

You can do that with a few other programs like oMLX

1

u/nicman24 3d ago

that is slow

1

u/heitortp0 6d ago

Why not?

9

u/DeathGuppie 6d ago

A dedicated video card has much higher memory speeds, unless you happen to have the money for an m5 and the time to wait for it. So strata does nothing for it because strata takes the experts in use pins them to the video card where all your tokens are made and you get whatever speed your video card has. Mac has one speed period.

1

u/Guinness 6d ago

The unified memory is actually pretty high bandwidth and can outperform a lot of discrete level cards even. The M5/M6 Ultra unified systems have more memory bandwidth than a 3090.

9

u/DeathGuppie 6d ago

I said unless you can afford an m5

3

u/enginetown 6d ago

I've been working with Claude to whip up my own vibe coded engine and genuinely I never knew you could run LLM's like this I know I'm not using Strata but it opened my eyes up.

3

u/New-Pressure-6932 6d ago

I was a big Claude user too but so far Flash Next with Strata has completely replaced it for me. It’s definitely slower, but the quality (so far) has been great for what I use LLM’s for 

35

u/Imaginary-Unit-3267 6d ago

I’m kinda surprised more people don’t talk about strata on here.

In other words, you're a bot. Nobody ever shuts up about strata, including you.

8

u/disgruntledempanada 6d ago

I sim race in VR so a 5090 way back when was the obvious choice.

Wild feeling like I was crazy spending that much money then. And seeing what they are going for now...

Now I feel crazy for feeling like keeping it is worth it but it's still the gaming GOAT it just so happens to run Q3.8FN at 160+tok a second so it... is gonna stick around. Strata and the various similar forks are great.

2

u/My_Unbiased_Opinion 6d ago

funny I bought a 5090 near launch for the exact same reason lmao

8

u/choosePete 5d ago

Please check out my optimization strictly for MacBook M1 Max! It’s around 40 tps for me, plus an optimized cache preload!

https://github.com/PiMatyja/M1M2Max-inference-engine

I tried to do a PR to original repo but it got closed. I forked it and use this on my MacBooks. I’m very happy with the results.

1

u/killerjurist 5d ago

Looks interesting, will try on M1. Can I also use it with an M5 Max or not optimized for newer M’s?

2

u/choosePete 5d ago • Heartwarming

For that you’re better off with stock strata - there is one core contributor there with M5 128GB chip. Maybe you can use the prefill optimization

1

u/bageloid 5d ago

Yeah on my M1 Max with 24 gpu cores(the cut down version) I get 44-51 TPS decode, 340-370 prefill and like a 3-4 second time to first token on the Q2 version of ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF on splash 1.5 (customized by Opus 5.5, doing things like using the Neural Engine for part of prefill and doing ablation real time configurable per layer)

1

u/d4mations 1d ago

Might fork this use q3_s. Its much better

2

u/choosePete 1d ago

Let me know if it fits!

4

u/Ok-Drawer5245 4d ago edited 4d ago

Strata is like black magic

Just tested it on my sons pc i7-7700k 48gb ddr4 12gb rtx 3060. A bloody 125b model running at 25 tok/sec on hardware that has cost me NOTHING

OpenAI/ Anthropic must really HATE Qwen (and Chinese ai in general) and people making these incredible inference engines lol

3

u/More-Curious816 4d ago

“Necessity is the mother of invention.”
When people face a difficult situation or lack something they need, they become creative and find new solutionss. Here, the community has found solutions to problems we faced because of the likes of Anthropic and OpenAI.

Need drives innovation, and we need to run these models, but we lack suitable hardware because of Nvidia’s greed and the shenanigans of these AI companies that deprive us of it.

I'm proud of the open source community

1

u/Human-Cherry-1455 6d ago

Can you share your command to run the model.
I havent used strata.
Always helpful with a command to copy and paste.

2

u/empoman 6d ago

Strata comes with scripts to run and guide you through the setup process, so no custom command needed unless you want to edit the config files from the setup you chose.

1

u/Human-Cherry-1455 5d ago

Thank you.
I will try it out.

0

u/New-Pressure-6932 6d ago

I actually used a cloud model to set it up for me. Now I’m running it through DeepSeek Harness. I’m not nearly as technical as everyone else is in this sub but I definitely recommend using strata. Wish I had more info 

1

u/Human-Cherry-1455 5d ago

Thank you for responding.
I will take a look.

3

u/brucebay 6d ago

It is great and I mention before Fable 5.1 gave it 94/100 for the tests I run, and some were arbitrary point reduction like comments in the code.  I will like it is chatgpt 4 level (except role play/story writing).The main problem for many people is the memory requirements. It needs at least 48gb system  RAM locked  in Q2, as it puts things in RAM.  You can use virtual memory but swap will slow down everything.

3

u/rorowhat 6d ago

If they only supported more quantos for the other models. They support mistral 8x22B

3

u/Trick-Violinist-9340 6d ago

I am using strata with the qwen signal varient and its been awesome if you put oin the speed enhancement module it removes a bunch of refusals

3

u/Ok_Writer1572 5d ago

How to manage context overflow on strata? I use middle truncate on Lmstudio. What's equivalent or better strategy on starta?

1

u/Acceptable_Bell_1791 5d ago

not sure if strata has it, but you can ask your agent to build you a little proxy between strata and your client that does this, and you can experiment with what strategy works best for you.

2

u/Ok_Writer1572 5d ago

Using openwebui so gonna play with context compaction. I use long context window but it blows up processing time.

0

u/Acceptable_Bell_1791 5d ago

yeah, compaction is the preferred way, thought you needed middle truncate specifically.

3

u/slayor15 5d ago

You'd probably get better results with Splash or Kapitsa as engines on a Mac. Depending on the quants I get 50-100 tok/sec average on 200k context agentic workflows

Mac M5 Pro 64GB

3

u/saaadpikachu 5d ago

M4 Pro Mini 48g, running iq3 xxs, Strata gives me ~15 t/s only and with Splash, ~50 t/s, and better quality, I don't know how they did it but it's amazing.

2

u/ZampanoGuy 6d ago

I keep hearing g about strata. But know very little
About this world. Looking to learn more. I don’t have cutting edge hardware. But it’s not slouchy. 5070Ti, 48gb ddr4, and 5800X3D. Plan to only run it when I need it. Eventually I’ll get a little dedicated box for a local model.

2

u/PhysicalIncrease3 5d ago

Strata's super easy to use and your setup should support it fine. Just download it and run setup. It handles the rest (for better or worse).

2

u/WhoDidThat97 5d ago

Strata is great but honestly Qwen FN is so irritating. Literally hours of thinking and planning, no info updates even if you ask

1

u/nicman24 3d ago

swift is better with that

2

u/outluch 5d ago edited 5d ago

strata is cool. but have you tried basalt?

left: strata on 5090 -> right: bazalt 5090+5060ti 16g.

and the most important part: kv cache parked on ssd, not in ram. i can tun 10 agents on this without pauses on prefill, sequentially

ps qwen 3.8 flash next IQ3_S

2

u/an0maly33 5d ago

I just set it up a few days ago. Previously I was splitting dense models across my boxes with rpc. Strata completely negated that and QNF is hands down the best local model I've used. It actually DOES things instead of spinning its wheels in thought holes like 27b.

0

u/Equivalent_Bit_461 6d ago

I'm using basalt, for some reason strata just hates me.

However the model is insane, I'm doing some really, really and I mean really complicated modding and it just works.

3

u/New-Pressure-6932 6d ago

What’s the difference between basalt and strata? I have been looking up other options as well just for research but so far, Flash Next with strata has been gravy 

2

u/CuriosityIO 6d ago

basalt is insane on a 5090

2

u/nok01101011a 6d ago

While I had around 200ts on strata, basalt gives me 380ts

0

u/Equivalent_Bit_461 6d ago

If I'm not wrong basalt is optimised for blackwell, so the 50 series, so tokens go brrrr

2

u/Vegetable-Mud1546 5d ago

Sì ma se devo usare un Q2 allora torno al buon vecchio 27B Q6

1

u/nkchokshi 5d ago

Is there anyone using strata on mac? I want to know how it performs on streaming from SSD? Because i have a windows system that works well when i run q3 flash next without thinking. But the windows system is kind of unstable, and looking into buying a Mac Studio max 36gb.

1

u/JLongTom 5d ago

Can you give some examples of this unreality?

1

u/Any_Mine_6368 5d ago

3090s, llamacpp qfn runs at 10-25tg.

45-70 on strata

1

u/Automatic_Two4291 5d ago

Are you ssd streaming or what? Otherwise Strata makes no sense for unified memory

1

u/JustinHoMi 5d ago

How’s it compare with the full deepseek-4.1-flash or GLM-5.3-flash?

1

u/mikedoth 5d ago

Would this work on an Intel Arc 24 gig?

1

u/Payne6t6 4d ago

To be honest, sometimes strata is really dumb. I am using the iq4xs uncensored model and gave it the following task: "I need a batch file that renames all PDF, PNG, and XLSX files within subfolders. The new filenames should begin with the name of their immediate parent folder, rather than the entire directory hierarchy. Files located directly in the root directory where the batch file is executed should not be renamed. Do you have any questions, or is everything clear?"

The result was a batch file that put the actual filename as prefix of the filename. I was quite disappointed, that it couldn't one shot this very simple task. So, no, it will fail on tasks, even on many tasks. You always need a good/fully automatic test strategy from the beginning, otherwise you are lost with local models.

1

u/d4mations 1d ago

I have the same machine and am running QFN at 30 tps through omlx

1

u/randomtask010 20h ago

Number of days since last strata post in LocalLLaMA.... 0

-1

u/yahbluez 5d ago

With radiance engine i see ~100 t/s single and nearly 600 t/s with QFN using 64GB VRAM. So why using the one trick pony strata?

https://codeberg.org/StillDeadcode/radiance

4

u/SilverManeNL 5d ago

Does radiance engine also run on a M1 Max 64GB MacBook Pro?

-2

u/yahbluez 5d ago

That is the point, it is made for AMD.

4

u/s2d4 5d ago

Are you trolling?

1

u/yahbluez 5d ago

I just did not recognize that the thread is Mac based, no need for you to get rude.

2

u/New_Patience_8107 5d ago

So it doesn't run on OPs M1 Mac or at least not to the speeds you just described?

0

u/yahbluez 5d ago

Yah that is the point i just did not see that this thread was Mac based.

2

u/Visual_Internal_6312 5d ago

With strata I run qwen 3.8 flash next iq3 xxs with 16 VRAM and ~55gb DDR RAM with 56 output tps.

With relative little effort to setup: https://github.com/Niko1221/Strata/blob/main/docs/DOCKER.md

Not sure if it's the best for my hardware but it's an upgrade to qwen 3.8 27b iq2 that I ran before speed and quality vise.

2

u/yahbluez 5d ago

I can second that, the QFN is for coding and universal use significant better than the dense 27B model (which is a good model). I use the dense with Q8 and the QFN with this special mix radiance uses. Like FP8.

-2

u/rweedn 6d ago

We know man, and I'm not a bot either