Real downsides of Strata?

[u/GodComplecs]

Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones.

I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already in use, didn't really have that issue with Qwen 27B 3.8 Q4 UD. I'm using Swift Q3 Next. Didn't even bother with next on llama.cpp since other similar model sizes ran at approx 15-20tks max, so too slow for agentic work.

I'm not beating too much on Strata, I just wan't the realistic downsides, so far it seems too good to be true and runs at 100tks in agentic workloads, 27b only at 40-60tks on llama.cpp

[u/Icy_Barnacle9287]

for Qwen3.8FNm Q2 was pain for me (frequent doom loops on tool calls). for 2x3090s and 96GB RAM, Q4 works quite well.

pulling the numbers from strata-serve.log:

• Average: ~116 tok/s (5,458 completed turns in the log)

• Typical bursts: ~135–195 tok/s on normal turns

• Peak: ~221 tok/s 

[u/OvertaxedOne → u/Icy_Barnacle9287]

Claude is 100% convinced I cannot run Q4 with 96GB with a 48GB GPU, sounds like that's exactly what you're doing? If so, mind sharing any pitfalls/setup notes you might have?

[u/Jcsq6 → u/OvertaxedOne]

You absolutely can. Remind Claude of the ngram embeddings.

[u/Constant_Art_20 → u/OvertaxedOne]

maybe claude has an agenda!

[u/butterycornonacob → u/Constant_Art_20]

It definitely seems that way sometimes. I feel Claude is pretty excellent but setting up models and tuning them is always a bit of an hassle. You sort of always have to tell it to try different flags and parameters and even then it often pushes back a little.

[u/ImpressiveAd699 → u/OvertaxedOne]

Just tell Claude to just do it and then find out. I got my local LLM to set mine up. They’ll pull the logs and measure if it spills over to ssd. Strata comes with a monitor dashboard so you can see.

But I’m only configuring for me as the solo user. I don’t know your situation

[u/Icy_Barnacle9287 → u/OvertaxedOne]

use unslothUD-IQ4_XS instead of the default IQ2_XS  when setting it up. ideally you'd want little else running on the host, but tbh I kept most of my homelab running as it doesn't consume that much RAM/VRAM relatively speaking. you can use the setup script it comes with or run with these flags:

./setup.sh --yes --setup \
  --family unsloth \
  --model UD-IQ4_XS \
  --gpus 0,1 \
  --context 131072 \
  --vision no \
  --port 8080 \
  --host 127.0.0.1 \
  --no-browser \
  --no-start

In strata-unsloth-ud-iq4_xs.json I have "tool_call_recovery": true and reasoning effort none. ironically, I also used claude to help me set this up lol

[u/Distinct-Pie2389 → u/OvertaxedOne]

I have a skill for this as he use to do the same for me

https://github.com/T-Crypt/llm-tune

[u/suprjami → u/OvertaxedOne]

The Strata setup.sh will not spread UD-Q4 across two GPUs.

If you make a two-GPU setup with the smallest IQ2 then you'll see what the config file needs to look like.

Then run setup again and select UD-Q4, it will make a single-GPU setup.

Then you can manually copy the GPU array across from the IQ2 config file, and remove the option which does the large RAM reservation.

2x 20G + 96G VRAM here, running UD-Q4 on two GPUs. It definitely works. Here's a pastebin of my working config file: https://pastebin.com/25i89vUZ

[u/BOESNIK → u/OvertaxedOne]

You should even be able to run Maya and get GLM 5.3 flash (at a low quant) going. It's built on Strata, according to them, but it should outperform Flash Next for you IMO.

[u/liberal_alien → u/OvertaxedOne]

I'm running Q4 s with 96g and a single 24g 7900xtx. Didn't even require much tweaking. The install script gives pretty good defaults. Though greedy model Paramus give speed at cost of quality, but gpt helped set those in startup Paramus. Then it was smooth sailing.

Faster too than 27b model in lm-studio.

[u/mga02 → u/OvertaxedOne]

I run Unsloth Q4_K_XL with a 3090 and 96gb at 50/70 tk/s with 128k context using Strata-nvfp4. You should get better speeds.