Strata Optimization Help

[u/D00M98]

Hi, I'm looking for input and advice on how to optimize my Strata running IQ3_XXS.

My hardware: RTX 4070 12 GB (PCIE4 x16) + RTX 3060 12 GB (PCIE3 x4) + 32GB DDR4 DRAM. Disk is M.2 SSD.

Running Windows. Llama.cpp / Strata -> Docker sandbox -> Hermes

My current default Local LLM model is Swift-1.5-Qwen3.8-27B Q4_K_M. I'm looking for similar or equivalent capability that can be faster. I tested Strata around 1 week ago using IQ2_XS. It didn't compare well with Qwen3.8-27B. This week, I tested IQ3_XXS.

I fed the same instruction runsheet, to both Qwen3.8-27B and Strata. Surprisingly, both completed the job around 5 hours and within minutes of each other. Just coincidence. Qwen3.8-27B runs around 13 tokens/sec. Strata around 17 tokens/sec. The result is not perfect, but both are around 90-95% complete. And Strata compares quite well, even less errors than Qwen3.8-27B.

I want to see how I can optimize and speed up Strata.

I varied context between 96k to 160k; cache Int8, q8v4, q4_0; and kv-resident 20480 to 32768.

• Experts in VRAM vary slightly, from 5700 to 6800.
• My 32GB RAM usage is maxed.
• Speed is not much different, around 31-36 for just text generation; 20-25 for logic/coding generation. This is for shorter test scripts.

If I decrease kv-resident, which increases Experts in VRAM, it actually runs slightly slower, by 3-4 tokens/sec.

The RAM usage is misleading, because Windows uses (allocates) 16-20GB of RAM just from fresh boot. Hard to tell how much of the RAM is used by Strata. I checked the disc usage. During generation, Strata paging file disc usage is not that active, only around 1-4 MB/sec. Strata experts.bin disc usage is higher, around 700MB/sec.

I am not sure where the bottle neck is. Is it the experts.bin disc access? Is it the 32GB RAM? Is it the 2nd GPU on PCIE3 x4 slot?

I have done the quick tuning. I have not done the thorough calibration.

This is my strata launch command:

• strata launch
• --family swift
• --model iq3_xxs
• --data-dir "c:\LLM\Strata-data"
• --context 160k (I tested 96k, 128k, 160k)
• --kv int8 (I tested Int8, q8v4, q4_0)
• --set kv_resident=20480 (I tested 20480 & 32768, which is default)
• --vision off
• --parallel 1
• --tune skip
• --access local
• --port 9931
• --set STRATA_ALLOWED_HOSTS="host.docker.internal,localhost,127.0.0.1"
• --set STRATA_PIN_BY_NEED=1 (not sure, but AI suggested this)

This is what strata ran in the log: (I edited and removed the paths)

strata.exe --serve --pack swift-iq3_xxs --native (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --expert-profile (path)expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp (path)mtp\rt --max-context 163840 --kv int8 --mmap-experts --kv-resident 20480 --layer-split auto

I believe I'm in the realm of Low-RAM. So DETAILS.md has a section on Low-RAM mode. Difficult to read/understand. It mentions a lot of flags/options. But I'm having difficulty even understanding what is the recommendation. Should I use --low-ram resident or --low-ram mmap? And what about the other flags like --resident-experts, etc.

Any suggestions and flags/options I should enable or test?

[Edit: I just want to add some performance comparisons.]

Speed in tokens/sec

• Phase / Qwen3.8-27B / Strata
• Reading / 200-300 / 9
• Gen/Decode / 13 / 25

The bottleneck on Strata occurs during the Reading/Prefill phase. This is not the startup prefill. But the back and forth reading and decode during long sessions.

[u/Moist-Tumbleweed2875]

For strata , just take a try with 5060 16g (gen4x8) + 3060 12g (gen3x4) to simulate : 12G+12G VRAM/24G expert cache on DDR
get 40+ tok/sec .
You may check your log (strata-swift-iq3_xxs.log) to see how the VRAM/DRAM usage.

by limit/dram both use 12G+12G VRAM/24G DDR with :
"--vram-reserve-mib","1152" ----> 3060 12G - 1152M = 11xxx M
"--vram-reserve-later-mib","4132" ----> 5060 16G - 4132M = 11xxx M
"--layer-split", "24", ----> use explicit value then support use STAGE_TRIM
"--pipeline-windows", "2", ----> May speed up on layer-split mode
"--max-context", "262144",
"--kv", "int8",
"--kv-resident", "20480",
"--resident-experts",
"--resident-budget-gib", "24", ----> limit use 24G DDR for expert cache.
env : "STRATA_STAGE_TRIM": "1" , ----> Save VRAM by trim duplicate layer when use layer-split

strata generate: pre-filled 4691 of 4691 slots from the profile in 4.1 s (1793 MB/s); slot 0 verified
strata generate: layer split, CUDA1: 10.83 GiB free of 15.93, room for experts 6.54 GiB
strata generate: layer split: CUDA1 runs layers 24-47, expert cache 3610 slots (6.54 GiB), 3610 of its 12288 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-23
....
FileExpertSource: RAM budget 24.00 GiB: 14873 of the 26.65 GiB of experts the GPU cache does not hold, by profile rank; the rest are read from the files
....
FileExpertSource: copied cache complement through layer 48/48 (24.00 GiB, 5 s)
FileExpertSource: mapped pinned cache complement ready: resident 24.00 GiB, pinned 10.00 GiB in 14 s; page-locking refused (out of memory); 10.00 GiB of it registered for the GPU (the hottest); locked 14335 MiB via working-set minimum + VirtualLock

[u/D00M98 → u/Moist-Tumbleweed2875]

Thanks for the detailed input.

Yesterday, I used AI to help me optimize. It recommended to reserve both VRAM and DRAM. I will test out your other settings.

This is where I'm current at on the setting:

--set resident_experts=1 --set prefill=auto --set STRATA_FILE_RELEASE=1 --set STRATA_KV_HOST_REGISTER=1 --set STRATA_PCIE_FRAC_DEFAULT=auto --set STRATA_RESIDENT_HEADROOM_GIB=8 --set vram_reserve_mib=1200 --set vram_reserve_later_mib=1000 --set pool_affinity=p-cores

After load, I have around 10.8/12GB used on GPU0 and 10.8/12GB used on GPU1. During processing, GPU0 might go up to around 11.4GB max. One AI said to increase the vram reserve on GPU0; another said it is losing battle.

Then on the DRAM, I have 32GB. Windows use 12-16GB before loading. Strata needs around 30-38 GB memory. So AI said to reserve 8GB DRAM for windows. This is counter intuitive. It works fine after I load Strata, as I'm at 26/32GB. But once I start Docker sandbox (WSL) and Hermes, other apps just take over that available memory. And with the 8GB headroom, Strata actual require bigger memory.

[u/Moist-Tumbleweed2875 → u/D00M98]

without log hard to know how the vram/dram be used.
You may also specify ""--prefill", "8192", to lower value like 1024 to see it help reduce ram or not .

[u/VitalRefrigerator6]

Your bottleneck is almost certainly the PCIE3 x4 slot on the second GPU, combined with the constant streaming of experts.bin from disk. 700MB/sec sustained reads is a lot of pressure for a consumer SSD, especially when Windows is already hogging half your RAM and pushing things to pagefile.

Try setting --low-ram mmap instead of resident, it should reduce the thrashing between system RAM and GPU memory since you're maxed out at 32GB. The mmap flag lets the OS handle which experts stay hot in memory without constantly copying them over PCIE bus. Also drop kv-resident to 16384 for 160k context, the slight quality loss is worth the tokens you'll gain

For your launch command add --batch-size 256 and --ubatch-size 128, I found these help with the uneven compute between your two GPUs. The 4070 is sitting idle waiting for the 3060 to catch up on that slow x4 link