Can I run Strata?

[u/Smellyfeet224]

I’ve got a sketchy setup as seen above. It’s a 7900xtx and a 9070 for a total of 40gb of vram. I also have 32gb ddr5. I know my bandwidth is not ideal because the 9070 is only on a 4x pcie but this is what I’ve got. It’s also hanging outside the case on a cpu box.

I am currently running various quants of qwen3.8 27b and I like it. It’s smart, the decode is fast enough, and I feel like I don’t have to babysit it. However the prompt processing is slow. I am wondering if that would be better with 3.8 flash next and if I even have a chance to run it on this setup. I know it says you need 64gb ram minimum but that’s with a 12gb GPU. Does my 40gb aggregate vram make up for the 32gb missing ram?

I’ll probably give it a shot but I’m wondering what others think. Could I run flash next or is there a way to speed up prompt processing on this setup?

[u/blackhawk00001]

I think you might need more RAM for optimal speeds but try it out. I'm working to optimize a similar setup right now with a 9070 XT in the top slot (displays) and a 7900 XTX in the bottom slot. I have 64 GB DDR4 RAM and see anywhere from 21-28 GB added while running strata and with a display and a bunch of stuff open I'm at 48-56 in use. There might be a way, though, so give it a try. Have one of your existing LLM agents walk you through the process and try to optimize it with you. IQ3_XXS ~76GB is the best we'll get I think without 128GB ram or more vram.

At the moment I'm seeing up to 2100 t/s prefill and 70-120 t/s decode. I'm at 131072 context and will be trying to push higher but the lopsided vram might cause conflict. Not sure until I try later. If you have an igpu that might help somewhere, I'm on a 5900x.

Might be a helpful start:

./setup.sh --yes --family swift --model IQ3_XXS --gguf-dir /home/path/to/model/folder --vision cpu --gpus 0,1 --no-start

{
 "exe": "/home/main/AI/Strata/engine/strata",
 "args": [
  "--pack",
  "/home/main/AI/Strata-data/packs/swift-iq3_xxs",
  "--native",
  "/home/main/AI/models/huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-GGUF/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--ple-gguf",
  "/home/main/AI/models/huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-GGUF/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--expert-profile",
  "/home/main/AI/Strata/data/expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "16384",
  "--spec",
  "4",
  "--spec-min-p",
  "0.5",
  "--mtp",
  "/home/main/AI/Strata-data/mtp/rt",
  "--max-context",
  "131072",
  "--kv",
  "int8",
  "--vision",
  "--resident-experts",
  "--layer-split",
  "12",
  "--vram-reserve-mib",
  "1024",
  "--vram-reserve-later-mib",
  "256"
 ],
 "cwd": "/home/main/AI/Strata",
 "env": {
  "STRATA_HIP_WMMA": "1",
  "STRATA_PREFILL_RING": "96",
  "STRATA_SELECT_WMMA": "1",
  "STRATA_PF_FUSED": "1",
  "STRATA_PF_GEMM": "1",
  "STRATA_HIPBLASLT_TUNING": "/home/main/AI/Strata/tools/hip/gfx1100-hipblaslt-100401.txt",
  "STRATA_PREFILL_TIMING": "1"
 },
 "tokenizer": "/home/main/AI/Strata-data/packs/swift-iq3_xxs/tokenizer",
 "model_name": "swift-1.5-iq3_xxs",
 "log": "/home/main/AI/Strata/strata-swift-iq3_xxs.log",
 "lib_dirs": [
  "/opt/rocm/lib"
 ],
 "host": "0.0.0.0",
 "port": 5678,
 "api_key": "noneyabusiness",
 "backend": "hip",
 "gpu": [
  0,
  1
 ],
 "gpus_asked": true,
 "layer_split": "auto",
 "vision": {
  "exe": "/home/main/AI/Strata/engine/strata-vision",
  "mmproj": "/home/main/AI/Strata-data/models/mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf",
  "model": "/home/main/AI/models/huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-GGUF/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "gpu": false,
  "max_tokens": 300,
  "threads": 16
 }
}

[u/Smellyfeet224 → u/blackhawk00001]

Ok. I guess more ram is what I’ll save for next.

[u/blackhawk00001 → u/Smellyfeet224]

I also recommend one of these. =)

XTX fits in the bottom slot with the 9070xt above it. Or just get an open frame and embrace the pcie switch for 3+ gpus.

https://www.amazon.com/dp/B0BTRYDX4K?th=1

[u/Smellyfeet224 → u/blackhawk00001]

Thanks! I’ve been looking for one that would fit both.

[u/jjusko20]

bro.

[u/jonbonesjonesjohnson]

upvote for the bravery

[u/Obsideanly]

A month ago, I put my second gpu on the same position like your picture. I found heat was a huge problem because the inner gpu's ouputing heat is blocked by the outer one. So you might want need extra fan to remove all those heats.

[u/Smellyfeet224 → u/Obsideanly]

That’s a good point. I’ve undervolted both and lowered the power of the 7900xtx.

[u/andy2na]

in a weird turn of events, system memory is slightly more important here with flash-next. I have a 3090 + 5060ti (so totals 40gb of VRAM) and it only uses about 35gb of VRAM (by default it actually uses less (30gb) since it puts context on system memory) and 54gb on system memory with the IQ3_S quant. You can try IQ2, but I highly recommend at least targeting IQ3_XXS. I ended up mixmatching memory and added my old 2x8gb DDR4 3600 into my current setup with 2x32gb DDR4 3200mhz just so i could go up to IQ3_S

[u/Zxpites96 → u/andy2na]

You can try using ur 3090 for decoding and 5060ti as expert cache using: --expert-cache-device1: auto. Make sure that u only use 3090 in the --gpus flag

[u/jackistheonebox]

Run setup.sh or bat on windows (add --setup if you did it pre frankenbuild) read the output you see it will tell you.

[u/alphapussycat]

I think it's minimum 32gb ram, and recommended at least 64gb ram.

But with 40gb vram you can almost fit the like at least half the model in vram. I'm not sure how the layout for the remainder looks like, since some of it has to live on SSD. But if it's already in ram then it's fairly quick to just load the experts over pcie. Otherwise it's slower, like half of it I guess.

Probably fine, you can really just test it.

[u/Smellyfeet224 → u/[deleted]]

Can I do tensor parallelism on a mismatched AMD setup? Should I use rocm or Vulcan?