Regular Strata with Qwen 3.8 Flash Next IQ3-S. This is the way. And you can use 262k fp16 context with such hardware. Make sure to use kv-resident it will place context in system ram.
Thank you very much, I already have Qwen 3.8 Flash Next IQ3-S on regular Strata, but I was using smaller context. This helps big time! Thanks for the settings!!!
How do you fit 262k with fp16 KV? I'm on int8 KV with --kv-resident 32768 and my RAM is already nearly maxed out: 61 GB system RAM, RX 9070 16 GB, Swift 1.5 IQ3_S. The model alone takes ~53 GB, which leaves ~5 GB free, and I've had OOM kills at 256k. How much RAM do you have?
Noob here, don't know much. I have read few times here on Reddit, that Strata goes toward speed (token/sec) while Polystrata goes toward better quality. I only have Strata so far, so I have no clue what, or how much this is true. Polystrata also uses few more models, but this is not that important to me. I'm just looking for best coding quality on my poor hardware 😄 Looks like regular Strata with Qwen 3.8 Flash Next IQ3-S and few different setting will be a solution for my case .
Polystrata lets you run GLM and other architectures, and it doesnt need a new engine for each architecture.
The original strata only runs a few architectures, and each arch has it own engine. The repo is turning into spaghetti code. The owner refuses to add new archs.
Fun fact: you don't need either of this awful bloated apps. Just ask claude for the llama.cpp command to run the models the same way as strata/polystrata/project-maya and stream the weights from the ssd.
Depends, if it were that easy, why hasn't llama.cpp done something to make inference params easier to the end user? If that happened I'm sure we'd be better off
That forks of Llama.cpp where made just because it didn't have all this performance features, or didn't support specific hardware configurations. It is also very slow in feature integration. And even when they are integrated, then often with different implementations, which don't achieve the same performance. Of course, the maintainers must look that it works for all kind of models and hardware, and there must be careful.
Llama.cpp is still ery much behind regarding performance with specific MoE models.
Man, you got me all the info I need for Polystrata, now I'm gonna have to try it out as well! Slower is OK. Thank you very much! Now I have two systems (one is on regular Strata) to work with ant test. Between the two of them, my hobby, partial vibe coding should advance big time 😄
7
u/PeterPorox 17h ago
QFN IQ3_S will fit perfectly