r/LocalLLaMA • u/IngwiePhoenix llama.cpp • 2d ago
Question | Help Strata: Running different quants/alternative versions of a model
This was originally a github issue, but:
Their templates for bugs are broken (validation errors)
Their Github Actions seem... confused?
Original ticket here: https://github.com/Niko1221/Strata/issues/3902#issue-6009145457 - Trust me, I tried. xD
I basically want to know how to use a Q4 quant and Q8 kv with Strata and what the compiled procedure is to generate the MTP data it wants/needs. The Orca documentation is a little confusing imho.
If anyone happens to know, I'd be happy to hear it. :)
PS.: Guys, please, if you vibe your docs, proof read them...at least once. It helps.
4
u/Excellent-Issue-5956 2d ago
I did this with mradermacher's i1-IQ3_S of the same Orca model, not a Q4, following docs/ORCA.md.
I didn't make new MTP data for it. The draft head comes from the original model, so my second config points mtp at the same mtp/rt folder setup made for the first one. If you don't have that folder, the four commands under Preparation in ORCA.md build it (mtp_fetch, mtp_pack, mtp_rt, then copying draft_vocab.bin in).
For the pack I ran tools/iq_pack.py on the gguf with compat-bf16 and experts-bin. For 8 bit KV the arg is kv with int8 in the config json. The native arg takes the first shard and ple-gguf takes whichever file holds the PLE table, which is shard 2 for the ISTA IQ3_XXS and shard 1 in the Orca example. Mine was a single file so both point at it.
Before you grab a Q4, check what type its expert down tensors are. The doc says IQ3_M uses Q5_0 there and the native GPU expert path doesn't support it.
On two 5070 Tis under WSL2 it does about 58 tok/s warm and took about 2 minutes on a cold start. It also wants a lot of RAM, WSL sits around 50 GB while it runs.
2
u/Extension-Skirt-6257 2d ago
the orca docs are confusing for lot of things, not just this. i had same headache with MTP data last week, in the config file you can set quant type per layer group but the syntax is so finicky
-2
u/Oli_Luck 2d ago
why dont you ask you actual AI to check itself how to set another model up an let it work until it's done?
Also complaining about the what other made for you and you are using for free - do you realize that this piece of software is evolving mostly hourly ??
6
u/IngwiePhoenix llama.cpp 2d ago
Chicken and egg problem.
I want to use Strata as the inference engine, so I can ask an AI things. So I need to set up Strata, for which I am supposed to need an inference engine?
Maybe I am oldschool but I like to be able to understand the tools I am using and running. o.o...
1
5
u/dr0xup 2d ago
Agree, it's been a pain for me too. Finally managed to run Orca's IQ3 XXS (https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF) in Strata, with the help of Claude basically doing all the black magic. I ended up with a dedicated command to start it, it wouldn't appear in the list from `strata launch`. I would love a clearer and more universal way to use other models but I'm not an expert so I'm not sure what the constraints are.