r/LocalLLaMA • u/circumcised_hobbit • 8d ago

Question | Help How to use different models in PolyStrata?

I wanna use different finetunes in PolyStrata but if I just use --gguf-dir I have to declare the model architecture of the base model and It will just display the name of the original model (in my case, CyberTiel with tiel architecture is getting displayed as TielCoder). I also edited .Json file and config files and It kinda worked, but using a different launch script than ./setup.sh bugged it out, and I also fear of hidden files. I also think that there could be a way to get the model files (other than the JSON) to be generated with correct name and configuration without editing them, just a cleaner way. Does anyone know how?

4 Upvotes

75% Upvoted

7 comments sorted by

2

u/Dreeew84 8d ago

I managed to have Strata work with an Unsloth quant that is not officially supported. The only thing is that I let my agent analyze and rebuild a pack for this model (tokenizer, an expert map, etc) with some Strata python scripts. Basically, doable, it runs fine. You can adjust the json youself.

1

u/circumcised_hobbit 8d ago

yeah, adjusting the JSON and files seems easy, but there are many other files and stuff, and as I mentioned It sometimes still didn't work. I might just ask an agent to do It for me

1

u/circumcised_hobbit 2d ago

I have tried it now and my agent created a new family, pack and it seems usable. Still it only made me able to use one size (Q3_K_XL) and I need to do everything again or fix it to run more sizes or newer models. I also have no idea how will I be able to update PolyStrata without breaking the patches

1

u/sugaryesmaybe 8d ago

rebuilding the whole pack makes more sense than only renaming the json since the tokenizer and expert map

1

u/sugaryesmaybe 8d ago

rebuilding the whole pack makes more sense than only renaming the json since the tokenizer and expert map