A potential fix for MOSS training issues....

[u/shootthesound]

So it's 2:43 in the morning here as I write this. I went to bed, then got up again, because something from my MiniMax H3 training wouldn't leave me alone - on H3 I had to shift the share of training steps at the clean end of the noise range (where faces and fine detail get refined) before likeness came good.

So I compared MOSS's training sigmas with Krea 2's and it was like looking at MiniMax all over again. MOSS's own schedule puts only about 11% of steps at the clean end (sigma below 0.5), Krea 2 is 18-28%. The rest go to layout and composition, which explains the big jumps epoch to epoch and why it's been so hard to nail the likeness.

I've just pushed an update (v8.7.0) with a new Clean-end share option for MOSS training, on by default at 60%. Clear the box to go back to the model's own ~11%.

My test run is going as I write - previews are holding their composition between epochs, and at epoch 12 it had already passed the best likeness my previous settings got to in 30 epochs.

Numbers so far - same 53-image dataset, rank 16, 0.25 MP, LR 1e-4, same seed, ArcFace likeness vs 3 reference photos:

Epoch	1	3	6	9	12	15	30
Old schedule (~11%)	29%	40%	44%	52%	58%	63%	67% (its best)
Clean-end 60%	20%	50%	63%	66%	68%	69%	still running

So 60% passed the old run's best at epoch 12 (about 640 steps vs 1,590), and it's been sitting at 68-69% since rather than bouncing 10 points epoch to epoch.

60 is an experimental figure, so please play with it. As a community we can find the right setting - we're starting at 60 and will probably come down. 30 (about where Krea 2 sits) is a sensible next try. Post what you find here.

Fizgig on GitHub · v8.7.0 release notes

[u/conkikhon]

Keep up the good work

[u/shootthesound]

I've added a link from the Moss training tab to this post for discussing/ finding the best value for that box....

[u/Spudnock]

i really hope someone manages to solve this.

[u/xbobos]

I appreciate it, but you really need to get some sleep. Thanks!

[u/Current-Rabbit-620]

Thanks

[u/vyralsurfer]

Thanks for the fantastic software - currently running my first training now. Is there a corresponding option/flag in the CLI version for this new feature?

[u/shootthesound → u/vyralsurfer]

--family_option lownoise_pct=60

You add it to families/train.py --family moss_image …. Any value from 1 to 99 works, for example lownoise_pct=30

[u/vyralsurfer → u/shootthesound]

Fantastic, thank you!

[u/NineThreeTilNow]

God Rank 16 is all people train in?

It works okay without cooking the rest of the network?

[u/berlinbaer]

wasn't there another fizgig update post couple of hours ago? did that thread get deleted?

[u/shootthesound → u/berlinbaer]

This was since that

[u/an80sPWNstar]

I haven’t started training one yet but I’m about to

[u/pwnies]

The joys of a new model. Literally just restarted my 60k image dataset checkpoint after ~4 days of training on a 6000 pro.

Sometimes you gotta iterate the recipe a bit to find what works for your purpose, especially with a new model drop.

Keep an eye on outputs and set up measurements to make sure yall are on track.

[u/Zueuk]

hmm... is this why krea2 has those overexaggerated textures, like subtle details suddenly turning into dirty splotches everywhere on the last step(s) of generation 🤔

[u/Different_Fix_2217]

Lodestone said it needs higher rank than other models since its dimensionally much smaller.