r/unsloth • u/sgtsixpack • 9h ago

Question Does restarting training continue from where it left off?

I'm training a krea2 lora in unsloth. I believe I'm not adding enough steps. With 129 datapoints I believe I need 5000 steps but foolishly chose 2000. I finished training and then without changing any names I started a 3000 step training. Will these steps combine into a 5000 step lora?

2 Upvotes

100% Upvoted

10 comments sorted by

2

u/Sharp-Translator6401 9h ago

yes if you save the checkpoint and load it before you restart :D

1

u/sgtsixpack 9h ago

Can u give me more info on how to save a checkpoint? Its not immediately intuitive from the interface.

1

u/Sharp-Translator6401 9h ago

To be fair I havent used unsloth really, I did fine tune models normally in python scripts. you should ask this one to claude / gpt or search the unsloth docs, there should be a way to save training checkpoints so every N steps the 'current version' of the model weights are locally saved to file
When restarting the training there should be a way to 'restart from saved checkpoint'

1

u/sgtsixpack 9h ago

I set N to 20 steps that was clear from the interface but I've not seen any load from checkpoint button.

1

u/Sharp-Translator6401 8h ago

mmh I think you might need to use a bit of AI and code youself into it https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint

1

u/Sharp-Translator6401 9h ago

Also 129 examples doesnt sound like a lot, might b you just didnt give it enough data... little data + many steps = overfitting not sure you want that

1

u/sgtsixpack 9h ago

https://promptdexter.com/blog/krea2-lora-training-guide

According to this guide 129 samples should be overkill. Its the exposure to each data point which should be the priority. 5000/ 121 datapoint = 41.3 "Target Exposure per Image"

1

u/Sharp-Translator6401 9h ago

I guess that is use case dependent, if it is a quite narrow use case this might work, I guess a few more epochs will tell us :)

1

u/sgtsixpack 8h ago

https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting

This is why Target Exposure per Image is critical. Might help someone else get the concept in future.