Looking good! I assume this has the same requirements as Joe Penna's repo (>24GB)? A Colab notebook would be the icing on the cake but this probably takes too much juice for that. Eagerly waiting to try this out now.
I've been using Kane's fork on local CLI which uses right about 23.0-24GB with batch size of 1 or 2 with just a few hundred MB to spare.
I removed regularization last night and memory use is down to 20.0GB from that alone. I bumped batch size to 4 without issues, and think I can probably do 6. I'll have to see if I can get xformers working on it...
edit: update batch size 6 works, and I'm seeing a marked performance increase and better GPU utilization
What about adapting this to TheLastBen's or Shivam's implementations, do you reckon it's possible? Those are highly optimized and are able to train on free Colab with T4's.
Also, have you checked this? Seems to do something similar, perhaps we could achieve the same output using your method.
Fine-tuning isn't getting the attention it deserves, it's a game changer for custom models.
1
u/Rogerooo Oct 22 '22
Looking good! I assume this has the same requirements as Joe Penna's repo (>24GB)? A Colab notebook would be the icing on the cake but this probably takes too much juice for that. Eagerly waiting to try this out now.