Repository navigation
on the textual inversion training colab, a way to output .bin file before the training is fully completed #524
Description
since the training usually takes few hours, and for different datasets, objects or styles, you can adapt with different training parameters, is there a way to output the .bin -let's say- every 1000 steps or so?
that would be very useful to check if the training is going in a good direction and maybe you can economize your gpu/usage if you are happy with the results.
it's very frustrating when you are 1000steps to complete a 10000steps training and runtime disconnects ending with hours of hours of training gone without a checkpoint.
also the ability to somehow save and resume the training would be a dream.
Metadata
Metadata
Assignees
Labels
No labels
Activity
patrickvonplaten commentedon Sep 17, 2022
cc @patil-suraj
isamu-isozaki commentedon Sep 25, 2022
HI! I implemented this for my project and I'm happy to make a pr here if you want! It can also log images at fixed intervals
isamu-isozaki commentedon Sep 26, 2022
I think I did it here #645
patil-suraj commentedon Sep 28, 2022
Thanks a lot for the PR @isamu-isozaki, taking a look now!
isamu-isozaki commentedon Sep 28, 2022
@patil-suraj Np! Glad to help.
patrickvonplaten commentedon Oct 25, 2022
@patil-suraj - re-pinging you here
isamu-isozaki commentedon Oct 25, 2022
@patrickvonplaten @patil-suraj hullo. I made a pr and I think it already got merged.
patrickvonplaten commentedon Oct 26, 2022
Super, closing this one then!