-
-
Notifications
You must be signed in to change notification settings - Fork 189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Overtrain fix #431
Overtrain fix #431
Conversation
now overtrain works as it should. after training for lowest_value+overtrain_threshold, if there is no decrease in lowest_value, it overtrains and the train stops.
added saving best_epoch every time lowest_value changes |
I think this doesn't work, I started training with 10 overtraining threshold and it started saving a lot of models randomly but then it removed all saved models and the training was so slow
|
You said this fix doesn't work but I would like to summarize how it works. From the screenshot I see that you have set it to save every 10 epochs. If you set the overtraining threshold to 10 epochs, this means. Save another 10 epochs after the last recorded best_epoch.pth file and if there is no improvement, finish training because the model is overtraining. and every time a new best_epoch is found it deletes the old best_epoch file because the previous best_epoch.pth file is no longer the best epoch. So to summarize, if current_epoch > best_epoch+overtraining_threshold_value stop training bc of overtraining. and every time a new best epoch is found, save best_epoch.pth and delete the previous best epoch file. |
We're gonna merge this pull request and give it a spin. If the overtraining detector looks sharper, we'll roll with the changes. |
now overtrain works as it should. after training for lowest_value+overtrain_threshold, if there is no decrease in lowest_value, it overtrains and the train stops.