Difficulty in Implementing Model Re-Training for Time-Series Data with Pre-Trained Model

[AyushBhardwaj321]

Hello everyone,

I'm encountering an issue with implementing incremental learning for time-series data using a pre-trained model, and I'm seeking guidance on how to address it.

Description:
I have a substantial dataset consisting of 2.5 million data points, structured as follows:

                                ds    y
0        2020-10-21 12:57:47+00:00  0.0
1        2020-10-21 12:57:48+00:00  0.0
2        2020-10-21 12:57:49+00:00  0.0
3        2020-10-21 12:57:50+00:00  0.0
4        2020-10-21 12:57:51+00:00  0.0
...                            ...  ...
2591996  2020-11-20 12:57:43+00:00  7.0
2591997  2020-11-20 12:57:44+00:00  7.0
2591998  2020-11-20 12:57:45+00:00  7.0
2591999  2020-11-20 12:57:46+00:00  6.0
2592000  2020-11-20 12:57:47+00:00  6.0

[2592001 rows x 2 columns]

Objective:
My aim is to train an initial model, let's call it Model_v1, on this historical data. However, due to the large size of the dataset, training the model from scratch is time-consuming. Therefore, I want to explore the possibility of using Model_v1 as a base model and incrementally updating it with new data obtained from a real-time source, which arrives approximately every minute. Ideally, I would like to re-train the model on a weekly or monthly basis to adapt to any changes in the underlying patterns.

Challenge:
Although I've managed to save the historical model (Model_v1), I'm encountering difficulties when attempting to load it and fit new data onto it. The incremental learning process seems to be failing, and I'm unsure about the correct approach to achieve this.

Request for Assistance:
I would greatly appreciate any insights, suggestions, or best practices on how to effectively implement incremental learning with pre-trained models for time-series data. Additionally, if anyone has encountered similar challenges or has successfully implemented such a solution, your input would be invaluable.

Thank you in advance for your assistance and guidance.

[madtoinou]

Hi @AyushBhardwaj321,

You can find an example on re-training/fine-tuning pretrained model here. We recommend using the method load_weights_from_checkpoint() as it provides a lot of control over the parameters used for the second training session.

[dennisbader]

And load_weights() from a manual save (model saved with model.save()).

[AyushBhardwaj321]

Hi @AyushBhardwaj321,

You can find an example on re-training/fine-tuning pretrained model here. We recommend using the method load_weights_from_checkpoint() as it provides a lot of control over the parameters used for the second training session.

@madtoinou
Thank you for providing the example on re-training/fine-tuning pretrained models with Darts. I have a couple of follow-up questions:

1. Does this approach apply exclusively to Torch models? If so, could you direct me to where I can find information on which Darts models are built on the Torch framework?
2. I'm currently utilizing forecasting models such as ARIMA, Prophet, XGBModel, and FFT-based models. Can the method mentioned above for re-training/fine-tuning be applied to these models as well? I ask because I haven't come across any parameters for initializing these models to save model checkpoints.

[AyushBhardwaj321]

And load_weights() from a manual save (model saved with model.save()).

@dennisbader
Thank you for the suggestion to use load_weights() from a manually saved model. I've experimented with this approach and successfully loaded the initial model, trained it, and made predictions. However, I noticed that the model seems to rely solely on the fitted values for predictions, disregarding the historical trends and patterns from the loaded model.

I'm seeking guidance on how to ensure that when re-training the model, it incorporates the historical trends and patterns from the loaded model, in addition to fitting the new data. Any insights or suggestions on how to achieve this would be greatly appreciated.

[madtoinou]

1. Yes, only the neural-network based models support this approach. You can find the list here, in the "PyTorch (Lightning)-based Models" section.
2. No, these models do not support fine-tuning, they are retrained from scratch every-time you call fit(); it's one of the inconvenient of regression models.

[dennisbader]

@AyushBhardwaj321, fine-tuning only works with our neural network models as @madtoinou pointed out. There you can also adapt the learning rate and other parameters for fine-tuning so your model won't overfit on the data used for fine-tuning.

[AyushBhardwaj321]

@madtoinou @dennisbader
Thank you for the clarification.

It seems that only the neural-network based models in Darts support the approach of re-training/fine-tuning pretrained models. You can find the list of these models in the "PyTorch (Lightning)-based Models" section.

Regarding the other forecasting models such as ARIMA, Prophet, XGBModel, and FFT-based models, it appears that they do not support fine-tuning or retraining from pretrained models. These models are retrained from scratch every time the fit() method is called, which may pose some inconvenience.

Certainly! I'll keep this issue open as I may require further support during the implementation process. If I encounter any challenges or have additional questions while working with the models, I'll be sure to reach out for assistance. Thank you for your continued support and guidance.

[AyushBhardwaj321]

@madtoinou @dennisbader
Is there a comprehensive example available for the complete implementation of incremental model training using PyTorch (Lightning)-based Models? By "complete implementation," I mean a step-by-step guide covering aspects such as data retrieval, preprocessing, model feeding, model or model checkpoint saving, prediction from the saved model checkpoint, and finally, incremental model training based on previously trained models' checkpoints using new data.
I would greatly appreciate any resources or guidance on this matter.

[madtoinou]

Hi @AyushBhardwaj321,

I don't think that the documentation includes such a tutorial but you could easily combine the code snippet mentioned above to load weights from a checkpoint with any of the other example notebook such as the quickstart, this one or this one.

I would recommend checking the various examples, I am sure that you will find something very close to what you want to achieve and use it as a base.