Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Scaler with rolling/expanding window to eliminate look ahead bias #1540

Open
tuomijal opened this issue Feb 6, 2023 · 11 comments · May be fixed by #2021
Open

[FEATURE] Scaler with rolling/expanding window to eliminate look ahead bias #1540

tuomijal opened this issue Feb 6, 2023 · 11 comments · May be fixed by #2021
Labels
feature request Use this label to request a new feature improvement New feature or improvement

Comments

@tuomijal
Copy link

tuomijal commented Feb 6, 2023 •
edited

Problem description
Currently, Scaler transforms input series "globally", meaning that all values of the input vector are considered:

from darts.dataprocessing.transformers import Scaler
from darts import TimeSeries

scaler = Scaler()

s = [1, 2, 3, 4, 5, 6, 7, 8, 9]
s = TimeSeries.from_series(s)

s_norm = scaler.fit_transform(s)
s_norm.pd_series()

0    0.000
1    0.125
2    0.250
3    0.375
4    0.500
5    0.625
6    0.750
7    0.875
8    1.000

This is not a problem if data is manually split into train and test sets and scaler is fitted with training set.

However, if we go on to use this vector as an input to historical forecasts, we risk introducing look ahead bias into analysis (at least when performing normalization). To eliminate this bias, rolling window approach is sometimes used https://arxiv.org/abs/1907.09452.

Describe proposed solution
One solution would be to add parameters to scaler like so:

class Scaler(InvertibleDataTransformer, FittableDataTransformer):
    def __init__(
        self, scaler=None, type="Global", window=None, name="Scaler", n_jobs: int = 1, verbose: bool = False
    ):
   """
   Parameters
   ----------
   type
        The type to scale the data. Options:
        "Global" uses all data points 
        "Rolling" uses rolling window with window size specified by 'window' parameter
        "Expanding" uses expanding window with initial window size specified by 'window' parameter
    window
        Size of window if type is either "Rolling" or "Expanding"
    """

Describe potential alternatives
Another option is to integrate this functionality to historical_forecasts and backtest functions. This might be convenient because parameters above could be inferred by the desired backtesting setup.

Additional context
Thank you again for excellent software!
@tuomijal tuomijal added the triage Issue waiting for triaging label Feb 6, 2023
@dennisbader
Copy link
Collaborator

dennisbader commented Feb 10, 2023 •
edited

Another way would be to allow users to pass some (of our existing) transformers for the target and covariates to historical_forecasts.
Then we could simply refit transform on each train eval split.

@hrzn
Copy link
Contributor

hrzn commented Feb 10, 2023 •
edited

Thanks for raising this excellent point @tuomijal. I personally like the solution of @dennisbader as it would remove the need for users to use the windowing exactly right - they wouldn't have to worry about it, only specify which kind of scaling they want when calling historical forecasts.

@tuomijal
Copy link
Author

I agree, the solution proposed by @dennisbader is the most elegant one 👍🏼

@dennisbader dennisbader mentioned this issue Feb 10, 2023
Open
@madtoinou madtoinou added feature request Use this label to request a new feature improvement New feature or improvement and removed triage Issue waiting for triaging labels Feb 22, 2023
@hrzn hrzn added this to To do in darts via automation Feb 22, 2023
@hrzn hrzn removed this from To do in darts Feb 22, 2023
@JanFidor
Copy link
Contributor

Hi @dennisbader @madtoinou ! The PR looks cool, could I pick it up?

@madtoinou madtoinou mentioned this issue Apr 4, 2023
Open
@madtoinou
Copy link
Collaborator

Hi @JanFidor,

historical_forecast() currently contains several bugs that are being fixed, and will undergo a considerable refactoring after the next release. I would recommend waiting for these changes to be merged before working on this very interesting feature.

@JanFidor
Copy link
Contributor

JanFidor commented Apr 4, 2023

Sure things, If there's a chance to avoid major merge conflicts, I'll happily take it. I'll keep my eyes peeled for the new release!

@madtoinou madtoinou mentioned this issue Jul 5, 2023
Closed
@madtoinou
Copy link
Collaborator

@JanFidor

Refactoring of historical forecasts has just been merged on the main branch, if you still have time to work on this, you can go ahead!

The logic is now found in two different places: ForecastingModel.historical_forecasts and RegressionModel._optimized_historical_forecasts, make sure to implement this in both. If you need, you can implement this feature in utils/historical_forecasts/utils.py and call it in the two methods.

@JanFidor
Copy link
Contributor

Hi @madtoinou, thanks for reminding me, this issue totally slipped my mind! Small heads up, I might have slightly less time going forward, but I'll happily give it a go. I already browsed the RegressionModel._optimized_historical_forecasts method and noticed that it's only used with pretrained models (I think that in this case data leakage would have to be prevented outside of historical_forecasts.), so I wanted to ask you whether I'm missing something or if I should just add a Scaler as an unused parameter for now.

@madtoinou
Copy link
Collaborator

Indeed, we decided to optimize this method step by step and the "retrain" logic was a bit harder to support directly.

I think that the main source of data leakage is the processing/transformation of the entire input series instead of just the part available/used for the latest historical forecast (for both retraining and/or inference). So the logic should be contained in historical_forecasts().

@madtoinou madtoinou mentioned this issue Sep 22, 2023
Closed
@JanFidor JanFidor linked a pull request Oct 6, 2023 that will close this issue
Open
@madtoinou madtoinou mentioned this issue Dec 27, 2023
Closed
@j-adamczyk
Copy link

Any news on this? Lack of this feature means that if backcast or other more involved testing procedures are needed, Darts is quite unusable, since those transforms are really necessary (e.g. differencing, scaling).

@Joseph-Foley
Copy link

Yeah, I'm also quite keen on this being part of darts. I was hoping the pipeline class would function more like sklearns pipeline where transformations and models can be bundled together. Then if the pipeline class had backtest / historical_forecast we could be sure of no data leakage during backtesting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request Use this label to request a new feature improvement New feature or improvement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/scalar with window JanFidor/darts
7 participants
@hrzn @tuomijal @madtoinou @Joseph-Foley @j-adamczyk @JanFidor @dennisbader