Does river handle varying feature spaces/feature evolvable streams?

[bking124]

I have recently stumbled upon River and am still learning about all the features, so please excuse me if I have missed an easy answer to my question in the documentation.

From my understanding, the models in River can handle all sorts of concept drift, but generally assume that the feature set stays the same size (as far as I'm aware). However, I am interested in handling situations where new features may pop up and old features may drop away over time. In recent literature, this seems to be called 'feature evolvable streams' or 'varying feature spaces'. Some recent research I've found:

• Learning with Feature Evolvable Streams
• Learning with Feature and Distribution Evolvable Streams
• Online Random Feature Forests for Learning in Varying Feature Spaces
• Online Learning from Data Streams with Varying Feature Spaces

So my question is: are river models capable of handling such streams? I found an old discussion (#625) on a similar topic--the consensus seemed to be that most models would ignore any new features observed at test/inference time and missing features would be handled in different ways depending on the algorithm. However, this thread was 2.5+ years ago, and I know river has likely changed significantly since then, so I thought it was worth raising this question again.

[gbolmier]

Hi @bking124, River does support dynamic feature sets. As you may know River models take dictionaries as inputs; models basically learn or predict from whatever features are provided (using the sparse nature of dicts).

There is a short mention of that in the introduction page of the docs, not sure if there's more detailed content elsewhere...

Just expect to sometimes run into cold start issues when the feature distribution changes in a way that makes the previous patterns learned by the model out of date. The model will naturally adapt to the new nature of the data after some time.

[brikin01]

Hi @gbolmier! Thank you for the response! It's very fascinating how River can support such dynamics, and I'm interested to play around with it more. I've tried out several models on a toy dataset and I do indeed see it adapting to a new feature. I'm still trying to wrap my mind around how this update is happening mathematically. Are there any good resources or articles which discuss how simple models like linear or logistic regression are able to adapt in an online fashion? I'm just trying to square what I see River doing with the theory and mathematics of linear/logistic regression that I've studied before. It's unclear to me how such adaptation could be represented in a more formal way, and a reference would be lovely. I saw a list of references about online linear model learning here, but wasn't sure if there was a particular one that really described the approach River takes.

[gbolmier]

Nice to see such enthusiasm 🙂 I personally haven't read much theoretical resources on the subject so unfortunately I don't really have any to suggest.

Taking a simple example, in the case of a linear model learning its parameters via SGD (Stochastic Gradient Descent), it's quite intuitive to see how the model adapts to a new feature:

So let's say our model is provided this new sample to learn from: x = {'a': 1, 'b': 2, 'c': 3}, where a and b are features known by the model (that is the model has learnt weights w_a and w_b for them) and c is a new feature unseen by the model so far.
The model will start by making a prediction out of this sample by calculating y_pred = intercept + 1*w_a + 2*w_b + 3*w_c. By default w_c will be initialized to zero¹. Then the model will calculate the error, the loss, and the gradient for this prediction. Using the calculated gradient it will update its different weights to correct itself, giving the new c feature a more or less important one depending on its prediction power relative to the other features.
The same process will repeat iteratively for each new sample, making the model adapt to whatever samples are provided to it.

To go further, you may have noticed that there are multiple factors that can play a role in the "sensitivity" of the model to adapt more or less fast. Coming to my mind: the learning rate, regularization, the data itself, the loss function, the optimizer, the nature of the model.

Footnotes

1. It's not really clear from the API docs but in this case the default weight initialization scheme uses zeros to initialize new feature weights. ↩

[brikin01]

That explanation helps a lot. It clarifies what occurs in the case of an added feature, but I was hoping you could also share some intuition about what occurs when a feature is no longer present. Following your example, say our model now knows features a, b, and c. Then a new sample is provided: x={'a':1, 'c':3}. In this setting, how is the prediction made and how will the SGD update be performed? Will w_b stay the same? Finally, will the model always keep track of w_b or will it eventually 'forget' that weight?

[gbolmier]

Glad it helps 😁
Great question, the model will only use the weights of the features present in the sample. So in your example, w_a and w_c. The model will not update w_b (no gradient entry for this parameter as not used in the prediction).
Now the model will not "forget" that weight in the sense that it is not erased or reset if not used for a while. Though it can be done manually pretty easily with del model._weights["b"].