Custom Loss function?

[savinay]

Is it possible to use a custom loss function with sklearn models?

[glemaitre]

Most of the time this is not possible. Could you mention which estimator where you would like to tune the loss function?

[kirk86]

Don't you think this should change, what's the purpose of the library if it doesn't allow for flexibility on the user part to tinker with different ideas, plus there's been almost 4 years since this post https://bit.ly/3xi18Gi?

I had to spent a couple of days just to figure our how to add a custom loss to GBDT, and still I'm not sure if I've done it properly since there's not documentation on this.

Also documentation on custom scoring vs custom loss should be more evident regarding their differences, what are their purposes and when and where are they used.

[lorentzenchr]

Don't you think this should change, what's the purpose of the library …

A plain NO.
The purpose of this library is, among others,

Simple and efficient tools for predictive data analysis

I would be interested in use cases for other loss functions.

Let me conclude with 2 points:

• Incorporating the pinball loss in linear models, i.e. QuantileRegressor, was highly non trivial as it needs completely different and more complex solvers!
• I don’t recommend it, but one could use/inherit from the private BaseHistGradientBoosting and initialize with one‘s own loss function inheriting from BaseLoss.

[savinay]

Hi,

I am looking to add custom loss function for Ridge Regression, Random Forest Regressor, SVM Regressor, Elastic Net Regression.

[lorentzenchr]

@savinay Which loss function do you have in mind?
Loss functions are hard coded to speed up the fitting time. For linear models, different estimators use different loss functions, see e.g. https://scikit-learn.org/stable/modules/linear_model.html#generalized-linear-regression and https://scikit-learn.org/stable/modules/linear_model.html#quantile-regression. There are also to specify predefined losses for random forests and gradient boosting.
BTW, ridge regression with another loss than squared error would not be (linear) ridge regression anymore:smirk:

[lorentzenchr]

Is there a reason for those loss functions? Can you tell what the Bayes rules are for those?

[leenamurgai]

Thank you @lorentzenchr. Yes, there is a reason... I think generalised entropy indices can be seen as the complete set of subgroup decomposable utility functions. Subgroup decomposable means that for any partition of the population, it can be additively decomposed into the sum of a between-group component (which measures the amount of variation between the subgroups) and a within-group component (which measures the amount of variation within the subgroups). It turns out that for $\alpha=1/2$, the relative contribution to the cost from the between-group component is maximised.

I'd like to compare group fairness metrics for the same model trained on the same data with different cost functions relating to different values of alpha (for reference $\alpha=0$ corresponds to the cross entropy loss and $\alpha=2$ corresponds to mse). I'm hoping that I will find that $\alpha=1/2$ results in the fairest model. That's the theory. I'd be grateful for your advice. Ideally I want to do this for linear and logistic regression for a variety of datasets and different values of $\alpha$. I have my own (not so well tested) versions of these models but if there's another option, I'd be grateful to know.

For general $\alpha$:

def clf_alpha_loss(y, p, alpha):
    if alpha == 0:
        return - (y * np.log(p) + (1 - y) * np.log(1 - p))
    if alpha == 1:
        return y * p * np.log(p) + (1 - y) * (1 - p) *np.log(1 - p)
    return (y * np.power(p, alpha) + (1 - y) * np.power(1 - p, alpha)) / (alpha * (alpha - 1))

def reg_alpha_loss(y, p, alpha):
    if alpha == 0:
        return -np.log(np.absolute(p - y))
    if alpha == 1:
        return np.multiply(np.absolute(p - y),np.log(np.absolute(p - y)))
    return np.power(np.absolute(p - y), alpha) / (alpha * (alpha - 1))

[lorentzenchr]

@leenamurgai Thanks for the explanation.
The problem with those losses is that it is unclear what they estimate (what property/functional of the true distribution $F_{Y|X}$). The clf_alpha_loss, for instance, does not estimate true probabilities anymore, i.e. the argmin of the expected loss is not $p=E[Y]$ but something else. In fact, of all power loss functions for binary classification of the form $L(Y, m)=Y (1-m)^r + (1-Y) m^r$ with model prediction $m$ and some positive power $r$, only $r=2$ gives a proper scoring rule, i.e. a loss function with Fisher consistent argmin for the true probability $p$, see Buja, Stuetzle, Shen (2005) "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications" Chapter 5 (they call $m$ as $q$, you called it $p$, I called $p=E[Y]$, so notation may be confusing.)

[leenamurgai]

Hi @lorentzenchr, makes sense, you want your optimised model to be an unbiased estimator, thanks for the link. Would have been nice if this magic cost function existed.

[lorentzenchr]

@leenamurgai If that paper interests you, you might also be interested in Gneiting, Raftery "Strictly Proper Scoring Rules, Prediction, and Estimation" (among others, gives connection to generalized entropy) and Gneiting "Making and Evaluating Point Forecasts" (consistent scoring functions for point forecasts).

[lorentzenchr]

The original question

Is it possible to use a custom loss function with sklearn models?

has been answered: No (not without a bit of hacking).

I'll close the discussion and thank for all contributing to it, in particular the persons contributing their use cases and reasons for it.