Using regression for orthogonality correction (Section 4.3 of paper)

[kweweli]

Hey, I read you guys paper and it is really interesting. Good work! I wanted to implement the concept for xgboost model object. I noticed that the current implementation is based on sklearn model object. I understood most of concept except the orthogonality correction.

Now, my question is, can I use regression for the orthogonality correction? I am thinking of running regression(Pii, Pjj) and Pij is the target variable. Then to get the corrected Pij, I extract the residuals by taking the difference between Pij and Pij' (the regression output).

Is there any disadvantages from using this method, or I am completely off?

Pii ---> shap interaction b/w feature i and feature i
Pjj ---> shap interaction b/w feature j and feature j
Pij ---> shap interaction b/w feature i and feature j

[j-ittner]

Dear Kweweli,

thank you so much for taking the time to read our paper and for reaching out with your thoughtful question. I appreciate your interest and it’s fantastic to hear that you want to extend this to XGBoost models.

The method you’re considering for the orthogonality correction using regression is an interesting one, but it might not be ideal for the task at hand. The reason we perform the orthogonalisation step in FACET is to separate the synergy and redundancy in SHAP interaction values, which are a combination of both. This is achieved by constructing a synergy vector that is perpendicular to the plane spanned by the two main effect vectors Pii and Pjj. This synergy vector represents the “pure” interaction effect not accounted for by the individual main effects. This vector can be determined using an orthogonal projection, a mathematically sound and appropriate operation for this kind of problem.

The regression method you suggested would indeed capture some sort of relation between Pij and the combination of Pii and Pjj. However, it might not guarantee the orthogonality required to fully separate the synergy from redundancy.

On another note, I’d like to suggest you consider experimenting with the XGBRegressorDF and XGBClassifierDF classes provided by sklearndf. This library is designed to be compatible with sklearn and provides a straightforward way to use XGBoost models directly with FACET. Thank you once again for your interest and I wish you the best of luck with your exploration of FACET. Please do not hesitate to get in touch if you have any further questions or if there’s anything more we can assist you with.

Jan