combined_ftest_5x2cv: accuracy vs error rates

[AlbertoImg]

Hi @rasbt,
First of all thanks for the implementations and the teaching material. It is really appreciated.
I want to use this test is r, so I was looking at your python implementation to get an idea. I would like to ask you a technical detail that is not clear to me:
I saw that in your function "combined_ftest_5x2cv", in a classification case, you use the "accuracy" as the default scoring. When looking at the papers Dietterich 1998 and Alpaydin 1998, they mentioned this scoring ($p_i^{(j)}$) as "error rates" or "observed proportion of test examples misclassified by algorithm".
Are you using the "accuracy" scoring because the math at the end does not change (considering accuracy = 1-error rate, and the difference between the algorithms cancel this "1-" operation) ?

Thanks in advance
Best
Alberto

Your implementation:
``
if scoring is None:
if estimator1._estimator_type == "classifier":
scoring = "accuracy" <-- HERE
elif estimator1._estimator_type == "regressor":
scoring = "r2"
else:
raise AttributeError("Estimator must " "be a Classifier or Regressor.")
if isinstance(scoring, str):
scorer = get_scorer(scoring)
else:
scorer = scoring

variances = []
differences = []

def score_diff(X_1, X_2, y_1, y_2):
estimator1.fit(X_1, y_1)
estimator2.fit(X_1, y_1)
est1_score = scorer(estimator1, X_2, y_2) <-- HERE
est2_score = scorer(estimator2, X_2, y_2) <-- HERE
score_diff = est1_score - est2_score <-- HERE
return score_diff
´´