Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract probability for a tresholded model #981

Closed
tpoisot opened this issue Nov 4, 2022 · 6 comments
Closed

Extract probability for a tresholded model #981

tpoisot opened this issue Nov 4, 2022 · 6 comments

Comments

@tpoisot
Copy link

tpoisot commented Nov 4, 2022

When calling predict on a BinaryThresholdPredictor, is there a specific incantation that can give the probability of the positive class? Is there a way to "unwrap" the underlying model?

@ablaom
Copy link
Member

ablaom commented Nov 6, 2022

No, I think if you want to get probabilisitic predictions for the original (atomic) model, then you need to train the atomic model on the data. You can't eat your cake and have it too.

I don't know your use-case, but I suppose you could roll your own wrapper that preserves the behaviour of predict (keeps it probabilistic) but overloads predict_mode to do the thresholding. You can specify predict_mode as the operation in things like evaluate! and TunedModel.

using MLJ
import CategoricalDistributions

mutable struct Thresholder <: ProbabilisticComposite
    model
    threshold::Float64
end

# to do the thresholding:
function deterministic(y_probabilistic, threshold)
    classes = CategoricalDistributions.classes(y_probabilistic)
    map(pdf.(y_probabilistic, classes[2]) .> threshold) do above
        above ? classes[2] : classes[1]
    end
end

function MLJ.fit(thresholder::Thresholder, verbosity, X, y)
    Xs = source(X)
    ys = source(y)
    mach = machine(thresholder.model, Xs, ys)
    y_probabilistic = predict(mach, Xs)
    y_deterministic = node(y -> deterministic(y, thresholder.threshold), y_probabilistic)

    network_machine = machine(Probabilistic(), Xs, ys;
                              predict=y_probabilistic,
                              predict_mode=y_deterministic)

    return!(network_machine, thresholder, verbosity)
end

X, y = make_moons()
DecisionTreeClassifier = @load DecisionTreeClassifier pkg=DecisionTree
thresholder = Thresholder(DecisionTreeClassifier(), 0.7)

mach = machine(thresholder, X, y) |> fit!
predict(mach, X)      # probabilistic predictions
predict_mode(mach, X) # thresholded predictions

julia> evaluate!(
       mach, 
       operation=[predict_mode, predict], 
       measure=[accuracy, log_loss],
       resampling=CV()
       )
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
PerformanceEvaluation object with these fields:
  measure, operation, measurement, per_fold,
  per_observation, fitted_params_per_fold,
  report_per_fold, train_test_rows
Extract:
┌────────────────────────────────┬──────────────┬─────────────┬─────────┬───────────────────
│ measure                        │ operation    │ measurement │ 1.96*SE │ per_fold         
├────────────────────────────────┼──────────────┼─────────────┼─────────┼───────────────────
│ Accuracy()                     │ predict_mode │ 0.9670.0345  │ [0.96, 0.92, 0.9 LogLoss(                       │ predict      │ 1.21.24    │ [1.44, 2.88, 2.8 
│   tol = 2.220446049250313e-16) │              │             │         │                  
└────────────────────────────────┴──────────────┴─────────────┴─────────┴───────────────────
                                                                            1 column omitted

@ablaom
Copy link
Member

ablaom commented Nov 6, 2022

(After JuliaAI/MLJBase.jl#853, the preferred way of exporting the learning network is to replace MLJ.fit definition above with the simpler

function MLJ.prefit(thresholder::Thresholder, verbosity, X, y)
    Xs = source(X)
    ys = source(y)
    mach = machine(:model, Xs, ys)
    y_probabilistic = predict(mach, Xs)
    y_deterministic = node(y -> deterministic(y, thresholder.threshold), y_probabilistic)

    (predict=y_probabilistic, predict_mode=y_deterministic)
end

and change the subtyping Thresholder <: ProbabilisticComposite to Thresholder <: ProbabilisticNetworkComposite.)

@tpoisot
Copy link
Author

tpoisot commented Nov 7, 2022

No, I think if you want to get probabilisitic predictions for the original (atomic) model, then you need to train the atomic model on the data. You can't eat your cake and have it too.

In the specific case I used to explore this, the atomic model was wrapped into a BinaryThresholdPredictor, and then into a TunedModel to optimize the threshold -- so I think (if I understand correctly what TunedModel and BinaryThresholdPredictor do from the documentation) that the atomic model is trained.

@ablaom
Copy link
Member

ablaom commented Nov 7, 2022

Yes, it is trained, but the public API doesn't give you a way to produce probabilistic predictions. It is possible to get them without retraining, but it's a hack (non-public API).

@tpoisot
Copy link
Author

tpoisot commented Nov 7, 2022

Got it, thanks!!

@ablaom
Copy link
Member

ablaom commented Jun 9, 2024

Closing as question resolved.

@ablaom ablaom closed this as completed Jun 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants