How to use Cleanlab as Pre-Processing step with time consuming model training?

[daMichaelB]

Hello everyone,

i am just getting started with Cleanlab and really enjoy it so far! Thank you for the awesome work first of all.

I am currently trying to integrate the LearningWithNoisyLabels(clf=MyPytorchModel()) into my Pytorch Training.
My idea was to

• use Cleanlab to filter out noisy labels with cross validation
• after filtering out the noisy labels i use the rest of the samples for my training routine

lnl = LearningWithNoisyLabels(clf=MyPytorchModel())
lnl.fit(index_list, noisy_labels)

# do magic to filter out noisy samples

# go on with my training routine and clean samples

The following is now unclear to me:

• Can i now use lnl.noise_mask to drop the noisy labels?
• Can i see which labels Cleanlab would suggest for the noisy labels ?

[daMichaelB]

OK i found that training on clean samples is done at the end of the .fit() method already:

self.clf.fit(x_pruned, s_pruned, sample_weight=self.sample_weight)

I think it can make sense to filter out the noisy samples with a time efficient training routine, like

• training on smaller images
• only train the CNN's head
• fewer epochs

And afterwards do the real time consuming training with clean labels on bigger images / more epochs / full CNN. Does that make sense?

[cgnorthcutt]

Hi @daMichaelB . This is a great suggestion. I think this is a great feature to add and we'll keep this in mind. One trick you can use for now is to obtain the label errors yourself via cleanlab.pruning.get_noise_indices() using a smaller model trained with fewer epochs. But note that if you use a significantly worse model, it will affect the quality of the errors found. As a rule of thumb, I recommend training a model within at least 5% accuracy of your fully-trained, top performing model.

In response to your original question, you can use packages like https://github.com/skorch-dev/skorch to auto-integrate pytorch models into cleanlab without any custom code.

[daMichaelB]

Hi @cgnorthcutt . That makes sense! I will try it the next days and hope to find a good compromise between the training time and quality of label errors. Thank you for your response and advice!

[addisonklinke]

I was wondering the same thing with regard to using models which are time-consuming to train. In the paper (pg. 12 / sec. 5.1) the authors explain their training procedure for the CIFAR-10 experiment

All models are trained using ResNet-50 with the common setting:
learning rate 0.1 for epoch [0,150), 0.01 for epoch [150,250), 0.001 for epoch [250,350)

Training ResNet-50 for 350 epochs has got to be fairly time-consuming even on a high-end GPU, so I am curious @cgnorthcutt did you repeat this entire process for each cross validation fold? If so, how many folds were used in total? I could not find those details in the paper, only this mention a few sentences later

For our method, we fix its hyper-parameter, i.e. the number of folds in cross-validation across different noise levels, and do not tune it on the validation set.

Are there alternative CL strategies besides cross validation? For instance

1. Run one training with one validation set (large enough to have many samples each of label)
2. Perform confidence counting and calculate the joint Q on this validation set only
3. Use in-sample probabilities to compute psx for the remaining training samples
4. Multiply the percentages in Q by the whole dataset size (train + val) to determine how many samples to prune

In this approach, I think the estimate of Q would be reasonably accurate (relative to the unknown latent) since it's still based on hold-out probabilities. However, I would be concerned about steps 3 and 4 doing a poor job of ranking samples for pruning due to the use of in-sample probabilities

[cgnorthcutt]

Hi @addisonklinke , Let's think through your approach together, especially step 3. In-sample probabilities are mostly going to be 1 if label, else 0 for many training paradigms and loss functions (like cross entropy with many epochs of training). So they won't provide much value -- you can read more about how models (esp neural networks) can overfit to random labels and produce 0 and 1 probabilities even for the wrong labels if you use in-sample probabilities here: https://arxiv.org/pdf/1611.03530.pdf

Some solutions instead are to use pre-built bagging solutions that train an ensemble by training on folds. This gets you two birds with one stone -- you can compute the out of fold probabilities and also train an ensemble. An example of a package that does this for you is https://github.com/awslabs/autogluon

Another option is to use a pre-trained classifier which will guarantee out of sample probabilities, but this fails if there is significant distribution shift so be careful.

Some good news: we are working on a way to automate finding out-of-sample probabilities efficiently for our CleanLab users and plan to release this sometime in late 2022.

[addisonklinke]

Yes that's about what I figured for in-sample probabilities. In your paper, did you retrain the ResNet procedure from scratch across multiple validation folds?

Unfortunately in my domain-specific computer vision task, a pre-trained classifier is not available. For image data, it seems that a CNN is typically the most accurate option for classifying out-of-sample probabilities, which I assume improves the final quality of the pruned dataset. However, for computational reasons it seems much more practical to go with a bagging solution (as you mention) or other traditional (non-deep) ML model so that the training time becomes feasible to repeat across multiple folds. Do you agree that is a valid interpretation of the tradeoffs? Of course, I am sure there are certain computer vision tasks where traditional ML outpeforms CNN