CleanLab with Pseudo Labels #84
Replies: 1 comment 2 replies
-
|
Hi @filippoBUO apologies for the late response, but just wanted to check if you're still interested in this topic! Your pseudolabeling use-case sounds super interesting, and cleanlab's label error identification algorithms definitely share similarities with ideas in pseudolabeling. If possible, could you share a minimal code example to help us make the discussion more precise? In particular, I'm not sure how exactly you are artificially injecting noise during the pseuodlabeling process. If you artificially inject noisy labels at the end where you randomly flip some pseudolabels to incorrect other labels, I would imagine cleanlab can easily discover these noisy examples. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jwmueller Sorry I think I didn't explain myself correctly when I referred to the injection of Noise. Now I'll try to be a little bit clearer. What I meant is that cleanlab might be of great help to make a pseudo-label algorithm more robust. In fact the labels generated in pseudo labelling will inevitably contain some noise. The idea would be to use cleanlab to detect part of that noise (either at the end of the all pseudo-labelling process or at every iteration of it) thinking of it as a supervisor to the process. This could be a more valid alternative to human annotation than a simple pseudo-labelling algorithm. Moreover to stay on topic, clenalab could also be used in an active learning process to correct possible errors made by the human annotator. At the time of that post I think I should have gone deeper on all the details of cleanlab but what I remember is that in this case when the noise is "created" by the classifier itself, cleanlab had some problems in identify it as compared to the case when the noise is injected artificially which as you said cleanlab can easily discovered it. I didn't mean that I inject noise in the pseudolabelling process, in this case the noise is there because the classifier is self-learning the labels. Unfortunately I'm not anymore in studying fast annotation of unlabeled dataset, but the idea might be worth investigating it a little bit more for you guys. |
Beta Was this translation helpful? Give feedback.
-
|
Ah yes we are definitely investigating both of these areas! In particular, we'll very soon be adding some new functionality for using cleanlab in active learning with multiple annotators. Stay tuned! If you're interested in contributing more wonderful ideas, you may want to check out our new Slack channel: |
Beta Was this translation helpful? Give feedback.
-
Hi!
I'd like to try this library in a self-learning scenario. At the moment I've a very big dataset with very few labeled samples, let's say 10% of all the samples, and 90% of unlabeled samples. The goal of self-learning is to label all the unlabeled part in an iterative manner. A chosen classifier is trained on the initial 10% of the labeled dataset and it predicts the labels for the unlabeled part. The K most confident prediction are added to the labeled training dataset and all the process continues up to when all the samples have been labeled. At the end I will have a vector of pseudo labels that is noisy.
My idea was to build a self-learning algorithm using one of the two options:
What I found is that in this setting CleanLab seems not to work as good as when I generate a Transition Matrix and Inject artificially the noise.
Even in the extreme case where with one iteration I want to label all the unlabeled samples at once and then stop, CleanLab is not able to find noise.
Am I missing something?
Thank you a lot and congratulations for this project!
Beta Was this translation helpful? Give feedback.
All reactions