Multi-Class Span Classification Support

[Steven-Yiran]

Summary

This pull request aims to add support for the multi-class span classification dataset proposed in #385. The multi-class support is an extension of the single-class functionalities in #982. The issue display format for display_issues() is modified to suit the span classification dataset. The new additions are backward compatible with single-class usage in #982.

An example of a multi-class span classification task:

import numpy as np
from cleanlab.experimental.span_classification import find_label_issues

# 3 span class with labels 1, 2, 3
tokens = [
    ['a', 'b', 'c', 'd'],
    ['e', 'f', 'g'],
]
labels = [
    [[0], [1, 2], [1, 3], [0]],
    [[1], [2, 3], [3]],
]
pred_probs = [
    np.array([[0.9, 0.2, 0.3], [0.9, 0.9, 0.2], [0.9, 0.1, 0.7], [0.1, 0.1, 0.1]]),
    np.array([[0.1, 0.9, 0.1], [0.1, 0.9, 0.9], [0.1, 0.9, 0.9]]),
]

issues = find_label_issues(labels, pred_probs)
# {1: [(0, 0), (1, 0)], 2: [(1, 0), (1, 2)], 3: []}

display_issues(issues, tokens, labels=labels, pred_probs=pred_probs)
# Span Class: 1
# Sentence index: 0, Token index: 0
# Token: a
# According to provided labels/pred_probs, token marked as outside span but predicted inside span with probability: 0.9
# ----
# a b c d
#
# ...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.25%. Comparing base (09245a9) to head (dbd9b06).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1034      +/-   ##
==========================================
+ Coverage   96.19%   96.25%   +0.05%     
==========================================
  Files          74       74              
  Lines        5841     5841              
  Branches     1043     1043              
==========================================
+ Hits         5619     5622       +3     
+ Misses        132      130       -2     
+ Partials       90       89       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.