Label Noise: When Your Ground Truth Is Wrong

In 2021, researchers at MIT published Pervasive Label Errors in Test Sets, showing that the most famous ML benchmarks have significant error rates. ImageNet's test set had at least 5.8 percent mislabeled examples. MNIST had 0.15 percent. When benchmark models differ by tenths of a percent, this matters.

Types of label noise

• Random noise: uniform mistakes across classes
• Systematic noise: certain classes often confused (golden retriever vs. labrador)
• Adversarial noise: deliberately mislabeled data (data poisoning)
• Label flip: the correct label exists but was swapped

How noisy labels hurt

1. 1Models learn the noise along with the signal
2. 2Accuracy plateaus even with more data
3. 3Benchmarks become unreliable ranking tools
4. 4Real-world deployment reveals embarrassing failures

# Detecting likely label errors with confident learning
from cleanlab import Cleanlab
import numpy as np

# pred_probs: model predictions for each class, shape (N, K)
# labels: given labels, shape (N,)

lab = Cleanlab()
issues = lab.find_label_issues(
    labels=labels,
    pred_probs=pred_probs,
    return_indices_ranked_by='self_confidence'
)

print(f'Likely mislabeled: {len(issues)} examples')
print('Top 10 suspects:', issues[:10])

Mitigations

• Double-labeling: two annotators per item, resolve disagreements
• Active learning: train model, find low-confidence predictions, re-label
• Noise-robust loss functions (e.g., symmetric cross-entropy)
• Confident learning to flag statistical outliers
• Publish cleaned splits alongside originals so benchmarks improve over time

The big idea: no label is sacred. Every dataset has errors. Building systems that can measure and tolerate label noise is a core skill in production ML.