Inter-Annotator Agreement: Measuring Reality

If two reasonable humans cannot agree on a label, neither can a model. Inter-annotator agreement tells you if a task is even well-defined.

28 min · Reviewed 2026

The Most Underused Metric

Before you train a model to do a task, check if humans can even do it. If two qualified annotators disagree 30 percent of the time, a model at 85 percent accuracy might actually be superhuman. If humans agree 99 percent and the model hits 85, the model is the problem.

Ways to measure agreement

Metric	Good for	Range
Raw percent agreement	Quick sanity check	0 to 100%
Cohen's kappa	Two annotators, categorical labels	-1 to 1
Fleiss's kappa	Many annotators	-1 to 1
Krippendorff's alpha	Missing data, any label type	-1 to 1

A simple calculation

from sklearn.metrics import cohen_kappa_score annotator_a = ['spam', 'not_spam', 'spam', 'spam', 'not_spam'] annotator_b = ['spam', 'not_spam', 'not_spam', 'spam', 'not_spam'] kappa = cohen_kappa_score(annotator_a, annotator_b) print(f'Cohen\'s kappa: {kappa:.2f}') # Agreement corrected for chance: 0.58 = moderateComputing annotator agreement

When disagreement is a signal

High disagreement is not always a bug. For subjective tasks (is this joke funny?), disagreement reveals real human diversity. Modern work in perspectivism argues that for some tasks, we should train models to predict a distribution over labels, not collapse to a single ground truth.

The big idea: the upper bound on model performance is human agreement. If humans cannot do a task consistently, no model can. Measure agreement before you measure accuracy.

End-of-lesson check

6 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-annotator-agreement

What is the main idea of "Inter-Annotator Agreement: Measuring Reality"?
1. If two reasonable humans cannot agree on a label, neither can a model. Inter-annotator agreement tells you if a task is even well-defined.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Inter-Annotator Agreement: Measuring Reality"?
1. kappa
2. inter-annotator agreement
3. labeling
4. Cohen's kappa
What should a careful learner remember about "Interpreting kappa"?
1. Use AI to draft or organize ideas about inter-annotator agreement, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about inter-annotator agreement be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about inter-annotator agreement.

← Back to interactive lesson

Tendril · Creators · AI Foundations

Inter-Annotator Agreement: Measuring Reality

If two reasonable humans cannot agree on a label, neither can a model. Inter-annotator agreement tells you if a task is even well-defined.

28 min · Reviewed 2026

The Most Underused Metric

Ways to measure agreement

Metric	Good for	Range
Raw percent agreement	Quick sanity check	0 to 100%
Cohen's kappa	Two annotators, categorical labels	-1 to 1
Fleiss's kappa	Many annotators	-1 to 1
Krippendorff's alpha	Missing data, any label type	-1 to 1

A simple calculation

from sklearn.metrics import cohen_kappa_score annotator_a = ['spam', 'not_spam', 'spam', 'spam', 'not_spam'] annotator_b = ['spam', 'not_spam', 'not_spam', 'spam', 'not_spam'] kappa = cohen_kappa_score(annotator_a, annotator_b) print(f'Cohen\'s kappa: {kappa:.2f}') # Agreement corrected for chance: 0.58 = moderateComputing annotator agreement

When disagreement is a signal

The big idea: the upper bound on model performance is human agreement. If humans cannot do a task consistently, no model can. Measure agreement before you measure accuracy.

End-of-lesson check

6 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-annotator-agreement

What is the main idea of "Inter-Annotator Agreement: Measuring Reality"?
1. If two reasonable humans cannot agree on a label, neither can a model. Inter-annotator agreement tells you if a task is even well-defined.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Inter-Annotator Agreement: Measuring Reality"?
1. kappa
2. inter-annotator agreement
3. labeling
4. Cohen's kappa
What should a careful learner remember about "Interpreting kappa"?
1. Use AI to draft or organize ideas about inter-annotator agreement, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about inter-annotator agreement be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about inter-annotator agreement.

← Back to interactive lesson