Representation Bias: Who Is in the Data?

If your training data is 90 percent men, your model will work worse for women. Representation bias is the most pervasive issue in AI.

32 min · Reviewed 2026

The Gender Shades Study

In 2018, Joy Buolamwini and Timnit Gebru tested commercial face recognition systems from IBM, Microsoft, and Face++. Accuracy was nearly perfect for light-skinned men but dropped to 65 percent for dark-skinned women. The reason was brutally simple: the training data was overwhelmingly light-skinned men.

Where representation bias hides

Speech recognition: worse for non-native accents, African American English, and children
Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled ceremony rather than wedding)
Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
Language models: fluent in English and Chinese, clumsy in Swahili and Tagalog

Detecting representation bias

import pandas as pd

df = pd.read_csv('face_dataset.csv')

# Check representation across demographic columns
print(df['skin_tone'].value_counts(normalize=True))
print(df['gender'].value_counts(normalize=True))
print(df['age_group'].value_counts(normalize=True))

# Cross-tab: are some combinations missing?
print(pd.crosstab(df['skin_tone'], df['gender']))

# Flag underrepresented groups
threshold = 0.05  # 5%
underrepresented = df['skin_tone'].value_counts(normalize=True)
print('Underrepresented:', underrepresented[underrepresented < threshold])A quick representation audit

Fixing it

Actively oversample from underrepresented groups during training
Use stratified sampling when collecting new data
Publicly report accuracy metrics per subgroup (not just overall)
Set a minimum accuracy floor before deployment (no subgroup below X%)
Invite audits by affected communities before release

The big idea: you cannot fix what you do not measure. Every serious ML deployment should report accuracy per group, not just an overall number that hides disparities.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-representation-bias

What is the core idea behind "Representation Bias: Who Is in the Data?"?
1. If your training data is 90 percent men, your model will work worse for women. Representation bias is the most pervasive issue in AI.
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which term best describes a foundational idea in "Representation Bias: Who Is in the Data?"?
1. Gender Shades
2. representation bias
3. disaggregated evaluation
4. stratified sampling
A learner studying Representation Bias: Who Is in the Data? would need to understand which concept?
1. representation bias
2. disaggregated evaluation
3. Gender Shades
4. stratified sampling
Which of these is directly relevant to Representation Bias: Who Is in the Data??
1. representation bias
2. Gender Shades
3. stratified sampling
4. disaggregated evaluation
Which of the following is a key point about Representation Bias: Who Is in the Data??
1. Speech recognition: worse for non-native accents, African American English, and children
2. Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled…
3. Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
4. Language models: fluent in English and Chinese, clumsy in Swahili and Tagalog
Which of these does NOT belong in a discussion of Representation Bias: Who Is in the Data??
1. Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
2. Speech recognition: worse for non-native accents, African American English, and children
3. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
4. Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled…
Which statement is accurate regarding Representation Bias: Who Is in the Data??
1. Use stratified sampling when collecting new data
2. Publicly report accuracy metrics per subgroup (not just overall)
3. Actively oversample from underrepresented groups during training
4. Set a minimum accuracy floor before deployment (no subgroup below X%)
Which of these does NOT belong in a discussion of Representation Bias: Who Is in the Data??
1. Use stratified sampling when collecting new data
2. Actively oversample from underrepresented groups during training
3. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
4. Publicly report accuracy metrics per subgroup (not just overall)
What is the key insight about "Representation bias defined" in the context of Representation Bias: Who Is in the Data??
1. Representation bias happens when some group is underrepresented in the training data.
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which statement accurately describes an aspect of Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. In 2018, Joy Buolamwini and Timnit Gebru tested commercial face recognition systems from IBM, Microsoft, and Face++.
3. Implement processes for data-subject rights requests
4. surveillance capitalism
What does working with Representation Bias: Who Is in the Data? typically involve?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Implement processes for data-subject rights requests
3. The big idea: you cannot fix what you do not measure. Every serious ML deployment should report accuracy per group, not just an overall numb…
4. surveillance capitalism
Which best describes the scope of "Representation Bias: Who Is in the Data?"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It was deprecated in 2024 and no longer relevant
4. It focuses on If your training data is 90 percent men, your model will work worse for women. Representation bias i
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Where representation bias hides
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Detecting representation bias
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Implement processes for data-subject rights requests
3. Fixing it
4. surveillance capitalism

← Back to interactive lesson

Tendril · Creators · AI Foundations

Representation Bias: Who Is in the Data?

If your training data is 90 percent men, your model will work worse for women. Representation bias is the most pervasive issue in AI.

32 min · Reviewed 2026

The Gender Shades Study

Where representation bias hides

Speech recognition: worse for non-native accents, African American English, and children
Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled ceremony rather than wedding)
Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
Language models: fluent in English and Chinese, clumsy in Swahili and Tagalog

Detecting representation bias

import pandas as pd

df = pd.read_csv('face_dataset.csv')

# Check representation across demographic columns
print(df['skin_tone'].value_counts(normalize=True))
print(df['gender'].value_counts(normalize=True))
print(df['age_group'].value_counts(normalize=True))

# Cross-tab: are some combinations missing?
print(pd.crosstab(df['skin_tone'], df['gender']))

# Flag underrepresented groups
threshold = 0.05  # 5%
underrepresented = df['skin_tone'].value_counts(normalize=True)
print('Underrepresented:', underrepresented[underrepresented < threshold])A quick representation audit

Fixing it

Actively oversample from underrepresented groups during training
Use stratified sampling when collecting new data
Publicly report accuracy metrics per subgroup (not just overall)
Set a minimum accuracy floor before deployment (no subgroup below X%)
Invite audits by affected communities before release

The big idea: you cannot fix what you do not measure. Every serious ML deployment should report accuracy per group, not just an overall number that hides disparities.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-representation-bias

What is the core idea behind "Representation Bias: Who Is in the Data?"?
1. If your training data is 90 percent men, your model will work worse for women. Representation bias is the most pervasive issue in AI.
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which term best describes a foundational idea in "Representation Bias: Who Is in the Data?"?
1. Gender Shades
2. representation bias
3. disaggregated evaluation
4. stratified sampling
A learner studying Representation Bias: Who Is in the Data? would need to understand which concept?
1. representation bias
2. disaggregated evaluation
3. Gender Shades
4. stratified sampling
Which of these is directly relevant to Representation Bias: Who Is in the Data??
1. representation bias
2. Gender Shades
3. stratified sampling
4. disaggregated evaluation
Which of the following is a key point about Representation Bias: Who Is in the Data??
1. Speech recognition: worse for non-native accents, African American English, and children
2. Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled…
3. Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
4. Language models: fluent in English and Chinese, clumsy in Swahili and Tagalog
Which of these does NOT belong in a discussion of Representation Bias: Who Is in the Data??
1. Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
2. Speech recognition: worse for non-native accents, African American English, and children
3. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
4. Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled…
Which statement is accurate regarding Representation Bias: Who Is in the Data??
1. Use stratified sampling when collecting new data
2. Publicly report accuracy metrics per subgroup (not just overall)
3. Actively oversample from underrepresented groups during training
4. Set a minimum accuracy floor before deployment (no subgroup below X%)
Which of these does NOT belong in a discussion of Representation Bias: Who Is in the Data??
1. Use stratified sampling when collecting new data
2. Actively oversample from underrepresented groups during training
3. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
4. Publicly report accuracy metrics per subgroup (not just overall)
What is the key insight about "Representation bias defined" in the context of Representation Bias: Who Is in the Data??
1. Representation bias happens when some group is underrepresented in the training data.
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which statement accurately describes an aspect of Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. In 2018, Joy Buolamwini and Timnit Gebru tested commercial face recognition systems from IBM, Microsoft, and Face++.
3. Implement processes for data-subject rights requests
4. surveillance capitalism
What does working with Representation Bias: Who Is in the Data? typically involve?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Implement processes for data-subject rights requests
3. The big idea: you cannot fix what you do not measure. Every serious ML deployment should report accuracy per group, not just an overall numb…
4. surveillance capitalism
Which best describes the scope of "Representation Bias: Who Is in the Data?"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It was deprecated in 2024 and no longer relevant
4. It focuses on If your training data is 90 percent men, your model will work worse for women. Representation bias i
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Where representation bias hides
2. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Detecting representation bias
3. Implement processes for data-subject rights requests
4. surveillance capitalism
Which section heading best belongs in a lesson about Representation Bias: Who Is in the Data??
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Implement processes for data-subject rights requests
3. Fixing it
4. surveillance capitalism

← Back to interactive lesson