Representation Bias: Who Is in the Data?

In 2018, Joy Buolamwini and Timnit Gebru tested commercial face recognition systems from IBM, Microsoft, and Face++. Accuracy was nearly perfect for light-skinned men but dropped to 65 percent for dark-skinned women. The reason was brutally simple: the training data was overwhelmingly light-skinned men.

Where representation bias hides

• Speech recognition: worse for non-native accents, African American English, and children
• Image classification: worse for non-Western contexts (a photo of a Nigerian wedding might be labeled ceremony rather than wedding)
• Medical AI: trained mostly on white adult patients, fails on darker skin or pediatric cases
• Language models: fluent in English and Chinese, clumsy in Swahili and Tagalog

Detecting representation bias

import pandas as pd df = pd.read_csv('face_dataset.csv') # Check representation across demographic columns print(df['skin_tone'].value_counts(normalize=True)) print(df['gender'].value_counts(normalize=True)) print(df['age_group'].value_counts(normalize=True)) # Cross-tab: are some combinations missing? print(pd.crosstab(df['skin_tone'], df['gender'])) # Flag underrepresented groups threshold = 0.05 # 5% underrepresented = df['skin_tone'].value_counts(normalize=True) print('Underrepresented:', underrepresented[underrepresented < threshold])

Fixing it

1. 1Actively oversample from underrepresented groups during training
2. 2Use stratified sampling when collecting new data
3. 3Publicly report accuracy metrics per subgroup (not just overall)
4. 4Set a minimum accuracy floor before deployment (no subgroup below X%)
5. 5Invite audits by affected communities before release

The big idea: you cannot fix what you do not measure. Every serious ML deployment should report accuracy per group, not just an overall number that hides disparities.