Data Cleaning: The Unglamorous 80 Percent

Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here is what that actually looks like.

30 min · Reviewed 2026

The Glamorous Picture vs. Reality

Social media shows data scientists training models and making slick dashboards. Reality shows them wrestling with encoding errors, removing duplicate rows, fixing dates that are stored five different ways, and hunting down why column Z has three million NaNs.

What cleaning actually involves

Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
Normalizing text (lowercase, strip whitespace, unicode normalization)
Removing duplicate rows (exact and near-duplicates)
Handling missing values intentionally
Detecting and removing outliers
Joining data from multiple tables
Reconciling different vocabularies (USA vs. US vs. United States)

A concrete pandas example

import pandas as pd

df = pd.read_csv('raw.csv', encoding='utf-8')

# Standardize text columns
df['country'] = df['country'].str.strip().str.lower()
df['country'] = df['country'].replace({
    'usa': 'united states',
    'us': 'united states',
    'u.s.a.': 'united states'
})

# Parse dates
df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce')

# Remove duplicate rows
before = len(df)
df = df.drop_duplicates()
print(f'Dropped {before - len(df)} duplicates')

# Save cleaned version
df.to_parquet('clean.parquet')A small but real cleaning pipeline

For LLM training specifically

Strip HTML and boilerplate (nav bars, cookie banners, ads)
Remove adult content and hate speech
Filter out pages that are mostly machine-generated spam
Remove near-duplicate pages (MinHash and LSH are standard)
Balance languages, domains, and topics

The big idea: the unglamorous work is where the quality lives. The best models are not just the biggest, they are the ones whose data was cleaned most carefully.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-cleaning-80-percent

What is the core idea behind "Data Cleaning: The Unglamorous 80 Percent"?
1. Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here is what that actually looks like.
2. standard deviation
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which term best describes a foundational idea in "Data Cleaning: The Unglamorous 80 Percent"?
1. ETL
2. data cleaning
3. preprocessing
4. deduplication
A learner studying Data Cleaning: The Unglamorous 80 Percent would need to understand which concept?
1. data cleaning
2. preprocessing
3. ETL
4. deduplication
Which of these is directly relevant to Data Cleaning: The Unglamorous 80 Percent?
1. data cleaning
2. ETL
3. deduplication
4. preprocessing
Which of the following is a key point about Data Cleaning: The Unglamorous 80 Percent?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
3. Normalizing text (lowercase, strip whitespace, unicode normalization)
4. Removing duplicate rows (exact and near-duplicates)
Which of these does NOT belong in a discussion of Data Cleaning: The Unglamorous 80 Percent?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
3. Normalizing text (lowercase, strip whitespace, unicode normalization)
4. standard deviation
Which statement is accurate regarding Data Cleaning: The Unglamorous 80 Percent?
1. Remove adult content and hate speech
2. Filter out pages that are mostly machine-generated spam
3. Strip HTML and boilerplate (nav bars, cookie banners, ads)
4. Remove near-duplicate pages (MinHash and LSH are standard)
Which of these does NOT belong in a discussion of Data Cleaning: The Unglamorous 80 Percent?
1. Remove adult content and hate speech
2. standard deviation
3. Filter out pages that are mostly machine-generated spam
4. Strip HTML and boilerplate (nav bars, cookie banners, ads)
What is the key insight about "The Anaconda survey" in the context of Data Cleaning: The Unglamorous 80 Percent?
1. A 2022 survey of over 3,000 data practitioners found they spend about 38 percent of their time on data preparation and c…
2. standard deviation
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
What is the recommended tip about "Build your mental model" in the context of Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. AI isn't magic — it's pattern recognition at scale. The more you understand how it works, the more effectively you can u…
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which statement accurately describes an aspect of Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. Social media shows data scientists training models and making slick dashboards.
4. Gender Shades
What does working with Data Cleaning: The Unglamorous 80 Percent typically involve?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. Gender Shades
4. The big idea: the unglamorous work is where the quality lives. The best models are not just the biggest, they are the ones whose data was cl…
Which best describes the scope of "Data Cleaning: The Unglamorous 80 Percent"?
1. It focuses on Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here i
2. It is unrelated to foundations workflows
3. It applies only to the opposite beginner tier
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. What cleaning actually involves
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which section heading best belongs in a lesson about Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. A concrete pandas example
4. Gender Shades

← Back to interactive lesson

Tendril · Builders · AI Foundations

Data Cleaning: The Unglamorous 80 Percent

Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here is what that actually looks like.

30 min · Reviewed 2026

The Glamorous Picture vs. Reality

What cleaning actually involves

Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
Normalizing text (lowercase, strip whitespace, unicode normalization)
Removing duplicate rows (exact and near-duplicates)
Handling missing values intentionally
Detecting and removing outliers
Joining data from multiple tables
Reconciling different vocabularies (USA vs. US vs. United States)

A concrete pandas example

import pandas as pd

df = pd.read_csv('raw.csv', encoding='utf-8')

# Standardize text columns
df['country'] = df['country'].str.strip().str.lower()
df['country'] = df['country'].replace({
    'usa': 'united states',
    'us': 'united states',
    'u.s.a.': 'united states'
})

# Parse dates
df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce')

# Remove duplicate rows
before = len(df)
df = df.drop_duplicates()
print(f'Dropped {before - len(df)} duplicates')

# Save cleaned version
df.to_parquet('clean.parquet')A small but real cleaning pipeline

For LLM training specifically

Strip HTML and boilerplate (nav bars, cookie banners, ads)
Remove adult content and hate speech
Filter out pages that are mostly machine-generated spam
Remove near-duplicate pages (MinHash and LSH are standard)
Balance languages, domains, and topics

The big idea: the unglamorous work is where the quality lives. The best models are not just the biggest, they are the ones whose data was cleaned most carefully.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-cleaning-80-percent

What is the core idea behind "Data Cleaning: The Unglamorous 80 Percent"?
1. Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here is what that actually looks like.
2. standard deviation
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which term best describes a foundational idea in "Data Cleaning: The Unglamorous 80 Percent"?
1. ETL
2. data cleaning
3. preprocessing
4. deduplication
A learner studying Data Cleaning: The Unglamorous 80 Percent would need to understand which concept?
1. data cleaning
2. preprocessing
3. ETL
4. deduplication
Which of these is directly relevant to Data Cleaning: The Unglamorous 80 Percent?
1. data cleaning
2. ETL
3. deduplication
4. preprocessing
Which of the following is a key point about Data Cleaning: The Unglamorous 80 Percent?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
3. Normalizing text (lowercase, strip whitespace, unicode normalization)
4. Removing duplicate rows (exact and near-duplicates)
Which of these does NOT belong in a discussion of Data Cleaning: The Unglamorous 80 Percent?
1. Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
3. Normalizing text (lowercase, strip whitespace, unicode normalization)
4. standard deviation
Which statement is accurate regarding Data Cleaning: The Unglamorous 80 Percent?
1. Remove adult content and hate speech
2. Filter out pages that are mostly machine-generated spam
3. Strip HTML and boilerplate (nav bars, cookie banners, ads)
4. Remove near-duplicate pages (MinHash and LSH are standard)
Which of these does NOT belong in a discussion of Data Cleaning: The Unglamorous 80 Percent?
1. Remove adult content and hate speech
2. standard deviation
3. Filter out pages that are mostly machine-generated spam
4. Strip HTML and boilerplate (nav bars, cookie banners, ads)
What is the key insight about "The Anaconda survey" in the context of Data Cleaning: The Unglamorous 80 Percent?
1. A 2022 survey of over 3,000 data practitioners found they spend about 38 percent of their time on data preparation and c…
2. standard deviation
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
What is the recommended tip about "Build your mental model" in the context of Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. AI isn't magic — it's pattern recognition at scale. The more you understand how it works, the more effectively you can u…
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which statement accurately describes an aspect of Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. Social media shows data scientists training models and making slick dashboards.
4. Gender Shades
What does working with Data Cleaning: The Unglamorous 80 Percent typically involve?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. Gender Shades
4. The big idea: the unglamorous work is where the quality lives. The best models are not just the biggest, they are the ones whose data was cl…
Which best describes the scope of "Data Cleaning: The Unglamorous 80 Percent"?
1. It focuses on Surveys consistently find data scientists spend 60 to 80 percent of their time cleaning data. Here i
2. It is unrelated to foundations workflows
3. It applies only to the opposite beginner tier
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. What cleaning actually involves
3. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
4. Gender Shades
Which section heading best belongs in a lesson about Data Cleaning: The Unglamorous 80 Percent?
1. standard deviation
2. Berkeley admissions 1973: overall lower acceptance rate for women, but women had…
3. A concrete pandas example
4. Gender Shades

← Back to interactive lesson