Data Cleaning: The Unglamorous 80 Percent

Social media shows data scientists training models and making slick dashboards. Reality shows them wrestling with encoding errors, removing duplicate rows, fixing dates that are stored five different ways, and hunting down why column Z has three million NaNs.

What cleaning actually involves

1. 1Fixing encoding (UTF-8 vs Latin-1 vs Windows-1252)
2. 2Standardizing date formats (2026-04-23 vs 4/23/26 vs April 23rd)
3. 3Normalizing text (lowercase, strip whitespace, unicode normalization)
4. 4Removing duplicate rows (exact and near-duplicates)
5. 5Handling missing values intentionally
6. 6Detecting and removing outliers
7. 7Joining data from multiple tables
8. 8Reconciling different vocabularies (USA vs. US vs. United States)

A concrete pandas example

import pandas as pd df = pd.read_csv('raw.csv', encoding='utf-8') # Standardize text columns df['country'] = df['country'].str.strip().str.lower() df['country'] = df['country'].replace({ 'usa': 'united states', 'us': 'united states', 'u.s.a.': 'united states' }) # Parse dates df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce') # Remove duplicate rows before = len(df) df = df.drop_duplicates() print(f'Dropped {before - len(df)} duplicates') # Save cleaned version df.to_parquet('clean.parquet')

For LLM training specifically

• Strip HTML and boilerplate (nav bars, cookie banners, ads)
• Remove adult content and hate speech
• Filter out pages that are mostly machine-generated spam
• Remove near-duplicate pages (MinHash and LSH are standard)
• Balance languages, domains, and topics

The big idea: the unglamorous work is where the quality lives. The best models are not just the biggest, they are the ones whose data was cleaned most carefully.