Where Training Data Actually Comes From

The weights get the credit, but the data does the work. Change nothing about the architecture and swap in better data, and the model gets noticeably smarter. That is why top labs spend huge sums on data curation.

Where text comes from

• Common Crawl: a free, massive scrape of the public web
• Wikipedia and other reference wikis
• Books: public domain plus commercial libraries
• Academic papers and preprints
• Code repositories on GitHub and elsewhere
• Synthetic data: generated by smaller models

The cleaning pipeline

Raw web data is a mess. Duplicates, spam, broken HTML, hateful text. Before training, teams spend months filtering and deduplicating. A rule of thumb: at least half of any raw crawl gets tossed.

1. 1Deduplicate near-identical pages
2. 2Filter by language and quality heuristics
3. 3Strip HTML and normalize Unicode
4. 4Remove PII and known toxic content
5. 5Up-sample high-value domains like science and textbooks

RLHF: humans in the loop

After pretraining, companies hire people to rate model outputs. Better responses are rewarded. Worse ones are penalized. This method, called reinforcement learning from human feedback, is what turns a raw text predictor into a helpful assistant.

Why the diet matters

• Biases in data become biases in the model
• Coverage gaps mean the model is weak in some languages or topics
• Recency is limited by the training cutoff date
• Synthetic data can cause a feedback loop if overused

“Garbage in, garbage out is the oldest rule in computing, and AI did not escape it.”

The big idea: an AI is a reflection of the text it was raised on. Understanding the data makes model behavior much less mysterious.