Quality Filtering: Separating Signal From Noise

The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of the highest-leverage steps in modern AI.

28 min · Reviewed 2026

Most of the Internet Is Garbage

If you scrape the open web, most of what you get is spam, SEO-generated junk, parked domains full of ads, auto-translated machine output, and boilerplate. Training a model on this raw mess makes a worse model than training on a smaller but cleaner subset.

Common filter techniques

Language ID (keep only target languages)
Length filters (drop pages under 50 words)
Character-level rules (drop pages with too much punctuation)
Repetition filters (drop pages where one line repeats 50 times)
Perplexity filters (use a small LM to score how natural the text looks)
Classifier-based filters (train a model on Wikipedia vs. random web)

The perplexity trick

Train a small language model on a known-good corpus like Wikipedia. Run it on any web page and compute its perplexity, a measure of how surprising the text is. Random spam has very high perplexity (unpredictable garbage). Genuine writing has low perplexity. Keep only the low-perplexity pages.

Classifier filters

Take a sample labeled as high-quality (curated lists, Wikipedia, reference books) and another sample of random web text. Train a binary classifier. Then run it on the full corpus and keep the pages it labels high-quality. GPT-3 famously used this approach with a simple logistic regression.

# Pseudocode for a simple quality classifier
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.linear_model import LogisticRegression

# Labels: 1 = high quality, 0 = random web
X_train = [...]  # list of documents
y_train = [...]  # 1s for curated, 0s for random

vec = HashingVectorizer(n_features=2**18)
clf = LogisticRegression()
clf.fit(vec.transform(X_train), y_train)

# Score a new document
score = clf.predict_proba(vec.transform(['some new page']))[0, 1]
if score > 0.7:
    keep_this_page()A GPT-3 style quality classifier

The big idea: quality filtering is where small labs and big labs actually differ. Anyone can download Common Crawl. Not everyone can filter it well.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-quality-filtering

What is the core idea behind "Quality Filtering: Separating Signal From Noise"?
1. The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of the highest-leverage steps in modern AI.
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which term best describes a foundational idea in "Quality Filtering: Separating Signal From Noise"?
1. perplexity
2. quality filter
3. classifier
4. boilerplate
A learner studying Quality Filtering: Separating Signal From Noise would need to understand which concept?
1. quality filter
2. classifier
3. perplexity
4. boilerplate
Which of these is directly relevant to Quality Filtering: Separating Signal From Noise?
1. quality filter
2. perplexity
3. boilerplate
4. classifier
Which of the following is a key point about Quality Filtering: Separating Signal From Noise?
1. Language ID (keep only target languages)
2. Length filters (drop pages under 50 words)
3. Character-level rules (drop pages with too much punctuation)
4. Repetition filters (drop pages where one line repeats 50 times)
Which of these does NOT belong in a discussion of Quality Filtering: Separating Signal From Noise?
1. Character-level rules (drop pages with too much punctuation)
2. encoding
3. Length filters (drop pages under 50 words)
4. Language ID (keep only target languages)
What is the key insight about "Classic result" in the context of Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. The Llama 1 team kept only about 7 percent of raw Common Crawl after filtering.
4. UTF-8
What is the key insight about "The quality paradox" in the context of Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. UTF-8
4. Perplexity filters favor text that looks like Wikipedia. This can silently bias models toward formal, encyclopedic langu…
Which statement accurately describes an aspect of Quality Filtering: Separating Signal From Noise?
1. If you scrape the open web, most of what you get is spam, SEO-generated junk, parked domains full of ads, auto-translated machine output, an…
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
What does working with Quality Filtering: Separating Signal From Noise typically involve?
1. encoding
2. Train a small language model on a known-good corpus like Wikipedia. Run it on any web page and compute its perplexity, a measure of how surp…
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which of the following is true about Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. Take a sample labeled as high-quality (curated lists, Wikipedia, reference books) and another sample of random web text.
4. UTF-8
Which best describes the scope of "Quality Filtering: Separating Signal From Noise"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It was deprecated in 2024 and no longer relevant
4. It focuses on The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of th
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. Common filter techniques
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. encoding
2. The perplexity trick
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. Classifier filters
4. UTF-8

← Back to interactive lesson

Tendril · Builders · AI Foundations

Quality Filtering: Separating Signal From Noise

The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of the highest-leverage steps in modern AI.

28 min · Reviewed 2026

Most of the Internet Is Garbage

Common filter techniques

Language ID (keep only target languages)
Length filters (drop pages under 50 words)
Character-level rules (drop pages with too much punctuation)
Repetition filters (drop pages where one line repeats 50 times)
Perplexity filters (use a small LM to score how natural the text looks)
Classifier-based filters (train a model on Wikipedia vs. random web)

The perplexity trick

Classifier filters

# Pseudocode for a simple quality classifier
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.linear_model import LogisticRegression

# Labels: 1 = high quality, 0 = random web
X_train = [...]  # list of documents
y_train = [...]  # 1s for curated, 0s for random

vec = HashingVectorizer(n_features=2**18)
clf = LogisticRegression()
clf.fit(vec.transform(X_train), y_train)

# Score a new document
score = clf.predict_proba(vec.transform(['some new page']))[0, 1]
if score > 0.7:
    keep_this_page()A GPT-3 style quality classifier

The big idea: quality filtering is where small labs and big labs actually differ. Anyone can download Common Crawl. Not everyone can filter it well.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-quality-filtering

What is the core idea behind "Quality Filtering: Separating Signal From Noise"?
1. The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of the highest-leverage steps in modern AI.
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which term best describes a foundational idea in "Quality Filtering: Separating Signal From Noise"?
1. perplexity
2. quality filter
3. classifier
4. boilerplate
A learner studying Quality Filtering: Separating Signal From Noise would need to understand which concept?
1. quality filter
2. classifier
3. perplexity
4. boilerplate
Which of these is directly relevant to Quality Filtering: Separating Signal From Noise?
1. quality filter
2. perplexity
3. boilerplate
4. classifier
Which of the following is a key point about Quality Filtering: Separating Signal From Noise?
1. Language ID (keep only target languages)
2. Length filters (drop pages under 50 words)
3. Character-level rules (drop pages with too much punctuation)
4. Repetition filters (drop pages where one line repeats 50 times)
Which of these does NOT belong in a discussion of Quality Filtering: Separating Signal From Noise?
1. Character-level rules (drop pages with too much punctuation)
2. encoding
3. Length filters (drop pages under 50 words)
4. Language ID (keep only target languages)
What is the key insight about "Classic result" in the context of Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. The Llama 1 team kept only about 7 percent of raw Common Crawl after filtering.
4. UTF-8
What is the key insight about "The quality paradox" in the context of Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. UTF-8
4. Perplexity filters favor text that looks like Wikipedia. This can silently bias models toward formal, encyclopedic langu…
Which statement accurately describes an aspect of Quality Filtering: Separating Signal From Noise?
1. If you scrape the open web, most of what you get is spam, SEO-generated junk, parked domains full of ads, auto-translated machine output, an…
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
What does working with Quality Filtering: Separating Signal From Noise typically involve?
1. encoding
2. Train a small language model on a known-good corpus like Wikipedia. Run it on any web page and compute its perplexity, a measure of how surp…
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which of the following is true about Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. Take a sample labeled as high-quality (curated lists, Wikipedia, reference books) and another sample of random web text.
4. UTF-8
Which best describes the scope of "Quality Filtering: Separating Signal From Noise"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It was deprecated in 2024 and no longer relevant
4. It focuses on The raw web is 99 percent garbage. Filtering it down to the 1 percent worth training on is one of th
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. Common filter techniques
2. encoding
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. encoding
2. The perplexity trick
3. Scope the audit: what question are we trying to answer?
4. UTF-8
Which section heading best belongs in a lesson about Quality Filtering: Separating Signal From Noise?
1. encoding
2. Scope the audit: what question are we trying to answer?
3. Classifier filters
4. UTF-8

← Back to interactive lesson