Deduplication: Why Repeats Hurt Models

The raw internet is full of copies. News articles get syndicated across hundreds of sites. License text appears in every open-source repo. A viral tweet gets quoted ten thousand times. If you train on raw Common Crawl without removing duplicates, your model sees the Wikipedia article on France not once, but a thousand times.

Research that proved it matters

A 2021 paper, Deduplicating Training Data Makes Language Models Better by Lee et al., showed that aggressive deduplication reduces memorization of training text by 10x and slightly improves model quality. Every major lab now deduplicates.

Two flavors of duplication

MinHash, the workhorse

MinHash is a clever algorithm that estimates document similarity in sub-linear time. Break each document into n-grams (say, 5-word shingles), hash them, keep the minimum from each of many hash functions, and compare the minimums. It lets you find near-duplicates across a trillion-token dataset in hours instead of centuries.

from datasketch import MinHash, MinHashLSH

def minhash_for(text):
    mh = MinHash(num_perm=128)
    for word in set(text.lower().split()):
        mh.update(word.encode('utf-8'))
    return mh

lsh = MinHashLSH(threshold=0.8, num_perm=128)
docs = ['cats are fuzzy', 'cats are quite fuzzy', 'dogs bark loudly']

for i, text in enumerate(docs):
    lsh.insert(f'doc_{i}', minhash_for(text))

# Find near-duplicates of doc 0
print(lsh.query(minhash_for(docs[0])))

The big idea: more unique data beats more total data. Deduplication is one of the quietest, most effective steps in modern AI.