Word2vec: Meaning Becomes Geometry

In 2013, Tomas Mikolov and colleagues at Google released word2vec, a pair of simple neural network algorithms for learning vector representations of words from raw text. Each word became a point in a few hundred dimensions, learned from the company it kept.

The magic trick that caught everyone's attention: vector arithmetic produced analogies. The vector for king minus man plus woman landed near queen. Paris minus France plus Italy landed near Rome. Structure in language became structure in geometry.

How it learned

• Skip-gram: predict surrounding words from the current word
• CBOW: predict the current word from surrounding context
• Negative sampling: pit true neighbors against random ones to sharpen learning
• Train on billions of words; adjust vectors until predictions improve

Embeddings were not brand new. Older work on latent semantic analysis and neural language models by Bengio had mapped words to vectors before. Word2vec's contribution was speed, scale, and a viral demo that made everyone understand what embeddings were good for.

“You shall know a word by the company it keeps.”

The big idea: meaning lives in patterns of use, and those patterns can be captured as geometry. This insight powers not just language models but also image models, retrieval, and recommendation systems.