Tokens and Embeddings: How AI Reads Words

When you type a sentence to an AI, it does not see the letters. The text goes through two conversions. First, it gets chopped into tokens. Second, each token gets replaced by a long list of numbers called an embedding.

Step one: tokenization

A tokenizer breaks text into chunks. Common chunks get their own token. Rare words are split into parts. One rule of thumb: about 4 characters of English text equals 1 token, roughly.

• cat → 1 token
• unbelievable → usually 2 or 3 tokens like un, believ, able
• 😀 → usually 1 to 3 tokens depending on the tokenizer
• spaces and punctuation often become their own tokens

Step two: embeddings

Each token gets replaced with a vector, usually hundreds or thousands of numbers long. This vector is the token's location in a huge invisible map of meaning. Tokens with similar meanings end up close together on that map.

A tiny example in code

tokens = tokenizer.encode("I love pizza")
# tokens might be [40, 1842, 21876]

embeddings = model.embed(tokens)
# each token now a 4096-number vector
# embeddings.shape == (3, 4096)

Why this matters for you

• Token counts decide how much you can fit in a prompt
• Embeddings power search, recommendations, and clustering
• Most AI tools charge per token, not per word

The big idea: text becomes tokens, tokens become vectors, and meaning becomes math. That is the doorway from human language into the inside of an AI model.