Tokens and Embeddings: How AI Reads Words

AI does not read letters. It reads tokens, which live as vectors in a space of meaning. Learn how text becomes numbers you can do math on.

30 min · Reviewed 2026

From Letters to Numbers

When you type a sentence to an AI, it does not see the letters. The text goes through two conversions. First, it gets chopped into tokens. Second, each token gets replaced by a long list of numbers called an embedding.

Step one: tokenization

A tokenizer breaks text into chunks. Common chunks get their own token. Rare words are split into parts. One rule of thumb: about 4 characters of English text equals 1 token, roughly.

cat → 1 token
unbelievable → usually 2 or 3 tokens like un, believ, able
😀 → usually 1 to 3 tokens depending on the tokenizer
spaces and punctuation often become their own tokens

Step two: embeddings

Each token gets replaced with a vector, usually hundreds or thousands of numbers long. This vector is the token's location in a huge invisible map of meaning. Tokens with similar meanings end up close together on that map.

A tiny example in code

tokens = tokenizer.encode("I love pizza") # tokens might be [40, 1842, 21876] embeddings = model.embed(tokens) # each token now a 4096-number vector # embeddings.shape == (3, 4096)Turning words into vectors before the model thinks about them.

Why this matters for you

Token counts decide how much you can fit in a prompt
Embeddings power search, recommendations, and clustering
Most AI tools charge per token, not per word

The big idea: text becomes tokens, tokens become vectors, and meaning becomes math. That is the doorway from human language into the inside of an AI model.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-tokens-and-embeddings

What is the main idea of "Tokens and Embeddings: How AI Reads Words"?
1. AI does not read letters. It reads tokens, which live as vectors in a space of meaning. Learn how text becomes numbers you can do math on.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Tokens and Embeddings: How AI Reads Words"?
1. embeddings
2. tokens
3. vectors
4. vocabulary
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. cat → 1 token
4. Use the first answer without checking it
What should a careful learner remember about "Why chunk at all?"?
1. Use AI to draft or organize ideas about tokens, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about tokens be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about tokens.
Which action would help you apply "Tokens and Embeddings: How AI Reads Words" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. unbelievable → usually 2 or 3 tokens like un, believ, able

← Back to interactive lesson

Tendril · Builders · AI Foundations

Tokens and Embeddings: How AI Reads Words

AI does not read letters. It reads tokens, which live as vectors in a space of meaning. Learn how text becomes numbers you can do math on.

30 min · Reviewed 2026

From Letters to Numbers

Step one: tokenization

A tokenizer breaks text into chunks. Common chunks get their own token. Rare words are split into parts. One rule of thumb: about 4 characters of English text equals 1 token, roughly.

cat → 1 token
unbelievable → usually 2 or 3 tokens like un, believ, able
😀 → usually 1 to 3 tokens depending on the tokenizer
spaces and punctuation often become their own tokens

Step two: embeddings

A tiny example in code

tokens = tokenizer.encode("I love pizza") # tokens might be [40, 1842, 21876] embeddings = model.embed(tokens) # each token now a 4096-number vector # embeddings.shape == (3, 4096)Turning words into vectors before the model thinks about them.

Why this matters for you

Token counts decide how much you can fit in a prompt
Embeddings power search, recommendations, and clustering
Most AI tools charge per token, not per word

The big idea: text becomes tokens, tokens become vectors, and meaning becomes math. That is the doorway from human language into the inside of an AI model.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-tokens-and-embeddings

What is the main idea of "Tokens and Embeddings: How AI Reads Words"?
1. AI does not read letters. It reads tokens, which live as vectors in a space of meaning. Learn how text becomes numbers you can do math on.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Tokens and Embeddings: How AI Reads Words"?
1. embeddings
2. tokens
3. vectors
4. vocabulary
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. cat → 1 token
4. Use the first answer without checking it
What should a careful learner remember about "Why chunk at all?"?
1. Use AI to draft or organize ideas about tokens, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about tokens be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about tokens.
Which action would help you apply "Tokens and Embeddings: How AI Reads Words" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. unbelievable → usually 2 or 3 tokens like un, believ, able

← Back to interactive lesson