AI Reads Tiny Word Chunks Called Tokens

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-explorers-foundations-AI-and-the-token-not-word-r10a5

1. Before AI can understand a sentence you type, it first breaks the sentence into what?

   1. Complete sentences
   2. Full dictionary words
   3. Tiny word chunks called tokens
   4. Pictures of letters

2. A token in AI is most similar to which of these?

   1. A whole paragraph
   2. An entire dictionary
   3. A syllable in a spoken word
   4. A complete book chapter

3. If you type the word 'cat' into an AI, how many tokens will it likely become?

   1. 10 tokens
   2. 5 tokens
   3. 1 token
   4. 3 tokens

4. The word 'unhappy' would most likely be broken into how many tokens by AI?

   1. 2 tokens: 'un' and 'happy'
   2. 1 token
   3. 5 tokens
   4. 10 tokens

5. When you count 100 words you typed, about how many tokens would an AI say it actually received?

   1. Around 130 tokens
   2. Around 20 tokens
   3. Exactly 50 tokens
   4. Exactly 100 tokens

6. Which of these is TRUE about how AI sees text?

   1. AI sees tokens, not full words
   2. AI sees pictures of each letter
   3. AI reads text exactly like humans do
   4. AI ignores spaces completely

7. You type 'pineapple' into an AI and ask how many tokens it is. What will the AI most likely tell you?

   1. 50 tokens
   2. 100 tokens
   3. Exactly 1 token
   4. 2 or 3 tokens

8. Why does AI split words into tokens instead of reading whole words?

   1. Tokens let AI draw pictures of words
   2. Tokens help AI process language more efficiently
   3. Tokens are easier for humans to read
   4. Tokens make the text longer

9. If someone says AI 'reads' words, what do they really mean?

   1. AI remembers every word perfectly
   2. AI sees the words as complete pictures
   3. AI speaks the words out loud
   4. AI breaks words into token chunks and processes those

10. The word 'building' might become which of these token combinations?

    1. 'building' (1 token)
    2. 'build' + 'ing' (2 tokens)
    3. 'b' + 'u' + 'i' + 'l' + 'd' + 'i' + 'n' + 'g'
    4. 'build' (1 token)

11. When you send a message to AI, how does it count the length of your message?

    1. By counting pictures
    2. By counting every letter
    3. By counting sentences only
    4. By counting tokens, not words

12. What is the MAIN thing you learned about how AI processes text?

    1. AI reads text as small chunks called tokens
    2. AI cannot read text at all
    3. AI only reads text written by humans
    4. AI reads text as complete sentences only

13. A token could be which of the following?

    1. A piece of a word like 'un' or 'ing'
    2. A sound recording
    3. An entire encyclopedia
    4. A video file

14. Two different AIs might split the word 'pineapple' differently. What numbers might they each give you?

    1. They would both always say 1
    2. They would both always say 100
    3. They would both always say 50
    4. One might say 2, another might say 3

15. If you ask an AI to count the 'words' in your sentence, what is it actually counting?

    1. Nothing - AI cannot count
    2. How many times you pressed keys
    3. Tokens that represent word pieces
    4. The actual letters