What a Token Actually Is (And Why It Matters for Your Prompts)

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-foundations-AI-and-what-a-token-actually-is-teen

1. What does an AI language model actually read when processing your text input?

   1. Letters
   2. Sentences
   3. Words
   4. Tokens

2. Which of these words is MOST likely to be split into multiple tokens by a tokenizer?

   1. happy
   2. antidisestablishmentarianism
   3. hamburger
   4. run

3. If a prompt contains 100 tokens, approximately how many words would that typically represent in English?

   1. About 100 words
   2. About 133 words
   3. About 50 words
   4. About 75 words

4. Why might writing a very concise prompt save a user money?

   1. Shorter prompts require less processing time and fewer tokens, which reduces costs
   2. Concise prompts are always more accurate
   3. The AI reads short prompts faster
   4. AI companies give discounts for short prompts

5. Which of these names would a tokenizer be MOST likely to split into multiple tokens?

   1. X Æ A-12
   2. Sarah Jones
   3. Michael
   4. John Smith

6. What is the process called that converts text into tokens for an AI to process?

   1. Word splitting
   2. BPE encoding
   3. Tokenism
   4. Tokenization

7. A student writes two prompts with the same meaning: Prompt A is 50 words, Prompt B is 30 words. If both use typical English, which is likely to use fewer tokens?

   1. Prompt B (30 words)
   2. They will use the same tokens
   3. Word count doesn't affect token count
   4. Prompt A (50 words)

8. The lesson mentions 'BPE' as a key term. What is BPE?

   1. A measurement of token efficiency
   2. A programming language for AI
   3. A tokenization method that merges common byte pairs
   4. A type of AI model

9. A user writes 'The quick brown fox jumps over the lazy dog.' How many tokens would this sentence most likely contain?

   1. About 9 tokens
   2. About 15 tokens
   3. About 4 tokens
   4. About 7 tokens

10. Which statement best captures the 'big idea' from this lesson?

    1. AI always understands exactly what you write
    2. Tokens are more important than meaning
    3. AI cannot read at all
    4. AI thinks in tokens, not words

11. A user includes this unusual word in their prompt: 'Supercalifragilisticexpialidocious'. How will the tokenizer likely handle this?

    1. As a single token
    2. As multiple tokens
    3. It will be spelled incorrectly
    4. The AI will not understand it

12. If you want to reduce the token count of a prompt without changing its meaning, which approach would help most?

    1. Remove unnecessary words and be more concise
    2. Add more examples
    3. Add polite phrases like 'please' and 'thank you'
    4. Write everything in all capital letters

13. The lesson suggests using a tokenizer demo. What would you observe when using such a tool?

    1. Whether your prompt is grammatically correct
    2. How many words are in your prompt
    3. How your prompt gets broken down into individual tokens
    4. How much the API will charge

14. Compare these two prompts with identical instructions: Prompt 1 uses simple common words, Prompt 2 uses technical jargon and rare terms. Which will likely use more tokens?

    1. Prompt 2 with technical jargon
    2. They will use the same tokens
    3. Technical terms use fewer tokens
    4. Prompt 1 with simple words

15. What happens when you write a prompt that is extremely long?

    1. You will use more tokens and incur higher costs
    2. The AI will summarize it for you
    3. The AI will only read the first part
    4. Long prompts are always better