How AI Models See Text: Tokens, Context, and Why It Matters

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-foundations-tokens-final1-creators

1. A student writes a prompt that is 200 words long. If approximately 1.33 tokens equal 1 word, about how many tokens is this prompt?

   1. Around 150 tokens
   2. Around 266 tokens
   3. Around 400 tokens
   4. Around 66 tokens

2. Why might the word 'GPTs' be tokenized as two separate tokens while 'GPT' is one token?

   1. All acronyms must be wrapped in special markers
   2. The 's' suffix is treated as a separate token because tokenizers learn to split common suffixes as their own units
   3. The tokenizer splits any word containing more than three characters
   4. The tokenizer has a built-in list of model names

3. What practical skill does the lesson say understanding tokenization improves?

   1. It improves your ability to debug code
   2. It helps you optimize prompts to use fewer tokens for the same result
   3. It teaches you how to build your own AI model
   4. It helps you write poetry that sounds more human

4. What is the 'lost-in-the-middle' problem in the context of very long AI inputs?

   1. The AI literally loses data during processing
   2. The AI forgets how to complete sentences when it reaches the middle of a response
   3. The model tends to ignore or poorly utilize information that appears in the middle of a long context window
   4. Tokens in the middle of input get corrupted more than tokens at the beginning or end

5. A 50,000-token context window can hold approximately how many words?

   1. Exactly 25,000 words
   2. Exactly 50,000 words
   3. Around 65,000 words
   4. Around 35,000 words

6. If you are working with a very long document that exceeds the context window, what does the lesson recommend?

   1. Delete the middle section of the document
   2. Use a faster computer to process the document
   3. Break the document into smaller chunks and retrieve only relevant sections
   4. Upload the document multiple times to different AI systems

7. Why is the phrase 'cost per token' significant for someone using AI?

   1. It determines how fast the AI responds
   2. It is the primary factor that determines how much you pay for using the AI service
   3. It affects the accuracy of the AI's responses
   4. It controls how many languages the AI can understand

8. A user pastes a 100-word paragraph and asks an AI to estimate its token count. What can the AI accurately do without using a tokenizer tool?

   1. Estimate the token count based on the approximate 1.33 tokens-per-word ratio
   2. Split the paragraph into individual characters
   3. Tell the user exactly which words will become single vs. multiple tokens
   4. Provide an exact token count

9. What is tokenization?

   1. The process of counting how much an AI service costs
   2. The process of generating text one token at a time
   3. The process of breaking text into tokens that the AI can process
   4. The process of teaching an AI to speak a new language

10. The lesson includes an exercise where a student pastes their own paragraph for token analysis. What is the main point of this exercise?

    1. To compare different paragraphs from different students
    2. To help the student see tokenization in action using their own text as a concrete example
    3. To test the student's typing speed
    4. To demonstrate that the AI can count words

11. What does the lesson mean when it says AI models 'see' text as tokens, not words?

    1. AI models only recognize words that are exactly one token long
    2. AI models have visual eyes that look at text
    3. The fundamental processing unit in AI models is the token, not the linguistic word, so the model operates on token sequences
    4. AI models cannot read text unless it is converted to images

12. Why might a very long document 'fail in subtle ways' when given to an AI?

    1. The AI will change the meaning of the document
    2. The document will be automatically shortened without your consent
    3. The AI will refuse to read it
    4. The document might exceed the context window, or information in the middle may be ignored due to the lost-in-the-middle problem

13. Which statement about context windows is accurate?

    1. Context windows only apply to output, not input
    2. Context windows are infinite for premium users
    3. Context windows measure how fast the AI generates text
    4. Context windows are hard limits that cannot be exceeded, though limits vary by model

14. What is one reason to optimize your prompts for token count?

    1. Optimized prompts make the AI run faster on your device
    2. Optimized prompts are required to get any response from the AI
    3. Fewer tokens mean lower cost and potentially better attention to the parts that matter
    4. The AI ignores prompts that are too long

15. The lesson mentions that different words may become different numbers of tokens. Which word is most likely to become a single token?

    1. Running
    2. Unconstitutionally
    3. Cat
    4. Internationalization