AI Benchmarks: What 'GPT Beats Human' Really Means

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-evaluation-benchmarks-teens-final2-teen

1. What does the term 'benchmark' refer to in AI development?

   1. The process of training an AI on large amounts of text data
   2. A type of AI model that learns from user feedback
   3. A physical computer server used to run AI models
   4. A standardized test used to compare AI model performance across different tasks

2. What is 'contamination' in the context of AI benchmarking?

   1. When benchmark questions accidentally appear in a model's training data
   2. The process of removing an AI model from a competition
   3. When two different AI models are tested simultaneously
   4. A method to clean up errors in benchmark datasets

3. Why might a headline claiming 'AI beats humans on X benchmark' be misleading?

   1. Headlines are always completely false
   2. Benchmarks are never updated after release
   3. AI models cannot actually beat humans at any task
   4. The benchmark might not measure skills that matter for real-world use

4. What type of questions does the GPQA benchmark contain?

   1. Simple yes or no questions about text comprehension
   2. Graduate-level science questions requiring specialized knowledge
   3. Basic arithmetic and math problems
   4. Questions about popular culture and current events

5. What does SWE-bench measure?

   1. Knowledge of software history and companies
   2. Speed of AI response times for coding requests
   3. The ability to write creative stories about software
   4. Real software engineering tasks like fixing bugs from actual GitHub projects

6. What is 'vibes-eval' as described in the lesson?

   1. An informal assessment of how well AI actually performs in real use, without numerical scores
   2. A test that measures whether users feel emotionally connected to AI
   3. A benchmark that assigns emotional scores to AI responses
   4. A method to evaluate AI based on user reviews and ratings

7. Based on the lesson, what should matter more than benchmark scores when choosing an AI model?

   1. How well the model performs on your own specific, real tasks
   2. The price the company charges for the model
   3. How many social media posts mention the model
   4. The year the model was originally released

8. A student finds that Model A scores 10% higher than Model B on a popular benchmark, but Model B actually works better for their coding homework. What explains this?

   1. Model B was contaminated during testing
   2. The benchmark tests different skills than the student's homework tasks
   3. The student is not using the models correctly
   4. The benchmark scores must be wrong

9. What does it mean if a benchmark has been 'leaked into training data'?

   1. Researchers published the benchmark answers online
   2. The benchmark creators shared their questions publicly before release
   3. Test questions from the benchmark were accidentally included when training the AI model
   4. The AI model memorized the entire internet

10. What is the difference between a 'benchmark' and a general 'evaluation'?

    1. Benchmarks are for beginners while evaluations are for experts
    2. A benchmark is a standardized, formal test with established questions, while evaluation can be any assessment of performance
    3. A benchmark measures intelligence but evaluation measures speed
    4. There is no difference—they are the same thing

11. The lesson suggests you should test AI models on tasks you actually care about. What is the main reason for this?

    1. AI companies require personal testing data
    2. You cannot trust any benchmark results
    3. Benchmark scores don't always predict how well a model will work for your specific needs
    4. Benchmarks are always more expensive than personal tests

12. What is a 'hype cycle' as referenced in the lesson?

    1. The cycle of training and testing AI models
    2. A technical process for measuring AI learning speed
    3. A pattern of excessive excitement followed by disappointment when claims don't live up to expectations
    4. A type of benchmark that tests emotional intelligence

13. Which of the following would be the strongest sign that an AI benchmark result might not reflect real progress?

    1. The AI model has a colorful logo
    2. The benchmark was created by a university
    3. The test uses multiple choice questions
    4. Researchers discover the test questions appeared in the model's training data

14. A company claims their new AI 'beats humans' on a test. What critical question should a careful reader ask?

    1. Was the test written in English?
    2. Is the company using a Mac or PC?
    3. Does this test measure skills that actually matter for real-world use?
    4. Did humans try their best on the test?

15. Why might testing two different AI models on your own tasks be more useful than just reading benchmark scores?

    1. Benchmarks are illegal to reference
    2. All benchmarks measure the same thing
    3. You need to pay for personal testing but benchmarks are free
    4. You see exactly how each model handles your specific needs rather than generic test questions