robots.txt and ai.txt: The Web's Consent Signals

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-robots-txt-ai-txt

1. What year was the Robots Exclusion Protocol originally created?

   1. 2010
   2. 2001
   3. 1994
   4. 1999

2. A website operator wants to allow search engine indexing but prevent AI companies from training on their content. Which file combination would achieve this?

   1. Only ai.txt since robots.txt is obsolete
   2. robots.txt configured for search + ai.txt configured to block AI training
   3. Only robots.txt with a special AI flag
   4. Both files set to block all crawlers

3. Which AI crawler has faced public reports of inconsistent adherence to robots.txt rules?

   1. Google-Extended
   2. PerplexityBot
   3. ClaudeBot
   4. GPTBot

4. A developer argues that robots.txt should legally force AI companies to respect its rules. What limitation exists in the current system?

   1. Robots.txt only works for European websites
   2. Neither robots.txt nor ai.txt has legal force on its own—they are signals only
   3. AI companies are legally required to ignore robots.txt
   4. Robots.txt has expired copyright protection

5. What technical mechanism would a photographer use to embed a machine-readable instruction that prevents AI training on their images?

   1. Adding a watermark to images
   2. Encrypting the image files
   3. IPTC metadata tag DataMining: prohibited
   4. Registering with a copyright office

6. Which organization published one proposed standard for ai.txt?

   1. The U.S. government
   2. Wikipedia
   3. Spawning.ai
   4. W3C

7. What does the HTTP header X-Robots-Tag allow website operators to do?

   1. Block all users from certain countries
   2. Automatically translate page content
   3. Encrypt all website traffic
   4. Specify crawler rules at the page or resource level rather than site-wide

8. A content creator wants to ensure their written articles cannot be used for AI training while still appearing in Google search results. What should they do?

   1. Add a password to their website
   2. Configure robots.txt to allow search crawlers + ai.txt to block AI training crawlers
   3. Remove all images from their articles
   4. Register as a nonprofit organization

9. Which company created the crawler named GPTBot?

   1. Google
   2. OpenAI
   3. Meta
   4. Microsoft

10. What is C2PA primarily designed to provide for digital content?

    1. Better search engine rankings
    2. Video compression
    3. Faster page loading times
    4. Cryptographically signed provenance and authenticity

11. Google-Extended behaves differently from other Google crawlers in what way?

    1. It only indexes video content
    2. It cannot be blocked
    3. It requires payment to access
    4. It respects robots.txt separately from Google's search crawler

12. What is the fundamental limitation that led to proposals like ai.txt?

    1. Robots.txt cannot distinguish between search indexing and AI model training
    2. Robots.txt requires payment to use
    3. Robots.txt is encrypted
    4. Robots.txt only works on English websites

13. The EU AI Act (2024) takes a step toward enforcement by requiring what?

    1. All websites to block AI access
    2. Crawlers to respect rights-reservations
    3. AI models to disclose training sources
    4. AI companies to pay royalties to all websites

14. What happens when a well-behaved crawler encounters a website's robots.txt file?

    1. It reports the website to authorities
    2. It reads and follows the rules specified within
    3. It indexes the site faster
    4. It deletes the file automatically

15. Which crawler is associated with Anthropic?

    1. CCBot
    2. Google-Extended
    3. ClaudeBot
    4. GPTBot