robots.txt and ai.txt: The Web's Consent Signals

Martijn Koster created the Robots Exclusion Protocol in 1994. The idea is elegantly simple: every website hosts a plaintext file called robots.txt at its root. Well-behaved crawlers read it and follow its rules. It has no teeth, but it has held the web together for three decades.

The basic syntax

User-agent: * Allow: / Disallow: /private Disallow: /admin User-agent: Googlebot Allow: / Sitemap: https://example.com/sitemap.xml

How AI crawlers handle it

• GPTBot (OpenAI, launched August 2023) — respects robots.txt
• ClaudeBot (Anthropic) — respects robots.txt
• Google-Extended (Google's Gemini training) — respects robots.txt separately from search
• CCBot (Common Crawl) — respects robots.txt
• PerplexityBot — respects robots.txt, but some reports suggest inconsistency

Enter ai.txt

Because robots.txt conflates search indexing with AI training, several proposals have emerged for a separate ai.txt file. Spawning.ai published one standard; others are under discussion at the IETF. The core idea is that publishers should be able to say yes to search and no to AI training without blocking all crawlers.

# Example ai.txt User-Agent: * Disallow: / # Allow specific AI uses Allow-AI: research-noncommercial Allow-AI: translation Disallow-AI: training Disallow-AI: generation

Alternative signals

• HTML meta tags: <meta name="robots" content="noai">
• HTTP headers: X-Robots-Tag: noai, noimageai
• IPTC photo metadata: DataMining: prohibited
• C2PA content credentials: cryptographically signed provenance

The big idea: the web's consent infrastructure was built for a different era. Updating it for AI is an open project, and every site maintainer is a small participant in the standard we end up with.