Standard Definition
AI crawlers are web crawlers of AI search and AI language model providers that collect web content for model training data or search functions. Important representatives: GPTBot (OpenAI training data), OAI-SearchBot (ChatGPT Search), Claude-Web and ClaudeBot (Anthropic), PerplexityBot (Perplexity AI), Google-Extended (training crawler for Gemini, separated from Googlebot), Applebot-Extended, Bytespider (ByteDance/TikTok), Bingbot (Microsoft Copilot). Control happens via robots.txt per User-Agent string — important is granular control: training crawlers and search crawlers of the same provider can be separately allowed or blocked. Strategic implications per provider differ substantially.
What this means in mandate practice
AI crawler strategy is an independent strategic decision — and is often neglected.
First, the standard robots.txt of most sites implicitly allows AI crawlers. Those who set no explicit Disallow instructions per AI bot allow crawling for training and search functions. This is in most cases the strategically right choice — the visibility advantages in AI answers outweigh the data protection concerns. Those who explicitly block all AI crawlers actively exclude themselves from the search perception of the next 5-10 years.
Second, the separation between training and search crawlers is strategically relevant. GPTBot is OpenAI's training crawler, OAI-SearchBot is the search crawler — separately controllable. Google analogously separates Googlebot (search index) and Google-Extended (Gemini training). Sites with concerns about training data use but wanting search visibility can block the training crawler and allow the search crawler. This differentiated control is the most-sensible strategy — blanket blockings are rarely optimal.
Third, AI crawlers evolve quickly — maintenance is necessary. New bots appear; existing bots get renamed or split. Those running a 2023 robots.txt unchanged have not specifically considered current AI crawlers. Calvarius recommends an annual robots.txt review in Q1 — simultaneously with the glossary update cycle. In mandates with an AI visibility focus, we additionally check server logs monthly for new bot user agents and decide deliberately on allowance or blocking for each new bot.