AI Search

AI crawlers

AI crawlers are automated bots operated by AI companies (OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Google-Extended) that fetch web pages to use as training data or as live citation sources.

Allowing AI crawlers in robots.txt is now a GEO requirement. The major bots to allow explicitly: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and Bingbot. Blocking them removes you from AI answer surfaces entirely — a far worse trade than the marginal infrastructure cost of serving them.

Related terms

llms.txt

A markdown file that tells AI engines what your site is about.

robots.txt

A file that tells crawlers which URLs to fetch.

Generative Engine Optimization (GEO)

Optimizing content so LLMs cite it verbatim in AI answers.

Ready to actually rank in AI search?

Local Pages generates GEO-optimized landing pages for any local business in minutes.

Start free