Local Pages
SEO

robots.txt

robots.txt is a plain-text file at the root of a website (/robots.txt) that tells crawlers which paths they're allowed to fetch, following the Robots Exclusion Protocol.

For GEO, robots.txt should explicitly Allow all major AI crawlers and reference your sitemap. Common mistake: a wildcard Disallow that accidentally blocks GPTBot or ClaudeBot. Audit yours at /robots.txt and confirm each AI user-agent has an Allow directive.

Last reviewed on by the Local Pages editorial team.

Related terms

AI crawlers

Bots that gather web content for LLM training and citations.

llms.txt

A markdown file that tells AI engines what your site is about.

Ready to actually rank in AI search?

Local Pages generates GEO-optimized landing pages for any local business in minutes.

Start free