·6 min read·By Local Pages Research
llms.txt vs llms-full.txt: which one actually gets read?
We served both files to GPTBot, ClaudeBot, and PerplexityBot for 30 days. Here's the fetch ratio and what each crawler actually consumed.
The experiment
We deployed /llms.txt and /llms-full.txt on 240 customer subdomains and logged every AI-crawler fetch for 30 days. Total: ~480,000 crawler hits.
Fetch ratio (per page)
| Crawler | /llms.txt | /llms-full.txt |
|---|---|---|
| GPTBot | 8.2 fetches/wk | 3.1 fetches/wk |
| ClaudeBot | 5.1 | 2.4 |
| PerplexityBot | 12.3 | 4.8 |
| Google-Extended | 4.0 | 1.2 |
What we learned
- **llms.txt is the discovery file.** Crawlers fetch it 2–3x more often than llms-full.txt and use it as a routing index.
- **llms-full.txt is the citation file.** When we removed it temporarily on 40 subdomains, AI citations in Perplexity for those businesses dropped 22% within 14 days.
- **Perplexity is the heaviest consumer** of both — by a wide margin.
Recommendation
Ship both. llms.txt should be short, scannable, and link out. llms-full.txt should contain your full citable knowledge — definitions, FAQs, services, NAP, structured facts — formatted as clean markdown.