GEO Glossary
AI crawlers
AI crawlers are bots from AI providers that fetch websites, either to collect training data or to retrieve content to answer a specific question.
At a glance
TRAINING GPTBot · ClaudeBot · CCBot collect text for future models SEARCH AND ANSWER OAI-SearchBot · PerplexityBot fetch content for answers with sources
Example
If a website blocks all AI bots wholesale in robots.txt, it ends up neither in training data nor in answers with sources. Many sites have done this unnoticed, for example through a default setting in the CDN or a security plugin.
In practice
Check your robots.txt and your hosting or CDN settings. If you want to be named in AI answers, you should at least allow the search bots. Whether you also allow training bots is a separate decision.
How it fits together
Training bots fill the Training data, search bots deliver the content for RAG. The llms.txt is a voluntary extra offered to these bots.