Whether an AI can cite your pages is decided long before the answer: at crawl time. AI providers send their own bots to your website, and these AI crawlers come with very different intentions. Some collect training data, others build a search index, others fetch a page because a user just asked about it.
Here you get an overview of the most important bots, a decision aid, robots.txt templates and two traps that make many websites invisible.
Key takeaways
- Three kinds of bots: Training, search and fetching on user request. Each can be controlled separately.
- Block training, allow search: That is possible. You stay visible in AI search without providing training data.
- User fetches: According to the providers, ChatGPT-User and Perplexity-User often don’t follow robots.txt.
- Google-Extended only controls training for Gemini, not Google Search.
- JavaScript: Most AI crawlers don’t execute JavaScript. They don’t see content that is only created in the browser.
On this page:
- Three kinds of AI bots
- What you should allow
- robots.txt templates
- The silent blocker: CDN and firewall
- JavaScript: what AI crawlers see
- FAQ
- Sources
Three kinds of AI bots
The big providers now separate their bots by purpose. That is the most important information for any decision, because each purpose has different consequences for your visibility.
TRAINING GPTBot · ClaudeBot → knowledge for new models SEARCH OAI-SearchBot · Claude-SearchBot PerplexityBot → index for AI search USER FETCH ChatGPT-User · Claude-User Perplexity-User → one page for one question
| Bot | Purpose and rule |
|---|---|
| GPTBot | OpenAI · training follows robots.txt |
| OAI-SearchBot | OpenAI · ChatGPT search follows robots.txt |
| ChatGPT-User | OpenAI · fetch on user request robots.txt often doesn’t apply, says OpenAI |
| ClaudeBot | Anthropic · training follows robots.txt |
| Claude-SearchBot | Anthropic · search follows robots.txt |
| Claude-User | Anthropic · fetch on user request follows robots.txt |
| PerplexityBot | Perplexity · search follows robots.txt |
| Perplexity-User | Perplexity · fetch on user request mostly ignores robots.txt |
| Google-Extended | Google · Gemini training no crawler of its own, only a token in robots.txt |
Sources: OpenAI, Anthropic, Perplexity, on Google-Extended Search Engine Roundtable.
Excluding training bots doesn’t mean giving up AI search.
What you should allow
Two questions are enough for the decision. The first is a marketing question, the second a business question.
BE NAMED IN AI SEARCH? ├─ yes → allow search bots └─ no → block search bots RELEASE CONTENT FOR TRAINING? ├─ yes → do nothing └─ no → block training bots
My recommendation: companies that want to be found always allow the search bots. Whether you block training is a question of what your content is worth, not an SEO question. For most service providers, the benefit of being part of the models’ knowledge outweighs it.
robots.txt templates
The robots.txt sits in the root directory of your domain. If there is no entry for a bot, it may access everything. According to OpenAI and Perplexity, changes take effect after about 24 hours.
1 ALLOW EVERYTHING add nothing2 BLOCK TRAINING allow search3 BLOCK EVERYTHING invisible in AI search
Option 1: allow everything
You don’t need to add anything. Just make sure there is no blanket ban such as User-agent: * with Disallow: / in the file.
Option 2: block training, allow search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Option 3: block everything
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /Watch out: option 3 makes you largely invisible in AI search. User fetches such as ChatGPT-User still can’t be reliably prevented this way.
The silent blocker: CDN and firewall
Many websites block AI bots without knowing it. The reason rarely lies in the robots.txt, but one layer before it: in firewalls, security plugins and content delivery networks.
AI BOT ▼ CDN / firewall ├─ 403 → blocked ▼ robots.txt ▼ server → 200 → visiblerobots.txt is only step two
The best-known example is Cloudflare. Since July 2025, Cloudflare blocks AI crawlers by default for new customers, as MIT Technology Review reported. If your site is delivered via Cloudflare, check the bot settings, even if your robots.txt allows everything.
Tip: check your server logs to see whether OAI-SearchBot, Claude-SearchBot and PerplexityBot arrive at all and which status code they get. A 403 means something in front is blocking.
JavaScript: what AI crawlers see
Googlebot executes JavaScript, most AI crawlers don’t. Vercel analyzed this for its network in December 2024: none of the big AI crawlers, including the bots from OpenAI, Anthropic and Perplexity, executed JavaScript. The exceptions were Gemini, which runs on Googlebot’s infrastructure, and AppleBot.
SERVER-SIDE RENDERED text is in the HTML → visible CLIENT-SIDE RENDERED text is created in the browser → empty page for AI bots
This mainly affects websites built with React, Vue or similar frameworks that load content only in the browser. The test takes a minute: open the page source (not the developer tools) and search for a sentence from your text. If it isn’t there, most AI crawlers don’t see it either.
Watch out: crawler behavior changes. Check the providers’ current documentation before major rebuilds. Google recommends extra SEO care for JavaScript websites anyway (guide to AI features).
From my practice
On bernard.digital I currently allow all AI crawlers, including the ones for training. My robots.txt doesn’t block a single bot. Honestly, I don’t have a firm opinion on training yet. Part of me wants to get into the models’ long-term memory so they know my definitions. Another part resists the idea of language models being trained on my content. It feels like plagiarism. That is why I keep the two questions apart: I always allow the search bots, and I decide on training separately.
FAQ
Does robots.txt keep AI crawlers out?
According to their own documentation, the training and search bots from OpenAI, Anthropic and Perplexity follow it. Fetches on user request such as ChatGPT-User or Perplexity-User often don’t.
Is robots.txt legally binding?
The robots.txt is a technical convention, not a law. For legal questions about reserving usage rights, ask a lawyer. This is not legal advice.
Does Google-Extended harm my Google ranking?
No. Google-Extended only controls whether content is used for Gemini and Vertex AI and, according to Google, has no effect on inclusion or ranking in Google Search.
How do I block AI bots in WordPress?
The easiest way is via the robots.txt in the root directory or via an SEO plugin that can edit the file. Then check at yourdomain.com/robots.txt whether the entries show up.
Next in the handbook: Besides robots.txt, two more file formats are said to matter for AI. What they really do, and what helps instead, is covered in the next chapter: llms.txt and the entity map.
Sources
- OpenAI: Overview of OpenAI Crawlers
- Anthropic: Does Anthropic crawl the web?
- Perplexity: Perplexity Crawlers
- Search Engine Roundtable: Google-Extended does not affect Google Search, February 2024
- Vercel: The rise of the AI crawler, December 2024
- MIT Technology Review: Cloudflare will now block AI bots by default, July 2025
- Google Search Central: AI features and your website, July 2026
