{"id":11752,"date":"2026-10-29T09:08:46","date_gmt":"2026-10-29T08:08:46","guid":{"rendered":"https:\/\/bernard.digital\/?p=11752"},"modified":"2026-10-29T10:14:36","modified_gmt":"2026-10-29T09:14:36","slug":"ai-crawlers-robots-txt","status":"publish","type":"post","link":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/","title":{"rendered":"Allow or block AI crawlers: robots.txt for GPTBot, ClaudeBot and co."},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Whether an AI can cite your pages is decided long before the answer: at crawl time. AI providers send their own bots to your website, and these AI crawlers come with very different intentions. Some collect training data, others build a search index, others fetch a page because a user just asked about it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here you get an overview of the most important bots, a decision aid, robots.txt templates and two traps that make many websites invisible.<\/p>\n\n\n\n<div class=\"wp-block-group bz-callout-yellow has-text-color has-background is-layout-constrained wp-container-core-group-is-layout-7ff68f20 wp-block-group-is-layout-constrained\" style=\"border-radius:15px;color:#161514;background-color:#ffff9e;padding-top:28px;padding-right:32px;padding-bottom:20px;padding-left:32px\">\n<h2 class=\"wp-block-heading\">Key takeaways<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Three kinds of bots:<\/strong> Training, search and fetching on user request. Each can be controlled separately.<\/li>\n\n\n<li><strong>Block training, allow search:<\/strong> That is possible. You stay visible in AI search without providing training data.<\/li>\n\n\n<li><strong>User fetches:<\/strong> According to the providers, ChatGPT-User and Perplexity-User often don&#8217;t follow robots.txt.<\/li>\n\n\n<li><strong>Google-Extended<\/strong> only controls training for Gemini, not Google Search.<\/li>\n\n\n<li><strong>JavaScript:<\/strong> Most AI crawlers don&#8217;t execute JavaScript. They don&#8217;t see content that is only created in the browser.<\/li>\n<\/ul>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>On this page:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list bz-toc\">\n<li><a href=\"#bots\">Three kinds of AI bots<\/a><\/li>\n\n\n<li><a href=\"#entscheidung\">What you should allow<\/a><\/li>\n\n\n<li><a href=\"#vorlagen\">robots.txt templates<\/a><\/li>\n\n\n<li><a href=\"#firewall\">The silent blocker: CDN and firewall<\/a><\/li>\n\n\n<li><a href=\"#javascript\">JavaScript: what AI crawlers see<\/a><\/li>\n\n\n<li><a href=\"#faq\">FAQ<\/a><\/li>\n\n\n<li><a href=\"#quellen\">Sources<\/a><\/li>\n<\/ul>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"bots\">Three kinds of AI bots<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The big providers now separate their bots by purpose. That is the most important information for any decision, because each purpose has different consequences for your visibility.<\/p>\n\n\n\n<figure class=\"bz-tx\"><figcaption>\/\/ What an AI bot comes for<\/figcaption><pre><span class=\"l k\">TRAINING<\/span><span class=\"l\">  GPTBot \u00b7 ClaudeBot<\/span><span class=\"l c\">  \u2192 knowledge for new models<\/span><span class=\"l\"> <\/span><span class=\"l k\">SEARCH<\/span><span class=\"l\">  OAI-SearchBot \u00b7 Claude-SearchBot<\/span><span class=\"l\">  PerplexityBot<\/span><span class=\"l y\">  \u2192 index for AI search<\/span><span class=\"l\"> <\/span><span class=\"l k\">USER FETCH<\/span><span class=\"l\">  ChatGPT-User \u00b7 Claude-User<\/span><span class=\"l\">  Perplexity-User<\/span><span class=\"l c\">  \u2192 one page for one question<\/span><\/pre><\/figure>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Bot<\/th><th>Purpose and rule<\/th><\/tr><\/thead><tbody><tr><td>GPTBot<\/td><td><strong>OpenAI \u00b7 training<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>OAI-SearchBot<\/td><td><strong>OpenAI \u00b7 ChatGPT search<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>ChatGPT-User<\/td><td><strong>OpenAI \u00b7 fetch on user request<\/strong><br>robots.txt often doesn&#8217;t apply, says OpenAI<\/td><\/tr><tr><td>ClaudeBot<\/td><td><strong>Anthropic \u00b7 training<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>Claude-SearchBot<\/td><td><strong>Anthropic \u00b7 search<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>Claude-User<\/td><td><strong>Anthropic \u00b7 fetch on user request<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>PerplexityBot<\/td><td><strong>Perplexity \u00b7 search<\/strong><br>follows robots.txt<\/td><\/tr><tr><td>Perplexity-User<\/td><td><strong>Perplexity \u00b7 fetch on user request<\/strong><br>mostly ignores robots.txt<\/td><\/tr><tr><td>Google-Extended<\/td><td><strong>Google \u00b7 Gemini training<\/strong><br>no crawler of its own, only a token in robots.txt<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">As of October 2026, according to the providers&#8217; documentation.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Sources: <a href=\"https:\/\/developers.openai.com\/docs\/bots\" target=\"_blank\" rel=\"noreferrer noopener\">OpenAI<\/a>, <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-the-web\" target=\"_blank\" rel=\"noreferrer noopener\">Anthropic<\/a>, <a href=\"https:\/\/docs.perplexity.ai\/docs\/resources\/perplexity-crawlers\" target=\"_blank\" rel=\"noreferrer noopener\">Perplexity<\/a>, on Google-Extended <a href=\"https:\/\/seroundtable.com\/google-extended-seo-36865.html\" target=\"_blank\" rel=\"noreferrer noopener\">Search Engine Roundtable<\/a>.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Excluding training bots doesn&#8217;t mean giving up AI search.<\/p>\n<\/blockquote>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"entscheidung\">What you should allow<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Two questions are enough for the decision. The first is a marketing question, the second a business question.<\/p>\n\n\n\n<figure class=\"bz-tx\"><figcaption>\/\/ Decision aid<\/figcaption><pre><span class=\"l k\">BE NAMED IN AI SEARCH?<\/span><span class=\"l y\">  \u251c\u2500 yes \u2192 allow search bots<\/span><span class=\"l\">  \u2514\u2500 no  \u2192 block search bots<\/span><span class=\"l\"> <\/span><span class=\"l k\">RELEASE CONTENT FOR TRAINING?<\/span><span class=\"l\">  \u251c\u2500 yes \u2192 do nothing<\/span><span class=\"l y\">  \u2514\u2500 no  \u2192 block training bots<\/span><\/pre><\/figure>\n\n\n\n<p class=\"has-background wp-block-paragraph\" style=\"background-color:#c3e6ff\">My recommendation: companies that want to be found always allow the search bots. Whether you block training is a question of what your content is worth, not an SEO question. For most service providers, the benefit of being part of the models&#8217; knowledge outweighs it.<\/p>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"vorlagen\">robots.txt templates<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The robots.txt sits in the root directory of your domain. If there is no entry for a bot, it may access everything. According to OpenAI and Perplexity, changes take effect after about 24 hours.<\/p>\n\n\n\n<figure class=\"bz-tx\"><figcaption>\/\/ Three options<\/figcaption><pre><span class=\"l k\">1 ALLOW EVERYTHING<\/span><span class=\"l\">  add nothing<\/span><span class=\"l y\">2 BLOCK TRAINING<\/span><span class=\"l y\">  allow search<\/span><span class=\"l k\">3 BLOCK EVERYTHING<\/span><span class=\"l c\">  invisible in AI search<\/span><\/pre><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Option 1: allow everything<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You don&#8217;t need to add anything. Just make sure there is no blanket ban such as <code>User-agent: *<\/code> with <code>Disallow: \/<\/code> in the file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Option 2: block training, allow search<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/\n\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Option 3: block everything<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: GPTBot\nDisallow: \/\n\nUser-agent: OAI-SearchBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: Claude-SearchBot\nDisallow: \/\n\nUser-agent: PerplexityBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/<\/code><\/pre>\n\n\n\n<p class=\"has-background wp-block-paragraph\" style=\"background-color:#fff2cc\">Watch out: option 3 makes you largely invisible in AI search. User fetches such as ChatGPT-User still can&#8217;t be reliably prevented this way.<\/p>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"firewall\">The silent blocker: CDN and firewall<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Many websites block AI bots without knowing it. The reason rarely lies in the robots.txt, but one layer before it: in firewalls, security plugins and content delivery networks.<\/p>\n\n\n\n<figure class=\"bz-tx\"><figcaption>\/\/ Where a bot gets stuck<\/figcaption><pre><span class=\"l k\">AI BOT<\/span><span class=\"l\">  \u25bc<\/span><span class=\"l\">  CDN \/ firewall<\/span><span class=\"l y\">  \u251c\u2500 403 \u2192 blocked<\/span><span class=\"l\">  \u25bc<\/span><span class=\"l\">  robots.txt<\/span><span class=\"l\">  \u25bc<\/span><span class=\"l\">  server \u2192 200 \u2192 visible<\/span><span class=\"l c\">robots.txt is only step two<\/span><\/pre><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The best-known example is Cloudflare. Since July 2025, Cloudflare blocks AI crawlers by default for new customers, as <a href=\"https:\/\/www.technologyreview.com\/2025\/07\/01\/1119498\/cloudflare-will-now-by-default-block-ai-bots-from-crawling-its-clients-websites\/\" target=\"_blank\" rel=\"noreferrer noopener\">MIT Technology Review<\/a> reported. If your site is delivered via Cloudflare, check the bot settings, even if your robots.txt allows everything.<\/p>\n\n\n\n<p class=\"has-background wp-block-paragraph\" style=\"background-color:#c3e6ff\">Tip: check your server logs to see whether OAI-SearchBot, Claude-SearchBot and PerplexityBot arrive at all and which status code they get. A 403 means something in front is blocking.<\/p>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"javascript\">JavaScript: what AI crawlers see<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Googlebot executes JavaScript, most AI crawlers don&#8217;t. Vercel analyzed this for its network in <a href=\"https:\/\/vercel.com\/blog\/the-rise-of-the-ai-crawler\" target=\"_blank\" rel=\"noreferrer noopener\">December 2024<\/a>: none of the big AI crawlers, including the bots from OpenAI, Anthropic and Perplexity, executed JavaScript. The exceptions were Gemini, which runs on Googlebot&#8217;s infrastructure, and AppleBot.<\/p>\n\n\n\n<figure class=\"bz-tx\"><figcaption>\/\/ What a crawler sees without JavaScript<\/figcaption><pre><span class=\"l k\">SERVER-SIDE RENDERED<\/span><span class=\"l\">  text is in the HTML<\/span><span class=\"l y\">  \u2192 visible<\/span><span class=\"l\"> <\/span><span class=\"l k\">CLIENT-SIDE RENDERED<\/span><span class=\"l\">  text is created in the browser<\/span><span class=\"l c\">  \u2192 empty page for AI bots<\/span><\/pre><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This mainly affects websites built with React, Vue or similar frameworks that load content only in the browser. The test takes a minute: open the page source (not the developer tools) and search for a sentence from your text. If it isn&#8217;t there, most AI crawlers don&#8217;t see it either.<\/p>\n\n\n\n<p class=\"has-background wp-block-paragraph\" style=\"background-color:#fff2cc\">Watch out: crawler behavior changes. Check the providers&#8217; current documentation before major rebuilds. Google recommends extra SEO care for JavaScript websites anyway (<a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noreferrer noopener\">guide to AI features<\/a>).<\/p>\n\n\n\n<div class=\"wp-block-group bz-callout-dark has-text-color has-background is-layout-constrained wp-container-core-group-is-layout-e8f27b5c wp-block-group-is-layout-constrained\" style=\"border-radius:15px;color:#f4f2ef;background-color:#161514;padding-top:32px;padding-right:36px;padding-bottom:28px;padding-left:36px\">\n<p class=\"bz-callout-eyebrow wp-block-paragraph\">From my practice<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On bernard.digital I currently allow all AI crawlers, including the ones for training. My robots.txt doesn&#8217;t block a single bot. Honestly, I don&#8217;t have a firm opinion on training yet. Part of me wants to get into the models&#8217; long-term memory so they know my definitions. Another part resists the idea of language models being trained on my content. It feels like plagiarism. That is why I keep the two questions apart: I always allow the search bots, and I decide on training separately.<\/p>\n<\/div>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"faq\">FAQ<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Does robots.txt keep AI crawlers out?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">According to their own documentation, the training and search bots from OpenAI, Anthropic and Perplexity follow it. Fetches on user request such as ChatGPT-User or Perplexity-User often don&#8217;t.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Is robots.txt legally binding?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The robots.txt is a technical convention, not a law. For legal questions about reserving usage rights, ask a lawyer. This is not legal advice.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Does Google-Extended harm my Google ranking?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. Google-Extended only controls whether content is used for Gemini and Vertex AI and, according to Google, has no effect on inclusion or ranking in Google Search.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I block AI bots in WordPress?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The easiest way is via the robots.txt in the root directory or via an SEO plugin that can edit the file. Then check at yourdomain.com\/robots.txt whether the entries show up.<\/p>\n\n\n\n<p class=\"bz-bridge wp-block-paragraph\"><strong>Next in the handbook:<\/strong> Besides robots.txt, two more file formats are said to matter for AI. What they really do, and what helps instead, is covered in the next chapter: <a href=\"https:\/\/bernard.digital\/en\/?p=11685\">llms.txt and the entity map<\/a>.<\/p>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"quellen\">Sources<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OpenAI: <a href=\"https:\/\/developers.openai.com\/docs\/bots\" target=\"_blank\" rel=\"noreferrer noopener\">Overview of OpenAI Crawlers<\/a><\/li>\n\n\n<li>Anthropic: <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-the-web\" target=\"_blank\" rel=\"noreferrer noopener\">Does Anthropic crawl the web?<\/a><\/li>\n\n\n<li>Perplexity: <a href=\"https:\/\/docs.perplexity.ai\/docs\/resources\/perplexity-crawlers\" target=\"_blank\" rel=\"noreferrer noopener\">Perplexity Crawlers<\/a><\/li>\n\n\n<li>Search Engine Roundtable: <a href=\"https:\/\/seroundtable.com\/google-extended-seo-36865.html\" target=\"_blank\" rel=\"noreferrer noopener\">Google-Extended does not affect Google Search<\/a>, February 2024<\/li>\n\n\n<li>Vercel: <a href=\"https:\/\/vercel.com\/blog\/the-rise-of-the-ai-crawler\" target=\"_blank\" rel=\"noreferrer noopener\">The rise of the AI crawler<\/a>, December 2024<\/li>\n\n\n<li>MIT Technology Review: <a href=\"https:\/\/www.technologyreview.com\/2025\/07\/01\/1119498\/cloudflare-will-now-by-default-block-ai-bots-from-crawling-its-clients-websites\/\" target=\"_blank\" rel=\"noreferrer noopener\">Cloudflare will now block AI bots by default<\/a>, July 2025<\/li>\n\n\n<li>Google Search Central: <a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noreferrer noopener\">AI features and your website<\/a>, July 2026<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Which AI bots exist, what they come for and how to control them via robots.txt: block training, allow search, with templates and the CDN and JavaScript traps.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_wp_convertkit_post_meta":{"form":"-1","landing_page":"0","tag":"0","restrict_content":"0"},"footnotes":""},"categories":[124],"tags":[],"class_list":["post-11752","post","type-post","status-publish","format-standard","category-geo"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Block or allow AI crawlers: robots.txt [Templates]<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Block or allow AI crawlers: robots.txt [Templates]\" \/>\n<meta property=\"og:description\" content=\"Which AI bots exist, what they come for and how to control them via robots.txt: block training, allow search, with templates and the CDN and JavaScript traps.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"bernard.digital\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-29T08:08:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-29T09:14:36+00:00\" \/>\n<meta name=\"author\" content=\"Bernard Zitzer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@BernardZitzer\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bernard Zitzer\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/\"},\"author\":{\"name\":\"Bernard Zitzer\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/#\\\/schema\\\/person\\\/f1805ea4f33fa8bb2024847266a4b81e\"},\"headline\":\"Allow or block AI crawlers: robots.txt for GPTBot, ClaudeBot and co.\",\"datePublished\":\"2026-10-29T08:08:46+00:00\",\"dateModified\":\"2026-10-29T09:14:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/\"},\"wordCount\":1072,\"articleSection\":[\"GEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/\",\"url\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/\",\"name\":\"Block or allow AI crawlers: robots.txt [Templates]\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/#website\"},\"datePublished\":\"2026-10-29T08:08:46+00:00\",\"dateModified\":\"2026-10-29T09:14:36+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/#\\\/schema\\\/person\\\/f1805ea4f33fa8bb2024847266a4b81e\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/ai-crawlers-robots-txt\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Start\",\"item\":\"https:\\\/\\\/bernard.digital\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Allow or block AI crawlers: robots.txt for GPTBot, ClaudeBot and co.\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/bernard.digital\\\/en\\\/\",\"name\":\"bernard.digital\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/bernard.digital\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/bernard.digital\\\/en\\\/#\\\/schema\\\/person\\\/f1805ea4f33fa8bb2024847266a4b81e\",\"name\":\"Bernard Zitzer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g\",\"caption\":\"Bernard Zitzer\"},\"description\":\"Ich bin Bernard Zitzer, SEO- und GEO-Berater aus Berlin. Ich bringe Marken in Google und in KI-Antworten nach vorn und arbeite daf\u00fcr mit h\u00f6chstens f\u00fcnf Dauerkunden und einem Projektkunden gleichzeitig.\",\"sameAs\":[\"https:\\\/\\\/bernard.digital\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/bernardzitzer\\\/\",\"https:\\\/\\\/x.com\\\/BernardZitzer\"],\"url\":\"https:\\\/\\\/bernard.digital\\\/en\\\/author\\\/bernard\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Block or allow AI crawlers: robots.txt [Templates]","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Block or allow AI crawlers: robots.txt [Templates]","og_description":"Which AI bots exist, what they come for and how to control them via robots.txt: block training, allow search, with templates and the CDN and JavaScript traps.","og_url":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/","og_site_name":"bernard.digital","article_published_time":"2026-10-29T08:08:46+00:00","article_modified_time":"2026-10-29T09:14:36+00:00","author":"Bernard Zitzer","twitter_card":"summary_large_image","twitter_creator":"@BernardZitzer","twitter_misc":{"Written by":"Bernard Zitzer"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/#article","isPartOf":{"@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/"},"author":{"name":"Bernard Zitzer","@id":"https:\/\/bernard.digital\/en\/#\/schema\/person\/f1805ea4f33fa8bb2024847266a4b81e"},"headline":"Allow or block AI crawlers: robots.txt for GPTBot, ClaudeBot and co.","datePublished":"2026-10-29T08:08:46+00:00","dateModified":"2026-10-29T09:14:36+00:00","mainEntityOfPage":{"@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/"},"wordCount":1072,"articleSection":["GEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/","url":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/","name":"Block or allow AI crawlers: robots.txt [Templates]","isPartOf":{"@id":"https:\/\/bernard.digital\/en\/#website"},"datePublished":"2026-10-29T08:08:46+00:00","dateModified":"2026-10-29T09:14:36+00:00","author":{"@id":"https:\/\/bernard.digital\/en\/#\/schema\/person\/f1805ea4f33fa8bb2024847266a4b81e"},"breadcrumb":{"@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/bernard.digital\/en\/ai-crawlers-robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Start","item":"https:\/\/bernard.digital\/en\/"},{"@type":"ListItem","position":2,"name":"Allow or block AI crawlers: robots.txt for GPTBot, ClaudeBot and co."}]},{"@type":"WebSite","@id":"https:\/\/bernard.digital\/en\/#website","url":"https:\/\/bernard.digital\/en\/","name":"bernard.digital","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bernard.digital\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bernard.digital\/en\/#\/schema\/person\/f1805ea4f33fa8bb2024847266a4b81e","name":"Bernard Zitzer","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24b78f3f96e5b588272554226fe171e525c02d1c70395a51588c831c55374a90?s=96&d=mm&r=g","caption":"Bernard Zitzer"},"description":"Ich bin Bernard Zitzer, SEO- und GEO-Berater aus Berlin. Ich bringe Marken in Google und in KI-Antworten nach vorn und arbeite daf\u00fcr mit h\u00f6chstens f\u00fcnf Dauerkunden und einem Projektkunden gleichzeitig.","sameAs":["https:\/\/bernard.digital","https:\/\/www.linkedin.com\/in\/bernardzitzer\/","https:\/\/x.com\/BernardZitzer"],"url":"https:\/\/bernard.digital\/en\/author\/bernard\/"}]}},"_links":{"self":[{"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/posts\/11752","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/comments?post=11752"}],"version-history":[{"count":7,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/posts\/11752\/revisions"}],"predecessor-version":[{"id":12148,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/posts\/11752\/revisions\/12148"}],"wp:attachment":[{"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/media?parent=11752"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/categories?post=11752"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bernard.digital\/en\/wp-json\/wp\/v2\/tags?post=11752"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}