# How to configure robots.txt for GPTBot, ClaudeBot and PerplexityBot | Illia BizTech > Based on official documentation from OpenAI, Anthropic, Perplexity and Google, here is what each AI crawler actually does and whether you should block it in robots.txt. Source: https://illia-biztech.de/en/blog/ki-crawler-robots-txt-erlauben 1. [Home](/en) 2. [Resources](/en/blog) 3. GPTBot, ClaudeBot and PerplexityBot in robots.txt. Allow or block ![Technician checking server access logs across multiple monitors](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_ki_crawler_robots_txt_erlauben_d862958530.png) ![Portrait of Illia Belichenko, technical specialist at Illia BizTech](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/small_Illia_Belichenko_author_81f88864f0.png) By: Illia Belichenko November 10, 2026 # GPTBot, ClaudeBot and PerplexityBot in robots.txt. Allow or block GPTBot, ClaudeBot and PerplexityBot are three different crawlers from three different companies, and each company also runs separate bots for model training and for live search. There is no single answer to allow or block, the right call depends on the specific bot. ## What GPTBot is and how it differs from OpenAI's other bots OpenAI runs four distinct crawlers, each described separately in [its official documentation](https://developers.openai.com/api/docs/bots). GPTBot collects content to train generative models, OAI-SearchBot indexes pages for ChatGPT's search results, ChatGPT-User fetches a page on a specific user's direct request, and OAI-AdsBot checks the safety of ad landing pages. Blocking GPTBot in robots.txt does not remove a site from ChatGPT's search results, the separate OAI-SearchBot handles that. It only means future content from the site will not go into training data for OpenAI's future models. ## Should you block ClaudeBot Anthropic also splits its crawlers by task. According to [Anthropic's help page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), ClaudeBot collects data for model training, Claude-User fetches a page when a Claude user asks about it directly, and Claude-SearchBot improves search quality inside Claude. All three bots honor robots.txt rules. Blocking only makes sense for the specific bot whose function a business does not want to support. If the goal is keeping content out of model training while staying reachable in Claude's live answers, only ClaudeBot gets blocked, Claude-User and Claude-SearchBot stay open. ## Should you allow PerplexityBot in robots.txt PerplexityBot collects pages for Perplexity's own index, Perplexity-User fetches a page the moment someone asks about it, both agents are described in [Perplexity's official documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), which also publishes IP lists for verifying a request's authenticity. Allowing PerplexityBot comes with one complication that GPTBot and ClaudeBot do not have. In August 2025 Cloudflare publicly accused Perplexity of continuing to fetch data after being blocked through robots.txt, disguising itself as an ordinary browser by rotating its user agent and IP addresses, a story covered in detail by [Search Engine Journal](https://www.searchenginejournal.com/cloudflare-delists-and-blocks-perplexity-from-crawling-websites/552899/). Perplexity called Cloudflare's accusation a marketing move and denied that the bots in question even belonged to the company. The practical takeaway is that a robots.txt rule formally stops a well-behaved bot, but it does not guarantee full protection from a company that has already been part of a dispute like this one. ## What actually happens if you block AI crawlers The consequences differ by bot, and it matters not to mix them up. Blocking Google-Extended has no effect on a site's presence in regular Google Search and is not a ranking signal, as stated directly in [Google's crawler overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers), because that token only controls whether content trains Gemini. Blocking CCBot cuts off the Common Crawl dataset that has trained many language models, including open ones, as described on [Common Crawl's own page](https://commoncrawl.org/ccbot). Blocking search bots such as OAI-SearchBot or PerplexityBot closes off that crawler's direct access to the site's own pages and stops the assistant from citing them directly, but it does not rule out the business being mentioned. The assistant can still learn about it from reviews, directories, and other sites it does crawl. That decision is worth making deliberately rather than by default. ## How an AI crawler differs from an ordinary search bot An ordinary search bot like Googlebot builds an index to show a page in results and sends the site traffic through a click. A training crawler such as GPTBot or CCBot fetches text once for training a model and sends no direct traffic at all, while a search crawler such as OAI-SearchBot or PerplexityBot behaves more like an ordinary search bot but fetches a page far less often. That leads to different decision logic. Blocking a training crawler makes sense if a business does not want to give away content for free, blocking a search crawler only makes sense if a business is willing to disappear from that assistant's answers entirely. ## How to check which bots visit a site The most reliable method is checking the server or hosting access logs and looking for User-Agent strings like GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. After that, it is worth confirming a bot is genuine. OpenAI, Anthropic and Perplexity all publish official IP ranges and recommend verifying requests through reverse DNS, stated in both [OpenAI's documentation](https://developers.openai.com/api/docs/bots) and [Perplexity's documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). ## What to check first * Open the hosting's access logs and look for User-Agent lines with GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and CCBot * Decide on each bot separately, not with one blanket Disallow: / rule * If the goal is appearing in assistant answers, keep the search bots open (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and block only the training bots (GPTBot, ClaudeBot, CCBot) if wanted * Verify unusual traffic against the official IP lists instead of trusting the User-Agent string alone **Share** [ ![Developer inspecting a website's structured data in browser developer tools](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_schema_markup_lokales_unternehmen_7a4925ced8.png) Previous Post Schema.org structured data for a local business site. What you actually need](/en/blog/schema-markup-lokales-unternehmen) [ ![Developer editing the llms.txt text file in a website's code on a laptop](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_llms_txt_kleines_unternehmen_bf0e7164f2.png) Next Post The llms.txt file on a website. Does a small business need it](/en/blog/llms-txt-kleines-unternehmen) ![Portrait of Illia Belichenko, technical specialist at Illia BizTech](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/small_Illia_Belichenko_author_81f88864f0.png) ##### Illia Belichenko Illia is responsible for the technical side of Illia BizTech: website development, website updates, forms, integrations, automation and technical support. He works on connecting websites, content, advertising and internal workflows so they work together without creating unnecessary manual work. ##### Recent Posts [![Checking a company's registration in the German Handelsregister on a laptop screen](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_handelsregister_impressum_agentur_pruefen_84f39c129a.png)](/en/blog/handelsregister-impressum-agentur-pruefen) #### [Checking a web agency before you pay: Handelsregister and Impressum](/en/blog/handelsregister-impressum-agentur-pruefen) November 13, 2026 [](/en/blog/handelsregister-impressum-agentur-pruefen) [![Person reading Google search results with an AI overview on a smartphone screen](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_google_ai_overviews_sichtbarkeit_edbe04f07f.png)](/en/blog/google-ai-overviews-sichtbarkeit) #### [What a website needs to appear in Google AI Overviews](/en/blog/google-ai-overviews-sichtbarkeit) November 12, 2026 [](/en/blog/google-ai-overviews-sichtbarkeit) [![Developer inspecting a website's structured data in browser developer tools](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/medium_schema_markup_lokales_unternehmen_7a4925ced8.png)](/en/blog/schema-markup-lokales-unternehmen) #### [Schema.org structured data for a local business site. What you actually need](/en/blog/schema-markup-lokales-unternehmen) November 11, 2026 [](/en/blog/schema-markup-lokales-unternehmen) GPTBot and OAI-SearchBot from OpenAI handle different jobs, and one blanket Disallow rule for every AI bot at once almost always blocks more than a business actually wants to block. Illia Belichenko configures robots.txt for each client bot by bot, rather than copying one generic template from a forum. Illia BizTech builds and maintains websites for companies in Germany in German, English, Russian and Ukrainian, from the first version through ongoing maintenance and search visibility work. One monthly plan covers the website, advertising and content, the scope is agreed in advance and can be moved between them when priorities change. The domain, hosting and ad accounts stay registered to the company, so the work can continue on the existing setup without a full rebuild. That fits businesses moving to Germany or expanding into the German-speaking market who want to explain their task once.