Dominate The World

Should you allow GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot and Bingbot in robots.txt?

Allow the crawlers that feed search answers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot) if you want to be cited, and decide separately about the training crawlers (GPTBot, ClaudeBot and the Google-Extended token), because the vendors say these are independent settings. OpenAI says "Each setting is independent of the others" and that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers"; Google says Google-Extended does not affect Search inclusion or ranking. Bots that act on a user request (ChatGPT-User, Perplexity-User) may not follow robots.txt at all. Read from each vendor's documentation on November 19, 2026.

What does each crawler do?

Vendor Name in robots.txt What the vendor says it does Allow to be cited?
OpenAI OAI-SearchBot Surfaces websites in ChatGPT's search features Yes: opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links
OpenAI GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models Not needed for search; disallowing means content should not be used in training
OpenAI ChatGPT-User Visits pages for certain user-initiated actions in ChatGPT and Custom GPTs OpenAI says because users initiate these, robots.txt rules may not apply; it is not used to decide Search eligibility
Google Googlebot Main crawler for Google Search; its preferences affect Google Search Yes: AI Overviews and AI Mode need the page indexed and snippet-eligible
Google Google-Extended (token only) Controls whether crawled content may be used for Gemini training and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI Does not affect inclusion in Google Search and is not a ranking signal; it has no separate user agent
Perplexity PerplexityBot Surfaces and links websites in Perplexity search results; "not used to crawl content for AI foundation models" Yes; Perplexity recommends allowing it and its published IP ranges
Perplexity Perplexity-User Visits a page when a user's question needs it Perplexity says it generally ignores robots.txt because a user triggers it
Anthropic Claude-SearchBot Analyzes content to improve search result relevance Blocking means content is not indexed for search, which may reduce visibility
Anthropic Claude-User Retrieves pages when a user's question requires it Blocking may reduce visibility in user-directed search
Anthropic ClaudeBot Collects public web content that may be used for model training Blocking excludes your future content from training datasets
Microsoft Bingbot Bing's standard crawler Bing's guidelines list "Blocking Bingbot in your robots.txt file" as something to avoid

User-agent strings seen on November 19, 2026 include OAI-SearchBot/1.4, GPTBot/1.4, ChatGPT-User/1.0, PerplexityBot/1.0, Perplexity-User/1.0 and bingbot/2.0; vendors say versions can change, so match on the name.

What can you control, and how fast?

Example: be cited in search, opt out of training

This sketch uses only the names in the table; check each vendor's page before deploying, because names and signals change.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Blocking training crawlers is a business decision with no effect we can source on whether you are cited in search answers. Blocking a search crawler removes you from that engine's answers.

What to do

  1. Decide per row: cited in search (allow the search bots) and training (your call).
  2. Leave Googlebot and Bingbot allowed; both vendors tie AI answers to their normal search index.
  3. Wait up to a day for OpenAI changes, then check server logs for the names above.
  4. Re-read the vendor pages every quarter; this table is dated.

Related

Sources

Read November 19, 2026:

Disclosure: Dominate The World is published by Neural Command LLC (NRLC.ai), which sells AI search visibility, schema and entity services.

Next step

Copy your live robots.txt next to the table above and mark each bot as allowed, blocked or unmentioned.