What does each crawler do?
| Vendor | Name in robots.txt | What the vendor says it does | Allow to be cited? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces websites in ChatGPT's search features | Yes: opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links |
| OpenAI | GPTBot | Crawls content that may be used to train OpenAI's generative AI foundation models | Not needed for search; disallowing means content should not be used in training |
| OpenAI | ChatGPT-User | Visits pages for certain user-initiated actions in ChatGPT and Custom GPTs | OpenAI says because users initiate these, robots.txt rules may not apply; it is not used to decide Search eligibility |
| Googlebot | Main crawler for Google Search; its preferences affect Google Search | Yes: AI Overviews and AI Mode need the page indexed and snippet-eligible | |
| Google-Extended (token only) | Controls whether crawled content may be used for Gemini training and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI | Does not affect inclusion in Google Search and is not a ranking signal; it has no separate user agent | |
| Perplexity | PerplexityBot | Surfaces and links websites in Perplexity search results; "not used to crawl content for AI foundation models" | Yes; Perplexity recommends allowing it and its published IP ranges |
| Perplexity | Perplexity-User | Visits a page when a user's question needs it | Perplexity says it generally ignores robots.txt because a user triggers it |
| Anthropic | Claude-SearchBot | Analyzes content to improve search result relevance | Blocking means content is not indexed for search, which may reduce visibility |
| Anthropic | Claude-User | Retrieves pages when a user's question requires it | Blocking may reduce visibility in user-directed search |
| Anthropic | ClaudeBot | Collects public web content that may be used for model training | Blocking excludes your future content from training datasets |
| Microsoft | Bingbot | Bing's standard crawler | Bing's guidelines list "Blocking Bingbot in your robots.txt file" as something to avoid |
User-agent strings seen on November 19, 2026 include OAI-SearchBot/1.4, GPTBot/1.4, ChatGPT-User/1.0, PerplexityBot/1.0, Perplexity-User/1.0 and bingbot/2.0; vendors say versions can change, so match on the name.
What can you control, and how fast?
- OpenAI says search-related robots.txt changes can take about 24 hours to apply. If a site allows both OAI-SearchBot and GPTBot, OpenAI says it may use one crawl for both purposes.
- Anthropic says its bots honor robots.txt and the non-standard Crawl-delay, advises against IP blocking because it stops the bots reading robots.txt, and publishes its crawler IPs. It also says that from October 30, 2026 it honors a machine-readable "no AI training" signal, rolling out gradually, and that a rule naming ClaudeBot takes precedence.
- Google says it plans to support the AIPREF training preference signal in early 2027. To limit what Search shows from a page, use nosnippet, data-nosnippet, max-snippet or noindex, not Google-Extended.
- Bing says robots.txt controls crawl access, not indexing; use noindex to keep a URL out of Bing, Copilot and grounding results, and note that noarchive prevents content being used in Copilot responses and grounding.
- Verify the visitor. Bing says user agents are easy to spoof and offers a Verify Bingbot tool; OpenAI, Perplexity and Anthropic publish IP lists.
Example: be cited in search, opt out of training
This sketch uses only the names in the table; check each vendor's page before deploying, because names and signals change.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Blocking training crawlers is a business decision with no effect we can source on whether you are cited in search answers. Blocking a search crawler removes you from that engine's answers.
What to do
- Decide per row: cited in search (allow the search bots) and training (your call).
- Leave Googlebot and Bingbot allowed; both vendors tie AI answers to their normal search index.
- Wait up to a day for OpenAI changes, then check server logs for the names above.
- Re-read the vendor pages every quarter; this table is dated.
Related
Sources
Read November 19, 2026:
- OpenAI, Overview of OpenAI crawlers
- Google Search Central, Google's common crawlers and AI features and your website
- Perplexity, Perplexity crawlers
- Anthropic, Does Anthropic crawl data from the web?
- Microsoft Bing, Which crawlers does Bing use? and Bing Webmaster Guidelines
Disclosure: Dominate The World is published by Neural Command LLC (NRLC.ai), which sells AI search visibility, schema and entity services.
Next step
Copy your live robots.txt next to the table above and mark each bot as allowed, blocked or unmentioned.