The short answer
If discoverability in AI-assisted search is your goal, review and intentionally configure the agents used for search and user-requested access. Decide separately whether you want to allow agents used for model training. Do not treat one blanket “AI bot” rule as if every agent has the same purpose.
The agent purposes below come from the publishers' current documentation. Recheck the linked sources before changing production controls because names and product behavior can change.
Map each agent to its published purpose
| Publisher and token | Published purpose | Relevant decision |
|---|---|---|
| OpenAI: OAI-SearchBot | Surfaces websites in ChatGPT search results. | Search discoverability. Source |
| OpenAI: GPTBot | Crawls content that may be used to train generative AI foundation models. | Model-training preference. Source |
| OpenAI: ChatGPT-User | May visit a page for certain user-initiated actions; it is not used to determine Search inclusion. | User-requested access. Source |
| Anthropic: Claude-SearchBot | Navigates the web to improve search-result quality. | Search discoverability. Source |
| Anthropic: Claude-User | Supports page access initiated by Claude users. | User-requested access. Source |
| Anthropic: ClaudeBot | Collects web content that may contribute to model training. | Model-training preference. Source |
| Perplexity: PerplexityBot | Surfaces and links websites in Perplexity search results; Perplexity says it is not used for foundation-model training. | Search discoverability. Source |
| Perplexity: Perplexity-User | Supports user-requested page visits. | User-requested access. Source |
| Google: Googlebot | Controls crawling for Google Search, including AI features in Search. | Google Search access and preview controls. Source |
| Google: Google-Extended | A product token for managing specified Gemini training and grounding uses. Google says it does not affect inclusion or ranking in Google Search. | Training and grounding preference, separate from Google Search. Source |
Use a purpose-based decision process
- List the products and discovery surfaces that matter to the business.
- Separate search agents, user-requested fetchers, training crawlers, and product-control tokens.
- Check whether a broad `User-agent: *` rule already allows or blocks the path.
- Review page-level `noindex`, snippet controls, authentication, and firewall rules separately.
- Document the business decision and source-review date.
- Verify the public robots.txt response and monitor legitimate crawler access where logs are available.
Example for an intentionally open public site
User-agent: * Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
This example does not express a training preference and is not a universal recommendation. Add or omit specific groups according to the site's policy and the publisher documentation.
Verify more than the file contents
- The robots.txt URL returns the intended public file.
- The sitemap URL is correct and public.
- No contradictory group overrides the intended rule.
- The page itself does not carry an unintended `noindex` directive.
- The firewall or hosting layer does not block an agent you intended to allow.
- The review date and documentation links are recorded.
Check the page after access.
Once the page can be reached, audit whether its answer, entity context, evidence, and structure are usable.
Audit a page free