Review AI crawler access by separating search discovery, model training, and user-requested page visits. Identify the documented control for each purpose, then check both your published rules and what your site actually serves.
An allowed crawl doesn't prove that a page was indexed, retrieved, or cited. A useful review records the policy you intended, the access you observed, and the conclusions the evidence supports.
Separate search access from model training
Discovery, training, and user-requested visits serve different purposes.
Start with a decision about the content. Which public pages should be available for search discovery? What are your preferences for the provider's documented training uses? Which pages require a signed-in user regardless of the visitor's software?
“Allow AI” is too broad to answer those questions. The controls and their effects depend on the provider and purpose. The examples below cover OpenAI and Google; they aren't a complete directory of AI crawlers.
OpenAI distinguishes search, training, and certain user actions
OpenAI's crawler documentation describes these roles:
| Name | Documented purpose |
|---|---|
| OAI-SearchBot | Crawling for ChatGPT search results |
| GPTBot | Crawling content that may be used to train OpenAI's generative AI foundation models |
| ChatGPT-User | Certain visits initiated by users of ChatGPT or custom GPTs, rather than automatic web crawling |
OpenAI says the settings are independent: publishers can allow search crawling while declining GPTBot's training use. ChatGPT-User doesn't control search eligibility; robots.txt may not apply to its user-initiated actions.
An OAI-SearchBot opt-out isn't an erasure request: opted-out sites may still appear as navigational links.
Google Search and Google-Extended address different uses
Google says Googlebot is the crawl control for Search, including its AI features. A page must be indexed and eligible for a Search snippet to qualify as a supporting link in AI Overviews or AI Mode; eligibility doesn't guarantee appearance. See AI features and your website.
Google-Extended addresses specified Gemini model-training and grounding uses. Grounding means supplying content to a model when responding to a prompt. Restricting this token therefore shouldn't be described as a training-only choice. Google says it doesn't affect inclusion or rankings in Google Search. See Google's common crawler documentation.
Google-Extended also has no separate HTTP user-agent string. It is a robots.txt control token. Searching your access logs for a crawler request with that name isn't a sufficient way to check the policy.
For a Google indexing problem, our page-indexing diagnosis guide covers the next investigation.
Check both the published rules and the response
A declared rule and a usable response answer different questions.
Review the deployed configuration for the actual host and URL you care about. Don't assume a local file, an editor setting, or a policy written for a different host describes the public site.
Have the person responsible for the site check:
- The relevant robots.txt rules and the provider's current interpretation of them.
- The final URL after redirects.
- What the CDN, firewall, and origin return for the relevant requests.
- Whether the returned content is the intended page, a sign-in screen, or a challenge.
- Any indexing and preview controls that apply to the intended search use.
Google's AI-feature guidance specifically includes allowing crawling through robots.txt and hosting or CDN infrastructure. A permissive file cannot, by itself, establish that the rest of the request succeeds.
Keep private content behind real access controls. Google's robots.txt introduction explains that robots rules depend on crawler cooperation and aren't a security mechanism. It also distinguishes preventing a crawl from keeping a URL out of Google Search.
A worked example: an explicitly hypothetical access audit
Imagine a small publisher that wants its public guides discoverable in search, restricts the documented training uses it has selected, and keeps account receipts private.
All paths, settings, responses, and observations in the following table are invented for this exercise. They are not HoneyWeRank test results or evidence of any provider visiting a real site.
| Item reviewed | Hypothetical observation | Supported conclusion and next step |
|---|---|---|
| A public guide requested in a controlled test using an OAI-SearchBot user-agent string | The robots rule allows the path. The test receives HTTP 200, but the body contains a browser challenge instead of the guide. | The rule permits access, but this test did not obtain the guide. Investigate the challenge and relevant edge rules. |
| A public ticket FAQ checked for Google Search access | The path is allowed and a controlled request returns the expected page. No verified Googlebot request is recorded in this exercise. | The configuration and test response look consistent with the intended access. A real Googlebot visit, indexing, and AI-feature appearance remain unverified. |
| GPTBot's rule for the public guides | The deployed rule disallows the guide path, as the publisher intended. | The recorded rule matches the selected OpenAI training preference. Don't turn that observation into a claim about every provider or past training data. |
| An account-receipt URL requested without signing in | The response requires authentication and exposes no receipt content. | This test supports the intended restriction for that request. Keep the authentication requirement; don't replace it with a robots rule. |
The first row is the important counterexample: a successful HTTP status doesn't establish that the useful content was delivered. Read the response as well as its code.
The second row limits a different conclusion. A page working in your own test doesn't establish that the provider fetched it. Write “our test obtained the page” until you have evidence for a stronger statement.
Distinguish a test request from a verified provider request
A user-agent string is a label, not sufficient proof of the sender. Google's request-verification guidance describes verification through published IP ranges or reverse and forward DNS checks.
OpenAI's crawler documentation links to its published IP ranges. Verify real requests against current provider information.
A controlled request can reveal how your site responds to that request. Label it honestly. It isn't evidence that the vendor has visited, indexed the page, or selected it for an answer.
Keep a record of what the evidence proves
Keep the policy, observation, and conclusion distinct.
For each reviewed URL or policy, save a compact record with the provider and purpose, intended decision, deployed rule, response observation, verification method, date, and unresolved question.
For the fictional guide blocked by a challenge, a useful handoff could read:
Purpose: Public search discovery.
Policy: The reviewed path is allowed for the relevant search crawler.
Observation: Our controlled request received a challenge instead of the guide.
Limit: This wasn't a verified provider request. Search inclusion and citations weren't tested.
Next action: Have the site operator review the relevant edge rule, then repeat the same check and record the result.
This gives the operator a specific problem to investigate. “AI can't see our site” doesn't identify a provider, URL, request, response, or desired policy.
After a change, repeat the relevant check. Keep an intentional denial marked as intentional rather than treating every blocked request as an error. Recheck private access independently from public search access.
Track access and visibility separately
Use different records for different questions:
| Question | Evidence to look for |
|---|---|
| Does the deployed policy match our decision? | The relevant published rule and its documented scope |
| Did a particular request receive the page? | The response and associated request details |
| Was the request really from the claimed provider? | Provider-appropriate request verification |
| Was the page indexed or used in an answer? | Evidence from the relevant search or answer system |
An access record can't answer the last question on its own. Our guide to measuring AI visibility explains how to keep mentions, citations, and accurate recommendations distinct.
For the hypothetical publisher, resolving the challenge would close one access issue. A later citation would be a separate observation, recorded with its question, system, date, and source link.