Skip to main content

AI Crawler Access Tester

Check how robots.txt and page directives treat search and AI crawlers

robots.txt

Test access sends this URL to CodingTool servers. They request the site’s robots.txt and, when page directives are on, the page itself, with an emulated browser user agent and no cookies. The target site receives these requests. Nothing is stored.

An emulated request is not an authentic crawler visit. Providers verify their crawlers by IP address or DNS, and sites can answer real crawlers differently.

Advanced
Crawlers to check

The default set covers Googlebot, Bingbot, OAI-SearchBot, GPTBot and ChatGPT-User.

Evaluated locally instead of the live file. Leave empty to fetch the live robots.txt.

Fetches the page once to read X-Robots-Tag and meta robots.

The robots.txt used for the test appears here.

Findings

No test yet
Enter a public page URL and press Test access, or load an example.

What this tool checks

AI Crawler Access Tester reads one robots.txt file and, optionally, the X-Robots-Tag header and meta robots tags of one page. It evaluates them for each selected search or AI crawler from a maintained registry and shows the robots policy, page fetch status, page directives and result side by side, with the exact rule and line that decided each result.

Crawl access is not indexing

robots.txt controls whether a crawler may fetch a URL; noindex controls whether a fetched page may be indexed. The tool keeps them in separate columns. A crawler blocked by robots.txt never sees a noindex tag, and an allowed crawler can still meet a noindex directive, which is reported as a potential indexability issue rather than a verdict about any search index.

Purposes and special agents

Crawlers are labelled by purpose: search, AI training, user-triggered, preview, ads or other. User-triggered fetchers act on a person’s request, and some providers document that robots.txt may not apply to them. Google-Extended is a usage-control token with no fetches of its own: blocking it limits AI model use without changing Google Search crawling.

Status codes and network use

Following RFC 9309, a 4xx robots.txt response means no restrictions and a 5xx response or network failure means complete disallow. Live tests send the URL to CodingTool servers, which request robots.txt and the page from the target site with an emulated browser user agent and store nothing. A pasted robots.txt is evaluated in your browser. Emulated requests are never authentic crawler visits.

Frequently asked questions
Does the test send my URL to a server?
Live tests send the URL to CodingTool servers, which request robots.txt and, when page directives are on, the page from the target site with an emulated browser user agent and no cookies. Private and internal addresses are blocked and nothing is stored. A pasted robots.txt with the page check off is evaluated entirely in your browser.
Why is ChatGPT-User or Perplexity-User marked as user-triggered?
These agents fetch a page when a person asks for it. Their providers document that robots.txt rules may not apply to such requests, so the tool still shows the robots.txt result but adds a note instead of promising the rule is enforced.
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a usage-control token with no fetches of its own. Blocking it limits how Google uses content for AI models, while Googlebot crawling for Search follows its own rules, which the tool shows in a separate row.

CodingTool is officially live on