AI Crawler Access Tester
Local processing with network featuresCheck how robots.txt and page directives treat search and AI crawlers
robots.txt
Test access sends this URL to CodingTool servers. They request the site’s robots.txt and, when page directives are on, the page itself, with an emulated browser user agent and no cookies. The target site receives these requests. Nothing is stored.
An emulated request is not an authentic crawler visit. Providers verify their crawlers by IP address or DNS, and sites can answer real crawlers differently.
Advanced
The default set covers Googlebot, Bingbot, OAI-SearchBot, GPTBot and ChatGPT-User.
Evaluated locally instead of the live file. Leave empty to fetch the live robots.txt.
Fetches the page once to read X-Robots-Tag and meta robots.
The robots.txt used for the test appears here.
Findings
Related Tools
What this tool checks
AI Crawler Access Tester reads one robots.txt file and, optionally, the X-Robots-Tag header and meta robots tags of one page. It evaluates them for each selected search or AI crawler from a maintained registry and shows the robots policy, page fetch status, page directives and result side by side, with the exact rule and line that decided each result.
Crawl access is not indexing
robots.txt controls whether a crawler may fetch a URL; noindex controls whether a fetched page may be indexed. The tool keeps them in separate columns. A crawler blocked by robots.txt never sees a noindex tag, and an allowed crawler can still meet a noindex directive, which is reported as a potential indexability issue rather than a verdict about any search index.
Purposes and special agents
Crawlers are labelled by purpose: search, AI training, user-triggered, preview, ads or other. User-triggered fetchers act on a person’s request, and some providers document that robots.txt may not apply to them. Google-Extended is a usage-control token with no fetches of its own: blocking it limits AI model use without changing Google Search crawling.
Status codes and network use
Following RFC 9309, a 4xx robots.txt response means no restrictions and a 5xx response or network failure means complete disallow. Live tests send the URL to CodingTool servers, which request robots.txt and the page from the target site with an emulated browser user agent and store nothing. A pasted robots.txt is evaluated in your browser. Emulated requests are never authentic crawler visits.
