Robots.txt Tester & Builder

Robots.txt Tester & Builder checks whether a crawler may fetch a URL under a robots.txt file. For every tested path it shows Allowed or Blocked by robots.txt, the applicable User-agent group, the matched rule with its line number and a full rule trace, so you can see why one rule beat another.

Edit & Test and Build run entirely in your browser. Test Live sends the URL you enter to CodingTool servers, which request /robots.txt from the target site with an emulated user agent; the target site receives that request, private and internal addresses are blocked, and nothing is stored. The fetched file opens read-only, and you can edit a copy to try changes.

Matching follows RFC 9309. A crawler uses the groups that name its product token, otherwise the User-agent: * groups. Within them the longest matching Allow or Disallow pattern wins, and Allow wins a tie. * matches any characters and a trailing $ anchors the end of the URL. A 4xx robots.txt response means no restrictions; a 5xx response or network failure means the whole site is disallowed.

robots.txt controls crawling, not indexing. A URL that is Blocked by robots.txt can still be indexed without its content if other pages link to it, and a noindex rule on a blocked page is never seen; treat such cases as a potential indexability issue. Provider behavior varies: Google ignores crawl-delay and processes only the first 500 KiB, and user-triggered agents may not follow robots.txt. Unusual but valid rules are reported, never rewritten.