Free tool · Technical SEO
Robots.txt tester
Fetch a live robots.txt or paste a draft, then test up to 100 URLs against Googlebot, Bingbot, GPTBot and other crawlers. Every result shows the exact rule that allowed or blocked it, matched the way Google does.
One per line, up to 100. Full URLs or paths like /blog/post?page=2.
How it works
Three steps, no signup.
Load a robots.txt
Enter a domain to fetch the live file, or paste a version you are editing before you deploy it.
Add URLs and bots
List the URLs or paths to test and pick the crawlers that matter to you.
Read the verdicts
See allowed or blocked for every URL and bot, with the matching line, plus syntax warnings for the file.
FAQ
Robots.txt Tester questions.
How does Google match robots.txt rules?
Google picks the most specific user-agent group, then the longest matching path rule. If an allow and a disallow rule match with the same length, allow wins. Wildcards (*) and end anchors ($) are supported. The tester follows the same logic.
Does blocking a URL in robots.txt remove it from Google?
No. Robots.txt stops crawling, not indexing. A blocked URL can still be indexed from links pointing to it. To keep a page out of search, allow crawling and use a noindex tag.
What happens if robots.txt returns an error?
A 4xx response such as 404 means crawlers treat the site as fully allowed. A 5xx error or timeout makes Google pause crawling the whole site, so a broken robots.txt can be worse than none.
Does Google support crawl-delay?
No. Googlebot ignores crawl-delay. Bing and some other crawlers respect it. The tester flags directives that Google does not use.
More free tools
Keep going.
Robots.txt Generator
Build a robots.txt, block AI training bots, keep AI search.
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
Sitemap Validator
Check any sitemap by URL: errors, noindex, redirects, robots.txt.
Htaccess Redirect Generator
301 and 302 redirects for Apache, Nginx, Netlify and Vercel.
Tools find the work. Lynx does it.
Lynx is an AI agent that builds your backlinks automatically across 2,000+ directories, forums and content platforms. Monitored for 90 days. Your first link is free.