Free tool · Technical SEO
Robots.txt generator
Set crawl rules for each user-agent, add allow and disallow paths and link your sitemap to generate a valid robots.txt you can drop at the root of your site.
One path per line
Optional, one per line
Adds a Disallow: / block for each AI bot (GPTBot, CCBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider)
Seconds, optional
Optional, your real sitemap URL
- No Sitemap URL set — add your sitemap so crawlers discover all your pages faster.
User-agent: * Disallow:
How it works
Three steps, no signup.
Set your rules
Choose which crawlers to target and which paths to allow or disallow.
Add your sitemap
Point crawlers to your sitemap URL so they find every page.
Deploy the file
Save the output as robots.txt at the root of your domain.
FAQ
Robots.txt Generator questions.
What is robots.txt for?
It tells search engine crawlers which parts of your site they may or may not request. It controls crawling, not indexing. To keep a page out of search results, use a noindex tag, not robots.txt.
Where does robots.txt go?
At the root of your domain, reachable at yourdomain.com/robots.txt. It must be at the root. Crawlers do not look for it in subfolders.
Can robots.txt hide a page from Google?
No. Blocking a URL stops crawling, but the URL can still appear in results if other pages link to it. Use a noindex meta tag or HTTP header to truly keep a page out of the index.
Should I add my sitemap to robots.txt?
Yes. A Sitemap line pointing to your XML sitemap helps crawlers discover all your URLs, even ones not linked prominently. The generator adds it for you.
Tools find the work. Lynx does it.
Lynx is an AI agent that builds your backlinks automatically across 2,000+ directories, forums and content platforms. Monitored for 90 days. Your first link is free.