Free tool · Technical SEO
Robots.txt generator
Set crawl rules for each user-agent, add allow and disallow paths and link your sitemap to generate a valid robots.txt you can drop at the root of your site.
One per line. * matches anything, $ marks the end of the URL.
Optional. Re-opens a path inside a disallowed folder; the longest matching rule wins.
AI crawlers
Training bots and AI search bots are separate. Most sites block training and keep search, so AI assistants can still cite and link them.
Choose bots one by one
Collect pages to train AI models. Blocking them has no effect on search rankings.
Index pages for AI answers that cite and link sources. Block these and you can't be cited.
Fetch a page when someone asks an assistant about it. Not bulk crawlers.
One per line, full URLs.
Seconds. Google ignores it; Bing and Yandex use it.
User-agent: * Disallow:
Upload the file to the root of your domain so it loads at yoursite.com/robots.txt. Then test a few important URLs against it before you go live.
How it works
Three steps, no signup.
Set your rules
Pick a preset or list the paths to allow and disallow, then choose which AI crawlers to block.
Add your sitemap
Point crawlers to your sitemap URL so they find every page.
Deploy the file
Save the output as robots.txt at the root of your domain.
FAQ
Robots.txt Generator questions.
What is robots.txt for?
It tells search engine crawlers which parts of your site they may or may not request. It controls crawling, not indexing. To keep a page out of search results, use a noindex tag, not robots.txt.
Where does robots.txt go?
At the root of your domain, reachable at yourdomain.com/robots.txt. It must be at the root. Crawlers do not look for it in subfolders.
Can robots.txt hide a page from Google?
No. Blocking a URL stops crawling, but the URL can still appear in results if other pages link to it. Use a noindex meta tag or HTTP header to truly keep a page out of the index.
Should I add my sitemap to robots.txt?
Yes. A Sitemap line pointing to your XML sitemap helps crawlers discover all your URLs, even ones not linked prominently. The generator adds it for you.
Should I block AI crawlers?
Block training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) if you don't want your content used to train models; that has no effect on rankings. Keep AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed if you want AI assistants to cite and link your pages.
More free tools
Keep going.
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
Sitemap Validator
Check any sitemap by URL: errors, noindex, redirects, robots.txt.
Htaccess Redirect Generator
301 and 302 redirects for Apache, Nginx, Netlify and Vercel.
HTTP Header Checker
Headers, redirect chain and a security header grade.
Tools find the work. Lynx does it.
Lynx is an AI agent that builds your backlinks automatically across 2,000+ directories, forums and content platforms. Monitored for 90 days. Your first link is free.