Free tool · Technical SEO
AI crawler access checker
Enter a URL to see whether AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others can access it. The checker tests robots.txt rules and sends real requests with each bot's user agent to catch CDN and firewall blocks.
Checks robots.txt for 19 AI crawlers and sends a live request as 17 of them, plus Googlebot for comparison.
How it works
Three steps, no signup.
Enter a URL
Use your homepage or any page you want AI assistants to read and cite.
Two checks per bot
Each crawler is tested against your robots.txt and with a live request using its user agent.
Allow or block on purpose
Decide which bots you want, then copy a robots.txt snippet that allows or blocks them.
FAQ
AI Crawler Checker questions.
Should I block AI crawlers?
It depends on your goals. Blocking training bots such as GPTBot or CCBot keeps your content out of future model training. Blocking search and user bots such as OAI-SearchBot or PerplexityBot also keeps you out of AI answers and their citations.
What is the difference between training and search bots?
Training crawlers collect pages to train models. Search crawlers index pages so assistants can cite them in answers. User-triggered fetchers load a page when a person asks the assistant about it.
Why would a bot be blocked if robots.txt allows it?
CDNs and firewalls such as Cloudflare can block AI bots by user agent, regardless of robots.txt. The live request catches this: a 403 or a challenge page means the bot is turned away at the server.
What are Google-Extended and Applebot-Extended?
They are robots.txt tokens, not separate crawlers. They control whether content fetched by Googlebot or Applebot can be used for Gemini or Apple Intelligence training, so they are checked against robots.txt only.
More free tools
Keep going.
Robots.txt Generator
Build a robots.txt, block AI training bots, keep AI search.
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
Sitemap Validator
Check any sitemap by URL: errors, noindex, redirects, robots.txt.
Htaccess Redirect Generator
301 and 302 redirects for Apache, Nginx, Netlify and Vercel.
Tools find the work. Lynx does it.
Lynx is an AI agent that builds your backlinks automatically across 2,000+ directories, forums and content platforms. Monitored for 90 days. Your first link is free.