AI Crawler Access Checker
Enter a website address and see which AI crawlers its robots.txt lets in: the ones that put you in ChatGPT, Claude, Perplexity and Google AI answers, the ones that fetch a page when a person asks, and the ones that only collect training data. Then copy a robots.txt block that sets the rules you want.
Check a site
Who can read it
A robots.txt block for the rules you want
robots.txt
Want to be cited by AI search, not only crawled?
Letting the right crawlers in is step one. We make sites readable for AI agents, add the structured data and Markdown copies they look for, and check how AI assistants describe your business.
How it works
Our server reads the site's robots.txt, exactly as a crawler would, and checks whether an llms.txt file exists. Your browser then applies the file's rules to 22 AI crawlers, following the same matching that Google documents and the robots.txt standard (RFC 9309) describes: a crawler uses the group written for it, or the rules for every bot (*) when there is none, and the longest matching path decides, with Allow winning a tie. You can test any path, not only the home page, and paste a draft robots.txt instead of checking a live site.
The crawlers are grouped by what they do. Answer engines such as OAI-SearchBot, Claude-SearchBot and PerplexityBot find and cite pages in AI answers, so blocking them takes you out of those answers. On-request fetchers such as ChatGPT-User read a page only when a person asks about it. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models, and blocking them changes nothing about how AI search finds you. The generated block follows that split: it lets the first two groups in, keeps the third out, and copies your existing rules for * into each new group, because a crawler with its own group ignores the * rules.
Paste the block at the end of your robots.txt. In WordPress, most SEO plugins have a robots.txt editor; otherwise edit the file at the root of your site. Then run the check again. If you want AI search to read your pages properly once they are in, our GEO readiness checklist covers the next steps.
Limitations
- robots.txt is a request, not a lock. A crawler can ignore it, and a host or CDN can block bots that robots.txt allows. Our guide to WordPress and AI crawlers explains where the real controls sit.
- The result shows what our server received. A CDN that answers bots differently, or a firewall rule, is invisible here. Check your server logs to see what a crawler actually got.
- The crawler list reflects the tokens each company documents today. Names change; a crawler not listed here still follows the rules for *.
- Only the first 100 KB of a robots.txt file is read, and an llms.txt file is reported as present, not checked for content.
What we store
The address you enter is sent to our server so it can read that site's robots.txt and llms.txt. We keep no record of the addresses checked or the files read. A pasted robots.txt never leaves your browser.