AI Crawler Robots.txt Checker
Paste your domain — we read your robots.txt and show which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) are allowed to read your site. If AI can't read you, it can't recommend you.
🔒 We only read your public robots.txt. No signup to run the check.
See the robots.txt we read (proof)
See the full crawler report
Unlock every AI & search bot's status plus the exact copy-paste fix to open your site to AI. We'll email you a copy.
Robots.txt AI Crawler Checker — FAQ
AI crawlers are bots from AI companies (OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity, Google-Extended) that read the web so their assistants can answer questions and cite sources. If your robots.txt blocks them, your site can't be used or recommended by AI.
Allow the AI search/answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-Web) so AI can cite you. Allowing AI training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) is a business choice — it can improve visibility but lets models train on your content.
Yes — reading your robots.txt is free and takes about 10 seconds. No signup to run the check.
Add an Allow rule for that bot in your robots.txt. This tool generates the exact snippet to paste. Note: a site can also block bots at the server/CDN (e.g. Cloudflare) — that isn't visible in robots.txt.