Free tool

robots.txt AI-crawler checker

See which AI crawlers — GPTBot (ChatGPT), ClaudeBot, PerplexityBot, Google-Extended, CCBot — your robots.txt allows or blocks, and whether you publish modern AI-usage signals. Instant, free, no account.

Why AI-crawler access matters

AI answer engines can only recommend and cite what their crawlers can read. A single overly-broad Disallow rule can silently remove your site from ChatGPT, Claude, and Perplexity answers — usually left over from a staging config or a blanket bot block.

What this checker looks at

Per-crawler allow/block status at your site root, whether a sitemap is declared, Cloudflare Content-Signal directives (search / ai-train / ai-input), and emerging IETF AIPREF-style usage preferences.

Blocking on purpose is fine

Some sites deliberately block AI training crawlers — that's a legitimate policy choice. The problem is blocking by accident. This tool shows you what your file actually says so the choice is yours.

Want the full picture — structured data depth, agent actionability, content extractability? Run the free AI-readiness scan.