AI crawler checker
Can AI bots read your website?
Enter a page address. Crawlerpeek shows which AI bots your robots.txt lets in, what a bot sees without JavaScript, and whether you have an llms.txt, with fixes you can copy.
Free, no account. 24 bots checked. We store nothing about the address you enter.
Result for
example.com
1 of 24 bots blocked: GPTBot
- OAI-SearchBotAllowed
- ClaudeBotAllowed
- PerplexityBotAllowed
- GPTBotBlocked
Without JavaScriptText is in the HTML
What the check tells you
Bot access
Whether OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others may fetch your page, per robots.txt. Training, search and user-triggered bots are shown separately.
What a bot sees
The raw HTML without JavaScript: title, description, headline, text length and links. It shows whether your content exists for a bot or appears only in the browser.
Fixes
A robots.txt block to paste and a prompt for Claude Code, Cursor or Lovable that lists only what the check found.
The bots we check
Every bot has its own page with its robots.txt token, owner, purpose and the official source. See all bots, or browse by operator.
- GPTBotOpenAI
- OAI-SearchBotOpenAI
- ChatGPT-UserOpenAI
- ClaudeBotAnthropic
- Claude-SearchBotAnthropic
- Claude-UserAnthropic
- PerplexityBotPerplexity
- Perplexity-UserPerplexity
- Google-ExtendedGoogle
- Applebot-ExtendedApple
- CCBotCommon Crawl
- meta-externalagentMeta
- BytespiderByteDance
- GooglebotGoogle
- BingbotMicrosoft
- ApplebotApple
- meta-externalfetcherMeta
- DuckAssistBotDuckDuckGo
- AmazonbotAmazon
- AhrefsBotAhrefs
- SemrushBotSemrush
- MJ12botMajestic
- facebookexternalhit
- Slackbot
Questions about the checker
What does the checker look at?
It reads your robots.txt and evaluates it for each AI and search bot we track. It fetches the page you entered the way a bot without JavaScript receives it, and it looks for /llms.txt. That is three requests, or up to five when a request has to be retried.
Does it prove a bot can reach my page?
No. robots.txt shows what you allow. A firewall or CDN can still block a bot. The checker does not send requests with other companies' bot user agents.
Is my address stored?
No. The address is in the link you can share, so it appears in short-lived server logs. We keep no database record of what you checked.
Why does the text length matter?
Many AI crawlers do not run JavaScript. If your text only appears after scripts run, they may see an empty page.
Guides
- How to let AI assistants read your site (and which bots to block)Which AI bots can read your site, how robots.txt controls them, and how to allow AI search while blocking training. With copy-paste examples.
- llms.txt: what it is and what the evidence saysllms.txt is a proposed Markdown file for AI systems. Here is the format, a template and what the data says about whether AI crawlers use it.
- robots.txt explained: syntax, matching rules and where the file goesHow robots.txt works: the four fields, how a crawler picks its group, wildcards, size limit and what happens when the file errors. Based on Google's documentation.
- robots.txt mistakes, and how to test your fileThe robots.txt mistakes that keep pages out of Google or fail to keep bots away, and how to check your file with a status code, a rule check and a bot-eye view.
All guides · See a page as a bot · robots.txt generator · Bot traffic data