See if AI assistants can read your site.
We check whether the crawlers behind ChatGPT, Claude, Perplexity and Apple can reach your pages, through your CDN and firewall.
We send each AI crawler to your site and see if it gets in.
Your robots.txt can let a crawler in while your firewall turns it away. So we check both, crawler by crawler. The live test comes from our servers, so treat it as a strong hint.
| Crawler | ||
|---|---|---|
| OAI-SearchBotOpenAI | Allowed | Allowed |
| Claude-SearchBotAnthropic | Allowed | Allowed |
| PerplexityBotPerplexity | Allowed | Allowed |
| ApplebotApple | Allowed | Allowed |
| DuckAssistBotDuckDuckGo | Allowed | Allowed |
| MistralAI-IndexMistral | Allowed | Allowed |
| Amzn-SearchBotAmazon | Allowed | Allowed |
| meta-webindexerMeta | Allowed | Allowed |
On requestNot scored. They open a page when a person asks an assistant to.
| Crawler | ||
|---|---|---|
| ChatGPT-UserOpenAI | Allowed | Allowed |
| Claude-UserAnthropic | Allowed | Allowed |
| Perplexity-UserPerplexity | Allowed | Allowed |
| meta-externalfetcherMeta | Allowed | Allowed |
| Amzn-UserAmazon | Allowed | Allowed |
TrainingNot scored. We show your rule and leave the choice to you.
| Crawler | ||
|---|---|---|
| GPTBotOpenAI | Allowed | Rules only |
| ClaudeBotAnthropic | Allowed | Rules only |
| Google-ExtendedGoogle | Allowed | Rules only |
| Applebot-ExtendedApple | Allowed | Rules only |
| CCBotCommon Crawl | Allowed | Rules only |
| meta-externalagentMeta | Allowed | Rules only |
| BytespiderByteDance | Allowed | Rules only |
| MistralAI-TrainingMistral | Allowed | Rules only |
| AmazonbotAmazon | Allowed | Rules only |
AI tools may read your page before its scripts load, and miss what the scripts add.
Obelisk compares every page before and after its scripts and flags anything that only appears later.
See how often AI answers name you, next to three competitors.
We estimate each site’s share of mentions in AI answers. Every figure is marked as an estimate, with its source and date.
We leave llms.txt out of your score, because Google says its search ignores the file.
Questions about AI search
Which AI crawlers do you check?
8 search crawlers, 5 that open pages when someone asks, and 9 training crawlers. They include the ones behind ChatGPT, Claude, Perplexity and Apple.
Does blocking AI training lower my score?
No. We show your rule for each training crawler and leave the choice to you. Only search access is scored.
Why test through my firewall?
Your robots.txt can allow a crawler while your firewall or CDN blocks it. The live test shows what a crawler meets.
Do I need an llms.txt file?
It won’t change your score. Google says its search ignores the file, so we don’t ask you to add one.
Is AI search part of the free audit?
Yes. Every audit checks AI search, the free one included.
Find out if AI search can read your site.
The free audit checks it, with no sign-up. Or book a call and we show you Obelisk on a business like yours.