ObeliskBot, our crawler.
It names itself, reads robots.txt and only opens public pages.
It names itself in every request.
User agent
ObeliskBot/1.0 (+https://theobelisk.app/crawler)
Robots token
ObeliskBot
What it does on your site.
- 01
Reads robots.txt first
It skips every page robots.txt keeps from it. If robots.txt can’t be read, it doesn’t crawl at all.
- 02
Opens public pages only
It sees what anyone on the internet sees. It never logs in and never opens private network addresses.
- 03
Waits when you ask
Give ObeliskBot a Crawl-delay and it waits that many seconds between pages.
Block it in robots.txt and it stays away.
Add the first two lines to block it. Use the second pair to slow it down instead. If your firewall blocks unknown bots, email support@theobelisk.app and we’ll tell you how to allow it.
Block it
User-agent: ObeliskBot Disallow: /
Slow it down
User-agent: ObeliskBot Crawl-delay: 5
When we test how AI crawlers see your site, each request carries ObeliskBot’s name too.
Bring the question your marketing data should answer.
On the call we show you Obelisk on a business like yours.