About our crawler

AgentReadyBot

You are probably here because you found this in your server logs:

AgentReadyBot/1.0 (+https://agentreadybot.com/bot)

What it is

Agent Ready Bot checks whether a website can be read by AI assistants — the ones people increasingly ask what to buy. Someone asked us to check this site, and we fetched a small number of its pages to see what an assistant would find.

We do not verify that the person who asked owns the site. If this scan was not you, it was someone evaluating your site — a competitor, a prospective customer, or an agency. Nothing we fetch is private: we read only what any visitor can read, never sign in, and never submit forms.

What it does

• One request per second, at most — to your site alone, never in parallel.
• At most 25 pages, chosen from your sitemap and the links on your homepage.
• Your robots.txt is honoured for which pages we read. We do always request two things regardless: /robots.txt itself, and your homepage — the homepage because there is no way to report on a site without looking at it once, and because we need it to tell you whether AI assistants are being blocked.
• We render pages in a browser to compare what a person sees with what a machine gets.
• We take one screenshot per page, kept privately, and delete it after 90 days.

Why you may also see other bot names from our address

This is the part worth explaining, because it looks alarming in a log. To find out whether your site blocks AI assistants, we have to ask it the way they would. So we send up to fifteen requests — five names across up to three pages — using these user-agent strings:

• GPTBot — OpenAI, collects data for training
• ChatGPT-User — OpenAI, visits because someone just asked a question
• ClaudeBot — Anthropic, collects data for training
• Claude-User — Anthropic, visits on behalf of a person
• PerplexityBot — Perplexity, answers a live search

We are not pretending to be those companies, and we do not use what comes back for anything but the report. It is a doorbell test: we need to know whether your server answers them or turns them away, and the only way to find out is to knock. All of it comes from our address, so you can tell ours apart from the real ones by the IP.

How to block us

Add this to your robots.txt and we will stop reading your pages:

User-agent: AgentReadyBot
Disallow: /

Being straight with you about what that does and does not stop: we will no longer read any of your pages, but we will still request /robots.txt and your homepage on a scan, and the assistant-name checks above still happen against the homepage. That is two or three requests instead of dozens. If you want zero traffic from us, block our requests at your firewall or CDN by that user-agent string — we will not work around it.

Contact

Agent Ready Bot is operated by Robots Drinking Tea LLC. Blocking us in robots.txt takes effect on the next scan and needs no request to us.