Every website has a file called robots.txt. It tells crawlers what they are allowed to read. For twenty-five years it quietly governed how Google indexed the web.
It now governs something else as well: whether an AI answer engine can read your site at all.
So we read it. Not the websites — just the file. 191 Australian digital agency websites, taken from public "best SEO agency in Melbourne / Australia" lists. No AI model was queried, no page was rendered. Two files per site: robots.txt and llms.txt.
Here is what came back.
Only 9 of 191 (5%) block at least one AI crawler
Not one of them blocks every crawler. There is no wildcard Disallow: / in the set.
| Crawler | Sites blocking it |
|---|---|
| CCBot (Common Crawl) | 7 |
| Bytespider (ByteDance) | 6 |
| PetalBot (Huawei) | 5 |
| meta-externalagent (Meta) | 4 |
| GPTBot (OpenAI) | 3 |
| Amazonbot | 3 |
| Applebot-Extended | 2 |
| ClaudeBot (Anthropic) | 2 |
| Google-Extended | 1 |
| PerplexityBot | 0 |
Look at which bots are being blocked. CCBot, Bytespider and PetalBot are scraper bots. The ones that decide whether your site gets cited inside an AI answer — PerplexityBot, Google-Extended, GPTBot — are blocked by 0, 1 and 3 sites respectively.
Half the sites publish an llms.txt. Most of them did not write it.
94 of 191 (49%) serve a file at /llms.txt — the proposed convention for telling AI systems which pages matter.
38 of those 94 files (40%) were generated by a WordPress SEO plugin. We found Rank Math and All in One SEO signatures in them. The site owner installed an SEO plugin for the usual reasons, and the plugin quietly shipped a file that reads like an AI strategy.
Another 6 sites serve a 200 HTML page at /llms.txt. To a crawler that is not a file. That is a web page, at a URL that promises a file.
The remaining ~50 look deliberately written.
What this actually means
Nobody is being reckless here. The opposite: almost nobody has made a decision at all.
That is not a criticism. llms.txt is a proposal, not a standard. There is no penalty for skipping it. Blocking GPTBot costs you nothing today and there is a reasonable argument for doing it. Silence is a perfectly defensible position.
But silence stops being neutral the moment an AI answer engine is choosing between two agencies for a query your client cares about. At that point "we never decided" and "we decided to be readable" are two different positions — and only one of them was chosen.
The genuinely useful finding is the plugin one. If your agency sells SEO and your own site publishes an AI-search file you did not know about, you are not alone: it is most of the industry. It also means the first agency to actually make the decision — and show its work — looks different from the rest.
How to check your own site in 60 seconds
- Open
yourdomain.com/robots.txt. - Search for
GPTBot,CCBot,PerplexityBot,Google-Extended. - If any of them sits under a
Disallow: /, that crawler cannot read your site. - Then check
yourdomain.com/llms.txt. If you get an HTML page, you do not have one — you have a soft-404.
If you want the version with a score attached: put any URL into our free checker and you get the technical read in about three minutes — leadranks.io/geo-checker.
The "AI Crawler Access" panel in a full LeadRanks report runs exactly the check in this article. It is the same parser. That is the point — we do not want a second opinion, we want the number.
Method and limits
- Two files read per site:
robots.txtandllms.txt. Nothing else was fetched. - Domains came from public agency directory lists. 192 attempted, 191 responded.
- We did not measure whether these sites are cited by AI systems. Blocking a crawler is a gate, not a score. Do not read this as a ranking.
- Counts can move by ±1 run to run when a site flakes on a single fetch.
- This is the check that had a false positive in our own tool last week — a bare
Disallow: /?line being read as "block everything". We found it, fixed it and shipped a regression test. The previous run of this sweep contained 1 false positive. This one contains 0.
If you run an agency site and want to know exactly where you sit against these 191, that is a 60-second check. Most of the answer is in a file you have probably never opened.