MACHINE 404 / Free live check

Can the AI readers actually reach your site?

Your browser gets a clean page, so everything looks fine. That tells you nothing about what your server says to something announcing itself as GPTBot. We ask as seven readers and as a browser, at the same moment, and show you all eight answers side by side.

This is the one thing you cannot check from the inside

Every other problem with a website shows itself eventually. This one does not. Your browser, your phone, your colleague's laptop and every uptime monitor you own send an ordinary browser user agent, get an ordinary 200, and report that the site is up. A rule that refuses one specific reader is invisible to all of them, because none of them is that reader.

We did not learn this from a blog post. Four of our own sites invite AI crawlers by name in robots.txt, and on 7 September 2026 the host was answering GPTBot with 429 and zero bytes while every browser got a clean page. Our own files said welcome. Our own server said no. We build websites for a living and we found it by writing a script to ask, which is the script behind this page.

The five ways a site turns an AI reader away

403 from a firewall. A WAF or a security plugin has a rule about bots and this reader matched it. Usually nobody remembers writing the rule, because it arrived as a default with a plugin or a plan.

429 from the host. Rate limiting by user agent, applied before your site is even consulted. This is the one we had, and it is the hardest to notice because it looks like politeness rather than refusal.

200 with a challenge page. The status code says success and the body is a Cloudflare, DataDome or Imperva check that expects JavaScript and cookies. No crawler solves it. This is a refusal wearing a success code, and a checker that only prints status codes will call it a pass. We read enough of the body to name the wall.

A Disallow nobody wrote on purpose. A staging flag, a plugin setting or a migration left Disallow: / inside a named group. The server is innocent here, the file is doing exactly what it says.

Sent somewhere else. The reader is redirected to a locale picker, a consent page or an app store, while your browser goes to the homepage. We follow one hop, and if a reader is sent to a different place than the controls, we say so.

Your robots.txt is a request. Your server is the answer.

So we read both, and the interesting result is the disagreement. A file that names User-agent: GPTBot and adds Allow: / while the server answers 403 is the worst of both worlds: you look open, you behave closed, and nobody tells you. Every row of the check says what your file asks for beside what your server actually did, so a refusal you chose reads differently from one you did not.

One trap worth repeating, because almost every site falls into it: a named User-agent group makes that crawler ignore the star group completely. Disallow your admin paths under the star, then add a polite named group for GPTBot underneath, and you have just given GPTBot the only unrestricted access on the site.

What this check proves, and what it does not

We send each reader's user agent string from our own address. A site that verifies crawlers by IP range or reverse DNS can refuse us and still admit the real GPTBot, so a refusal here is not proof that OpenAI is being turned away. We print that beside every result rather than at the bottom in small type.

What a refusal does prove, in every case, is that anything which merely announces itself as an AI reader is turned away at your door. That is exactly how a user agent rule and a host throttle behave, it is what happened to us, and it is worth ten minutes of somebody's attention either way.

And a pass is a real pass. If all eight answers are a clean page, nothing at your door is refusing AI readers and you can stop worrying about this particular failure and go look at whether there is anything worth reading once they are through, which is what the scan measures.

What this looks like across 42 Greek business websites

We asked exactly this of the 42 sites in our measured index, in one pass on 2026-09-13. Aggregates only: no business is named here and none will be.

2 of 42 have said anything at all about AI crawlers in robots.txt. Not blocked, not allowed: nothing. The other 40 left the decision to somebody else's default, which is the single most common position in the Greek market and is not a position.

4 of 42 serve Googlebot and an ordinary browser from the same address in the same second, and refuse every AI reader. Those doors are matching names, not checking who anybody is, and the real readers send the same names. That is the finding, and it is the one a site can fix this afternoon.

17 of 42 refused our browser control as well. That is a posture about automated traffic in general, not an AI policy, so we do not count it as blocking AI and neither should anybody else. 1 refused Googlebot too while serving a browser, which looks like checking identity by address rather than by name: our probe cannot be verified, so that one proves nothing about the real crawlers and is counted apart. 1 did not refuse at all, it rate limited: refused once, served on a retry two seconds later.

One more number, about the file rather than the door: we could read robots.txt on 28 of the 42. Exactly 1 answered a clean 404, and 13 refused us the file itself, which is its own kind of answer.

Measured from one address in Greece. A site can answer a different address differently, which is exactly why the check above exists: run it from here and you learn what our address is told, run it and read the caveat and you know what that is worth.

If something is being refused, the fix is at the server

Not in a text file. A robots.txt cannot undo a firewall rule, and adding a friendlier line to it changes nothing about a 403. The fix is a rule at the host, the WAF or the CDN, written by whoever has access to that panel, and then this same check re-run to confirm it rather than assumed.

It also comes back. A host changes a default, a plan gets upgraded, a security plugin updates its bot list, and the door closes again without a word to anybody. Run this monthly, or have somebody run it for you.