MACHINE 404 / The decision

GPTBot, ClaudeBot, PerplexityBot: allow or block?

Most sites have never made this decision, so it was made for them by silence. Here is what each crawler actually does, what you give up either way, and the one line of robots.txt syntax that quietly opens your private paths to exactly the bots you were trying to control.

Who is at the door

GPTBot is OpenAI's crawler. OAI-SearchBot is the one that builds what ChatGPT search can show. ChatGPT-User is a live fetch, made at the moment a person asks about your page. ClaudeBot, Claude-Web and Claude-SearchBot are Anthropic's. PerplexityBot is Perplexity's. CCBot is Common Crawl, which feeds a long list of models rather than one company. Bytespider is ByteDance.

They are not interchangeable, and that matters: blocking the crawler that builds the training set is a different decision from blocking the one that fetches your page because a customer just asked about you.

🚨 The robots.txt trap almost everybody falls into

A named User-agent group makes that crawler ignore the star group completely. So if your file disallows your admin paths under User-agent: * and then adds a polite User-agent: GPTBot / Allow: / underneath, you have just told GPTBot that none of those disallows apply to it. Every Disallow has to be repeated inside the named group, or the bot you were being careful about is the one bot with full access.

Google-Extended is not what most people think

It is not a crawler. It is a control over whether your content is used for Gemini and related grounding. Blocking it does not remove you from Google Search and does not remove you from AI Overviews, which are built on the ordinary search index. People block it believing they are opting out of AI answers in Google, and they are not.

So: allow or block?

If you sell something and want to be recommended, allow, and allow deliberately by naming them rather than by saying nothing. Silence is not a decision, it just leaves the outcome to somebody else's defaults. If you publish content that is your product, an archive, a paid library, research, then the calculation is different and blocking the training crawlers while allowing the live search fetchers is a coherent position.

And know what robots.txt is: a request, not a lock. It is honoured by the companies that publish their crawler names and ignored by everything that does not. If you need to stop something, that happens at the server, not in a text file.

One more thing that is invisible from the outside

Your robots.txt can say welcome while your host says no. We have measured sites whose file invites a named AI crawler and whose server answers that crawler with a refusal, which is the worst of both worlds: you look open and behave closed, and nobody tells you. The scan checks the file and the actual response, because only one of them is the truth.