Robots.txt Is No Longer the Gate - Your CDN Decides Who Gets In
Over two weeks, four independent sources and Google said the same thing: robots.txt tells crawlers what is allowed, but whether they get in is now decided by infrastructure - CDN, WAF and bot rules.
Signal in the data
Independent sources on this topic: 4 in the last week, 5 in 30 days, 6 in 90 days. We have tracked it since 2020-05-21; 50 claims in our database in total.
Number of independent sources per week (last 12 weeks).
What is happening
On 14 November, the robots.txt file of the airline United did not contain a single rule aimed at an AI crawler or agent. Yet the company had still decided that search engines may index its content, while AI agents have to ask first. That decision does not live in the file - it lives in the layer that lets requests through.
Around the same time, Google Search Central advised targeting specific bots rather than blocking all automated traffic. A separate reminder noted that roughly 15% of sites still have no robots.txt at all.
Two Polish write-ups added a practitioner's view: what actually blocks indexing, and why WordPress doesn't give you control over the file out of the box.
Where the sources agree
Robots.txt is a request, not a lock. The trade press and Google's own documentation agree: it is an instruction that a well-behaved crawler reads and honours, not an enforcement mechanism.
Enforcement has moved into infrastructure. Real blocking happens in the CDN, in the web application firewall and in bot management services. The catch is that these rules are often written without involving the SEO team - and can contradict the text file.
Separating the roles matters. OpenAI uses separate bot names for search and for training. Blocking the search bot removes a site from ChatGPT results, not just from training. Those are two business decisions, not one technical one.
Where they differ
Google describes robots.txt as a crawling control and reminds you that noindex only works if the bot can see the page in the first place. Combining a robots.txt block with a noindex tag is a mistake, because the crawler never reads the tag.
Some sources treat robots.txt as the main gate for AI bots and cite the share of sites that unknowingly cut themselves off from AI traffic. Others counter that when agents query on a user's behalf, the file guarantees nothing.
The third disagreement concerns side effects: a server that returns a 503 when asked for robots.txt can stop the entire site from being indexed. A block meant to protect content then does more harm than no block at all.
What to do about it
- Review three places together: robots.txt, your CDN/WAF rules and the list of user agents you allow. A mismatch between them is the most common failure.
- Check that your firewall isn't blocking the bots you actually want - search and training bots separately.
- Decide whether you want to block training only or drop out of AI answers as well. It's a business call, so document it before someone sets a rule in the CDN panel.
- Make sure the file returns a proper 200 or 404. A server error on robots.txt costs more than the file itself.
Our take
Robots.txt remains a layer of declarations, but control over who gets into the site has shifted to the CDN, the WAF and bot rules. If you want your site to show up in AI answers, what matters more than the file is whether your infrastructure blocks search bots. Confidence: medium - four independent sources point in the same direction, but none says how many companies actually have a gap between the file and the firewall.
Confidence: medium. We will change our view if: The assessment would change if Google or OpenAI published a technical standard for enforcing agent rules outside robots.txt, or if a study of a representative sample of sites showed that a gap between the file and the CDN is rare.
Sources
- Search Engine Journal, 2026-11-15: United Lets Search Engines Index Its Site, Makes AI Agents Ask First
- Search Engine Journal, 2026-11-14: AI Bot Management: Why SEOs Need More Than Block Lists
- Paq Studio - agencja SEO, 2026-11-13: SEO sklepu NIE ZACZYNA SIĘ od tekstów ani linków
- Top Online, 2026-11-13: SEO News #264 – SEOwcy boją się o pracę, spam UGC, Reddit z ograniczeniami
- Google Search Central Blog, 2026-11-12: Moving to a CDN without overwhelming your origin (or blocking Googlebot)
- Search Engine Journal, 2026-11-08: The 11 SEO Features Every CMS Should Ship With (Most Still Don’t)
- Search Engine Journal, 2026-11-03: Google: można blokować wybrane zasoby JavaScript przez robots.txt (Mueller)