> ## Content Index
> Fetch the complete content index at: https://bielawski.pro/llms.txt
> Use this file to discover other available public pages before exploring further.

# Robots.txt Is No Longer the Gate - Your CDN Decides Who Gets In
- URL: https://bielawski.pro/en/robots-txt-is-no-longer-the-gate-your-cdn-decides-who-gets-in/
- Published: 2026-11-16T06:09:11.000Z
- Updated: 2026-11-16T06:09:11.000Z
- Description: Over two weeks, four independent sources and Google said the same thing: robots.txt tells crawlers what is allowed, but whether they get in is now decided by infrastructure - CDN, WAF and bot rules.
- Author: Bartosz Bielawski
- Tags: SEO, #temat-72, #en

**Signal in the data**

Independent sources on this topic: **4** in the last week, **5** in 30 days, **6** in 90 days. We have tracked it since 2020-05-21; 50 claims in our database in total.

wk 350

wk 360

wk 370

wk 380

wk 391

wk 400

wk 411

wk 422

wk 430

wk 441

wk 454

wk 460

Number of independent sources per week (last 12 weeks).

## What is happening

On 14 November, the robots.txt file of the airline United did not contain a single rule aimed at an AI crawler or agent. Yet the company had still decided that search engines may index its content, while AI agents have to ask first. That decision does not live in the file - it lives in the layer that lets requests through.

Around the same time, Google Search Central advised targeting specific bots rather than blocking all automated traffic. A separate reminder noted that roughly 15% of sites still have no robots.txt at all.

Two Polish write-ups added a practitioner's view: what actually blocks indexing, and why WordPress doesn't give you control over the file out of the box.

## Where the sources agree

**Robots.txt is a request, not a lock.** The trade press and Google's own documentation agree: it is an instruction that a well-behaved crawler reads and honours, not an enforcement mechanism.

**Enforcement has moved into infrastructure.** Real blocking happens in the CDN, in the web application firewall and in bot management services. The catch is that these rules are often written without involving the SEO team - and can contradict the text file.

**Separating the roles matters.** OpenAI uses separate bot names for search and for training. Blocking the search bot removes a site from ChatGPT results, not just from training. Those are two business decisions, not one technical one.

## Where they differ

Google describes robots.txt as a crawling control and reminds you that noindex only works if the bot can see the page in the first place. Combining a robots.txt block with a noindex tag is a mistake, because the crawler never reads the tag.

Some sources treat robots.txt as the main gate for AI bots and cite the share of sites that unknowingly cut themselves off from AI traffic. Others counter that when agents query on a user's behalf, the file guarantees nothing.

The third disagreement concerns side effects: a server that returns a 503 when asked for robots.txt can stop the entire site from being indexed. A block meant to protect content then does more harm than no block at all.

## What to do about it

- Review three places together: robots.txt, your CDN/WAF rules and the list of user agents you allow. A mismatch between them is the most common failure.
- Check that your firewall isn't blocking the bots you actually want - search and training bots separately.
- Decide whether you want to block training only or drop out of AI answers as well. It's a business call, so document it before someone sets a rule in the CDN panel.
- Make sure the file returns a proper 200 or 404\. A server error on robots.txt costs more than the file itself.

## Our take

Robots.txt remains a layer of declarations, but control over who gets into the site has shifted to the CDN, the WAF and bot rules. If you want your site to show up in AI answers, what matters more than the file is whether your infrastructure blocks search bots. Confidence: medium - four independent sources point in the same direction, but none says how many companies actually have a gap between the file and the firewall.

**Confidence:** medium. **We will change our view if:** The assessment would change if Google or OpenAI published a technical standard for enforcing agent rules outside robots.txt, or if a study of a representative sample of sites showed that a gap between the file and the CDN is rare.

## Sources

- Search Engine Journal, 2026-11-15: [United Lets Search Engines Index Its Site, Makes AI Agents Ask First](https://www.searchenginejournal.com/united-ai-agents-permission-search-indexing/597418/?ref=bielawski.pro)
- Search Engine Journal, 2026-11-14: [AI Bot Management: Why SEOs Need More Than Block Lists](https://www.searchenginejournal.com/ai-bot-management-why-seos-need-more-than-block-lists/597148/?ref=bielawski.pro)
- Paq Studio - agencja SEO, 2026-11-13: [SEO sklepu NIE ZACZYNA SIĘ od tekstów ani linków](https://www.youtube.com/watch?v=csqsrmZkuMo&t=1576s&ref=bielawski.pro)
- Top Online, 2026-11-13: [SEO News #264 – SEOwcy boją się o pracę, spam UGC, Reddit z ograniczeniami](https://toponline.pl/blog/seo-news-264?ref=bielawski.pro)
- Google Search Central Blog, 2026-11-12: [Moving to a CDN without overwhelming your origin (or blocking Googlebot)](https://developers.google.com/search/blog/2026/11/crawling-cdn?ref=bielawski.pro)
- Search Engine Journal, 2026-11-08: [The 11 SEO Features Every CMS Should Ship With (Most Still Don’t)](https://www.searchenginejournal.com/the-11-seo-features-every-cms-should-ship-with-most-still-dont/594074/?ref=bielawski.pro)
- Search Engine Journal, 2026-11-03: [Google: można blokować wybrane zasoby JavaScript przez robots.txt (Mueller)](https://www.searchenginejournal.com/google-says-its-okay-to-block-some-javascript-resources-with-robots-txt/595736/?ref=bielawski.pro)