SEO

Blocking AI bots: the decision most companies made by accident

Cloudflare and hosting providers block AI crawlers by default, often without the marketing team knowing. On top of that, OpenAI runs its own access gate, called Aura, that ignores robots.txt. The result: a company drops out of AI answers and ads without noticing.

Bartosz Bielawski

Bartosz Bielawski

· 3 min read
Share
Blocking AI bots: the decision most companies made by accident

Signal in the data

Independent sources on this topic: 0 in the last week, 4 in 30 days, 6 in 90 days. We have tracked it since 2025-11-17; 13 claims in our database in total.

wk 332
wk 340
wk 350
wk 360
wk 371
wk 382
wk 390
wk 400
wk 410
wk 421
wk 433
wk 440

Number of independent sources per week (last 12 weeks).

What happened

Over the past two weeks, four independent sources have described the same mechanism from different angles: blocking AI bots is no longer a deliberate strategic decision but a default setting nobody in the company remembers choosing.

Cloudflare often blocks AI crawlers out of the box, and the setting can be switched on without the marketing team knowing. In one documented case, a Shopify store was blocking AI crawlers even though nobody had decided to do so. Separately, OpenAI confirmed that it uses a central safety gate called Aura to decide whether its browsing tools may connect to a given site, and that the gate ignores robots.txt and user settings when doing so.

Two different mechanisms, one outcome

It's worth separating them, because they get confused all the time. robots.txt is a declaration on the site's side: please don't crawl here. A Cloudflare block is an infrastructure decision: the request never reaches the server. Aura is a third layer, a decision by the model provider about whether to let the site into its tool at all.

The outcome is the same: the site disappears from AI answers. The difference is who can undo it. A webmaster can change robots.txt in a minute, but a Cloudflare setting requires access to the infrastructure panel, and the person with that access is often not the one responsible for SEO.

There is also a side effect that hits paid campaigns: a site that blocks OpenAI bots cannot serve ads in that channel. So a block set up to stop content theft cuts off both organic AI visibility and paid reach.

Where the real debate lies

Blocking is not always a mistake. Publishers make a serious case: if unique content is your only asset, handing it to models unconditionally means giving away your advantage. Two positions in this debate hold up.

First: for big media outlets with unique content, blocking AI bots is a mistake, because models will cite competitors instead - a loss of visibility for zero gain. Second: instead of blocking, put unique content behind a paywall and sell it as a paid product. A block then reflects a business model rather than a technical setting.

There is also a trap on the other side: blocking AI bots through Cloudflare can accidentally stop Googlebot and Bingbot from indexing the site. The damage is then not limited to AI - you disappear from classic search too.

What to check, in order

Start with the actual state, not the strategy. First, check whether robots.txt names AI bots, and with which directive. Then check whether your CDN or host blocks them independently of robots.txt, because that decision may have been made without the visibility team in the room. Finally, check whether any of those blocks affects Googlebot or Bingbot.

Only then is it worth discussing whether to block deliberately. For most companies that depend on visibility, the answer is: don't block, and make sure your brand is described correctly in AI answers. Blocking makes sense where content is a product sold behind a paywall, and even there it should be a conscious decision, not a default.

Our take

Today, blocking AI bots is mostly a default setting, not a strategy. Four independent sources in two weeks show the same pattern: Cloudflare blocks AI crawlers without the marketing team knowing, OpenAI runs its own gate that decides whether it can access a site, and the result is dropping out of AI answers and ads. For companies that depend on visibility, the sensible order is to audit the actual state first and then make a decision. A block has a business case only where content is the product.

Confidence: medium - the mechanisms are well documented, but the scale among Polish companies has not been measured. We will change our view if: Measurements show that blocking AI bots has no effect on brand visibility in AI answers, or Google and OpenAI start respecting robots.txt in a verifiable way.

Sources

Read more