Table of Contents
Google is facing growing pressure from publishers, regulators and internet infrastructure providers over the way its web-crawling system supports both traditional search and artificial intelligence services.
The controversy centres on Google’s use of a multi-purpose crawler. Critics argue that publishers should be able to allow Google to index their websites for Search while separately controlling whether their content can be accessed for AI-related purposes.
The issue has become increasingly important as AI-generated answers and agents change how people discover and consume information online.
Why Google’s Crawler Is Causing Concern
Googlebot has traditionally crawled websites so Google can discover and index pages for Search. However, the distinction between content gathered for search and information used by Google’s AI products has become increasingly difficult for publishers to manage.
Cloudflare CEO Matthew Prince has said Google’s crawler sees significantly more of the web than competing AI crawlers, with estimates putting its reach at 3.2 times that of OpenAI’s crawler and 4.8 times Microsoft’s.
This creates a difficult choice for publishers. Blocking Google’s crawler could protect content from certain AI uses, but it could also affect their visibility in Google Search.
Cloudflare has argued that publishers need more meaningful control over these different uses of their content.
Publishers Face Declining Search Traffic
The debate comes as publishers are already experiencing changes in traffic generated by search engines.
The supplied report cites media company Reach as reporting a 50% decline in Google referral traffic, while a SparkToro analysis found that fewer than one-third of Google searches resulted in a click to an external website.
The concern is that AI-generated answers can provide users with information directly inside search results, potentially reducing the need to visit the original publisher’s website.
For publishers that depend on advertising, subscriptions or referral traffic, fewer visits can have a direct financial impact.
AI and Bot Traffic Is Reshaping the Web
The pressure is also being driven by the rapid growth of automated web activity.
Cloudflare data cited in the report indicates that bots and AI agents now account for more than 57% of web traffic, compared with about 42% attributed to humans.
That shift highlights why crawler policies are becoming more important. Websites are no longer dealing only with traditional search-engine crawlers. AI systems and autonomous agents increasingly access online information for search, retrieval and other automated tasks.
Regulators Push for Publisher Choice
Regulators are also becoming involved in the dispute.
The UK’s Competition and Markets Authority (CMA) has pushed Google to provide publishers with greater control over whether their content is used by its AI features while allowing them to remain available through traditional Search.
Cloudflare has supported the underlying principle, arguing that publishers need a meaningful way to opt out of Google’s generative AI uses without sacrificing their search presence.
Google has also been testing an approach that gives publishers greater control over AI-related use.
The broader issue is whether publishers should have to make an all-or-nothing choice between Google Search visibility and participation in Google’s AI ecosystem.
Cloudflare Sets New Crawler Rules
Cloudflare is taking a more direct approach.
From September 15, 2026, Cloudflare’s new defaults will classify automated traffic into categories including Search, Agent and Training. On pages displaying advertisements, Training and Agent crawlers will be blocked by default under the applicable new settings, while Search will remain allowed by default.
This distinction is important because it means Cloudflare is not simply proposing to block all automated search activity.
Instead, the company wants publishers to have separate controls for search indexing and AI-related access.
Mixed-purpose crawlers create a particular challenge because a single crawler can perform more than one function.
Why Google Is Under Pressure
Google’s dominant position in search makes the dispute especially significant.
The company remains the leading search provider globally, meaning publishers cannot easily walk away from Google’s ecosystem without potentially losing a substantial source of traffic.
At the same time, publishers are increasingly questioning whether appearing in Search provides enough value if AI-generated answers reduce the number of users who click through to their websites.
The issue therefore extends beyond technology. It concerns the economic relationship between search engines, AI companies and the publishers whose content helps power the web.
What Could Happen Next?
The next stage of the dispute could depend on whether Google separates its crawling functions or provides publishers with more granular controls.
A system that clearly distinguishes Search from AI training and agent activity could allow publishers to remain visible in search results while setting different policies for AI-related access.
Cloudflare’s new rules increase pressure on companies that use mixed-purpose crawlers to make those distinctions clearer.
For publishers, the outcome could determine how much control they have over the use of their content in the expanding AI ecosystem.
Frequently Asked Questions
What is a Google AI crawler?
The term refers to Google’s web-crawling activity associated with collecting and accessing online content for AI-related services. Google’s crawler infrastructure can serve multiple purposes, including traditional Search.
Why are publishers concerned?
Publishers are concerned that AI features could use their content while reducing the number of visitors sent back to their original websites.
What is Cloudflare changing?
Cloudflare is introducing separate classifications for Search, Agent and Training traffic, with new defaults taking effect on September 15, 2026.
Will Cloudflare block Google Search?
Not by default under the new settings. Search remains allowed by default, while Training and Agent traffic will be blocked by default on applicable ad-supported pages.
Conclusion
The growing dispute surrounding the Google AI crawler reflects a broader transformation in how the web is accessed and monetised.
Publishers want the ability to remain visible in Google Search without automatically granting the same access for AI-related uses. Regulators are pushing for greater choice, while Cloudflare is introducing technical controls that distinguish Search from Training and Agent traffic.
As AI-generated search results and autonomous agents become more common, the question of who controls access to web content is likely to become increasingly important for Google, publishers and the wider internet.
