Google is under increasing pressure for using its extensive search infrastructure to power AI tools, as concerns grow about reduced website traffic and publishers’ revenues. Users are turning to AI tools for information more than they directly visit original sources.
Over the years Google has cemented its dominance in online search and provided broad access to free information for users. This model was based on an implicit agreement between Google and website owners: sites allowed Google’s crawler to scan and index their pages, and in return Google directed users to those sources.
Imbalance
But that agreement is changing as Google expands its AI activities. The infrastructure the company relies on to collect and index web content is now also being used to support AI-based search capabilities; a development that has raised concerns about upsetting the balance between the search engine and content owners.
This practice, largely hidden from public view, gives Google a significant advantage over other AI players and at the same time increases automated traffic on the internet at the expense of human users’ share.
However, Google now faces resistance from a UK regulator and one major industry player has launched a broad campaign against these practices; actions that could ultimately pave the way for necessary changes.
For years Googlebot, the company’s automated crawler, has roamed the web collecting website pages to add to Google’s vast index. This process allows Google’s search engine to show the most relevant and up-to-date pages in response to users’ queries.
However, in recent years the boundary between content Google gathers for its search engine and content used to generate AI responses has blurred. The same infrastructure is now used to supply content for both traditional search results and AI-powered generative search features.
Integrating search systems
According to Matthew Prince, CEO of internet security company Cloudflare, Google’s crawler accesses about 3.2 times more internet content than OpenAI’s crawler and about 4.8 times more than Microsoft’s crawler.
Prince runs a company through whose infrastructure about one-fifth of the internet’s traffic flows. Cloudflare manages that traffic and protects it from malicious attacks.
Prince has criticized Google for integrating its search system with its web-content collection system. He says this approach makes it difficult for publishers to prevent Google from using their content in AI-generated summaries without simultaneously losing the traffic they receive from traditional search results.
This advantage has broader consequences. The more AI relies on reading, summarizing and rewriting internet content without directing users to human-created pages, the weaker the economic incentive to produce original content will become.
Signs of this impact are now visible in the metrics of some websites and publishers. Reach, one of the UK’s largest journalism groups, said this week that visits to its sites from Google have almost halved, citing the expansion of “AI summaries” as one factor in the decline.
The dead internet
According to a Cloudflare report, AI proxy programs accounted for more than 57 percent of internet traffic in 2026, while human traffic was about 42 percent. Thus, for the first time in the internet’s history, machine traffic has surpassed human traffic.
Prince expects this gap to widen in the coming years; a trajectory aligned with what is called the “dead internet theory.” According to this theory, machines will gradually take a larger share of producing and consuming internet content.
Prince says human activity on the internet could become negligible compared with the activity of apps and proxy bots.
So far, use of AI proxy programs has been largely limited to software developers and companies. However, Mark Zuckerberg announced last week that Meta Platforms plans to soon offer a consumer-facing version of this technology through Facebook Messenger, WhatsApp and Instagram; a product he said will be ready and simple for billions of users.
Using bots to perform useful tasks is not inherently a problem, but if human creators no longer receive economic rewards for the content they produce, the internet risks becoming shallower, less authentic and ultimately less useful. In such a scenario, incentives to produce high-quality content will gradually decline.
The main threat from Google’s new search system arises at this point: humans create content, machines copy and summarize it into AI-generated responses, and the economic returns from those responses do not necessarily reach the original creators.
Accessibility
Google could have addressed this problem by offering websites an option to request that their content not be used in AI-based search features, without affecting its appearance in traditional search results.
According to internal Google slides revealed during a US antitrust case, the company considered offering such an option before launching “AI summaries” in April 2024, but ultimately abandoned it because the feature was revenue-generating for the company.
Since then Google has faced the consequences of that decision. Prince says Cloudflare warned the company that from 15 September it would automatically block Google’s multi-purpose crawlers for customers reliant on advertising revenues.
As a result, Google could lose access to millions of websites that use Cloudflare’s services.
Rising pressure
Pressure from UK antitrust authorities on Google has also increased. In June the Competition and Markets Authority (CMA) asked Google to give website owners a clear choice to prevent their content being used in AI-based search features without stopping their appearance in traditional search results.
Google was also barred from penalizing sites that choose such an option by lowering their ranking in search results.
Prince says there are signs Google may implement this standard not only in the UK but globally.
He also said a Google spokesperson confirmed the company is testing new settings that would allow websites to exclude their content from AI responses without reducing their ranking in search results. The feature is expected to be rolled out globally after testing in the UK.
This move could signal a change in current practices; practices in which the world’s largest tech companies largely set the rules for using internet content.
Google’s options
However, Google’s crawlers remain technically integrated. The crawler collects website content for indexing in the search engine, and that same content is then made available for use in AI-based search features.
One proposed solution is to split the system into two completely separate crawlers: one for indexing content in traditional search results and another for AI-based search features.
In that case, websites could directly and via the same infrastructure block AI from using their content without having to rely solely on Google’s promises; promises that internal slides from 2024 showed might change if revenues were at risk.
This solution could still be a middle ground compared with stricter approaches; including a blanket ban on data collection for training AI models or requiring AI companies to pay publishers directly.
Google still holds about 90 percent of the search market and has effectively become one of the main gateways to the internet. However, the company’s effort to leverage this dominance in the AI era has created new challenges. Increasing regulatory pressure and industry criticism could push Google to change its practices regarding the collection and use of web content.