GoDaddy integrates Cloudflare’s AI Crawl Control into hosting to let sites block, permit, or even charge AI crawlers

It has been reported that Cloudflare and GoDaddy are teaming up to give website owners more say over the bots roaming the internet. GoDaddy will integrate Cloudflare’s AI Crawl Control into its hosting platform, allowing site owners to block crawlers, permit them, or potentially charge for access. These crawlers—allegedly run by the likes of Anthropic, OpenAI, and Google—have been scooping up content to feed large language models while often sending back little referral traffic.

The deal and the tech

Cloudflare already moves and shields traffic for roughly 20% of the web; GoDaddy remains the largest domain registrar. Together they have a real shot at shifting norms. It has been reported that the companies are also backing standards such as Agent Name Service and Web Bot Auth to help verify the identity and behavior of automated agents. This builds on Cloudflare’s prior steps to block AI crawlers by default and to roll out tools that let publishers monetize or limit bot access.

What this means for creators and AI firms

Why should anyone care? Because the old web business model—index, click, monetize—has been strained by AI answer engines that return answers straight to users without sending them back to the source. The result: creators and small businesses watch traffic and revenue decline while their work trains models elsewhere. The emotional core here is obvious: people who produce original content feel exploited. Who pays for the work that powers the next chatbot?

The next act is messy and high-stakes

Implementation details, adoption rates, and whether major AI companies will accept paid access remain unclear; it has been reported that this is still early, and much depends on standards adoption and negotiation. Still, Cloudflare and GoDaddy are betting that identity, control, and compensation infrastructure can nudge the web toward a permission-based future—one where creators get a seat at the table, not just crumbs from the AI feast. Welcome to the internet trying to grow up.

Sources: businessinsider.com