Cloudflare and GoDaddy team up to curb AI bot brigades

The deal

It has been reported that Cloudflare and GoDaddy are partnering to give site owners more control over how AI crawlers access web content — starting with GoDaddy integrating Cloudflare’s AI Crawl Control into its platform. The tool, the companies say, will let site operators allow, block or even require payment from crawlers seeking access. “By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model,” Cloudflare’s Stephanie Cohen said, framing the move as protection for creators and a way to let legitimate AI agents play by the rules.

Why it matters

AI agents have been gobbling up the web for training and answering queries, and it has been reported that Cloudflare has already experimented with measures to blunt unauthorized scraping — from an AI that feeds bad data to scrapers to proposals that would require bots to pay for access. The worry is existential for many sites: when an AI serves the answer, readers don’t always click through. Who gets paid then? For small publishers and indie creators the stakes are personal. This isn’t just a tech tug-of-war; it’s money, attention and the future of the “human-first” web.

Standards and the road ahead

The pair also expressed support for standards they say will separate good bots from bad ones. It has been reported that they back the Agent Name Service (ANS) — allegedly built by GoDaddy and available on GitHub — and Cloudflare’s Web Bot Auth, which uses cryptographic signatures to identify agent requests. The pitch: an open, protocol-agnostic registry and signed requests can create trust and transparency on an increasingly agentic web. Skeptics will ask: can standards beat abuse? The companies are betting yes — and that keeping websites monetizable keeps everyone in the ecosystem alive.

Sources: The Register