Cloudflare experiment will block AI bot scrapers unless they pay a fee

Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether. The company released a free tool in 2024 to block AI bots, but with this change, users can block them by default without having to tinker with their settings. Several big publishers, including Condé Nast, TIME and The Associated Press have already signed up to block crawlers. In addition, Cloudflare has launched a private beta experiment called "pay per crawl," which would only allow crawlers to access a website's content if they pay for it. 

Matthew Prince, Cloudflare's CEO, recently went on record to say that publishers are facing an existential threat, because people aren't clicking on chatbots' source links. If users don't visit those sources, the websites don't get the ad revenue they need to be able to keep running. "Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it," Prince said in a statement released with the company's latest updates. "AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."

Cloudflare AI Audit.
Cloudflare

Cloudflare believes publishers should be able to charge AI bots for access if they want to, and pay per crawl is its first experiment for that particular purpose. "Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing," Cloudflare explained. The company records those transactions and provides the underlying technical infrastructure. Publishers will be able to allow certain crawlers to access their content for free if they want to, and they can define a flat, per-request price across its websites for other crawlers.

The company says pay per crawl is still in its very early stages, and it expects the tool to evolve in the future. It also says that it supports the development of other marketplaces and ways to charge AI crawlers for content. A marketplace could, for instance, allow dynamic pricing that enable publishers to charge different rates for different types of content. 

This article originally appeared on Engadget at https://www.engadget.com/ai/cloudflare-experiment-will-block-ai-bot-scrapers-unless-they-pay-a-fee-121523327.html?src=rss https://www.engadget.com/ai/cloudflare-experiment-will-block-ai-bot-scrapers-unless-they-pay-a-fee-121523327.html?src=rss
Created 4h | Jul 1, 2025, 2:30:22 PM


Login to add comment

Other posts in this group

Xbox's first Game Pass additions for July include Tony Hawk’s Pro Skater 3 + 4

Xbox has confirmed the first batch of Game Pass additions for July. The headliner this

Jul 1, 2025, 4:40:40 PM | Engadget
Apple TV's MLS Season Pass is half off for the rest of the season.

If you've put off buying an

Jul 1, 2025, 4:40:38 PM | Engadget
Apple claims former engineer shared Vision Pro secrets in new lawsuit

Apple is suing one of its former design engineers for allegedly stealing a trove of trade secrets that he then provided to his new employer, Snap. As reported by

Jul 1, 2025, 4:40:37 PM | Engadget
The best Prime Day Apple deals on iPads, AirPods, Apple Watches and more

Amazon’s Prime Day doesn’t officially begin until Tuesday, July 8, but we’re already seeing deals on some of our

Jul 1, 2025, 4:40:36 PM | Engadget
The Running Man trailer: Edgar Wright adds comedy to Stephen King's sci-fi dystopia

Edgar Wright is remaking The Running Man, and there's a ">trailer to prove it. Arnold Schwarzene

Jul 1, 2025, 4:40:35 PM | Engadget
Pick up one of our favorite Ninja air fryers for $50 off this Prime Day

You may be spending more time outside during the summer, especially if you have a grill, but the gadgets in your kitchen will come in handy this season, too. Prime Day deals have already discounted

Jul 1, 2025, 4:40:33 PM | Engadget
Donkey Kong Bananza was made by the team behind Super Mario Odyssey

Nintendo has confirmed that the top-notch development team behind Super Ma

Jul 1, 2025, 4:40:32 PM | Engadget