How Cloudflare declared war on AI scrapers

Cloudflare supports more than 20% of total internet traffic. The company recently made headlines with breakthrough technology that blocks AI companies from scraping online content with impunity. Cofounder and CEO Matthew Prince shares how the new tools are poised to dramatically impact AI firms, publishers, and the future of the internet. 

This is an abridged transcript of an interview from Rapid Response, hosted by Robert Safian, former editor-in-chief of Fast Company. From the team behind the Masters of Scale podcast, Rapid Response features candid conversations with today’s top business leaders navigating real-time challenges. Subscribe to Rapid Response wherever you get your podcasts to ensure you never miss an episode.

You released a new tool that’s got a lot of folks buzzing: a blocker for AI crawlers—the bots that scrape content from websites without their consent. You’ve called this new tool “the biggest thing” you or the company has ever accomplished?

Yeah. I feel incredibly fortunate to have built what today is a $60 billion company on the back of the internet. And we became aware about 18 months ago of a new threat to the internet, to content creators, which was being posed by these AI companies. When we realized that there was something we could do about it, we spent about a year talking to everyone in the content creation space, everyone in the AI space. . . . We’re going to change the rules of the road, and say that if you’re not paying for content as an AI company, then you don’t get that content.

Today it’s almost 10 times harder to get actual traffic from Google for the same amount of content you created. The minute you show an AI overview, it’s less likely that people click on links. And again, that is better for the Google user, but it is worse for the content creator because it means that you can’t sell a subscription, you can’t sell ads, and you can’t even get the ego boost of knowing that people are reading your stuff.

Today, OpenAI is 750 times harder to get traffic from than the Google of old. Anthropic is 30,000 times harder to get traffic from than the Google of old. And so, if content creation is struggling today [when it’s] 10 times harder, I worry that it won’t survive [if it’s] 750 times or 30,000 times harder [to read] original content. . . . And if people don’t have the incentive to create content, they’re not going to create content. So there needs to be some business model behind the future of the web, and it’s not going to be around traffic because an AI-driven web doesn’t drive traffic.

And the irony is that the AI itself needs the content to be able to make those answers. Now who knows where they’re going to get their answers from.

That’s the key: 80% of the major AI companies use Cloudflare in their infrastructure. What they have all said, with a few exceptions, is “We agree, content creators need to get paid for content, but it has to be a level playing field.” What nobody wants to do is pay for content where all of their competitors get it for free. So, creating that level playing field is incredibly important. 

Just Anthropic will scrape a site 60,000 times for every one visitor that’s there. Someone has to pay for that traffic. Just from a pure fairness perspective, they should be compensating creators that they’re pulling that content from.

We started as a cybersecurity company. We go to war every day with Russian hackers, Iranian hackers, North Korean hackers, Chinese hackers who are trying to get in and thwart our systems. So when we first started talking to publishers about this, it was almost this sort of nihilistic, “Oh my gosh, what are we possibly going to do? There’s no way we can stop it. These guys are so smart, they’re a bunch of nerds in Palo Alto. . . . We can’t ever possibly block them.” And I remember thinking, We block the North Koreans every day. AI companies are a piece of cake.

Before you release the first round of this tool, did you give the AI companies a heads-up?

I think there are some bad actors out there, and I think it’ll surprise some people who the bad actors are. We’re monitoring them, and very soon we will publish and we will name and shame who is actually a bad actor in this space. And we will take from what has been basically posting a speed limit sign that says “Don’t drive more than 55 miles an hour” . . . and we’ll make it into something that is actually much more strict. We’re saying, “Listen, we’re taking away your car, you’re not allowed to drive on the road anymore.”

I understand you’re exploring sort of a pay-per-crawl model with some of the content publishers, which to me sounds a little bit like a toll on the highway—that you have to pay a toll if you want to come through.

If you are generating a huge amount of cost by crawling somebody, but you’re not giving them any benefit, then step one is block them. Then once you’ve created scarcity, then there can be a market, right?

There has to be some compensation for taking content, and it’s not going to be traffic anymore, it’s going to be something else. Now the question is, “Okay, how do you pay?” And I think a lot of times, big AI companies and big publishers are just going to negotiate deals themselves. So if you’re Condé Nast, you go out and do an OpenAI deal, or a Google deal, or something else, and you negotiate it yourself. We don’t have any role in that. I think for the smaller AI companies, or for the long tail of publishers, Cloudflare can hopefully sit in between and help negotiate what is the best deal. And we don’t know exactly what that will look like yet. It could be a micropayment every time a page is accessed. It could be something that’s closer to a Spotify model where there’s a pool of funds and that gets distributed out to all of the different content providers. . . . That will develop, but step one in any market has to be scarcity. If you don’t have scarcity, you don’t have a market.

I’m actually optimistic [that] all of us are going to have subscriptions to a certain number of AI agents that are out there. And how AI companies will differentiate themselves is access to unique content that they have and they have alone. So, imagine Taylor Swift is about to release a new album, and she does an interview with some journalists, and they are willing to give that interview to one AI company exclusively for a week. How much is that worth? Probably quite a bit, right? A lot of people are going to sign up. And so, I’m actually optimistic that we might be at the precipice of a golden age of content creation.

If we do this right, and we get the incentives right, it might be that instead of us all worshiping the deity that Google taught us to worship, which is traffic, which has always been a really bad proxy for value, if instead we find a way to compensate creators based on when they actually create something which is worthwhile and advances human knowledge, we can actually do some real good in the world, at the same time that we help the content creators get paid more.

https://www.fastcompany.com/91376582/how-cloudflare-declared-war-on-ai-scrapers?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss