Pay Per Use Is the New Paywall
2026-07-02 · 5 min read · Janaina Maia
Cloudflare just did something that should make every product designer working with AI sit up and pay attention. They gave publishers a deadline: by September 15, AI companies must separate their search crawlers from their training and agent crawlers, or get blocked by default on any page that runs ads. Alongside that, they evolved their Pay Per Crawl marketplace into Pay Per Use, where publishers get paid when their content creates value in AI outputs, not just when a bot fetches a URL.
This is not a small policy tweak. It is an infrastructure company reshaping the economic relationship between the people who make content and the companies that consume it to train models and run agents. And the design implications are enormous.
What actually happened?
Cloudflare announced that starting September 15, 2026, its default settings will block mixed-use crawlers on pages that host ads. A mixed-use crawler is one that blends search indexing, AI training, and agent use into a single bot. Right now, most AI companies do exactly this. Google's Googlebot crawls for search results, AI Overviews, and AI Mode all at once. Cloudflare is saying: pick a lane.
CEO Matthew Prince framed it around a milestone that arrived earlier than expected: bot traffic now exceeds human traffic on the internet. When most of the traffic hitting your servers is machines training models and running agents, the old bargain — let us index you for search and we will send you readers — is broken.
The companion announcement is Pay Per Use, an evolution of last year's Pay Per Crawl. Instead of charging per page fetch, it charges when content actually creates value in an AI product. Initial partners are Ceramic.ai and You.com. Publishers opt in, and they get paid when their content surfaces in results or when an AI agent accesses premium content.
Why this is a design problem, not just a business problem.
The framing in most coverage is about money. Publishers want to be paid. AI companies want free access. Cloudflare is playing toll booth. That framing is not wrong, but it misses the product design challenge that sits underneath it.
Right now, when an AI product shows you a summary, a recommendation, or an answer that draws on web content, the user has almost no way to know where that information came from, whether the original creator consented, or whether money changed hands. The provenance is invisible. The economics are invisible. The user just sees a result.
This is the same problem I wrote about two days ago: labels are not enough. Slapping an AI badge on content tells you nothing about where it came from. The same applies on the supply side. Telling publishers they can now get paid does not solve the problem unless the AI products that consume their content build surfaces that make that payment visible, traceable, and meaningful to the end user.
What is Pay Per Use?
Pay Per Crawl was exactly what it sounds like: publishers charge AI bots per page fetch. The problem is that fetching a page is not the same as using it meaningfully. Over 50% of AI crawler traffic re-fetches unchanged pages, according to Cloudflare's own data. Charging per fetch incentivises crawling behaviour, not value creation.
Pay Per Use tries to fix this by connecting payment to outcomes. When your content appears in a Ceramic.ai search result, you get paid. When You.com accesses your premium article, you get paid. The model is closer to how royalties work in music or how affiliate links work in commerce: you get paid when value is created, not when the raw material is moved.
This is a better model for publishers and a fairer one for AI companies too. It means they pay for what they actually use, not for 50% redundant fetches. But making it work requires something that does not exist yet: a standard way to trace which content contributed to which AI output, and a product surface that shows that trace to the user.
The product implications.
- Attribution surfaces are now a competitive advantage. When an AI product can show you, the user, where a recommendation or answer came from and that the original creator was compensated, that is a trust signal. Not a compliance checkbox, a trust signal. The products that build visible attribution first will win with both publishers and users.
- Crawler separation is a design decision, not just an engineering one. Cloudflare is forcing AI companies to split their bots. But the harder question is: how does the product behave differently depending on which bot fetched the data? If search results and AI answers come from the same source material but are treated differently at the crawl level, the product needs to surface that distinction to the user. Is this a search result with a source link, or an AI-generated summary? The UX needs to make that clear.
- Pay Per Use needs measurement infrastructure. You cannot pay for value creation without measuring it. And measuring value contribution in a multi-source AI output is a genuinely hard problem. Who gets paid when an AI answer draws on five articles and synthesises them into one paragraph? How do you weight contribution? The companies that build transparent measurement will attract better publisher partnerships. The ones that hand-wave it will get the same distrust that search engines get today when they keep ad revenue share opaque.
- The opt-in model changes product dynamics. Pay Per Use is opt-in for publishers. That means some content will be compensated and some will not, based on who opted in. AI products will need to handle the UX of showing users that some sources are compensated partners and others are not, without creating a two-tier credibility system that penalises smaller publishers who lack the sophistication to opt in.
- This is the beginning, not the end. Cloudflare handles roughly 20% of web traffic. It is influential but not universal. The real question is whether this becomes the standard or remains one company's policy. If Google, which Cloudflare explicitly calls out as the biggest beneficiary of mixed-use crawling, decides to separate its bots, the market shifts. If it does not, publishers on Cloudflare get a lever they did not have before, but the broader ecosystem stays asymmetric.
My take.
I wrote recently that content provenance is a product surface, not a label. Cloudflare's announcement proves the point from the other direction. The labels are coming regardless. What matters is whether the products that consume this content build surfaces that make the provenance, the economics, and the consent visible to the people who create it and the people who see it.
Pay Per Use is the right idea at the right time. Bot traffic exceeding human traffic is the signal that the old bargain — index me for free and send me readers — is over. But the product design challenge is still ahead of us. Who builds the dashboard that shows a publisher what their content earned in AI products today? Who builds the attribution trail in the AI answer that shows which sources contributed and whether they were compensated? Who makes the crawler separation visible to the user, so they understand what they are seeing?
Cloudflare built the payment rail. Someone still needs to build the product surface. That is where the design opportunity is.