Tag Archives: Content Economy

Who pays the writer when the reader is an agent? A writer's page sends content to an AI agent, with a dashed return arrow and a dollar coin asking whether the agent pays the writer.

Who Pays the Writer When the Reader Is an Agent?

AI, , , , , , , , ,

Objective

The goal of this post is to show, with data from Cloudflare and TollBit, what is already happening to the web’s content economy now that AI agents do much of the reading for people. Today, AI companies pay a handful of large publishers through private licensing deals, while most other creators get little or nothing. I walk through why the old bargain broke, what the industry is building to replace it, why micropayments might finally work this time, and what still has to happen for creators to keep a reason to publish on the open web.

The Bargain That Built the Web

For about 25 years, the open web ran on an unwritten deal. You published something, a search engine crawled it, and in return the search engine sent you readers. What you did with those readers was up to you. Some creators showed ads. Some put the good parts behind a paywall and converted readers into subscribers. Many simply wanted to be read: to build a reputation, find customers, or share what they knew.

All three payoffs depended on one thing: a human visiting the page. The crawl was the price; the visit was the payment.

Two diagrams. Left: creator, search engine and reader in a loop where links bring visits back to the creator. Right: creator feeds an AI agent, the agent answers the reader directly, and only a dashed "rare clicks" arrow returns to the creator.
In the search web, the visit paid the creator. In the agent web, the answer stays with the agent, and so do the ads.

How the Bargain Broke

When people ask ChatGPT, Claude, Perplexity or Google’s AI Overviews instead of clicking through search results, the crawl still happens, but the visit mostly doesn’t.

Cloudflare sees a large share of web traffic and measures this directly as a crawl-to-referral ratio: how many pages a platform fetches for every visitor it sends back. In June 2025, Google crawled roughly 14 pages per referral, OpenAI about 1,700, and Anthropic about 73,000.

Bar chart on a log scale showing pages crawled per visitor referred in June 2025: Google 14 to 1, OpenAI 1,700 to 1, Anthropic 73,000 to 1.
Search engines still send visitors back. AI platforms mostly don’t.

These numbers move a lot, and that matters for how you read them. Anthropic’s ratio was around 286,000:1 in January 2025 and fell to about 38,000:1 by July, after Claude added web search with clickable citations. In the same Cloudflare sample, Mistral sent back more referrals than it made crawls. The ratio depends heavily on whether a platform’s crawling is for training, which never sends anyone back, or for answering live questions with links.

TollBit measures the same problem from the publisher side, using the sites that run its analytics. At the end of 2024, it found click-through rates from AI chatbots were 95.7% lower than from traditional Google search. Through 2025 it kept getting worse.

Line chart of click-through rate from AI apps in 2025. Sites with AI licensing deals fell from 8.8% in Q1 to 1.33% in Q4. All TollBit partner sites fell from 0.8% in Q2 to 0.27% in Q4.
Licensing deals bring money, not readers. Even publishers with deals saw click-through fall about 85% in a year.

The line I keep coming back to is the top one. It tracks how often people clicked through from AI apps to the sites of publishers that had signed AI licensing deals. Those publishers started 2025 with far better click-through than everyone else, at 8.8%, and by the end of the year it had fallen to 1.33%. A deal pays you, but it doesn’t bring the readers back.

Bots are also becoming a meaningful share of traffic in their own right. By Q4 2025, TollBit’s partner sites saw one AI bot visit for every 31 human visits, up from one in 50 earlier that year. One publisher, Digital Trends, measured 966 bot scrapes for every human referral.

The effect on a large publisher is concrete. People Inc. (formerly Dotdash Meredith) said Google search went from 54% of its traffic two years earlier to 24%, and blamed AI Overviews.

Ads didn’t disappear. They moved.

It is tempting to say the ad model is dying. It isn’t. It moved up to the agent layer. OpenAI launched ads in ChatGPT in early 2026, and less than 200 days later announced a $1 billion annualized run rate. That is a projection from one month of revenue, not money collected, but the direction is clear. Someone is still looking at ads. The difference is that the interface owner sells them, and nothing flows down to the people whose work made the answer possible.

This also explains why GEO (generative engine optimization) is a symptom rather than a fix. GEO helps your content get cited in an AI answer. That’s good for reach and reputation, but a citation without a click earns a publisher nothing. It is optimizing for a currency that doesn’t pay yet.

Not All Content Breaks the Same Way

The damage isn’t uniform. I find it useful to split the web into four kinds of content, because each one breaks differently.

Content that wants to be found. Product documentation, marketing pages, open-source READMEs. These sites never monetized the visit directly. Agents are simply a new distribution channel for them, and they will happily optimize for it.

Commerce. Here the ad was really a toll on discovery, and agents are already replacing it with a commission. OpenAI and Stripe’s Agentic Commerce Protocol (ACP) lets you buy from a merchant inside ChatGPT without visiting the merchant’s site. The merchant stays the seller of record, Stripe issues a payment token limited to that merchant and that cart total, and the merchant pays OpenAI a small fee on completed sales. The money flows from merchant to agent, because the agent brought the buyer. Commerce will mostly sort itself out: pay-per-click becomes pay-per-sale.

The one thing to watch in commerce is paid influence. If a merchant can pay to rank higher in an agent’s shortlist, and the agent buys on your behalf without showing you the alternatives, the agent quietly becomes the merchant’s salesperson. The line I’d draw is simple: influence that changes the deal for the buyer (a lower price, faster delivery) is competition; influence that changes the ranking without changing the deal is a conflict of interest. OpenAI says ads do not influence ChatGPT’s answers today. That is a promise worth holding it to.

User-generated content platforms. Reddit and Stack Overflow have sold their archives to AI companies; Reddit’s OpenAI deal is reportedly worth about $70 million a year. I don’t think bulk licensing is a durable model, because it undermines its own supply. Stack Overflow’s monthly questions fell from over 200,000 at the 2014 peak to under 50,000 by late 2025, and by December 2025 were back at 2008 levels. Over the same period its revenue reportedly doubled, partly from AI licensing. The platform is selling the stock of past answers while the flow of new ones dries up.

Content-as-product. News, analysis, reviews, reference, and the work of independent creators. This content was paid for by the visit, and there is no commission to replace it. This is where the hole is, and it is the focus of the rest of this post.

What Is Being Built

Over the past year, a fairly complete stack has emerged for letting a publisher charge an agent. Cloudflare is the most visible builder, but most of the pieces are open standards.

Four stacked layers: Declare (robots.txt, Content Signals, RSL), Identify (Web Bot Auth), Enforce (CDN or WAF edge, app middleware) and Pay (pay per crawl or per use, x402). A side panel notes that the edge can enforce access, but use after fetch depends only on law and contracts.
Each layer is weak alone. Together they make a working toll booth for honest agents.

Declare: what may be done with the content. Cloudflare’s Content Signals Policy extends robots.txt with three uses a site can allow or refuse separately: search (a traditional index with links), ai-input (using the content to generate an answer at query time) and ai-train (training a model). RSL (Really Simple Licensing) 1.0, released in December 2025, goes further and lets a publisher state licensing and payment terms in a machine-readable file. Over 1,500 media organizations support it, and it is managed by the RSL Collective, a nonprofit modeled on music royalty bodies like ASCAP and BMI. The catch: no major AI company has committed to honoring it.

Identify: which bot is asking. Today a crawler just claims an identity in a header, and anyone can fake it. Web Bot Auth, a Cloudflare-led IETF draft built on HTTP Message Signatures (RFC 9421), has bots sign their requests so the server can verify who is asking. It is still a draft, but Cloudflare, Anthropic and OpenAI moving to production together has made it the de facto standard. You can’t charge someone you can’t identify, so this is the foundation for everything else.

Enforce: allow, block or charge. Enforcement happens wherever the traffic flows. A CDN or firewall at the edge is the natural place, because it sees every request before your server does: it checks the signature, applies the rule for that bot, and returns HTTP 402 Payment Required if the bot hasn’t paid. Cloudflare says its network now returns over a billion 402 responses to AI crawlers per day. The same logic can also run as middleware inside the application, which is slower to reject unwanted traffic but knows things the edge doesn’t, such as whether an article is premium or breaking news. I expect most publishers to use both: the edge for coarse filtering and identity, the application for fine-grained pricing.

Pay: settle the charge. Cloudflare launched pay per crawl in July 2025, acting as merchant of record and letting a site set a flat price per request. In July 2026 it announced Pay Per Use, which pays publishers when their content actually creates value, for example when it appears in an AI answer, rather than when it is fetched. It also announced a Monetization Gateway that can charge for any web page, dataset, API or MCP tool, settling in stablecoins over x402, an open payment protocol that Coinbase created and handed to the Linux Foundation in 2026.

From September 15, 2026, Cloudflare also blocks “mixed-use” crawlers by default on ad-supported pages. That change is aimed squarely at Google, whose crawler collects for both search and AI Overviews in one pass. Until now, a publisher couldn’t refuse one without disappearing from the other.

Who pays whom

Once you look at real flows, there isn’t one payment model. There are three, and they differ in how far the user is from each payment.

Three columns on a spectrum. User's card: agent shops for you, the statement shows the merchant and the merchant knows the buyer. User-funded agent wallet: agent spends your top-up on small digital purchases, the seller sees only the agent. Agent operator's wallet: agent pays for its own inputs, you pay a subscription and the publisher sees only the agent.
In commerce the user still pays directly. In the content economy, the agent pays and the user stays anonymous to the seller.

With the user’s card, as in ChatGPT’s checkout today, the agent orchestrates but the money is yours: the merchant appears on your statement and knows who you are. With a user-funded agent wallet, you top up a balance and the agent spends it on small things on your behalf, like a paid article you want to read or a dataset for a research task. With the agent operator’s wallet, the agent pays for its own inputs, such as the 20 articles it reads to answer your question, and recovers the cost through your subscription. The content economy mostly lives in the last two, which is why it needs new rails rather than card networks.

Why Micropayments Could Work This Time

Paying a few cents per article has been tried since the 1990s and has failed every time. The usual explanation is fees, and fees were real: one recent analysis notes that traditional payment processors struggle with anything under about $0.50. But the deeper reason was named by Nick Szabo in the late 1990s. He argued that the binding cost of a micropayment isn’t technological, it’s mental: every tiny purchase forces a person to decide whether it’s worth it, and that decision costs more attention than the item is worth. So people chose subscriptions, or free content with ads.

An agent doesn’t have that cost. Given a budget and a policy, it can make thousands of small purchase decisions without fatigue. Szabo himself anticipated this. He suggested micropayments might work if a person could hand their budget and preferences to software that completes the purchasing for them. That is a reasonably precise description of an AI agent with a spending policy.

Stablecoins remove the other half of the problem. A card payment carries a fixed fee that swamps a one-cent charge. A stablecoin transfer on a fast network costs a tiny fraction of a cent (one source puts a Solana transfer at about $0.0001) and settles in seconds, globally, without each publisher needing a merchant account. That is why x402 settles in stablecoins. In the 30 days before the x402 Foundation’s July 2026 launch, the protocol processed 75.41 million transactions worth $24.24 million, an average of about $0.32 per transaction. These are mostly API and data payments, not article fees, but they show that machine-to-machine micropayments are already running at scale.

I don’t want to overstate this. Stablecoins don’t carry the chargeback protections people are used to on cards, regulation varies a lot by country, and for anything large the user’s card with a signed authorization will remain the better choice. But for the specific case of an agent buying a fraction of a cent’s worth of content, both of the old obstacles, the fee and the decision, are gone at the same time. That has not been true before.

The Hard Problems

Access is not use

robots.txt and Content Signals are declarations, not locks. A well-behaved bot chooses to comply; nothing stops one that doesn’t. TollBit found that AI scrapes bypassing robots.txt grew by over 40% in a single quarter, and in August 2025 Cloudflare removed Perplexity from its verified bot list for disguising its crawlers to evade blocks.

So enforcement comes from three places working together: the publisher declares, the edge enforces access, and the law covers use. But no protocol can see what happens after a page is fetched. Content can be cached and reused indefinitely, and training is the ultimate cache: one crawl, permanent value. That is why training is fought in court and settled with lump sums, while live answers are where per-request pricing can hold. It is also why the mixed-use crawler matters: if you allow a crawler for search, you can’t technically stop its owner using the same content for AI answers.

Who sets the price

Cloudflare’s Matthew Prince offered a useful picture: an AI model’s knowledge is a block of Swiss cheese. Content that fills a hole, such as new reporting, niche expertise or first-hand data, is worth far more than the hundredth article repeating what the model already knows. Pricing by new knowledge instead of traffic would reward originality over the clickbait the ad model encouraged.

But it is hard to measure. Only the AI company can see what its model already knows, the same article may fill a hole for one model and not another, and the first source captures nearly all the value. Supply and demand won’t fix this on their own: there are only a handful of buyers, only the buyer knows which content was useful, and content is trivially copyable. TollBit’s CEO put it bluntly: AI companies say they can’t pay every publisher the asking rate.

The toll booths are concentrating too. Every protocol here is open, but the practical position belongs to the few CDNs already in front of much of the web. Open protocols don’t guarantee an open market.

The toll booth is built, but few drive through

Cloudflare’s Pay Per Use launched with two partners and the Monetization Gateway is still a waitlist. Of the more than 7,000 publisher sites on TollBit’s network, nearly 20% are earning revenue from its bot paywall, ranging from hundreds to tens of thousands of dollars a month. The real money still moves through private deals like News Corp’s with OpenAI, reportedly worth more than $250 million over five years, which only large publishers can negotiate.

There are signs the open model can work. People Inc. said blocking non-paying bots brought almost everyone to the table, and it joined Microsoft’s pay-per-use marketplace. But for most creators, the traffic either turns around at the toll booth or passes on someone else’s negotiated pass.

What Needs to Happen

  • Separate crawlers by purpose, Google included. Most AI companies now use different crawlers for training, search and live user requests, so a publisher can treat each one differently. Google’s dual-purpose crawler is the largest exception, and until it splits, no publisher can refuse AI answers without losing search.
  • Separate enforcement from settlement. Enforcement has to sit where the traffic flows, which favors CDNs. Payment doesn’t. Publishers and AI companies should insist on open settlement through x402 or similar, so whoever controls access doesn’t automatically control the money.
  • Build a standard for usage reporting. Pay Per Use only works if AI companies report honestly when and how content was used. Today that depends on trust. This is the missing piece of the stack, and the one that makes the Swiss cheese idea measurable.
  • Make collective licensing the default for the long tail. An independent writer can’t negotiate with OpenAI. A collective like the RSL Collective can, the same way ASCAP lets songwriters get paid for radio play without counting every broadcast.
  • License flow, not stock, and share it with contributors. UGC platforms should sell ongoing access to fresh content rather than one-time dumps of their archives, and pass part of the revenue to the people producing it. Otherwise the contributors stop contributing.
  • Invest in spending-policy infrastructure. Users will only let agents spend on their behalf if setting, auditing and revoking limits is simple: a budget, a per-transaction cap, allowed categories, and a threshold above which the agent must ask. This is the layer Szabo was describing, and it gets far less attention than payment protocols.

Summary

The lesson is uncomfortable: the web’s old bargain can’t be bought back, so payment has to replace traffic. AI platforms crawl thousands of pages for every visitor they send back, click-through from AI apps fell to 0.27% by the end of 2025, and even publishers with licensing deals lost most of theirs. The pieces of a replacement exist: signed agent identity, machine-readable terms, 402 responses at the edge, and stablecoin micropayments that agents can make without the decision fatigue that killed micropayments before. But participation is uneven. Google hasn’t separated its crawlers, AI companies prefer private deals to open marketplaces, and small creators have no bargaining power. The rails are being built faster than the incentives, and creators who leave rarely come back. The real question is whether they will still be here when the market is ready.

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude.ai, which helped me research and check the sources, brainstorm the structure, draft the text and produce the figures.

References

Cloudflare

Standards and protocols

Data

Industry developments

Background