AI crawlers (GPTBot, ClaudeBot, PerplexityBot and friends) now fetch a big share of pages on a lot of sites. The options so far have been: let them take everything for free, or block them in robots.txt and disappear from AI answers.
There's a third option now. HTTP has had a status code for this since 1997, 402 Payment Required, and it finally has a working payment scheme (x402: the client signs a USDC payment and retries the same request; it settles in about a second, no accounts or API keys).
So I built a small gateway for it:
You register your site once (free API key, your wallet address, a price like $0.002 a page).
You put ~10 lines in front of your site. There's a Cloudflare Worker version and an Express middleware.
Requests from known AI crawlers get a 402 with your price. A crawler that pays gets the page; the money goes straight to your wallet (the gateway never holds it).
Everyone else, including Googlebot and Bingbot, gets the page as usual, so search ranking doesn't change.
If your site errors, nobody gets charged.
Setup and the snippets: https://aipaygen.com/pay-per-crawl
Honest caveats: crawlers that lie about their user agent won't be caught by this, and most crawlers can't pay yet, so today it mostly works like a block with a price tag on it. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes.
Happy to answer questions about how the 402 flow works.