AI agents are blocked from half the web. Who's going to fix that?
I’m a heavy AI user. Earlier this month I was using Claude to research a network firewall product, and as part of combing through the evidence I pointed my little army of agents at Reddit, because that’s where the good reviews are. Amazon listings, app store install counts, a couple of company-data sites too.
A lot of them came back empty-handed. A 503 from Amazon, 403s from a help center and the data sites, and Reddit, which now blocks anyone it hasn’t signed a deal with. So I had Claude Code run the same searches through a browser via an MCP server, and it visited every page and got the info I needed. Same agent, same questions. The only difference was that the second time it looked like me: my Chrome, my cookies, my home IP.
Using a browser, taking screenshots, parsing the page - that’s a lot of work for a few facts. And it got me thinking. Everybody is using AI agents to research and to act on their behalf, and yet the agents are blocked from the key sites. Surely there’s a better way to do this. Even if there isn’t a solution right now, this can’t be the state of things in two years. So who’s in the best position to solve it, and how are they doing it?
Here’s what I found once I dug in.
The sites aren’t blocking AI. They’re blocking traffic they don’t recognize. My agent and a lab’s training crawler look the same from the outside: datacenter IP, no history, requests coming in at a pace no human types. The site can’t tell them apart, so it blocks both. Cloudflare’s July report says more than half the traffic it sees is now non-human, and more than half of the crawler traffic is for AI training. A crawler that takes your content and sends nobody back to your site isn’t a visitor, so of course they block it.
The big assistants don’t have this problem, and once you see why, the whole thing makes sense. Meta’s fetcher, OpenAI’s, Google’s - they show up with a name (meta-externalfetcher, ChatGPT-User), from published IP ranges, and they’re on the lists of known bots that Cloudflare and the other CDNs keep. Sites have let Meta’s bots in for fifteen years because that’s how link previews work. Meta’s docs even say its fetcher can ignore robots.txt if a user asked for the page. So there is a way in. It’s just that only a handful of companies get to write their own pass.
Which brings me to - Who fixes it for people doing their own research? Cloudflare sits in front of about a quarter of all websites, so it’s already where the checking happens, and it has been busy. Signed requests so a bot can prove who it is (Web Bot Auth). A registry of signed agents. Pay Per Crawl, which turns a request into an HTTP 402 with a price, and now a payments layer on x402 so an agent can pay a fraction of a cent per page. And from September 15, its default for new sites with ads is to block agent traffic unless the owner opts in or gets paid. Cloudflare decides who gets in and collects the money.
Meta is the other one to watch. On September 8 it launched Muse, a personal agent that runs in a browser in Meta’s cloud and books, buys and fills forms for you. It’ll hit the same walls my agents hit, unless Meta cuts deals. Meta also has over 200 million businesses on WhatsApp (its 2023 number) who never need a website visit at all, so a lot of bookings will just happen inside WhatsApp.
Cloudflare has the position, Meta has the users, and nobody has built the piece in the middle yet - proof of which human an agent is working for, and who’s on the hook when it gets something wrong.
If you’re thinking about how the tech will evolve in this space, share your thoughts at my email.