Passport Control for AI Agents

Cloudflare sits in front of more than 20% of web domains and decides which requests reach them. Starting September 15, every new domain that joins Cloudflare will block AI agents by default on any page that carries ads.

The new defaults split automated traffic into three categories:

▸ Search (indexes your content to answer queries later): allowed ▸ Agent (acts in real time on a person's behalf): blocked on pages with ads ▸ Training (absorbs your content into a model's weights): blocked on pages with ads

Why are ads the trigger? Because an ad is a signal that the page was meant for a person to land there. When an agent reads the page instead, no human ever sees the ad, so the site earns nothing from that visit.

Existing domains keep their current settings, and Cloudflare is notifying owners before the date, so nothing goes dark on day one. The shift is slower than that: every new domain starts at "no", most owners never touch their defaults, and over time the default becomes the landscape.

For legitimate agents, the same announcement pushes an identity rail called Web Bot Auth. The idea: your agent signs every request with a private key, you publish the matching public key in a directory where sites can look it up, and the site (or the proxy in front of it) checks the signature. A request can also declare who it acts for and why: Forwarded: for="openai"; use="reference". Signing does not lift the block by itself, but it turns your agent from anonymous traffic into a verified identity that the site can choose to allow.

The catch is that identity cuts both ways. Cloudflare's bot-management systems already score requests by where they come from, so a datacenter IP starts with low trust before any signature is read. And Cloudflare itself admits that this trust system only reaches those who can afford to be identifiable, while small traffic sources need privacy. In practice, big labs clear the bar easily, while smaller builders get pushed toward hiding, which is exactly what reputation systems punish.

If you build agents that touch the web, four checks this week:

1. Classify your traffic honestly (search, agent, or training), because mixed-purpose crawling gets the most restrictive rule. 2. Check how your agent handles a 403 (access denied): it should fail clearly instead of retrying and burning your IP's reputation. 3. Follow the request-signing work (Web Bot Auth). The specs are early drafts, but the direction is set. 4. My read: don't invest in camouflage like residential proxies. That arms race favors the infrastructure.

For two decades the web has filtered bots by their behavior (rate limits, CAPTCHAs, IP reputation). It is now starting to filter them by identity, and an agent that cannot prove who it is will keep getting blocked with no clear way to fix it.