One in Forty 'Trusted' Bots Is an Impostor. Your Allow-List Lets It Right In.

Roughly one in every forty requests claiming to be PerplexityBot is a fake.

Not a rounding error. Not a theoretical risk. DataDome measured it across 7.9 billion AI agent requests in January and February of this year: nearly 2.4% of everything wearing Perplexity’s name was an impostor. Meta’s agent had it worse, with 16.4 million spoofed requests in those same two months.

Now the uncomfortable part. If your bot policy has an allow-list, and almost every bot policy does, those impostors are not sneaking past your defenses. Your defenses are holding the door for them, because the allow-list checks names, and a name is text anyone can type.

We watched this play out in a client’s logs a few weeks ago. Four requests, one morning, one pricing endpoint, all four introducing themselves as PerplexityBot. One came from Perplexity’s published IP ranges. One arrived through a residential proxy. One came from an anonymous cloud host and pulled four thousand URLs in an hour. The allow-list admitted all four.

There is a check the three fakes could not have passed. It takes milliseconds, the standards behind it shipped over the last eighteen months, and by the end of this article you will know exactly how to turn it on, which agents can already pass it, and how to rebuild your allow-list around it in 30 days without breaking production.

The short version: automated traffic passed half the web in 2025 (53%, per the Thales Bad Bot Report), and about half a percentage point separates benign automation from malicious. No behavioral rule splits a gap that fine, and no user-agent string proves anything. What changed in 2026 is that verification grew teeth: published IP ranges and reverse DNS for legacy crawlers, cryptographic signatures under the IETF’s Web Bot Auth work for modern agents, and Visa’s Trusted Agent Protocol for agents that spend money. Rebuild the allow-list on proof instead of names, run a policy ladder for everything else, and measure it like the revenue system it is.

You are outnumbered on your own website

Humans became a minority of web traffic in 2025. The Thales Bad Bot Report puts automation at 53% of everything, with bad bots alone at 40%, and human visitors down to 47%.

The growth is even more lopsided than the totals:

• Automated traffic is growing eight times faster than human traffic, and traffic from autonomous web-acting agents grew 7,851% year over year (HUMAN Security’s 2026 State of AI Traffic & Cyberthreat Benchmark Report, drawn from over a quadrillion interactions).
• AI bot traffic on Akamai’s network is up more than 300% since tracking began. Commerce alone absorbed 25 billion AI bot requests in one two-month window.
• AI-driven attacks grew 12.5-fold in a year, from about 2 million blocked incidents per day to 25 million (Thales).

So “the bots” are not one crowd, and treating them as one crowd is where policies go wrong. Four different visitors are at your door, and each deserves a different reception.

Training crawlers (GPTBot, ClaudeBot, Meta’s crawlers) bulk-read your content to train models. Mostly polite, but the trade is worsening: Akamai’s AI Pulse research found AI chatbot referrals send roughly 96% less traffic than traditional search, and users click cited sources about 1% of the time.

Fetchers (ChatGPT-User, OAI-SearchBot) grab your pages on demand because a real person asked a real question. Closer to a customer than a crawler.

Agentic browsers barely existed two years ago and now generate close to three-quarters of all agentic traffic, per HUMAN’s April reading. By June, Perplexity’s Comet alone was 47.6% of it, Claude 20.8%, OpenAI’s Atlas 16.5%, and e-commerce was their number one destination at 43.8% of volume. These agents shop, compare, and book for a specific person, often from that person’s own device and home IP address. Remember that detail. It breaks half the verification techniques on the market.

And the attackers dress up as all three.

Here is the number that defines the whole problem: across HUMAN’s network, about half a percentage point separates the rate of benign automation from malicious automation. Your detection has to split a 0.5% gap, at line rate, without insulting the customers hiding inside it.

Behavior alone cannot split that gap. Identity can.

Names are free, and attackers know it

Every bot policy that trusts a user-agent string rests on an assumption nobody says out loud: that the client told the truth about itself.

Nothing checks that string. Nothing ever has.

That is why the spoofing numbers look the way they do. Meta-ExternalAgent: 16.4 million fake requests in two months, the most impersonated identity on the internet. ChatGPT-User: 7.9 million. PerplexityBot: the highest fake rate, at nearly 2.4%. The impostors cluster in e-commerce, real estate, and travel, exactly where pricing and inventory data is worth stealing.

Sit with what that means for your allow-list. By trusting the name “PerplexityBot,” you signed up to admit a population where one request in forty is an attacker who picked that name because you trust it. Your allow-list stopped being a convenience. It became a published invitation.

It gets worse, because name-checking fails even when nobody is attacking you:

• In August 2025, Cloudflare caught Perplexity itself running undeclared stealth crawlers that rotated IPs and ASNs to dodge no-crawl directives, across tens of thousands of domains, millions of requests per day.
• robots.txt, the tool most teams reach for, is codified in RFC 9309 as a preference. A “no trespassing” sign with no fence. More than 2.5 million websites have used Cloudflare’s managed tools to refuse AI training, and the crawlers that honor those signals were never the ones to worry about.

Which leaves both lazy policies broken. Block everything automated, and you turn away the shoppers behind that 43.8% of agent volume, traffic we showed in our agentic commerce field guide now converts better than human visitors. Allow the known names, and you wave through millions of impostors a month.

The way out: stop asking traffic what it is called. Start asking what it can prove.

The check a spoofed bot cannot pass

Here is the fix, in three tiers, from oldest to strongest.

Tier 1: network proof. The legacy method, and still worth running today. Google has supported it for years: resolve the requesting IP back to a googlebot.com or google.com hostname, forward-confirm it, done. OpenAI publishes IP ranges for GPTBot, OAI-SearchBot, and ChatGPT-User; Perplexity does the same for its crawlers. Two structural limits, though. It only scales to big operators who maintain published infrastructure. And it collapses for agentic browsers, which run on your customer’s device and arrive from your customer’s home IP, not Perplexity’s data center.

Tier 2: cryptographic proof. This is what changed the game, under the name Web Bot Auth. The mechanics are pleasantly boring: the agent operator generates an Ed25519 key pair, publishes the public key in a hosted directory, and signs every request using HTTP Message Signatures (RFC 9421), a standard the IETF finalized back in 2024. Your edge fetches the key and checks the signature.

A spoofer can type “PerplexityBot” into a header forever. Without the private key, it cannot produce a valid signature. The check fails in milliseconds, and mathematics does not accept excuses.

The plumbing went mainstream this year. The IETF chartered a Web Bot Auth working group in early 2026, with a best-current-practice document targeted for August. Cloudflare moved signature validation into production at its edge in March. Agents from Anthropic, OpenAI, and Perplexity already sign requests. And Akamai built its June 2026 Agentic Security Framework around the same primitives, connecting agent identity, observability, and edge policy into one real-time decision, which matters to those of us who deploy and tune Akamai’s stack for a living.

Tier 3: transaction proof. A different question: not “which company built this agent” but “which human authorized this purchase.” Visa’s Trusted Agent Protocol lets an agent prove itself at payment time, and Akamai integrated TAP with its edge in June so merchants can verify agents before a transaction touches backend systems. The Know Your Agent framework being built with Skyfire and Experian adds the link between agent, platform, and person. If agents move money on your site, your fraud team should be reading about this tier tonight.

The practical map we use with clients:

Read the bottom row twice. A request that claims a trusted name and fails the check has just handed you the highest-quality signal in your logs: a deliberate impersonator, revealed by its own costume.

Who gets in, and how far

With proof available, the allow-list stops being a flat file of names and becomes a ladder. The stronger the proof, the more generous the policy. Four rungs cover nearly everything we see in production.

Verified and on-script gets the fast path. Identity proven, behavior matching its declared purpose: a fetcher fetching, a shopping agent checking out. Generous rate limits, no challenges, direct access to what it legitimately needs. For commerce and travel platforms this rung is a revenue channel, and friction here is money lost. Be generous. You can afford to be, because the identity is proven.

Verified but off-script gets throttles and budgets. The identity checks out, but a training crawler is suddenly hitting your priced API, or a fetcher is requesting at crawler volumes. Verification does not mean obedience. This rung is also where you decide the economics of your content: given that 96% referral drop, you may reasonably conclude that training crawlers earn some content and not the rest, or that access has a price. Cloudflare’s pay-per-crawl mechanism, built on the HTTP 402 status code, exists for exactly that negotiation, and we expect 2027 to bring more of these compensation rails, not fewer.

Declared but unverifiable gets challenges and a sandbox. The long tail: honest names with no signatures yet. Some of it is tomorrow’s customer channel from a startup that has not implemented signing; some of it is abuse. Graduated challenges, access to non-sensitive content, observation. Resist blanket-blocking it. But nothing on this rung touches checkout, login, or pricing APIs without earning trust.

Impersonators and hostiles get blocked and recorded. Claimed a verified name, failed the check? That actor just revealed capability and intent in a single log line. Treat it as threat intelligence, not noise.

Two things run underneath the whole ladder. Agents call APIs rather than filling forms, so everything in our piece on API security applies with more force now. And a verified agent inside a stolen session is still a stolen session, so the account takeover checklist stays in service. It just gains a new disguise to watch for.

The four numbers that make it stick

A ladder nobody measures quietly rots into the old allow-list. Worse, its failures are invisible: block a legitimate shopping agent and your dashboard reports success while the sale completes at your competitor. Challenges are not free either; we priced out what happens when challenge-based defenses meet real volume, and per-request billing under automated traffic gets ugly fast.

So instrument it. Four numbers, reviewed weekly, in the same meeting as the rest of your bot management tuning:

• Verified coverage. The share of automated traffic with proven identity. It should climb monthly as more operators sign requests; a plateau means your edge is ignoring proof that agents are already offering.
• False positives on verified agents. Every one is a bug with a dollar value attached.
• Spoof catch rate. Requests that claimed a trusted name and failed. Your evidence the ladder earns its keep, and a free threat feed.
• Origin relief. Well-tuned deployments in our experience cut origin load 15 to 30% by keeping unwanted automation out. That saving usually funds the whole effort.

Ruslan made the case on this blog that bot management is a tuning practice, not a checkbox. Agent verification inherits that completely. Signing agents launch monthly, key directories move, and the agent that was read-only in July starts transacting in October. The review cadence is the control.

Your first 30 days

Almost everything below runs in observe mode first, which is why this does not break production.

Week 1: inventory. Segment automated traffic by claimed identity, then test every claim against published ranges, reverse DNS, and signatures. Log, don’t act. This week produces the number that reorders priorities: how much of your “trusted bot” traffic cannot prove it is who it says. In our engagements this year, that number has never been zero, and it is usually large enough to end the meeting early.

Week 2: verification. Turn on signature validation at the edge, forward-confirm rDNS for legacy crawlers, and tag every automated request with its rung. Still observing. Argue about edge cases now, while they are free.

Week 3: policy. Write the ladder into rules for money endpoints first: checkout, login, booking, account creation, pricing APIs. The homepage can wait. This is also the week to decide your position on training crawlers deliberately, instead of inheriting whatever your robots.txt said in 2024.

Week 4: enforcement. Move from observe to action, stand up the four weekly numbers, put the review on the calendar. From here on, the work looks like tuning, which is exactly how it should look.

The old model asked traffic a question it could answer with a lie, and for years the lie rate was low enough that nobody noticed. That era is over. The truth-telling rate on trusted bot names is now measurably below 100% and falling, while the tooling that demands proof instead of promises shipped within the last eighteen months.

Which means the current moment is a gap. The impostors are already at work, the verification standards are already live, and most security teams have not connected the two. The teams that close that gap first get something rare in security: a control that blocks fraud and recovers revenue at the same time.

Your allow-list is either a list of proofs or a list of costumes. As of this year, that is a choice.