At a glance: Web Bot Auth is a proposed method for AI agents to cryptographically sign their HTTP requests, so a website can verify who is calling instead of trusting a user-agent string. As at 12 September 2026 the IETF’s webbotauth working group has adopted the protocol as a working-group Internet-Draft — a working document, not an RFC — while Cloudflare already ships an implementation. For most site operators it is useful today only through a CDN that verifies signatures on your behalf.
The problem: a user-agent string is just text
Anyone can send ChatGPT-User/1.0 in a header. There is no check, no key, nothing to forge. That mattered little when the worst outcome was a skewed analytics report. It matters now, because operators make real access decisions on the strength of that string: allow this crawler, block that one, serve the paywall or don’t.
How often is that badge worn dishonestly? Nobody publishes a clean global figure, and any site owner quoting their own logs is quoting an unauditable sample — so treat single numbers with suspicion. What is documented is that it happens at scale. On 4 August 2025 Cloudflare published an investigation finding that Perplexity “uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked”, using “multiple IPs not listed in Perplexity’s official IP range” and rotating through them, plus requests from different ASNs, “across tens of thousands of domains and millions of requests per day” (Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives).
Two cautions before you go auditing your own access log. Published IP lists lag reality: vendors add and retire addresses, and the JSON files — OpenAI’s gptbot.json, searchbot.json and chatgpt-user.json, for instance — are snapshots you have to refetch. And “unverified” is not “proven fake”: traffic legitimately traverses proxies and intermediaries, and a miss against a stale list proves only that the list missed it. A failed check is a reason to look harder, not a verdict.
What Web Bot Auth actually is
Web Bot Auth is a scheme in which an automated client signs its outbound HTTP requests with a private key, and the receiving server verifies that signature against a public key the client publishes. It is built on RFC 9421, HTTP Message Signatures, published February 2024 as a Standards Track RFC, which describes “a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message”. RFC 9421 is the settled foundation. Web Bot Auth is the bot-specific profile layered on top, and that part is not settled.
The current protocol document is draft-ietf-webbotauth-httpsig-protocol-00, dated 1 September 2026 and due to expire 5 March 2027, authored by Thibault Meunier of Cloudflare and Sandor Major of Google. It replaced the individual submission draft-meunier-webbotauth-httpsig-protocol-02, which the IETF datatracker now marks Replaced (checked 12 September 2026). Mechanically it defines:
- A
Signature-Agentrequest header — a dictionary structured header whose values are HTTPS URIs pointing at where the agent’s key material lives. - A key directory served at the well-known path
/.well-known/http-message-signatures-directory, with the media typeapplication/http-message-signatures-directory+json. - Required signature parameters:
createdandexpiresfor time bounds,keyidcarrying a base64url JWK SHA-256 thumbprint, andtagset to"web-bot-auth".
The draft is admirably blunt about its own limits. It states that the protocol “does not authenticate human users, does not provide anonymous authentication, and does not define authorization or delegation”. A valid signature proves that a holder of a key published at a given URL signed the message. It does not prove the agent is well-behaved, that it respects your robots.txt, or that it is entitled to your content. Identity, not permission.
Be honest about the standards status
This is where most write-ups overstate things, so let us be precise about what the IETF datatracker shows as at 12 September 2026.
An IETF working group called webbotauth does exist and is active, in the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef. Its charter covers cryptographically authenticating automated clients including search crawlers, archivers, AI training crawlers and agents fetching content for a user.
The working group has now adopted that protocol draft — and an adopted draft is still not an RFC. On 1 September 2026 the individual submission was replaced by the working-group document draft-ietf-webbotauth-httpsig-protocol-00. The datatracker gives its document type as “Active Internet-Draft (webbotauth WG)”, records “Replaces draft-meunier-webbotauth-httpsig-protocol”, places it on the IETF stream, and gives its “Intended RFC status” as “(None)” — checked 12 September 2026. Adoption means the working group has taken the document on as its own and will develop it; it confers no standards status. The draft’s own boilerplate states what that is worth: “Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ‘work in progress.'” This one expires on 5 March 2027, and its header still reads “Intended status: Standards Track” — an aspiration, not an achievement.
The rest of the work is still individual. The working group’s document page carries exactly one draft-ietf-webbotauth-* document — the protocol above — alongside unadopted individual submissions including draft-illyes-webbotauth-cbcp, draft-illyes-webbotauth-jafar, draft-nottingham-webbotauth-use-cases, draft-meunier-webbotauth-registry and draft-popov-webbotauth-semantic-anchor.
Signs of churn are visible in the detail: Cloudflare’s 1 July 2025 announcement told operators to host their key set “under /.well-known/http-message-signature-directory of your website” — singular — whilst the current draft registers the well-known suffix http-message-signatures-directory, plural. Anyone calling Web Bot Auth “the new standard” is describing where it is heading, not where it is.
How signature verification differs from rDNS and IP ranges
We covered the incumbent techniques in Verifying AI Bot Traffic: rDNS and Published IP Ranges. Both bind a claim to network infrastructure: reverse DNS asks whether the IP resolves to a vendor hostname that resolves back to the same IP, and IP ranges ask whether the address is on the vendor’s list. Both work, and both are awkward — rDNS costs a DNS round trip in the request path, and IP lists must be refetched as they change.
A signature binds the claim to a key instead of an address. That decouples identity from hosting, which matters enormously for user-directed agents that may run from a laptop, a residential connection or an ephemeral cloud sandbox — traffic that can never be captured by a fixed IP list.
| Method | Spoofable? | Verification cost | Coverage of AI traffic | Available today? |
|---|---|---|---|---|
| User-agent string | Trivially — it is free-text | Nil | Everything self-identifies (honestly or not) | Yes, and near worthless alone |
| Reverse DNS (forward-confirmed) | No, if done forward-confirmed | Two DNS lookups per check; cacheable | Only vendors that set PTR records | Yes, widely |
| Published IP ranges | No, but lists go stale | Fetch and refresh JSON; fast lookups | Only fixed-infrastructure crawlers | Yes — OpenAI, Google and others publish |
| Cryptographic signature (Web Bot Auth) | Not without the private key | Key-directory fetch (cacheable) plus per-request signature check | Only agents that have implemented it | Partly — via Cloudflare; the protocol is an adopted working-group Internet-Draft, not an RFC (re-checked 16 September 2026) |
Note the honest trade in the last two columns. Signatures are the strongest guarantee and the thinnest coverage. IP ranges are weaker and far broader. As at August 2026, OpenAI’s own bot documentation still points operators at IP ranges — “we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below” — with no mention of message signatures. Meanwhile the IP-range approach is itself being standardised in the same working group, via Gary Illyes’ draft-illyes-webbotauth-jafar-00, “A JSON-Based Format for Publishing IP Ranges of Automated HTTP Clients”, dated 21 April 2026. These are complementary, not rivals.
What a site operator can do today
If you sit behind Cloudflare, most of this is already done for you. Cloudflare added HTTP Message Signatures to its Verified Bots Program in a 1 July 2025 announcement, and its verified bots documentation now lists two acceptable routes — “Web Bot Auth” or “IP validation” — with honest self-identification demonstrated by “a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS”. On 28 August 2025 Cloudflare introduced a signed agents category for user-directed agents, launching with agents including ChatGPT agent, Goose from Block, Browserbase and Anchor Browser. Its documentation states: “As of July 1, 2026, the distinction between a Verified bot and a signed agent is expressed by a new metadata field tracked in BotBase: Direct versus Intermediary access, which tracks who can operate the bot”. Practically: you write rules against the category, not against strings.
If you don’t, do not rebuild this yourself yet. Three sensible steps:
- Log the
Signature-Agentheader. It costs nothing and tells you when signed traffic starts arriving. Most sites are not capturing it, so they have no baseline. - Keep doing forward-confirmed rDNS and IP-range checks. They cover the overwhelming majority of AI crawler traffic today and will for some time.
- Do not block on signature absence. Almost no agent signs yet. Treat a valid signature as a positive signal that earns trust, never treat its absence as evidence of bad faith.
What it means for legitimate AI agents
The interesting half is the other direction. As buying research shifts to agents that browse on a person’s behalf — a shift we wrote about in When the Buyer Is an Agent — those agents need a way to say “I am genuinely acting for a real user” that a defensive site will believe. Without it they get lumped in with scrapers and served a challenge page, and the buyer never reaches your form. Signed identity is the plausible way out, and the draft’s scope note is the right division of labour: the signature establishes who, and the site decides what they may do. Signing is necessary but not sufficient — an agent still has to behave.
A note on where we sit. Zian AI builds autonomous sales agents that work over phone, SMS, email and WhatsApp, so our version of this problem is the telephony equivalent — disclosing clearly that a caller is an AI agent, and getting the brand onto the caller-ID screen. We are not a bot-verification vendor and Web Bot Auth is not something we sell. We track it because the same principle recurs: agents that prove who they are get through, and agents relying on unverifiable self-description eventually get blocked.
Sources & ownership
| Claim in this post | Owner | Owner URL (fetched 26 Aug 2026; IETF rows re-checked 12 Sep 2026) |
|---|---|---|
| Working-group protocol draft: name, revision 00 dated 1 September 2026, expiry 5 March 2027, authors, “Active Internet-Draft (webbotauth WG)” document type, “Replaces draft-meunier-webbotauth-httpsig-protocol”, “Intended RFC status (None)” | IETF Datatracker | https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/ |
| The superseded individual submission draft-meunier-webbotauth-httpsig-protocol-02 is now in state Replaced | IETF Datatracker | https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/ |
Exactly one draft-ietf-webbotauth-* document on the WG document page; the other listed drafts are unadopted individual submissions |
IETF Datatracker | https://datatracker.ietf.org/wg/webbotauth/documents/ |
| webbotauth WG is active in the Web and Internet Transport area; chairs; charter scope | IETF | https://datatracker.ietf.org/wg/webbotauth/about/ |
| RFC 9421 HTTP Message Signatures, February 2024, Standards Track, and its abstract | IETF / RFC Editor | https://www.rfc-editor.org/rfc/rfc9421.html |
Signature-Agent header, well-known directory path, media type, required signature parameters, the scope limitation quote, and the Internet-Draft six-month boilerplate |
IETF (Meunier, Major) | https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt |
| IP-range publishing draft: title, version 00, dated 21 April 2026, author Gary Illyes | IETF Datatracker | https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/ |
| Verified Bots validation methods (Web Bot Auth or IP validation); honest self-identification wording; 1 July 2026 Direct/Intermediary metadata field | Cloudflare | https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ |
1 July 2025 cryptographic verification announcement and the singular /.well-known/http-message-signature-directory path |
Cloudflare | https://blog.cloudflare.com/verified-bots-with-cryptography/ |
| Signed agents category, 28 August 2025, and launch participants | Cloudflare | https://blog.cloudflare.com/signed-agents/ |
| Documented stealth crawling: undeclared Chrome-on-macOS user agent, IPs outside the published range, ASN rotation, scale | Cloudflare | https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ |
| OpenAI still directs operators to robots.txt plus published IP ranges, with no mention of message signatures | OpenAI | https://developers.openai.com/api/docs/bots |
FAQ
Is Web Bot Auth an approved internet standard?
No — it is an Internet-Draft, not an RFC. As at 12 September 2026 the IETF’s webbotauth working group, active in the Web and Internet Transport area, has adopted the protocol as draft-ietf-webbotauth-httpsig-protocol-00, dated 1 September 2026, which the datatracker shows as an “Active Internet-Draft (webbotauth WG)” with “Intended RFC status” of “(None)”. Adoption is the start of the standards process, not the end of it: the draft states that “Internet-Drafts are draft documents valid for a maximum of six months” and that it is “inappropriate to use Internet-Drafts as reference material or to cite them other than as ‘work in progress.'” Its intended status is Standards Track and it expires on 5 March 2027.
What underlying standard does it use?
It builds on RFC 9421, HTTP Message Signatures, published by the IETF in February 2024 as a Standards Track RFC. RFC 9421 is a finished standard; Web Bot Auth is the bot-specific profile on top of it, and that profile is still in draft.
Does a valid signature mean I should allow the agent?
No — it answers a narrower question. The draft states explicitly that the protocol “does not authenticate human users, does not provide anonymous authentication, and does not define authorization or delegation”. A signature tells you which key signed the request. Whether that party may crawl, scrape or purchase is your policy decision, expressed in robots.txt and your access rules.
Should I block bots that don’t sign their requests?
Not in 2026. Adoption is early and the vast majority of legitimate AI traffic still identifies via IP ranges and reverse DNS. OpenAI’s bot documentation currently directs operators to its published IP ranges and does not mention message signatures at all. Blocking unsigned traffic today would block nearly every crawler you actually want.
How do I check whether my current bot traffic is genuine?
Compare source IPs against the vendor’s published list, or run forward-confirmed reverse DNS. We walk through both, with commands, in Verifying AI Bot Traffic: rDNS and Published IP Ranges. Expect some hits to fail the check, and read those failures carefully: published IP lists go stale, and legitimate traffic can arrive via an intermediary, so “unverified” is not “proven fake”. That impersonation genuinely happens is documented — Cloudflare’s 4 August 2025 investigation found a crawler using a generic Chrome-on-macOS user agent and “multiple IPs not listed in Perplexity’s official IP range” — but the honest inference from one failed lookup is “investigate”, not “spoofer”.
What should a bot operator do to get verified?
Cloudflare’s 1 July 2025 announcement sets out the sequence: generate a signing key, host a JSON Web Key Set containing the public key at the well-known message-signature directory path, sign requests using a Web Bot Auth library, and register the directory URL through Cloudflare’s Verified Bots form.
Where this lands
User-agent guessing is finished as a trust mechanism; the only question is what replaces it. The answer is not yet a standard — it is an active working group, a fast-moving draft, and one large CDN shipping ahead of the specification. The correct response is cheap: log the header, keep your rDNS and IP checks running, and treat signatures as a bonus signal rather than a gate.
If you are building outbound or inbound AI agents and want to talk about how identity, disclosure and trust play out across web, phone and messaging channels, Apply For Partnership. Zian AI is in partnership-application beta.