Quick answer: As at 10 October 2026, no independent, reproducible benchmark tests Sierra, Decagon and Fin on the same tickets. Sierra’s τ-bench scores AI models, not Sierra’s product; none of its 67 leaderboard entries is any of the three. Fin’s 76% is Intercom’s own customer average. Decagon’s DuetBench tests its agent-building tool. Run your own same-ticket test.
People searching “Sierra vs Decagon vs Fin benchmark” usually want a league table. No one has published one, and the numbers that do exist measure different things on different data. This page sorts them out. It covers Sierra’s τ-bench family (τ-bench, τ²-bench, τ³-bench and τ-voice, with code on GitHub and papers on arXiv) and Intercom’s Fin figures (fin.ai’s 76% average, its top-10 benchmark tool and its $0.99-per-outcome pricing). It also covers Decagon’s DuetBench and DuetBench-2 and the customer-story figures on all three vendors’ sites. For each, we ask who measured it, on what data, and whether you could check it yourself. Then we lay out a test you can run on your own tickets, with the sample-size arithmetic that tells you when a gap between two vendors is real. Every vendor figure comes from that vendor’s own page, read on 10 October 2026. We name none of their customers.
Disclosure: Zian AI, which publishes this page, builds autonomous AI agents. Its digital team includes a 24/7 Customer Support Agent alongside its sales agents, so it competes with the products discussed here. It is not a helpdesk platform and is not scored here.
Is there a Sierra vs Decagon vs Fin benchmark?
No. As at 10 October 2026 we found no published test that runs all three products on one shared ticket set with a published method. We did not stop at the first empty search. This is what we checked, and the positive control we ran on each source before recording an absence:
- The τ-bench leaderboard. taubench.com loads its results from a public manifest. We downloaded it and opened every one of the 67 submission files (29 current text, 22 voice, 16 legacy). The model organisations listed are AI labs and agent builders: OpenAI, Anthropic, Google, xAI, Qwen and others. Not one entry is Sierra’s production agent, Intercom’s Fin or Decagon. Sierra itself submitted 51 of the 67, all of them other companies’ models. Control: known entries such as Claude Opus 4.5 parsed correctly.
- Intercom’s comparison pages. fin.ai publishes several “Fin vs” guides. They quote Intercom’s own figures and describe rivals, but none of them gives a shared ticket set, a sample size or a scoring rule. Control: the 76% figure appears on each page.
- Decagon’s site. We went through all 573 URLs listed in its sitemap. Its benchmark and evaluation pages (DuetBench, DuetBench-2, Simulations, its evaluation-engine post) test Decagon’s own tooling and do not compare it with other vendors. Neither DuetBench post links a public task set, dataset or code repository. Control: each post’s other in-body links, to Decagon’s own product and blog pages, parsed correctly.
- Sierra’s site. We went through all 393 URLs listed in its US-English sitemap. Its benchmark posts cover the τ family and, judging by their titles, multilingual transcription and agents that build agents. None tests a commercial support product.
- Third parties. The comparison pages that rank for this query come from vendors in the same market. A QA platform’s Sierra-vs-Decagon comparison leaves Fin out and relies on its own “simulation data from comparable deployments”. A rival support agent’s Fin-vs-Decagon page reports that rival’s own 500-ticket benchmark, not a test of Fin or Decagon. One published compilation, Aissist’s 2026 benchmark, says it outright: “We are not aware of any independent audit or multi-vendor study measuring these products’ real-world resolution rates.”
Claims on the vendors’ own pages point the other way, and we state them. Intercom writes that “in independent head-to-head testing conducted by customers, Fin achieved a 73% resolution rate” (fin.ai). It lists lower figures for two named rivals, which we do not repeat because no method is published. The page does not name the customers, the ticket count or the definition of resolution, so nobody outside can reproduce it. The same page says that “in independent testing, Fin handles 2x more complex queries than competitors”, also with no published method. Sierra writes that leaderboard submissions with trajectories are “verified, meaning they have undergone independent validation that confirms the reported results” (Sierra). That is true of the entries, but the entries are models, not these products.
The quotable version: as at 10 October 2026 there is no independent, reproducible three-way benchmark of Sierra, Decagon and Fin; there are three vendors measuring three different things.
What each published number measures: the evidence-quality table
One rule applies to every row. A number counts as reproducible only if its task set and scoring code are public, so an outsider could re-run it. It counts as independent only if someone other than the vendor being described produced it.
| Published number | Who measured it | On what data | Reproducible? Independent? | What it tells an enterprise buyer | What it does not tell you |
|---|---|---|---|---|---|
| τ-bench (June 2024): GPT-4o agents succeed on under 50% of tasks; pass^8 under 25% in retail | Sierra’s research team (arXiv 2406.12045) | Synthetic retail and airline databases, policy documents, LLM-simulated customers | Reproducible: yes, MIT-licensed code. Independent of the three products: it measures models | Models that look fine in one run fail when the same task is repeated | Anything about Sierra’s, Fin’s or Decagon’s production agents |
| τ²-bench (June 2025): success drops by up to 25 points when the user must act too | Sierra’s research team (arXiv 2506.07982) | Telecom troubleshooting tasks with dual control | Reproducible: yes. Measures models | Guiding a customer through steps is harder than acting for them | How any vendor’s agent handles your troubleshooting flows |
| taubench.com leaderboard: top τ²-bench text pass^1 of 87.9%; top τ³-banking pass^1 of 55.2% | Run by Sierra; 51 of 67 entries submitted by Sierra | Standard τ task sets; trajectories published | Reproducible: yes. Ranks models and a few agent builds, none of the three products | Which underlying model is strongest at policy-bound tool use | Which vendor to buy |
| τ-voice: voice frontier rose from 30% (Aug 2025) to 67% (Apr 2026), per Sierra’s May 2026 post; the live leaderboard’s top voice pass^1 is now 81.7% | Sierra | 278 tasks, English only, accents generated by text-to-speech | Reproducible: yes. Sierra calls absolute numbers “indicative rather than definitive” | Voice agents still lose task accuracy compared with text | How any vendor’s phone agent performs on your callers |
| Fin: 76% average resolution rate across 12,000+ customers | Intercom | Its own customer deployments | Not reproducible from outside; self-reported. Intercom calls it “independently verifiable” | Fin’s fleet average under Intercom’s definition | Your expected rate; how much of the average comes from easy queries |
| Fin benchmark tool: top-10 per industry average 85% resolution, 78% automation, 91% involvement | Intercom | 110M+ conversations, last three months, top 10 per industry | Not reproducible; self-reported; best performers only | A ceiling that some Fin customers reach | The median, or the bottom of the distribution |
| Fin: 73% in “independent head-to-head testing conducted by customers” | Intercom, citing unnamed customers | Not published | Not reproducible; independence cannot be checked | That Intercom has run comparisons | The ticket set, sample size or scoring rule |
| Fin Apex 2.0: +8.6% confirmed resolutions vs Apex 1.0; Apex for Procedures +15% vs Sonnet 5 | Intercom | Method not stated on fin.ai | Not reproducible; self-reported | Intercom is improving its own models | How Fin compares with Sierra or Decagon |
| DuetBench (9 June 2026): Duet Autopilot passed 93% of 90 diagnostic tasks vs an 83% human average; 45.5% of 44 build tasks vs a 23% human pass rate within 3 hours | Decagon | Simulated customer deployments; humans were recently certified Decagon employees | No public task set found; self-reported | How well Decagon’s tool builds and repairs agents | Customer-facing resolution rates |
| DuetBench-2 (9 Sept 2026): 81.9% pass rate; one Fortune 50 customer’s resolution rose from 30% to 55% | Decagon | Harder build tasks plus production data from unnamed customers | No public task set found; self-reported | That changes made by Autopilot can lift a live agent’s numbers | A cross-customer resolution average |
| Decagon home page stories: 70% chat and voice resolution; 80% deflection rate | Decagon, per named customer | One customer each | Not reproducible; self-reported | What a strong deployment can reach | Typical results; Decagon’s own glossary gives 25–45% deflection for financial services and healthcare |
| Sierra customer stories: 80% resolution rate; 70%+ resolution rate | Sierra, per named customer | One customer each | Not reproducible; self-reported | What a strong deployment can reach | A fleet average, which we did not find on sierra.ai |
Read down the “Reproducible? Independent?” column and you can see the problem. Every number you could re-run measures models. Every number about the actual products comes from the vendor that sells them.
What τ-bench measures, and why it is not a Sierra product score
τ-bench is Sierra’s most important contribution here, and it is easy to misread. In the original paper, an AI agent gets a policy document and API tools. A second language model plays the customer. The run is scored on whether the final database state matches the expected one, not on how polished the conversation sounded. The paper also introduced pass^k: the chance that an agent succeeds on every one of k repeat attempts at the same task. Sierra’s launch post explains why that matters: a GPT-4o agent “drops to ~25% on pass^8 in τ-retail”.
Four things limit what τ-bench can tell you about a vendor.
- It ranks models, not products. Its leaderboard entries are named after the models they test, such as Claude Opus 5, GPT-5.5 and Gemini 3 Pro. Sierra wrote at launch that it plans to use τ-bench to “help compile, tailor, and fine-tune our constellation of models”, but no leaderboard entry is Sierra’s production agent.
- The customers are simulated. An independent study, “Lost in Simulation” (arXiv, January 2026), ran τ-bench retail tasks with real people in the US, India, Kenya and Nigeria. Agent success rates varied “up to 9 percentage points across different user LLMs”. Simulation underestimated performance on hard tasks and overestimated it on moderate ones, and it worked worst as a stand-in for speakers of African American Vernacular English and Indian English.
- Versions are not comparable. The tau2-bench README says that after a July 2026 grading fix, “results produced with tau2-bench < 1.0.1 are not comparable with >= 1.0.1” on the banking domain. Earlier task fixes changed more than 50 airline and retail tasks.
- The tasks are synthetic. Retail, airline, telecom and banking databases built for the benchmark are not your order system, your refund policy or your identity checks.
What τ-bench does give an enterprise buyer is a method. Score the end state, not the transcript. Repeat each task to measure consistency. Publish the transcripts. Those three ideas carry straight into the test plan below. They also explain why a vendor’s single-run resolution rate says little about consistency. An agent that solves a task 90% of the time solves it four times in a row only about 66% of the time (0.94).
How Intercom counts a Fin resolution
Intercom publishes more data about its product’s performance than either rival, and its definitions are on the page. That is a real point in its favour. The detail is what lets you test it. Here is what Intercom’s own pages say, as at 10 October 2026:
- Definition on the benchmark tool: “A resolution is counted when the customer confirms their issue is solved, or doesn’t follow up” (fin.ai/benchmarks). The second half of that sentence is the one to probe. A customer who gives up and phones you instead may still count as a resolution.
- Definition on the learning page: Fin “only counts genuine, positive resolutions”, and excludes conversations where the customer expresses dissatisfaction or “returns with the same problem”. The page does not say how long the return window is.
- Denominators: the 76% average sits beside “12,000+ customers” on the fin.ai home page, “8,000+ customers” on its resolution-rate guide and both “12,000+” and “7,000+ customers” on its Fin-vs-Sierra page. They may be older and newer versions of the same claim; ask which period the 76% covers.
- Billing unit: Fin is priced at $0.99 per outcome with a 50-outcome monthly minimum (fin.ai/pricing). Resolutions, Procedure handoffs, disqualifications and self-serve routings are $0.99 each, and qualifications are $9.99. Intercom’s pricing page lists Fin Voice “via sales at $1.99 per voice outcome” (intercom.com/pricing).
The billing point matters for any benchmark. An outcome is not always a resolution: a correctly completed handoff is billed too. If you compare vendors on cost, divide what you paid by the resolutions you verified, not by the outcomes on the invoice.
What Decagon’s numbers measure
Decagon’s most-cited research numbers come from DuetBench, which it calls “the first benchmark in the customer service domain designed to evaluate agent self-improvement”. It does not measure how often Decagon’s agent resolves customer conversations. It measures whether Duet Autopilot, Decagon’s agent-building assistant, can diagnose problems, write Agent Operating Procedures (AOPs) and validate its own test simulations. In the first round (9 June 2026), the human comparison group was “recently certified Decagon employees”. DuetBench-2 (9 September 2026) adds production evidence. At one Fortune 50 enterprise, Autopilot changes lifted a Decagon agent’s resolution rate from 30% to 55%. Decagon also checked its LLM judge against human labels and reports 99.5% accuracy. That is more production evidence than most build-tool claims offer. Its only quantified production figure is still one unnamed customer. For “a broader cross section of customers” it reports “consistent outcome gains seven days after going live” but gives no number, and there is no public task set.
Decagon’s customer-story figures on its home page (70% chat and voice resolution, 80% deflection) are best read alongside Decagon’s own deflection-rate glossary. The glossary gives industry ranges of 55% to 75% for e-commerce, 40% to 60% for SaaS, and 25% to 45% for financial services and healthcare. It also warns that the common shortcut of counting a contact as deflected “if no human ticket opens within 24 to 72 hours” “can overstate deflection when a customer gives up and contacts an untagged channel later”. That is the same weakness as Fin’s “doesn’t follow up” rule, described by a different vendor. A test you run yourself has to close that gap.
Neither Decagon nor Sierra publishes a price list. decagon.ai/pricing and sierra.ai/pricing both returned 404 on 10 October 2026, and neither vendor’s sitemap lists a pricing page. Sierra’s own post on outcome-based pricing says that if a conversation is unresolved, “in most cases, there’s no charge”. Decagon’s pricing post (December 2024) describes two models without rates: per-conversation, “a fixed rate for every incoming conversation”, and per-resolution, “a higher fixed rate for each fully resolved conversation, with no charge for escalations”. It says most of its customers choose per-conversation. Your cost-per-verified-resolution comparison therefore needs a written quote from each.
Where Zian AI fits. Zian is an autonomous AI sales-agents platform whose digital team includes a 24/7 Customer Support Agent, working across phone, SMS, email and WhatsApp in 30+ languages. It is in partnership-application beta. Its AI customer support agents page sets out where it differs from helpdesk-first suites such as Fin. If you want one agent layer across sales and support, Apply For Partnership.
How to benchmark Sierra, Decagon and Fin yourself: the Same-Ticket Protocol
Since nobody has published a three-way test, the honest substitute is one you run yourself. We call it the Same-Ticket Protocol. Every vendor gets the same real tickets, each run is scored on the end state you can verify, and each vendor is tested more than once. Every vendor offers a simulation harness (Sierra’s Agent OS simulations, Decagon’s Simulations, Fin’s testing suite), but each harness is that vendor’s own. Use the harnesses to build and tune each agent, then score all three in one place you control. Intercom’s own evaluation guide recommends the same core step: “Give every vendor the same knowledge base, the same test conversations, and the same scoring rubric.” The protocol below adds an answer key, repeat runs, harm scoring and a sample-size check. The same principle applies in sales software, which we covered in why there are no trustworthy AI SDR benchmarks.
| Step | What you do | Metric | How to score it |
|---|---|---|---|
| 1. Freeze the set | Sample 300 resolved tickets from the last 90 days, stratified by your top intents, plus 50 that needed a refund, an identity check or an account change | Coverage by intent | Each intent’s share of the sample within 2 points of its share of live volume |
| 2. Write the answer key first | For each ticket, record the correct end state (refund issued, plan changed, escalated) and any policy that must not be broken | Answer key | Two reviewers agree on each key before any vendor runs |
| 3. Same knowledge, same tools | Give every vendor the same help-centre export, policies and sandbox APIs | Parity checklist | No vendor gets data the others lack |
| 4. Run and replay | Replay each ticket as a scripted customer, ideally your own agents playing customers for the 50 action tickets | Verified resolution rate | End state matches the key AND no policy breach; a vendor’s self-reported “resolved” does not count |
| 5. Repeat for consistency | Re-run the 50 action tickets 4 times each | pass^4 | A ticket passes only if all 4 runs pass |
| 6. Score harm separately | Flag wrong actions (refund to the wrong account, change made without authentication) | Wrong-action rate | Any wrong action on a regulated step fails that ticket outright |
| 7. Check escalations | For tickets the key says need a human, check the hand-off | Escalation precision | Escalated when it should, with the context a human needs |
| 8. Price it | Apply each vendor’s quoted billing unit to the run | Cost per verified resolution | Billed amount ÷ step-4 verified resolutions |
Worked cost example (hypothetical inputs). Suppose a month’s Fin invoice shows 6,000 billed outcomes at $0.99: 6,000 × $0.99 = $5,940. Your blind review verifies 5,100 of those conversations as true resolutions. Cost per verified resolution is $5,940 ÷ 5,100 = $1.165, or about $1.16, not $0.99. Do the same with Sierra’s and Decagon’s quoted units. Only then are the three prices comparable.
How many tickets do you need before a gap is real?
Most pilots are too small to separate vendors. Here are the 95% margins for a resolution rate near 70%, using the standard normal approximation for a proportion (1.96 × √(p(1−p)/n)). The gap column assumes each vendor runs its own sample of n tickets.
| Tickets per vendor (n) | Margin on one vendor’s rate | Margin on the gap between two vendors | What you can conclude |
|---|---|---|---|
| 100 | ±9.0 points | ±12.7 points | Only a gap of 13+ points means anything |
| 200 | ±6.4 points | ±9.0 points | Separates a weak vendor from a strong one |
| 300 | ±5.2 points | ±7.3 points | A 7-point gap is about the smallest you can trust |
| 500 | ±4.0 points | ±5.7 points | Good enough for a procurement decision |
| 1,000 | ±2.8 points | ±4.0 points | Needed only if vendors are within a few points |
Running the same tickets through every vendor narrows the gap margins somewhat, because ticket difficulty is shared, so treat the gap column as a conservative ceiling. The rule of thumb: at 300 tickets each, a gap of under 7 points between Sierra, Decagon and Fin is noise.
Which should an enterprise choose, and who is each wrong for?
Benchmarks will not choose for you, so choose on the evidence each vendor publishes and then test. Based only on the vendors’ own pages:
- Fin suits you if you want to calculate a price before talking to sales, because it is the only one of the three with a public per-unit price, and if you want a definition you can test. It is a poor fit if your measure of success is something other than the outcomes Intercom bills, because you will be reconciling two counts every month.
- Sierra suits you if you care how the research is done: its team publishes open, reproducible benchmarks and the pass^k reliability metric. It is a poor fit if you need a published fleet-wide resolution rate or a public price list, as we found neither on sierra.ai. For regulated-industry controls, see our Sierra vs Lorikeet regulated-industries scoring.
- Decagon suits you if most of your cost is the work of building and maintaining an agent, since DuetBench is aimed squarely at that. It is a poor fit if you need a published cross-customer resolution average before shortlisting. For card-data and residency questions, see Lorikeet vs Decagon for fintech support. For HIPAA and business associate agreements, see the healthtech alternatives to Decagon and Sierra.
The quotable version: choose a support agent on the evidence it publishes, then on a test of 300 or more of your own tickets each; a vendor’s headline rate is a ceiling, not a forecast.
Zian AI is not a helpdesk platform. Its agents book and qualify on the sales side and handle tier-1 support questions, across phone, SMS, email and WhatsApp. If one agent layer across both is your gap, Apply For Partnership.
Frequently asked questions
Is there an independent benchmark comparing Sierra, Decagon and Fin?
No, as at 10 October 2026. None of the 67 entries on the τ-bench leaderboard is any of the three products, and Intercom’s head-to-head claim publishes no ticket set or method. The only fair comparison is a test on your own tickets.
Does τ-bench measure Sierra’s own product?
No. τ-bench is Sierra’s open benchmark, but its leaderboard ranks AI models such as Claude, GPT and Gemini on synthetic retail, airline, telecom and banking tasks. None of its entries is Sierra’s production agent.
Is Fin’s 76% resolution rate independently verified?
Intercom calls it “independently verifiable”, but it is Intercom’s own average across its customers. On its benchmark tool, a resolution counts when the customer confirms the issue is solved “or doesn’t follow up”. Ask which definition and period apply to you.
What does Fin cost per resolution?
As at 10 October 2026, Intercom charges $0.99 per outcome with a 50-outcome monthly minimum, according to fin.ai/pricing. Outcomes include resolutions and Procedure handoffs; qualifications cost $9.99, and Fin Voice is $1.99 per voice outcome via sales.
How many tickets do I need to compare AI support agents fairly?
About 300 per vendor. At a 70% resolution rate that gives roughly a 5-point margin on each vendor and 7 points on the gap between two, so smaller gaps are noise. Use 500 if the vendors are close.
Can Sierra, Fin or Decagon submit to the τ-bench leaderboard?
Yes. The submission guide accepts custom agent systems as well as models, and Sierra invites voice platforms to send in adapters. As at 10 October 2026, none of the three vendors has a listed entry.
Where every figure on this page comes from
| Figure | Who published it | Link | Date read |
|---|---|---|---|
| τ-bench: under 50% task success; pass^8 under 25% in retail | Sierra research (Yao et al.) | arXiv 2406.12045 | 10 Oct 2026 |
| GPT-4o “drops to ~25% on pass^8 in τ-retail” | Sierra | sierra.ai blog | 10 Oct 2026 |
| τ²-bench: drop of up to 25 points, solo to interactive | Sierra | sierra.ai blog; arXiv 2506.07982 | 10 Oct 2026 |
| 67 leaderboard submissions; 51 submitted by Sierra; none from the three products; top pass^1 87.9% (τ²), 55.2% (τ³-banking) and 81.7% (τ³-voice) | Sierra (taubench.com manifest and submission files) | taubench.com | 10 Oct 2026 |
| Verified submissions “have undergone independent validation” | Sierra | sierra.ai blog | 10 Oct 2026 |
| τ-voice: 278 tasks; 30% to 67%; “indicative rather than definitive” | Sierra | sierra.ai blog | 10 Oct 2026 |
| Results before 1.0.1 not comparable; 50+ tasks fixed | Sierra (tau2-bench maintainers) | GitHub tau2-bench; taubench.com | 10 Oct 2026 |
| Simulated users vary success “up to 9 percentage points” | “Lost in Simulation” authors | arXiv 2601.17087 | 10 Oct 2026 |
| Fin 76% across 12,000+ customers; Apex +8.6% and +15% | Intercom | fin.ai | 10 Oct 2026 |
| Top-10 85% resolution, 78% automation, 91% involvement; 110M+ conversations; resolution definition | Intercom | fin.ai/benchmarks | 10 Oct 2026 |
| 76% across 8,000+ customers; 73% head-to-head; “2x more complex queries”; “genuine, positive resolutions” | Intercom | fin.ai/learn/ai-resolution-rate | 10 Oct 2026 |
| 76% across 7,000+ customers; “independently verifiable” | Intercom | fin.ai/learn/fin-vs-sierra | 10 Oct 2026 |
| $0.99 per outcome; 50-outcome minimum; $9.99 qualifications | Intercom | fin.ai/pricing | 10 Oct 2026 |
| Fin Voice $1.99 per voice outcome | Intercom | intercom.com/pricing | 10 Oct 2026 |
| DuetBench: 93% vs 83%; 45.5% vs 23%; 90 and 44 tasks | Decagon | decagon.ai/blog/duetbench | 10 Oct 2026 |
| DuetBench-2: 81.9%; 30% to 55%; 99.5% judge accuracy | Decagon | decagon.ai/blog/duetbench-2 | 10 Oct 2026 |
| 70% chat and voice resolution; 80% deflection rate (customer stories) | Decagon | decagon.ai | 10 Oct 2026 |
| Deflection ranges 55–75%, 40–60%, 25–45%; 24-to-72-hour proxy | Decagon | decagon.ai glossary | 10 Oct 2026 |
| 80% and 70%+ resolution rates (customer stories) | Sierra | sierra.ai/customers | 10 Oct 2026 |
| “In most cases, there’s no charge” for unresolved conversations | Sierra | sierra.ai blog | 10 Oct 2026 |
| Per-conversation and per-resolution pricing models; no charge for escalations under per-resolution | Decagon | decagon.ai blog | 10 Oct 2026 |
| 500-ticket benchmark of a rival support agent | Fini | usefini.com | 10 Oct 2026 |
| “Not aware of any independent audit or multi-vendor study” | Aissist | aissist.io | 10 Oct 2026 |
| Sample-size margins; $1.16 worked example; 0.94 ≈ 66% | Zian AI calculation (method shown on this page) | This page | 10 Oct 2026 |