The 2026 voice-agent benchmark gap

A clean demo answers the question the team chose to ask. A release decision has to answer harder questions: whether the agent follows a domain policy, calls the right tools in the right order, communicates what actually happened, copes with imperfect audio, and leaves the business system in the intended state. Recent research makes more of that behavior measurable. It does not make a public benchmark interchangeable with a production-readiness test.

Three research artifacts are especially useful in 2026. τ-Voice combines grounded tasks, realistic audio, and full-duplex interaction. EVA-Bench combines validated bot-to-bot simulations with separate accuracy and experience measures across voice-agent architectures. VoiceAgentBench probes structured tool use, multilingual generalization, and adversarial safety through spoken queries.

All three are arXiv preprints, not peer-reviewed standards or production audits. τ-Voice is arXiv v1, submitted March 14, 2026. EVA-Bench is arXiv v2, revised May 27, 2026; its record labels it “work in progress.” VoiceAgentBench is arXiv v3, revised February 13, 2026. Their reported results are paper measurements. The test plan later in this article is our engineering interpretation, not a claim made by the authors and not evidence that Voxeval or any other product improves an outcome.

What the newest benchmarks actually measure

τ-Voice extends the τ²-bench customer-service environment into voice. Its 278 tasks span retail, airline, and telecom workflows with domain rules, tool calls, and verifiable database goals. A tick-based orchestrator sends audio in both directions, while configurable personas, noise, channel effects, interruptions, and backchannels vary the interaction. Task completion is assessed from the final environment state and required communications; responsiveness, latency, interruption, and selectivity describe conversational behavior. That joint view matters because a fluid call can end in the wrong state, while a correct state can hide a poor interaction.

EVA-Bench evaluates live bot-to-bot audio conversations over 213 scenarios across three enterprise domains. It validates the simulated user’s behavior and speech before scoring the agent. EVA-A groups task completion, faithfulness to policy and tool results, and audio-level speech fidelity. EVA-X groups conversation progression, spoken conciseness, and turn-taking timing. Diagnostic measures sit outside those composites. The paper also reports repeated-trial statistics: pass@1 for average trial performance, pass@k for whether a scenario succeeds at least once, and pass^k for repeated success. Its 12 evaluated systems include cascade, hybrid, and speech-to-speech architectures, although some metric implementations differ because each architecture exposes different intermediate signals.

VoiceAgentBench, in its current v3 record, contains more than 6,000 synthetic spoken queries across English and six Indic languages: Hindi, Bengali, Marathi, Tamil, Telugu, and Malayalam. It tests fixed single-tool calls, tool retrieval, parallel calls, sequentially dependent calls, multi-turn tool invocation, and refusal of unsafe requests. Expected structured outputs allow separate scoring of tool selection, call structure, and parameter filling. This is valuable diagnostic coverage for speech-to-tool behavior; it is a static-query benchmark rather than an end-to-end simulation of a live business workflow.

These scopes overlap, but they are not a leaderboard with a shared denominator. They use different systems, prompts, tasks, simulators, languages, audio conditions, tools, graders, and definitions of success. Declaring a winner across the papers would collapse those differences.

Scroll diagram horizontally on smaller screens.

What the numbers do and do not prove

The most quotable result needs its boundary in the same sentence: within this benchmark, across 278 tasks, the audio-native agents evaluated by τ-Voice achieved 31–51% pass@1 in its clean condition and 26–38% in its realistic condition. Clean meant American-accented synthesized speech without background noise or interruptions, though both conditions used telephony conversion. Realistic combined diverse synthesized accents, environmental and burst noise, frame drops, muffling, interruptions, backchannels, and other speech events. The range compares three named provider systems under the paper’s setup. It is not a production failure rate, a forecast for a different stack, or evidence that every added acoustic factor causes the same change.

τ-Voice also states important limits. It evaluates English only using synthetic TTS personas rather than recorded callers. The simulator has perfect memory and instantaneous tool calls. Its user simulator receives the provider transcript rather than transcribing the agent audio. The authors report that annotators found the agent speech intelligible in all 91 sampled simulations; even so, bypassing agent-side ASR still simplifies the simulated loop. The paper does not measure partial task credit, user satisfaction, agent tone or naturalness, tool-call latency, non-English behavior, or human-user dynamics. The reproducible environment helps isolate factors; it does not reproduce every delay, behavior, or population in a deployed service.

EVA-Bench broadens measurement, but its boundaries remain material. All scenarios are English. Tools use a declarative mock executor rather than live APIs, so the study does not cover live-service failure, latency variance, partial responses, or schema drift. Its reported perturbation experiment uses a randomly sampled 90-scenario subset, 30 per domain, to test French-accented speech, coffee-shop noise, and their combination, not all 213 scenarios. That is one French accent and one coffee-shop noise condition rather than the diversity of either category. The user simulator does not systematically interrupt, and some metrics rely on language- or audio-language-model judges. The authors also caution that 12 systems are insufficient for broad claims about architecture classes and that bot-to-bot results may not transfer to real callers.

VoiceAgentBench contributes multilingual and tool-sequence coverage, but its paper says it does not include background noise, omits multi-turn dialogues for some Indic languages, excludes closed-source assistants for cost reasons, and does not study real-time tool invocation with an interactive user. Its queries are synthetically generated. A multilingual tool-call score tests the paper’s corpus and output contract, not the complete experience of a caller using a live service.

Taken together, the preprints show that transcript correctness or a generic conversation rating leaves consequential behavior unobserved. They do not establish a universal readiness threshold. Architecture differences also matter: a cascade can expose transcripts and component timing that an audio-native system may not, while a speech-to-speech system may preserve paralinguistic information that a text intermediary discards. Compare systems only on observable, matched outcomes, and document where measurement methods differ.

Turn benchmark dimensions into domain evals

Engineering inference: use the papers as a coverage map, then replace their generic domains, policies, tools, voices, and thresholds with your own. Start with a high-risk workflow, such as changing a booking or authenticating an account, and define the authoritative state before generating audio variations. Before retaining recordings or call-derived data, have the responsible privacy and security teams define lawful collection and consent requirements for the relevant jurisdiction and context. Responsible teams should minimize or redact sensitive data, restrict access, and apply a documented retention and deletion policy. Each case should preserve only the approved recording or excerpt, timeline, tool trace, policy assertions, scorer version, and final state needed to reproduce and review a failure.

Domain policy

Company eval. Exercise allowed, forbidden, and ambiguous requests against the exact policy version. Evidence to retain. Policy clause, decision trace, response, and tool calls. Example release rule. No prohibited action or skipped mandatory step.

Tool sequence

Company eval. Test independent, parallel, and output-dependent calls, including retries and stale intents. Evidence to retain. Ordered requests and results, arguments, IDs, and cancellation events. Example release rule. Dependencies are respected, and mutations are idempotent.

Grounded task completion

Company eval. Compare the requested outcome and permitted alternatives with authoritative end state. Evidence to retain. Initial state, expected state, final state, and spoken claims. Example release rule. Required state and user-facing communication both match.

Accent and noise

Company eval. Stratify consented or representative speech by accent, rate, device, codec, noise, and packet loss. Evidence to retain. Source provenance, transformation settings, and per-stratum results. Example release rule. Every approved stratum meets its lane-specific floor.

Interruptions

Company eval. Interrupt before confirmation, during disclosure, and around tool commitment. Evidence to retain. Mixed audio, VAD events, playback stop, and tool status. Example release rule. Obsolete speech stops, and abandoned intent causes no side effect.

Latency

Company eval. Measure connection, first audio, interruption response, tool completion, and total task time separately. Evidence to retain. Monotonic timestamps and percentile distributions by condition. Example release rule. Each critical path stays within its approved budget.

Disclosure

Company eval. Place required language amid questions, overlap, and retries. Evidence to retain. Audible recording, exact policy text, timing, and acknowledgement. Example release rule. Required disclosure is complete and not falsely confirmed.

Handoff

Company eval. Trigger transfer by request, confidence, policy, and system failure. Evidence to retain. Trigger reason, context packet, queue result, and caller experience. Example release rule. Transfer occurs once with the required context and no invented status.

Downstream state

Company eval. Reconcile what the agent said with records, notifications, charges, and tickets. Evidence to retain. Audit log, before-and-after records, and side-effect IDs. Example release rule. There is no unauthorized, duplicate, missing, or contradicted mutation.

Do not collapse these dimensions into one average. A critical unauthorized charge cannot be offset by nine pleasant conversations. Report results by workflow, severity, and condition; retain the denominator and number of independent trials. Repetition is particularly useful for stochastic agents: distinguish “succeeded once” from “succeeded on every required run,” borrowing EVA-Bench’s reliability question without importing its thresholds.

A release-readiness scorecard

Use one scorecard per workflow and release candidate. Fill it from replayable evidence, not a launch meeting’s memory.

• Candidate. Model, prompt, voice, transport, tool, policy, and scorer versions.
• Scope. Workflow, risk tier, languages, devices, networks, and explicitly untested populations.
• Corpus. Case IDs, provenance, transformation settings, attempts per case, and known coverage gaps.
• Must-pass gates. Policy, disclosure, authorization, handoff, and downstream-state assertions.
• Performance budgets. Latency and interaction budgets by condition, including tail percentiles.
• Results. Numerator and denominator by gate, severity, and stratum, plus repeated-run consistency.
• Failures. Linked recording, timeline, trace, final state, owner, and regression ID.
• Decision. Proceed, qualify, or block, with approver, date, rationale, and residual risk.

Block when any critical must-pass gate fails, evidence needed to assess a critical gate is missing, or the candidate creates an unauthorized or falsely reported side effect. Qualify only when critical gates pass and a bounded non-critical gap has an explicit exposure limit, owner, mitigation, and retest date. Proceed when the scoped gates and budgets pass at their required repetition level and reviewers accept the documented residual risk. These are governance defaults, not source-derived universal thresholds; regulated or high-consequence workflows may need stricter rules and independent review.

Before signing, ask four questions: Would the decision change if results were split by accent, device, or workflow? Can every failure be replayed from retained evidence? Does the spoken claim match the downstream state? Is an untested condition being described as a pass? A “no” to the first three, or “yes” to the fourth, means the scorecard is incomplete.

Voxeval is intended to help teams organize domain cases, aligned voice and tool evidence, regressions, and release decisions. The benchmark interpretation, eval matrix, and scorecard above do not depend on Voxeval; teams can implement them in an internal harness or other tooling. The essential move is to convert public research dimensions into evidence for the workflow actually being released.