How to evaluate a voice agent: 10 metrics beyond word error rate

Your team has a voice agent that sounds polished. The transcript is clean, the first response arrives quickly, and the demo ends with a cheerful confirmation. Now you need to decide whether that agent is ready to speak with real customers and touch real business systems.

Word error rate cannot make that decision for you.

A useful voice-agent evaluation has to measure the whole session: what the caller wanted, what the agent heard, what it said, which tools it used, which policies applied, how the conversation unfolded, and what changed in the system of record. This guide gives engineering and product teams ten practical metrics for doing that. By the end, you will have a scorecard that can separate a slightly awkward call from a genuinely unsafe release.

The short answer

The best way to evaluate a voice agent is to score business outcome and conversation experience separately, then join them in one release decision. Measure task success, critical entities, tool execution, policy compliance, final system state, latency, turn-taking, audio quality, caller effort, and handoff completion.

Do not compress them into one average too early. A friendly voice should not hide an unauthorized refund. A perfect tool call should not excuse a required disclosure that never happened. A low word error rate should not pass a call that updated the wrong account.

That is the core opinion behind this Voxeval framework: a voice agent is not a transcription product with a mouth. It is a timed actor inside a business workflow.

Why word error rate misses production failures

Word error rate, usually shortened to WER, compares a transcript with a reference and counts substitutions, deletions, and insertions. It is useful. If a speech recognizer turns “fifteen” into “fifty,” WER can help locate the failure.

The trouble starts when teams treat that diagnostic as a product score.

Imagine a fictional appointment-rescheduling agent. A caller says, “Move my Wednesday appointment to Friday morning.” The speech recognizer captures every word correctly. The agent cancels Wednesday, the booking tool fails before creating Friday, and the agent says, “You’re all set.”

The WER is zero. The customer now has no appointment.

Recent voice-agent research is moving toward broader measurement for the same reason. τ-Voice evaluates grounded task completion and full-duplex interaction across 278 tasks instead of judging transcripts alone. EVA-Bench separates accuracy from experience, combining task, faithfulness, speech, progression, conciseness, and turn-taking measures. VoiceAgentBench looks inside tool use by scoring tool selection, call structure, parameters, multi-step behavior, and safety.

These are research preprints with their own simulators, systems, and limits. They are not universal production standards. Their useful shared lesson is narrower: voice-agent quality has several independent layers, and a transcript covers only some of them.

Scroll diagram horizontally on smaller screens.

The ten metrics that belong in a voice-agent scorecard

1. Grounded task success

Question: Did the caller accomplish the permitted goal?

Task success should be grounded in something stronger than the agent’s own claim. For an appointment change, inspect the scheduling system. For a claim-status call, compare the spoken status with the authoritative claim record. For account recovery, verify that the approved identity and recovery steps completed.

Define success for each workflow before running the agent. “Helpful conversation” is too vague. “Existing appointment moved to the requested available slot, with a confirmation issued to the correct patient” is testable.

Track partial success as a diagnostic, but keep the final release rule clear. An agent that finds an available slot but never books it has made progress. It has not completed the task.

2. Critical-entity accuracy

Question: Did the system capture the exact details that control the workflow?

Not every word carries the same risk. A missed filler word rarely matters. One wrong digit in a policy number can move the agent onto another customer’s record.

Create a critical-entity set for each workflow. It may include names, dates, times, medication names, addresses, amounts, confirmation codes, flight numbers, or account identifiers. Score those entities separately from general transcription.

Preserve the chain from audio to transcript to tool argument. That shows whether the speech recognizer, the reasoning layer, or the tool adapter changed the value. A single end score will tell you that something failed. This chain tells you where.

3. Tool and side-effect correctness

Question: Did the agent call the right tool, with the right arguments, at the right time, exactly as many times as intended?

Tool success has at least four parts:

• The selected function matches the caller’s approved intent.
• Required arguments are complete and correct.
• The call happens only after the workflow has enough information and authorization.
• Retries, interruptions, and timeouts do not create duplicate side effects.

A tool can return 200 OK and still be wrong. The wrong account may have been updated perfectly. A duplicate payment may have been processed without an infrastructure error. Test business idempotency and downstream state, not just API status.

Tool assertions are useful before audio testing too. LiveKit’s current testing guidance separates behavioral checks, tool usage, grounding, error handling, and misuse resistance. Those lower-cost checks help isolate logic failures, while end-to-end audio tests cover the complete path callers will use.

4. Policy and disclosure compliance

Question: Did the agent follow the rules that applied to this specific call?

Policy evaluation is contextual. The same sentence can be required in one lane, optional in another, and prohibited in a third. Build the evaluator from the workflow’s approved policy source, not from a generic idea of what a good call sounds like.

Check prerequisites, ordering, wording where exact language matters, forbidden claims, escalation triggers, and permission boundaries. Also support not applicable. Penalizing an agent for missing a disclosure after the caller hung up before the disclosure point will train the team to distrust the evaluator.

For consequential workflows, a critical policy failure should block release even if every other metric passes. Do not let nine green boxes average away one unsafe action.

5. Verified final system state

Question: Does the business system contain the intended result after the conversation ends?

Task success describes the caller’s goal. Final-state verification checks the durable record left behind. The distinction matters when an agent speaks before a tool finishes, receives a late failure, or runs several dependent actions.

Capture relevant state before and after the session. Compare the actual mutation with the expected one. Record any additional effects such as messages, transfers, holds, refunds, or audit entries.

This metric catches the dangerous gap between “the agent said it happened” and “it happened.” In production voice AI, spoken confidence is not evidence.

6. Latency by stage and condition

Question: Where did the caller wait, and how often did the delay cross the workflow’s budget?

One end-to-end average hides too much. Measure connection setup, end-of-turn detection where relevant, first model output, first audible response, interruption detection, playback stop, tool completion, and recovery after failure.

Report distributions, not just averages. Break them out by device, network, codec, geography, language, workflow, and tool path. A fast median can coexist with a painful tail.

The right target depends on the moment. A brief acknowledgement and a final transaction result do not need the same budget. Set thresholds from the intended interaction and measured user behavior rather than copying one industry number.

7. Turn-taking and interruption recovery

Question: Did the agent know when to listen, speak, yield, wait, and recover?

Turn-taking is more than silence detection. Callers pause to think, speak over background noise, correct themselves mid-sentence, and interrupt an answer that is heading in the wrong direction.

Measure premature responses, clipped user speech, unwanted overlap, interruption-detection time, playback-stop time, obsolete generation, and stale tool actions. Review the audio because a cleaned transcript may make a chaotic exchange look orderly.

Full-duplex systems make this dimension even more important. Our guide to full-duplex evaluation covers backchannels, overlap, barge-in, silence, and tool timing in more detail.

8. Input and output audio quality

Question: Could the system understand the caller, and could the caller understand the system, under the conditions that matter?

Evaluate more than a clean browser microphone. Include representative phones, codecs, packet loss, room echo, car noise, nearby speech, hold-music bleed, and the speech patterns of the population the product is meant to serve.

On output, review intelligibility, pronunciation of critical terms, clipping, volume stability, unwanted artifacts, and whether prosody changes the meaning. A warm voice that reads “$1,500” as “fifteen dollars” is not high quality.

Synthetic audio is useful for controlled coverage. It does not prove performance with real speakers. Document which voices, languages, devices, and acoustic conditions remain untested.

9. Conversational repair and caller effort

Question: When something went wrong, did the agent recover without making the caller do all the work?

Count repetitions, rephrases, corrections, unnecessary confirmations, abandoned turns, and requests for a human. Then inspect why they happened. A caller repeating an account number may expose poor recognition, weak confirmation design, or a tool that rejected a correctly captured value.

Good repair is specific and economical. “I heard Friday the seventeenth. Is that right?” is better than restarting the entire flow. A caller should not need to discover the agent’s hidden state through trial and error.

Caller effort is not a substitute for outcome measurement. Some high-risk workflows should include deliberate confirmation. The goal is to remove avoidable work without removing necessary safety.

10. Handoff completion

Question: When the agent transfers the call, does the customer reach the right human with the right context?

An escalation decision is only the start. Test the destination, transfer type, queue behavior, context payload, caller explanation, hold experience, fallback, and human resolution. Track whether the caller has to repeat information the agent already collected.

A successful handoff ends in a verified next state, not a transfer_started event. If the queue is closed or the transfer fails, the agent needs an approved recovery path.

This metric also reveals a product truth that teams sometimes miss: choosing a human at the right moment can be a successful agent outcome. Containment is not the goal for every call.

Turn the metrics into release rules

The ten metrics are not a recipe for one universal score. Start by separating three kinds of decision.

Critical gates

These are failures that block a release in the relevant workflow. Examples include an unauthorized side effect, a missed mandatory disclosure, a wrong-account action, a fabricated tool result, or a failed handoff with no safe fallback.

Critical gates should be explicit and scenario-specific. “Policy score above 95%” is not enough if the remaining 5% includes a severe violation.

Quality budgets

These are distributions and rates the team expects to keep within an approved range. Examples include latency by stage, interruption recovery, critical-entity accuracy, caller repetitions, or successful repair.

Set the first budget from a measured baseline. Tighten it as the system and evidence improve. Publish the tested conditions beside the result.

Diagnostic signals

These help locate a failure but may not decide release on their own. General WER, model-token cost, raw call duration, or sentiment estimates often belong here. They become decision metrics only when the team can connect them to a user or business outcome.

Run each release candidate more than once where models, networks, or tools can vary. EVA-Bench highlights why average trial performance, best-of-several performance, and repeated success answer different questions. A workflow that succeeds once in five attempts is capable of success. It is not reliable.

A practical evaluation record

For every test session, keep enough information to reproduce the decision:

• Workflow and scenario version
• Caller intent and permitted outcome
• Input audio and condition labels
• Agent, prompt, model, voice, transport, and tool versions
• Transcript and aligned event timeline
• Tool requests, results, and downstream effects
• Policy assertions and evidence
• Scores for each applicable metric
• Severity, likely owning layer, and reviewer notes
• Final release decision and approver

This sounds like a lot until the first serious regression. Then it becomes the shortest path back to an explanation.

Our earlier article on the voice-agent benchmark gap explains how to adapt public research dimensions without treating a benchmark as a production audit. The guide to voice-agent release gates shows how to turn the resulting evidence into a release decision.

Where Voxeval fits

Voxeval is being built to help technical voice-AI teams organize domain workflows, run their actual agents, align audio with tool and policy evidence, and make workflow-level release decisions. The framework in this article does not depend on Voxeval. A team can implement it with internal fixtures, trace storage, audio review, scripts, and CI.

The important move is architectural: stop asking one metric to represent an entire voice system. Keep outcome, policy, tools, conversation behavior, and final state visible long enough to see what actually failed.

Common questions

What is the most important metric for a voice agent?

For a consequential workflow, start with grounded task success and verified final system state. The agent must accomplish the permitted goal and leave the authoritative business system in the intended state. Conversation quality still matters, but it cannot rescue an incorrect outcome.

Is word error rate still useful?

Yes. WER is useful for diagnosing transcription performance and comparing controlled speech-recognition conditions. Pair it with critical-entity accuracy and downstream outcome checks. Do not use it as the overall voice-agent quality score.

Should every voice-agent metric have the same weight?

No. Weighting depends on workflow risk, and some failures should be hard gates rather than weighted inputs. A missed disclosure or unauthorized payment should not disappear inside a strong average.

How many test calls are enough before release?

There is no universal count. Start with every critical workflow and failure path, then cover representative caller and audio conditions. Repeat nondeterministic cases enough to measure reliability, and keep adding redacted, approved production failures to the regression set.