Why voice-agent evaluation has to happen at the conversation level

The call passes every component check and still feels broken. The transcript is accurate, the tool calls succeed, and median latency looks fine. Yet the caller gets asked for the same account number three times, starts speaking over a late confirmation, then gives up during a repair loop.

For teams running voice AI engineering, product, or contact-centre operations, closing that gap means turning a long call into timed evidence, detecting recurring conversation patterns, and creating release rules that aggregate metrics cannot smooth away.

A call is not a bag of turns

Most QA systems score each turn, then average the results. That is convenient. It is also how a bad thirty-second experience earns a respectable score.

A June 2026 r/MachineLearning discussion described the practical problem plainly: manual review of long traces does not scale, and small timing mistakes or repeated confirmations accumulate. The thread had limited participation, so it is not research evidence. It is useful as a sharp description of what builders are feeling.

The research is moving the same way. EVA-Bench evaluates complete simulated conversations and separates accuracy from experience. Its 213 scenarios cover task completion, faithfulness, speech fidelity, progression, concision, and turn timing. τ-Voice combines grounded tasks, realistic audio, and full-duplex behavior across 278 tasks. Although they take different approaches, both warn that a voice agent can be locally correct and globally exhausting.

Scroll diagram horizontally on smaller screens.

Start with a timed evidence record

Do not begin with a judge prompt that asks, “Was this a good call?” Begin with events.

For each conversation, preserve:

• Caller speech start and end.
• What the agent actually played, including the cut-off prefix when the caller interrupted.
• Transcript revisions, not only the final transcript.
• Tool requests, responses, retries, and durable side effects.
• State changes such as identity_verified, handoff_requested, or order_committed.
• The final business record.

That record lets you ask useful questions. Did the agent repeat a question because it forgot the answer, because the transcript changed, or because the tool rejected the value? Those are three different bugs.

Keep timestamps on one clock. If the media server, model gateway, and tool service each use unrelated local time, your trace will look orderly while the call was not.

The record also needs a clear distinction between evidence and interpretation. “Caller speech started at 14.22 seconds” is evidence. “Caller became impatient” is an interpretation that may be useful, but it should point back to an audio span and a rubric. When both are stored as plain text in the same event stream, reviewers cannot tell what the system observed and what a grader inferred.

A practical event record usually has four layers. The raw layer holds permitted audio references and provider events. The normalized layer translates provider-specific names into your shared schema. The workflow layer records business state, including which values were current, superseded, proposed, or committed. The evaluation layer adds detector outputs, reviewer labels, severity, and the release rule that consumed the result.

This separation makes re-scoring possible. If a rubric changes, Voxeval can rerun the evaluation layer without pretending the underlying call changed. If a provider changes its endpointing event name, the normalized layer can absorb the difference without breaking every historical query. And when a reviewer challenges a failure, the team can walk from the label back to the exact audio, state transition, and tool result.

Because these layers hold unusually sensitive evidence, retention deserves the same care as instrumentation. A conversation trace can contain phone audio, identifiers, account details, and tool payloads that were never meant for a general analytics warehouse. Store only the evidence your test and incident process needs, redact or tokenize sensitive values where possible, separate access by role, and document deletion windows with the team responsible for privacy and security. Better debugging is not a reason to create an uncontrolled copy of every customer conversation.

Run three kinds of checks

At Voxeval, we recommend three passes because one evaluator should not carry the whole job.

1. Deterministic sequence checks. Use code for timestamps, event order, state transitions, duplicated side effects, missing tool results, and entity version conflicts. These checks are cheap, explainable, and ideal for hard failures such as a success claim that precedes the tool result.

2. Narrow model graders. Ask small questions against bounded evidence. Did the agent respond to the caller’s correction? Was a repeated confirmation necessary? Did the conversation resume at the right workflow step? Give the grader the relevant audio slice, delivered text, state, and rubric. Do not dump a forty-minute call into one prompt and hope for wisdom.

3. Human calibration. Sample passes and failures across languages, workflows, and severity bands. Reviewers should label both the detector result and the underlying conversation, including cases where the automated result was correct. When people disagree, fix the rubric before tuning a threshold. A threshold cannot rescue a question that two careful reviewers interpret differently.

LiveKit’s testing guide makes a useful distinction between turn-level behavioral tests and full simulations. Text tests are good for deterministic tool and behavior checks. Full audio remains necessary for timing, turn-taking, and speech failures.

Detect patterns, not awkward sentences

Turn the failure motifs your reviewers notice into detectors. A detector can be deterministic, model-graded, or a mix. The useful unit is a recurring sequence that an engineer can reproduce, not a sentence that merely sounded awkward during review.

The confirmation carousel happens when the agent asks for the same fact more than once after receiving a valid answer. A first pass can compare requested slots with captured values, then a narrow model judge can decide whether the repeat was justified. “Please say that again, the line broke up” is not the same as silently forgetting the postcode.

The premature victory lap is a cleaner case. The agent says the task is complete before the authoritative tool result arrives. Compare the spoken-success timestamp with tool completion and final-state verification. This check should be deterministic because the system either had evidence for the claim at that moment or it did not.

The repair loop starts when the caller corrects a detail, the agent acknowledges it, then uses the old value anyway. Track entity versions. When “Tuesday” becomes “Thursday,” mark Tuesday as superseded and fail any later tool argument that still contains Tuesday.

The slow squeeze is cumulative. No single pause crosses the latency threshold, yet a series of small waits makes the exchange drag. Measure dead air per task stage and count caller probes such as “hello?” or repeated fragments. Sample the audio before promoting a phrase into a detector because a repeat can come from noise, emphasis, or impatience.

The handoff amnesia appears after the AI gathers useful context, transfers the call, and the human starts from zero. A telephony transfer may have succeeded while the customer experience failed. Score whether the receiving agent had the issue, verified facts, attempted actions, failure reason, and suggested next step when the call arrived.

The detector definitions should be narrow enough that two engineers can reproduce the result. For every pattern, write down its trigger, the minimum evidence needed, exclusions, severity, and expected owner. A “repair loop” might require two unsuccessful attempts to resolve the same slot after the caller supplied a valid value. A single clarification over a noisy line would be excluded. Without that boundary, the dashboard will measure reviewer mood rather than a stable product failure.

Voxeval’s conversation-pattern QA registry gives each pattern a definition, evidence requirement, detector type, false-positive review, severity, owner, and release rule.

Run new detectors against a hand-labeled calibration set before they become release gates. Look at false positives as carefully as misses. A detector that catches every repeated question by flagging necessary identity re-verification will create alert fatigue and teach teams to ignore the report. The goal is not maximal flag volume. It is a small set of signals that consistently points to work someone can fix.

A fictional call, because the sequence matters

Consider this fictional call, designed to show the sort of trace a test runner should generate. A caller wants to move a broadband installation from Friday to Monday. The agent captures Monday, checks availability, and hears an interruption: “Actually, afternoon if possible.” It checks afternoon slots, finds one, and asks for confirmation. Then the old Friday date leaks back from the conversation summary, the booking tool receives Friday afternoon, and the agent says Monday afternoon.

Each component looks plausible in isolation: speech recognition heard both dates, the availability lookup succeeded, the booking call returned success, and the closing sentence sounded confident.

Conversation-level voice-agent evaluation sees the version chain: Friday was superseded and Monday was confirmed, yet Friday reached the write tool and the final state disagreed with the spoken result. That sequence must fail despite every component-level success.

Cluster the unknowns

Rules catch patterns you already know, but production will invent new ones. Embed or otherwise represent redacted conversation traces, then cluster by behavior signals, not only topic. Useful inputs include repeated intent transitions, correction count, overlap timing, tool retry shape, escalation reason, and final outcome.

Review a small sample from each growing cluster, name the pattern, and add a detector. Then promote both a clean example and a nasty one into regression coverage. This is how manual review scales without pretending humans disappear from the loop: people investigate new families while automation watches the families the team has already named.

Sampling still matters after the detectors are running. Review a mix of severe failures, clean passes, low-confidence judgments, new workflows, and calls from strata with little traffic. If reviewers only listen to flagged calls, they can estimate precision but not how many failures the detector missed. If they only listen to the highest-volume English lane, the long tail of routes, devices, languages, and accessibility needs stays invisible.

Keep a stable holdout set for release-to-release comparison, then add a rotating sample from recent traffic. The stable set reveals regressions against known behavior. The rotating set catches new call patterns and operational changes that the fixed set cannot anticipate. When production evidence becomes a regression case, preserve the original failure characteristics while replacing or masking customer data under your approved process.

Release rules should preserve the sequence

Do not reduce the result to “conversation quality: 87.” Keep separate gates:

• Unsafe or incorrect side effect?
• Confirmed correction lost later?
• Mandatory handoff context missing?
• Repeated repair beyond the workflow limit?
• Latency tail over budget for this lane?

Then report pattern rates by workflow, language, route, release, and agent version. A booking agent can pass address changes and fail cancellations. That split matters.

Always keep the numerator and denominator next to the percentage. “Repair loops fell to 1.2%” is hard to interpret without knowing whether that means 12 of 1,000 completed calls, 12 of 1,000 sampled calls, or 12 of 37 calls where a correction occurred. The relevant denominator for a lost-correction detector is conversations containing a correction opportunity, not every call in the product.

Use severity before averages. An occasional extra confirmation is annoying. A confirmed account correction that later reaches a payment tool with the old value is a different class of failure. The first may have a rate threshold and an improvement target. The second should usually block the affected lane until the team understands and fixes it.

The release record should name the detector and rubric versions, coverage window, sample size, workflow slices, known blind spots, and an owner for every open exception. That turns conversation-level evaluation from a dashboard into an operating process. Product can decide whether a bounded issue is acceptable, engineering can reproduce it, and support can recognize it when a similar call appears.

Start with five patterns from your worst calls, not fifty. Choose five you can explain, reproduce, and connect to a release decision. Use the post-interruption recovery method when a correction lands mid-turn and the repeat-yourself rate when the call moves to a person. Once those first detectors are calibrated and owned, the next five will be much easier to add without creating a graveyard of noisy alerts.

Subscribe to Voxeval for field-tested voice-agent evaluation methods, templates, and teardown notes.