Your barge-in test stops too early: evaluate what happens after the interruption

The agent stopped talking in 180 milliseconds, then forgot the caller’s correction and charged the old amount. A fast stop made the recording sound responsive while the workflow kept moving in the wrong direction.

That is why barge-in latency is not an interruption test. It is one timestamp inside a longer recovery problem. For voice engineers and QA teams who already measure playback stop but still see bizarre behavior after callers cut in, the method below combines a recovery rubric, a runnable public benchmark, and a test pack for company-owned workflows.

Playback stopped. Now what?

Most teams ask whether the caller was detected, how quickly synthesis stopped, and whether the agent began listening again. Those checks matter, but they do not establish recovery.

The next response has to understand why the caller interrupted, preserve the right state, discard anything the caller never heard, and continue without replaying a speech.

IHBench, released in June 2026, is aimed at that exact gap. Its public dataset contains 45 synthetic, verified conversations across 10 enterprise domains, with 428 interruption points and six interruption types: normal, impatient, correction, topic switch, filler, and pushback. Rather than asking only whether barge-in worked, the benchmark asks whether the agent recovered.

Scroll diagram horizontally on smaller screens.

Preserve the delivered prefix

Suppose your model generated:

“Your refund of forty-two dollars has been approved and will arrive within five to”

The caller heard only:

“Your refund of forty-two dollars”

Those are not the same conversation history. Store both assistant_generated_text and assistant_delivered_text, then give the next turn the delivered prefix rather than the sentence the system intended to play. Otherwise the model may assume the caller heard a deadline, disclosure, or question that never reached the phone.

IHBench models this explicitly. Its conversation rows keep the delivered assistant transcript and, when a turn was cut off, the original full text. That small distinction can determine whether the recovery is coherent or feels like conversational gaslighting.

The delivered prefix should be captured from the playback path, not estimated from the text generation stream. Audio may sit in a provider buffer after synthesis, and a stop request may arrive after another few words have reached the caller. If your history says the caller heard only “your refund,” while the phone actually played the amount too, the next response can repeat information or contradict what was audible.

Keep the generated text, synthesized chunks, playback acknowledgements, and stop event linked by stable IDs. For sensitive language, add a reviewable audio slice so a tester can confirm the boundary. This also exposes a common measurement mistake: reporting the time to send a stop command as barge-in latency while buffered audio continues to play.

Six interruptions, six different jobs

Do not put every overlap into one barge_in bucket because each interruption type gives the agent a different job.

A normal answer is the easiest case. The agent asks, “Would you like morning or afternoon?” and the caller says, “Afternoon.” The agent should accept the answer and advance. Repeating the whole question wastes time and makes a routine exchange feel broken.

Impatience carries a different intent. The agent begins a long policy explanation and the caller cuts in with, “Yes, yes, just tell me the fee.” The response should shorten and answer the requested point while preserving any required disclosure. Speed does not cancel policy, but policy does not require replaying every word from the beginning.

A correction changes authoritative state. When the caller says, “Not fifteen. Fifty,” the old amount becomes invalid. The next response should confirm the corrected value where risk warrants it, and every later tool call should use the new version.

A topic switch suspends one job and opens another. “Wait, before that, can I change the delivery address?” should pause the current workflow, enter the address flow, and return only if the original goal still applies. Test the workflow stack, not just whether the next sentence mentions the address.

A filler or backchannel may not be an interruption at all. “Mm-hm” can mean “keep going,” not “yield the floor.” A system that stops on every backchannel turns a natural listener into a conversation breaker and leaves required explanations half delivered.

Pushback cancels the assumption behind the request. If a caller says, “No, I am not giving you my full card number,” the agent should not resume the interrupted prompt as if nothing happened. It should address the concern, explain the approved path, or move to a human.

One detector cannot infer all six safely from timing alone, so model the recovery as a state transition rather than a fresh prompt. Before the interruption, record the active workflow, the current entity versions, any proposed tool action, and whether a mandatory statement was fully delivered. The interruption then produces a classified event, such as correction(amount: 50) or topic_switch(delivery_address), and the orchestrator decides what must be cancelled, retained, or resumed.

That boundary keeps the language model from carrying the whole burden. A model can interpret “not fifteen, fifty,” but the workflow layer should mark fifteen as superseded and prevent it from reaching a later write. A model can recognize a topic switch, but the application should decide whether the previous task may remain suspended or must be closed. The spoken recovery and the business transition are connected, yet they are not the same operation.

The third-party problem

Classification also depends on who spoke. On speakerphone, a child may ask, “Where are my shoes?”; a colleague in the room may say, “Do not approve that yet”; or the television may mention a number.

TPI-Bench studies whether spoken-language models can distinguish third-party interruptions from the primary user’s speech. Its companion training set contains 88,000 instances designed to push models toward acoustic speaker cues instead of semantic shortcuts.

Your production test does not need 88,000 samples to begin. Start with a primary user interrupted by background speech, background speech that sounds like a valid command, and the same words spoken by both the primary user and a third party. Add speaker changes during sensitive workflows and household or office noise from the markets you serve.

Speaker attribution should also have an explicit uncertain state. On a noisy speakerphone call, forcing every utterance into “primary caller” or “third party” can create false authority. When the system cannot establish who spoke a consequential command, the approved response may be to ask the primary caller to repeat it, perform another authentication step, or move the action to a human. The correct behavior depends on the workflow; silent acceptance should not be the default, and third-party speech should never quietly become authorization in a banking or healthcare lane.

Score the next action on two axes

IHBench separates task fulfillment from recovery quality. That is a sensible pattern for company-owned tests too.

Task state asks whether the agent moved the workflow to the right next state. The test should look for concrete transitions:

• Correction applied to the authoritative entity.
• Topic switch suspended the previous write.
• Pushback prevented the restricted request.
• Backchannel left the agent speaking.

Recovery behavior asks whether the next response fits what the caller actually experienced. Check that it:

• Addresses the interruption.
• Does not repeat already delivered content without reason.
• Does not assume unheard content was delivered.
• Resumes at the correct point.
• Keeps required disclosures and confirmations.

Score both because a polite recovery that writes the wrong value still fails. Delivery consistency belongs inside recovery behavior: the agent should act as if the caller heard only the words that were actually played. This catches a response that begins with “as I mentioned” and refers to information cut off by the interruption. It also catches the opposite problem, where the agent repeats a full disclosure even though only its final few words were interrupted and a shorter continuation was approved.

Build one case from every real failure

The following electricity-service call is fictional and did not happen, but it shows the template to fill from every approved real failure.

The caller says, “Move the payment date to the fifteenth.” The agent begins confirming the date. Halfway through, the caller says, “Sorry, the sixteenth.” The agent stops quickly, confirms the sixteenth, then calls the tool with the fifteenth because the tool payload was prepared before the correction.

Your regression case should preserve:

1. The audio up to the correction.
2. The delivered assistant prefix.
3. The state before interruption.
4. The corrected entity and superseded value.
5. The tool payload that must never execute.
6. The final account state.

Together, these fields enforce a strict rule: interruption recovery is complete only when the final business state reflects the recovered conversation.

Tool races need their own assertions. A correction can arrive while a lookup is pending, while a write is queued, or after a write has committed. If the operation can be cancelled, verify that cancellation reached the owner and no late callback resurrected the stale result. If it cannot be cancelled, reconcile the durable state and apply a controlled follow-up rather than starting a blind retry.

Idempotency protects against a different failure. An orchestrator may retry after losing the response even though the first write succeeded. The interruption path should carry the same operation identity when retrying the same intent and a new identity when applying a genuine correction. Otherwise the recovery can create two refunds, two bookings, or two order changes while the conversation sounds perfectly normal.

Run the public benchmark before adapting it

Once you know what a company-owned regression case must preserve, use the official IHBench quickstart to establish a public baseline. The dataset page currently gives these steps:

git clone https://github.com/boson-ai/ihbench
cd ihbench
pip install -e ".[api]"
cp .env.example .env

Add the provider keys required by the model and judge configurations, then:

python scripts/run_inference_api.py \
  --config configs/gpt-realtime-2.yaml \
  --output responses/model.jsonl \
  --epochs 3

python scripts/run_evaluate.py \
  --responses responses/model.jsonl \
  --output results/model.jsonl \
  --judge-config configs/judge.yaml

python scripts/summarize_results.py results/

Those commands come from the current dataset page. They require paid model access, so pin the toolkit commit and record model versions before comparing runs.

The public benchmark is synthetic, English-only, and judge-dependent. Its authors say so. Use it to learn the shape of the problem, then replace the workflow, language, policies, and interruption audio with yours.

When adapting the structure, build an interruption matrix rather than choosing one timestamp. Take the same caller intent and inject it near the start of agent speech, around the key entity, immediately before a question ends, while a tool is pending, and just after a durable side effect. The words are identical, but the required recovery may change because the caller heard different content and the application reached a different state.

Run each case more than once when the agent or endpointing model is stochastic. Keep the attempt count and report both average behavior and repeat consistency. A case that succeeds once in three attempts is evidence that the path can work, not evidence that it is dependable.

A release gate that cannot be gamed

For each interruption type, track:

• Detection and playback-stop timing.
• Correct speaker attribution.
• Correct interpretation of the interruption.
• State transition.
• Next-response quality.
• Tool cancellation or update.
• Verified final outcome.

Keep a critical lane for lost corrections, unauthorized actions, and third-party commands rather than averaging them with a fluent response. Report the remaining results by interruption type, injection point, workflow state, language, audio route, and outcome severity. A single barge-in success rate will hide whether the agent handles normal answers well but fails corrections during tool latency. Keep timing percentiles beside the functional pass rate, since a very conservative endpointing policy may improve speaker classification while making normal interruption feel unresponsive.

The release evidence should include the audio the caller heard, the delivered prefix, the pre- and post-interruption state, the tool timeline, the next spoken response, and the verified final record. A reviewer should be able to explain a failure without reconstructing it from six provider dashboards.

Interruption failures also expose timing and memory bugs. Trace the delay with the voice latency event model, then keep the recovered sequence inside conversation-level evaluation. The post-interruption recovery test pack supplies fictional starter cases and the fields needed to add your own. Take the nastiest approved correction pattern your team saw this week, cut the recreated audio at three different points, and run all three before the next release. The timing changes will often reveal separate bugs hiding behind one phrase.

Subscribe to Voxeval for more practical voice-agent test packs and production QA methods.