From SOP to voice-agent eval suite: a practical conversion guide

Your operations team already knows how the call should work. The knowledge lives in standard operating procedures, policy documents, scripts, escalation charts, and the heads of experienced agents. Your voice agent lives somewhere else: prompts, tools, model settings, and telephony configuration.

The gap between those worlds is where weak eval suites are born. Teams copy a few approved phrases into test prompts, run pleasant happy-path calls, and miss the decisions that make the procedure safe.

This guide shows product, operations, and engineering teams how to convert an SOP into versioned voice-agent scenarios, assertions, evidence requirements, severity, and release rules. The payoff is an eval suite that tests the workflow the business approved, not one model’s ability to recite a script.

The short answer

Convert an SOP in five passes: map decisions, generate scenarios, turn obligations into assertions, name the evidence that proves each assertion, and assign a release consequence. Keep the source clause attached to every test.

Do not feed a long PDF to an evaluator and ask, “Did the agent comply?” That hides ambiguity, makes failures hard to reproduce, and lets the judge invent policy. The eval case should say which clause applies, what action is allowed, what must happen first, what evidence proves it, and how severe a failure would be.

At Voxeval, we think of this as compiling operational intent into testable behavior.

Why an SOP is not an eval suite

An SOP is written to guide people inside an organization. It can rely on training, judgment, shared vocabulary, other documents, and exceptions that everyone on the team “just knows.” A test runner has none of that context unless you make it explicit.

Consider a fictional returns procedure:

Refunds under $200 may be approved after order verification. Refunds at or above $200 require supervisor review. Refunds to a new payment method are not permitted.

That short rule contains several test dimensions:

• An amount boundary at exactly $200
• A prerequisite called order verification
• A permission boundary for the agent
• A human escalation path
• A prohibited destination
• A downstream side effect
• A claim the agent may make only after the refund system responds

One happy-path conversation will not cover it. Neither will a semantic similarity score against the paragraph.

NIST’s AI Risk Management Framework calls for teams to define supported tasks, map risks across components, measure system behavior, and document human oversight. It is a voluntary risk framework, not a voice-agent test format. Its useful lesson here is that scope, impact, and oversight have to be explicit before measurement can mean much.

Scroll diagram horizontally on smaller screens.

Pass 1: map decisions before writing conversations

Start with the smallest approved unit of source material. It may be one SOP section plus the policy and system documentation that section depends on.

Mark five things:

1. Triggers: What caller request or system condition activates the procedure?
2. Prerequisites: What must be known or verified before the agent proceeds?
3. Branches: Which values, outcomes, or caller choices change the path?
4. Actions: What may the agent say, read, write, send, transfer, or stop?
5. Terminal states: What counts as completed, escalated, declined, abandoned, or failed safely?

Use decision language, not conversation language. “Ask for the order number” is a conversational instruction. “A verified order must be associated with the caller before refund eligibility is checked” is a state constraint. The second statement can survive a change in wording or voice.

Resolve ambiguous clauses now

If operations and compliance disagree on what “large refund” means, do not let the model decide. Record the ambiguity, name an owner, and pause that lane until the rule is approved.

An eval suite should expose policy gaps. It should not quietly fill them.

Keep a stable source identifier such as RETURNS-4.2.1. When the procedure changes, you can find every dependent scenario and assertion.

Pass 2: generate a scenario matrix, not a bag of prompts

A scenario defines initial state, caller intent, permitted path, and expected terminal state. Dialogue is one possible realization of that scenario.

For the fictional refund rule, start with equivalence classes and boundaries:

Scenario	Amount	Order verified	Requested destination	Expected state
Standard approval	$75	Yes	Original method	Refund completed
Verification missing	$75	No	Original method	Verification requested; no refund
Boundary below	$199.99	Yes	Original method	Refund completed
Boundary exact	$200	Yes	Original method	Supervisor review
Boundary above	$200.01	Yes	Original method	Supervisor review
Prohibited destination	$75	Yes	New method	Request declined or safe alternative offered
Tool failure	$75	Yes	Original method	No success claim; approved recovery
Caller correction	$75 to $250	Yes	Original method	Path updates to supervisor review

Then vary the voice conditions that could change behavior: a correction mid-turn, a long pause, background speech, a spelled identifier, an interruption during confirmation, or a language lane your product actually supports.

Do not multiply every variable with every other variable on day one. Use risk to select combinations. A correction near the $200 boundary deserves more coverage than ten different greetings.

Research benchmarks follow the same broad direction. VoiceAgentBench separates tool choice, structure, parameters, multi-step behavior, and safety. τ-Voice combines task completion with voice-interaction measures. Their datasets and simulators do not define your production cases, but they show why one transcript-level label is too coarse.

Pass 3: turn obligations into atomic assertions

An assertion should answer one question with a clear applicable state. Avoid “the call was good” and “the agent followed policy.”

For the refund example:

• The agent verified an eligible order before invoking the refund tool.
• The amount passed to the tool matched the caller-confirmed amount.
• The destination was the original payment method.
• Amounts at or above $200 did not invoke the refund tool directly.
• The agent initiated the approved supervisor path when required.
• The agent did not claim completion before a successful tool result.
• A failed tool left the order and payment systems unchanged.

Use deterministic assertions where the evidence is structured. A programmatic check is better than an AI judge for exact amounts, tool names, call counts, state transitions, and required event order.

Use human or calibrated model review for meaning that genuinely requires interpretation: whether a correction was acknowledged, whether the explanation was clear, or whether the agent pressured a caller after a refusal.

LiveKit’s testing docs make a similar separation. Its helpers support behavioral and tool assertions in text mode, while the docs point to end-to-end tools for the audio path. That is a useful layering pattern even if you do not use LiveKit.

Give assertions three outcomes

Use pass, fail, and not_applicable where applicability depends on the path.

A supervisor-disclosure assertion should not fail when the caller hangs up before a supervisor path is reached. Marking that call as a failure corrupts the metric. Marking it as a pass hides that the obligation was never tested.

Pass 4: name the evidence before choosing the evaluator

Every assertion needs an evidence contract. This prevents the evaluator from guessing from whatever artifact happens to be available.

Assertion type	Strong evidence
Caller supplied an exact amount	Aligned audio, final transcript, and captured entity
Agent confirmed the amount	Agent audio or text event after the caller value
Correct refund tool was called	Structured tool request with stable event ID
No duplicate refund occurred	Tool requests plus authoritative payment state
Supervisor path completed	Transfer events, destination, queue result, and final state
No false completion claim	Agent audio aligned with tool result timing

Transcripts are convenient but lossy. They can hide overlap, clipped speech, pronunciation, and when the caller heard a statement. Tool logs prove a request happened, but not that it was authorized. A 200 response proves transport success, not the intended business state.

For consequential actions, join conversation evidence to the system of record.

Preserve the causal chain

Link the source clause to the scenario, assertion, evidence, result, and release decision. A practical record might look like this:

case_id: refund_boundary_exact_v3
source:
  procedure: returns_sop_v7
  clauses: [RETURNS-4.2.1, PAYMENTS-2.4]
initial_state:
  verified_order: ORD_FIXTURE_104
  refundable_balance: 200.00
caller_goal: refund 200 dollars to the original payment method
expected_terminal_state: supervisor_review_started
assertions:
  - id: no_direct_refund_at_threshold
    evidence: [tool_events, payment_state]
    severity: critical
  - id: explain_supervisor_review
    evidence: [agent_audio, timeline]
    severity: major

This is an illustrative format, not a Voxeval API contract. Use the schema that fits your system.

Pass 5: assign severity and a release consequence

Severity describes impact. A release rule describes what the team does about it.

Those are related, but they are not identical. One critical wrong-account refund may block a release immediately. A minor wording issue may enter a quality budget. A repeated cluster of minor repair failures may still block because the caller experience has collapsed.

Use a small, shared severity scale:

• Critical: unsafe or unauthorized action, serious policy breach, wrong-person or wrong-account effect, unrecoverable sensitive-data exposure
• Major: task fails, required escalation fails, agent fabricates completion, caller is materially misled
• Moderate: avoidable repetition, confusing repair, slow recovery, context lost during handoff
• Minor: cosmetic language or voice issue with no material effect

Then write the release rule beside the assertion. Examples:

• Zero critical failures in refund mutation lanes
• Every mandatory disclosure assertion passes when applicable
• At least 99% critical-entity accuracy across the approved test set, with no account-identity error
• P95 transfer connection time stays inside the measured lane budget
• No regression from the current production baseline in caller repetitions

These numbers are examples, not recommended universal thresholds. Set them from workflow risk, baseline evidence, user research, and operational tolerance.

Build the first useful suite in one workshop

Bring together one experienced operator, one product owner, one engineer who knows the tools, and the policy or compliance owner for the workflow. Ninety focused minutes can produce a credible first slice.

Minutes 0 to 20: choose one terminal outcome

Pick a workflow with a real business state, such as appointment moved, refund requested, claim status read, or call transferred. Avoid “answer questions” as the first target.

Minutes 20 to 45: draw decisions and unsafe shortcuts

Map prerequisites, boundaries, branches, tool effects, and human paths. Ask the operator what a new employee gets wrong and what an experienced employee checks automatically.

Minutes 45 to 70: write eight to twelve scenarios

Cover the happy path, each policy boundary, corrections, missing information, a tool failure, a caller exit, and an escalation failure. Keep the first set small enough to review deeply.

Minutes 70 to 90: attach assertions and evidence

For every critical scenario, decide what must be proven and which artifact proves it. If the evidence does not exist, that is an observability requirement, not a reason to write a vague judge prompt.

After the workshop, run the cases manually once. You will find source ambiguity, fixture gaps, and evidence you forgot to capture. Fix those before automating at scale.

Common conversion mistakes

Testing the script instead of the outcome

Exact phrasing matters when policy requires exact language. Elsewhere, a word-for-word expectation can punish a clear answer and reward a memorized one that performs the wrong action.

Letting the model infer the policy

An evaluator should receive the applicable rule and evidence. It should not browse an entire policy corpus and decide which clause mattered after the call.

Ignoring negative space

Some of the most important requirements are things the agent must not do: reveal a record before verification, run a mutation twice, promise a result before confirmation, or continue after consent is withdrawn.

Treating every failure as a prompt problem

A failed scenario may belong to speech recognition, endpointing, reasoning, a tool schema, backend state, telephony, or the evaluator. Preserve enough evidence to assign an owning layer.

Losing the source version

When an SOP changes, stale tests can enforce yesterday’s policy. Version the source and review affected cases before release.

Where Voxeval fits

Voxeval is being built around this traceable chain: domain source, scenario, agent version, evidence, assertion, failure attribution, and release decision. The framework does not depend on Voxeval. A disciplined team can build it with documents, fixtures, logs, review tools, and CI.

The hard part is not generating more simulated conversations. It is preserving why each case exists and what evidence is allowed to decide it.

Next, use the ten-metric voice-agent scorecard to make sure the suite covers both business outcome and conversation experience. The guide on turning production failures into regression cases shows how this library should grow after launch.

Common questions

How many eval cases should one SOP produce?

There is no fixed ratio. A short paragraph with a monetary boundary and several exceptions may produce more cases than ten pages of explanatory material. Count decision boundaries, failure paths, and material voice conditions instead of pages.

Can an LLM generate the scenarios?

It can propose variations, but a domain owner should approve the decision map, expected state, and severity. Never let generated cases quietly invent policy.

Should the SOP text be included in every judge prompt?

Include the smallest applicable clause or structured rule, plus any approved interpretation. Large policy dumps raise cost and ambiguity. Deterministic checks should not need a judge prompt at all.

What should happen when the SOP is unclear?

Record the ambiguity, assign an owner, and keep the affected lane out of automated release decisions until the business approves the rule. An unclear source cannot produce a trustworthy pass or fail.