How to run an STT bake-off on your own voice-agent calls

Names, confirmation codes, or order modifiers keep getting mangled, so someone opens three speech-to-text dashboards and asks which provider has the best accuracy. A leaderboard looks tempting. It is also the wrong place to make the production decision.

The best STT system is the one that handles your callers, audio path, vocabulary, latency budget, and failure cost. This guide shows voice-AI engineers how to run a reproducible bake-off using consented, representative calls. You will compare final transcripts, critical entities, streaming stability, endpoint timing, reliability, and cost without turning one average WER into a winner.

The short answer

Send the same untouched audio through every candidate under documented settings. Score final word error rate, critical-entity accuracy, partial-transcript churn, finalization and endpoint timing, API reliability, and effective cost. Break the results out by call condition and workflow.

Do not declare a winner from a pooled average. One provider may lead on clean English and fail on your 8 kHz phone route. Another may have a slightly higher WER but preserve every account number. For a voice agent, the second result can be safer and more useful.

This is a bake-off, not a beauty contest.

What word error rate can and cannot tell you

Word error rate compares a machine transcript with a human reference:

WER = (substitutions + deletions + insertions) / words in the reference

Microsoft’s current speech evaluation guidance uses this formula and points to NIST’s SCTK tooling for local scoring. WER is useful for controlled comparison. It gives every word roughly the same value, which is not how a business workflow behaves.

Imagine a fictional food-ordering agent. The reference says, “one large paneer pizza, no onion, deliver to flat fifty.” A transcript that misses “one” filler-like word may score similarly to one that turns “no onion” into “onion.” Only one of those errors changes the order.

Keep WER. Add metrics that know what the call is for.

Scroll diagram horizontally on smaller screens.

Step 1: write the decision before collecting audio

State what the bake-off will decide. Examples:

• Choose the default streaming recognizer for English appointment calls.
• Decide whether Hindi-English calls need a separate model or route.
• Compare two models after changing from browser audio to PSTN.
• Test whether vocabulary adaptation improves product names without raising false positives.

Name the workflow, production audio route, supported languages, candidate configurations, metrics, and tie-breaker. Freeze them before looking at results.

A useful decision statement is narrow:

Select the streaming configuration for UK and India delivery-status calls. Critical order IDs and dates are hard gates. Among passing candidates, choose the lowest P95 finalization time, then compare effective cost.

That is far more useful than “find the most accurate STT.”

Step 2: build a representative, lawful audio set

Use calls you are allowed to process for this purpose. Confirm consent, contractual rights, retention, access, and cross-border handling with the appropriate owners. Redact or replace sensitive details when possible, and keep raw audio out of a casual shared folder.

If real calls are not available, start with recorded speakers under approved conditions. Synthetic audio can add controlled variation, but it should not be labeled representative of real callers.

Build strata that reflect the deployment:

• Phone, browser, or app audio
• Sample rate, codec, channel layout, and carrier route
• Quiet room, street noise, car, office, television, or nearby speech
• Accent, speech rate, age range, and voice characteristics that the product serves
• Short answers, long explanations, interruptions, and self-corrections
• Workflow vocabulary, names, addresses, codes, dates, amounts, and product terms
• Monolingual, multilingual, and code-switched lanes that are actually supported

Do not “balance” the set into something unlike production. Keep a production-weighted view and a risk-weighted view. Rare wrong-account errors deserve visibility even if they barely affect the overall average.

The 2026 Voice of India preprint makes a related point for Indian speech: clean, scripted benchmarks miss unscripted telephony and regional variation, while strict single-reference WER can punish valid spelling variation in code-mixed terms. The benchmark is closed-source and should not be treated as a substitute for your corpus. Its critique is a useful warning about dataset realism and transcript normalization.

Step 3: create a reference transcript with an annotation policy

Human labels are not automatically ground truth. Two careful annotators can disagree about fillers, fragments, punctuation, code-switching script, abbreviations, numbers, or an inaudible word.

Write a short annotation guide before labeling:

• Whether fillers such as “um” and “haan” are included
• How numbers, currencies, dates, and times are normalized
• How partial words and false starts appear
• How overlapping speakers are marked
• Which script represents code-switched words
• How inaudible sections are labeled
• Whether punctuation and capitalization are ignored for WER
• How names and domain terms are verified

Use a second reviewer for a sample and every critical entity. Resolve disagreement without showing annotators which provider produced which output.

Keep two reference layers when needed:

1. Lexical reference for reproducible WER normalization
2. Business reference for entities, intent, and expected downstream values

That separation prevents transcript formatting from deciding whether the agent captured a date correctly.

Step 4: freeze the input and configuration

Every candidate receives the same audio bytes. Do not denoise one provider’s input, resample another, and call the result fair.

Google’s Speech-to-Text documentation advises sending existing audio at its native sample rate rather than resampling it merely to reach 16 kHz. It also notes that legacy 8 kHz telephony can produce lower accuracy than higher-quality input. Treat audio preparation as a tested configuration, not invisible cleanup.

Record for every run:

• Provider, model, region, API version, and date
• Language and alternate-language settings
• Encoding, sample rate, channels, and stream chunking
• Punctuation, formatting, diarization, and profanity settings
• Vocabulary hints, adaptation, or key terms
• Endpoint, utterance-end, and interim-result settings
• Retry policy, timeout, and connection reuse

If a provider offers an adaptation feature, run both default and adapted configurations. Google’s guidance warns that stronger phrase boosting can reduce misses while increasing false positives. Measure both.

Step 5: capture the full streaming trace

Saving only the final transcript throws away the behavior that a voice agent experiences.

Capture each interim result, its arrival time, covered audio interval, final flag, endpoint or utterance-end signal, confidence where available, and any API error. Use a monotonic clock and preserve provider request IDs.

Deepgram’s documentation shows why. Interim transcripts can change as more audio arrives. Its endpointing uses voice activity and configurable silence to mark a speech boundary, while final transcript segments and speech-final events have different meanings. Other providers expose different concepts.

Normalize them into your own event model without pretending the semantics are identical.

Step 6: score six metric families

1. Final lexical accuracy

Calculate WER for the fully collected final transcript after the same normalization. Report substitutions, deletions, and insertions separately. A deletion-heavy result may point to weak audio or clipping. Insertions may reveal background speech or hallucinated text.

Break out WER by stratum. A single corpus number is a weighted average of your sampling choices.

2. Critical-entity accuracy

Define exact entities per workflow: order IDs, names, addresses, dates, amounts, medication names, policy numbers, or product modifiers.

Track exact match, normalized match, deletion, substitution, and false insertion. For repeated digits, keep positional error. A six-digit code with one wrong digit is not “mostly correct” for account access.

Also inspect the chain from final transcript to the entity passed into the agent’s tool. STT may be correct while later normalization breaks the value.

3. Partial-transcript stability

Count how often previously emitted words change before finalization. Weight changes near critical entities more heavily.

A simple diagnostic is:

partial churn = changed committed tokens / tokens emitted before final

Define “committed” for your consumer. Some agents wait for stable text. Others begin reasoning on partials. Measure the behavior you actually implement.

4. Timing

Measure at least:

• Time to first useful partial
• Streaming lag behind the audio cursor
• Time from last spoken phoneme to final transcript
• Time from last spoken phoneme to endpoint signal
• Tail distributions by condition

Do not combine these into one latency number. A fast partial can coexist with slow finalization. An aggressive endpoint can feel quick while chopping callers who pause.

5. Reliability

Track connection failures, timeouts, rate limits, empty results, dropped final segments, reconnection behavior, and duplicated text. Repeat long streams and silence-heavy calls.

The agent cannot recover from a recognizer that quietly loses an utterance unless the integration detects it.

6. Effective cost

Record provider charges under the exact tested plan and date, plus adaptation, region, logging, and minimum-duration rules. Add engineering and operational costs for stream management, fallbacks, and correction handling.

Do not put a timeless price table in the eval result. Store the pricing source and observation date.

Step 7: use hard gates before weighted scores

A weighted score can be useful after critical requirements pass.

Example release logic for the fictional delivery agent:

• Zero wrong-account matches in the critical test set
• No omitted negation in product-modifier cases
• At least the approved order-ID accuracy by phone route
• P95 endpoint time inside the lane budget
• Error rate below the operating threshold

Among candidates that pass, the team can weigh overall WER, stability, latency, and cost. The weights should reflect the workflow, not make a preferred vendor win.

When no candidate passes, do not lower the bar quietly. Options include confirmation for risky values, DTMF capture, spelling, a separate language route, vocabulary adaptation, a human fallback, or narrowing the supported workflow.

Step 8: challenge the result

Before deciding, run three checks.

Bootstrap or resample the corpus

Estimate how stable the ranking is under different samples. If two candidates trade places easily, report a tie rather than false precision.

Hold out a final set

Do not tune vocabulary and thresholds on every file you use to decide. Keep a blinded slice for the final comparison.

Replay through the voice agent

The best standalone transcript may not produce the best call. Run the candidates through the real turn detector, model, tools, and response path. Measure outcome, interruption, and caller repair.

This is where the ten metrics beyond WER become the final decision frame.

What the report should say

Publish enough detail for another engineer to reproduce the comparison:

• Decision and tested date
• Corpus size, source, consent basis, and exclusions
• Condition and language distribution
• Annotation and normalization policy
• Provider configurations
• Metric definitions and release gates
• Results by stratum with uncertainty
• Known data gaps
• Selected configuration and fallback
• Re-test triggers

Avoid “Provider A is 30% more accurate” without naming the corpus, metric, baseline, and conditions. That sentence will travel farther than its limitations.

Where Voxeval fits

Voxeval can help teams keep audio fixtures, provider configurations, transcripts, entities, timing events, agent outcomes, and release decisions tied together. The bake-off method does not require Voxeval. It requires discipline about sameness, provenance, and limits.

Our strongest opinion is this: use public benchmarks to choose candidates. Use your own calls to choose a production configuration.

For India-focused lanes, continue with the guide to testing Hindi-English voice agents. For release decisions, see production voice agents need release gates.

Common questions

How many calls are needed for an STT bake-off?

There is no universal count. Start with enough labeled audio to cover every critical workflow and condition, then estimate ranking uncertainty. Add data where the ranking or hard-gate result is unstable.

Should we compare default models or tuned models?

Compare both if tuning is realistic for production. Keep a default baseline, apply the same domain information where features are comparable, and document when provider-specific features make the configurations different.

Is confidence score comparable across providers?

Not automatically. Providers calculate and expose confidence differently, and some fields may be absent. Calibrate each score against correctness on your own labeled data before using it as a shared threshold.

Can synthetic speech replace real calls?

No. Synthetic speech is useful for controlled coverage, privacy-preserving fixtures, and rare cases. It does not establish performance for real speakers, devices, networks, or conversational behavior.