Build a streaming TTS torture test for dates, money, phone numbers, and promo codes

Try this in your streaming TTS stack:

“Your pickup is on 03/04/26. The total is ₹1,050.50. Use code SAVE20 at example.com/r/7B.”

Now listen without reading the text. Did the date mean March fourth or April third? Did the voice say rupees? Did it spell the code? Did the URL become mush?

This tutorial gives voice engineers a repeatable bake-off for the strings that polished demos avoid. You can run it across providers, voices, locales, and normalization strategies, then keep the winning cases as regression tests.

Naturalness scores will not save you

The April 2026 text-normalization discussion on r/MachineLearning started from a familiar frustration: streaming TTS benchmarks often focus on how pleasant a voice sounds while everyday structured text breaks.

Pleasant voices matter, but correctness comes first when the agent is reading an amount, date, dosage, address, or confirmation code. Official TTS documentation exposes the same classes through SSML: Google, Microsoft, and Amazon all document ways to guide readings for at least some dates, numbers, characters, phone numbers, or currency values, but their supported tags, locales, and engine behavior differ. Test the complete stack you plan to ship.

Scroll diagram horizontally on smaller screens.

Build cases from your business, not a trivia list

Start with the strings your agent reads to customers and the regions it actually serves:

Product context	High-value cases
US retail	Dollar amounts, apartment numbers, ZIP codes, order IDs, URLs, and email addresses
Indian payments or commerce	Rupee amounts in Indian grouping, PIN codes, UPI handles, IFSC-like alphanumeric strings, and dates in English, Hindi, and Hinglish lanes
European travel	Day-month dates, decimal commas, booking codes, local phone formats, and city or station abbreviations

Regional texture belongs in the suite for the markets you serve, not in a generic trivia collection. Mine cases from production templates, catalog data, failed-call reviews, support tickets, and the structured fields your tools return. Remove or replace customer data under your approved process, but preserve the shape that made the string difficult: leading zeroes, mixed scripts, repeated characters, unusual separators, or an abbreviation next to a number. A tidy collection invented in a spreadsheet will miss the awkward strings your systems actually produce.

Balance frequency and consequence. Common amounts and dates deserve broad coverage because callers hear them every day. A rare medication dosage, refund total, or authentication code may deserve just as much attention because one wrong reading carries more risk. Tag both dimensions so release sampling does not select only the most frequent classes.

Include ordinary sentences around each entity. A provider may read ₹1,050.50 correctly in isolation and rush it when it arrives between a long introduction and a follow-up question. Add cases where two structured values appear together, such as a date followed by a confirmation code, because punctuation and prosody can leak across the boundary.

Ambiguous text should stay ambiguous in the test

Do not quietly choose an answer for 03/04/26. Create at least three cases:

• Locale resolves it to month-day-year.
• Locale resolves it to day-month-year.
• Context is missing, so the expected behavior is a clarification before speech.

The third case tests the application as well as TTS. Consider a fictional hotel agent serving London and New York: the backend sends 03/04/26 without a locale, the voice confidently reads “March fourth,” and the London booking was for 3 April. TTS produced exactly what it was given, but the product still failed. The example is fictional, and the test should mark the missing locale as an upstream defect.

Keep upstream failures visible in the same report. If the normalizer receives an untyped value, the application drops the locale, or the model changes the canonical digits, labeling the final audio as a TTS error sends the team to the wrong component. Record the last trustworthy representation at every boundary: source data, display text, normalized text or SSML, synthesized audio, and what the listener recovered.

Include code-switching and pronunciation locks

Test a Hindi sentence that contains an English product name, a rupee amount, a coupon that must be spelled, a date, and a PIN code. Run the same case with Hindi-dominant and English-dominant voices if both are in production, and mark where language tags or lexicon entries are needed.

Keep a versioned pronunciation lock file for brand names, medicines, place names, and acronyms, and rerun it when the voice model changes. Also review whether the voice changes accent or rhythm around an inserted entity. A Hindi sentence that pauses awkwardly before every English product name can remain technically intelligible while sounding like stitched clips. That may be acceptable for a critical code and poor for a common menu item, so the rubric should separate semantic correctness from flow.

Give each case a testable contract

Each test case needs seven core fields. This example shows the contract:

Field	Value
Canonical value	INR 1050.50
Display text	₹1,050.50
Entity type	currency
Locale	en-IN
Intended speech	“one thousand fifty rupees and fifty paise”
Allowed variant	“one thousand and fifty rupees, fifty paise”
Forbidden	“one rupee, fifty point five zero”

Allowed variants prevent a brittle test, while forbidden readings protect meaning. Add the source of truth and the reason for the expected reading as provenance metadata. If the date comes from an ISO field and the caller locale is en-IN, store both. If a product code must be spelled because the brand team approved that form, link the lexicon version. A reviewer should not have to infer why one reading passes and another fails.

For values a caller may repeat or type elsewhere, include a write-back task. Play the audio without showing the text and ask the reviewer to enter what they heard. Exact reconstruction is a stronger signal for a PIN code or phone number than a general intelligibility score. Keep this separate from preference questions about whether the voice sounded pleasant.

Run four input modes

A reproducible bake-off compares four input modes rather than raw text alone.

Raw display text is the control. Send the exact string the application would show on screen so the result reveals what the TTS engine handles by itself, without help from the application.

Deterministically normalized text tests the production rewrite. Convert known entities into reviewed spoken text, keep the canonical value beside it, and verify that the rewrite did not alter the data.

SSML tests the controls a provider exposes. Use supported say-as, phoneme, substitution, or language tags. Microsoft’s current guide, for example, documents characters, alphanumeric, number_digit, date, telephone, and currency categories. Amazon Polly’s guide documents a different support matrix and engine caveats. Record the exact markup because “SSML mode” is not reproducible without it.

Adversarial stream chunks expose timing and boundary failures that completed text hides. Feed the value across bad token boundaries:

Currency:
₹
1,
050
.50

Code:
SAVE
20

Your production system receives streams rather than completed benchmark sentences, so the boundary is part of the test. Generate chunk patterns systematically by splitting before and after symbols, decimal marks, slashes, hyphens, language changes, and transitions between letters and numbers. Then replay boundaries observed from the production model, since a theoretical worst case and the provider’s actual token stream may differ.

Keep token arrival timing as part of the fixture. A buffer can handle four fragments that arrive within 20 milliseconds and fail when the last fragment arrives 300 milliseconds later. Test steady streams, bursty streams, and a stalled entity followed by a new sentence. The expected behavior may be to wait, clarify, or stop the phrase, but it should be defined.

Control the comparison before you render

For each provider:

• Pin the voice and model version where possible.
• Use equivalent audio encoding.
• Keep speed and style settings fixed.
• Test supported SSML, then raw text.
• Separate first-run and warm-run latency.
• Note unsupported tags instead of treating them as model failures.
• Save the rendered audio.

Use blind listening for the final comparison. Randomize filenames and case order so reviewers do not reward a familiar vendor or anticipate the next value.

Capture the timing honestly

For every render, record:

• Time to first audio.
• Time to first entity audio.
• Full render time.
• Buffer time added by normalization.
• Number of re-synthesized chunks.
• Whether the agent spoke before the entity was resolved.

Run several trials because a single warm request says very little about the tail. Use the same region, network path, audio format, sample rate, and concurrency, and log provider request IDs when available. Warm the connection in a controlled way and label the run. First-call latency matters for low-volume workflows, while warm latency may dominate a busy restaurant line. Mixing both into one percentile creates a number that cannot be reproduced. Also separate provider synthesis time from application buffering and phone delivery so a slow result reaches the right owner.

Grade meaning before beauty

Use a four-gate ladder so a pleasant voice cannot compensate for altered data.

Gate 1, semantic correctness: did the audio preserve the value? For sensitive fields, this is pass or fail. The rendered amount, date, address, or code must match the canonical input after permitted formatting differences.

Gate 2, intelligibility: could a listener write the value back correctly? Use several reviewers where accents or language switching matter, and hide the expected answer until after transcription.

Gate 3, pacing: are digits rushed, or are pauses so wide that one code sounds like several unrelated fields? Review both the whole phrase and the entity span because natural sentence pacing can still crush a confirmation number.

Gate 4, naturalness: only now score voice quality. This can guide provider or voice selection after the first three gates pass, but it should never rescue a wrong or unrecoverable value.

Use at least two reviewers for high-consequence or mixed-language cases and resolve disagreements against the written rule. Reviewer familiarity can bias results: someone who already knows the code may “hear” the intended characters through unclear audio. Include decoy variants that differ by one digit or letter.

Automated transcription can screen a large corpus, but it should not be the only judge. An ASR model may recover a number that a caller cannot, or fail a correct pronunciation because it expects another written form. Compare normalized meaning, write-back accuracy, and human review for the sensitive slice.

Make the release decision by entity and locale

Summarize the bake-off by entity class and locale rather than announcing one provider winner. One voice may handle Indian currency and English product names well but struggle with URLs; another may spell codes clearly and add too much startup delay. The right choice can vary by lane, or it can require your own normalization layer in front of more than one provider.

Before release, set non-negotiable gates for meaning-sensitive classes. A wrong amount, date, dose, identity code, or address component should not be balanced by ten natural product names. Lower-risk pronunciation issues can carry an owner and retest date, but the report should retain the exact failing audio and input path.

When a case fails, inspect the streaming normalization boundary before blaming the voice. If it only fails under pressure, the voice latency trace will show whether chunk timing changed the reading. The spoken-data TTS torture pack includes fictional starter rows and columns for all four input modes, latency evidence, allowed speech, forbidden speech, reviewer notes, and release status.

Replace at least half the sample rows with your own strings before trusting the result. The starter pack teaches the method; your tool outputs, menus, codes, locations, and customer language mix are where the useful failures live. For more voice AI test methods you can run against your own stack, subscribe to Voxeval.