Hindi-English voice agents: how to test code-switching

The Hindi voice-agent demo works beautifully when a tester reads a clean Devanagari script. Then a real caller says, “Mera order kal deliver hona tha, but address mein flat number twenty-six hai,” and three systems make three different guesses about the language, date, and number.

If you are building voice AI for customers in India, that is not an edge case to patch later. It is the product.

This guide shows engineering and product teams how to define a Hindi-English test lane across code-switching, names, numbers, scripts, telephony, tool arguments, pronunciation, and caller repair. You will get a practical matrix without pretending that one “Hinglish” dataset represents every bilingual speaker in India.

The short answer

Test Hindi-English voice agents as an end-to-end workflow, not only an ASR language setting. Preserve the chain from audio to transcript, normalized entities, tool arguments, agent speech, and final business state. Break results out by speaker, switch pattern, script policy, audio route, and workflow.

Do not use one bilingual score as a proxy for India. Hindi-English is one defined product lane. Speakers differ by region, first language, education, age, occupation, topic, and personal habit. Some switch entire turns. Others insert English product terms into Hindi syntax. Some expect a Devanagari transcript, some Romanized text, and many never see the transcript at all.

The system should work for the lane you claim to support, and the release evidence should say what remains untested.

Start by defining “Hindi-English” for your product

“Hinglish” is a convenient label, but it can hide several different behaviors:

• A Hindi sentence with an English noun: “Mujhe refund status chahiye.”
• An English sentence with a Hindi phrase: “The payment is showing pending, ab kya karun?”
• A switch between turns: the caller asks in Hindi, then confirms details in English.
• A switch inside an entity: an English brand name, Romanized address, or alphanumeric code inside Hindi speech.
• Indian English with no Hindi words, spoken by someone whose first language may be Hindi or another language.
• Hindi mixed with a third language, which is not a Hindi-English-only case.

Decide which of these are supported. Name the expected response behavior too. Should the agent mirror the caller’s current language, keep one language, ask a preference, or route to another lane?

There is no polite universal default. Test the choice with the people your product serves.

Scroll diagram horizontally on smaller screens.

Why clean monolingual testing gives false confidence

India’s speech diversity is not a minor accent variation around one standard speaker. The IndicVoices project collected 7,348 hours across 22 Indian languages, 16,237 speakers, and 145 districts, with read, extempore, and conversational speech. The authors built the dataset to capture linguistic, cultural, and demographic breadth.

That scale is a reminder, not proof that your agent is covered. IndicVoices is a research dataset with its own collection and transcription rules. It does not reproduce your callers, telephony, workflow, or language mix.

VoiceAgentBench evaluates agentic tasks in English and six Indic languages, with separate attention to tool selection, call structure, parameters, and safety. It is based on synthetic queries and does not settle how a deployed Hindi-English call will behave.

The 2026 Voice of India preprint focuses on unscripted telephonic speech across 15 languages and regional clusters. It argues that strict single-reference WER can penalize valid spelling variation in code-mixed terms. The benchmark is closed-source, so teams cannot treat it as a reproducible public replacement for their own data.

The shared lesson is clear: test natural speech, real audio conditions, and business behavior together.

Build a matrix around switch behavior

Do not start by collecting random bilingual sentences. Start with the workflows and moments where a switch changes risk.

For a fictional courier-support agent in Bengaluru, a caller might say:

“Delivery kal honi thi. Address is 26, Second Cross, near Indiranagar Metro. Please rider ko call karne bolo.”

This is an invented example, not a real call. It contains several test targets:

• kal needs context because it can refer to yesterday or tomorrow.
• 26 arrives as an English number inside a Hindi request.
• “Second Cross” and “Indiranagar Metro” should stay as address entities.
• “rider ko call karne bolo” asks for an action, not only information.
• The agent must not claim it contacted the rider unless the tool succeeds.

Build cases across at least these switch patterns:

Pattern	Example target	Main risk
English noun inside Hindi	refund, delivery, premium	Domain term deleted or translated incorrectly
Hindi phrase inside English	ab kya karun	Intent weakened or ignored
Whole-turn switch	Hindi question, English confirmation	State or language route resets
Number-language switch	“flat number twenty-six”	Wrong normalized digits
Script switch	Romanized Hindi input or output	Invalid matching and search
Correction across languages	“Friday nahi, Thursday”	Stale value survives
Third-language intrusion	Hindi-English lane receives Marathi or Tamil	False confidence instead of safe recovery

Add natural pauses, self-corrections, interruptions, and background speech. Code-switching in a read sentence is not the same as code-switching while someone is checking a message, standing near traffic, or correcting an address.

Test names as business-critical data

Names in India can expose every weak assumption in a speech pipeline: varied phonology, multiple common spellings, initials, honorifics, compound names, and transliteration between scripts.

Create an approved name set that reflects the people and records in your lane without using live personal data. Include:

• Short and long names
• Initials spoken separately
• Names with several accepted Roman spellings
• Similar-sounding names that map to different fixtures
• Names embedded in Hindi and English sentences
• Caller corrections and spelling
• Family and given-name order used by the workflow

Do not score only the transcript string. Check whether the correct record was retrieved and whether the agent confirmed ambiguity before revealing data.

A phonetic match can help find candidates. It should not become identity proof.

Test numbers in the forms callers use

Build cases for:

• English digits: “two six zero five”
• Hindi digits: “do chhe shunya paanch”
• Mixed sequence: “two six shunya five”
• Grouped mobile numbers
• Lakhs and crores where the workflow uses them
• Decimal amounts and rupees
• Dates expressed with month names or relative words
• Times in 12-hour and 24-hour form
• Alphanumeric booking and policy codes

For high-risk numbers, separate four checks:

1. What the caller said in the audio
2. What the recognizer produced
3. What normalization stored
4. What the tool received

Then verify the final record. A perfect transcript can still fail if a normalizer changes O to 0 or drops a leading zero.

Confirmation design matters. Reading back an entire 16-digit sequence can increase effort and expose sensitive data. Use the workflow’s approved masking and chunking rules.

Choose a script and normalization policy

The same spoken Hindi-English phrase can appear in Devanagari, Roman script, or a mix. Decide what each system boundary expects.

Questions to answer:

• Is the transcript for agents, search, audit, or only internal model context?
• Are Hindi words stored in Devanagari, Romanized form, or both?
• How are English brand and product terms preserved?
• How are valid alternate spellings matched?
• Does tool input use original text or a normalized identifier?
• Can reviewers search the call using either script?
• Is the transformation reversible enough to audit a failure?

Do not hide script conversion inside a generic cleanup step. Version it and test it.

Provider multilingual behavior varies. For example, Google’s current troubleshooting page says its referenced Speech-to-Text mode supports one language per request for mixed inputs and recommends checking model and language configuration. Other providers expose multilingual or language-detection modes with different limits. Verify current official docs for the exact model and region, then test the claim on your audio.

LiveKit’s February 2026 multilingual tutorial shows one implementation pattern: read the language attached to speech events and update the text-to-speech configuration when it changes. That is a useful build technique, not evidence that the listed languages, mixed utterances, pronunciations, or tools pass your workflow. Implementation and evaluation remain separate jobs.

Measure ten layers, not one bilingual WER

1. Intent and permitted goal

Did the system understand what the caller wanted, even when the action phrase crossed languages?

2. Code-switch boundary preservation

Did words disappear, translate, or attach to the wrong phrase around switch points?

3. Critical names and entities

Were names, places, products, dates, amounts, and identifiers correct?

4. Script and normalization

Did the stored form preserve meaning and map to the right business value?

5. Tool selection and arguments

Did the language switch change which function ran or which value it received?

6. Final business state

Did the correct delivery, booking, claim, or account record change as intended?

7. Output pronunciation

Could the caller understand names, amounts, abbreviations, and mixed-language phrases after synthesis?

8. Conversational repair

When the system missed something, did it ask a targeted question in a language the caller understood?

9. Caller effort

How many repeats, spellings, language switches, or human requests were needed?

10. Timing and turn-taking

Did language detection, translation, or model routing add delay or cause early turn endings?

Keep results by stratum. One Hindi-English average can hide a complete failure for mixed number sequences or callers on a particular phone route.

Test the voice, not just understanding

The agent’s output can be wrong even when its reasoning is right.

Listen for:

• Hindi phonemes flattened into English pronunciation
• English product names distorted inside a Hindi sentence
• Wrong stress that changes a name or number
• Awkward pauses at language boundaries
• A voice that switches accent or identity mid-call
• Prosody that makes a confirmation sound like a completed action
• Unreadable speed for codes and addresses

Use human reviewers who understand the language mix and context. An evaluator fluent in formal Hindi but unfamiliar with the lane’s everyday switching may grade natural speech poorly or miss an unnatural phrase.

When exact pronunciation matters, include audio examples and accepted variants in the rubric. Do not reduce the judgment to “sounds native.” That label is vague and can encode bias.

Separate browser and telephony results

A laptop microphone over WebRTC and a mobile call over PSTN are different products from the recognizer’s point of view.

Run the same core cases through:

• Clean local audio
• Your browser or app transport
• Your real inbound and outbound phone routes
• Representative mobile networks and regions
• Background conditions that match use

Track clipping, packet impairment, echo, channel mix, and the recognizer’s finalization behavior. If an 8 kHz route damages mixed-language entities, a clean 48 kHz browser demo should not average that problem away.

Create a safe recovery policy

The agent will meet speech it does not support. Decide what good failure looks like.

It may:

• Ask a narrow confirmation
• Offer Hindi or English explicitly
• Switch to DTMF for a critical number
• Transfer to a language-skilled human
• Schedule a callback
• Explain that it cannot complete the request safely

It should not fake understanding and act on a guess.

Test the escalation itself. Confirm destination, queue hours, context transfer, caller explanation, and fallback when the queue is unavailable. Our guide to AI-to-human handoff testing covers the complete path.

Report the claim honestly

A useful release note might say:

This agent version passed the approved Hindi-English delivery-status suite across the listed browser and India phone routes. The suite covers whole-turn and within-sentence switching, approved names and addresses, English and Hindi number forms, caller correction, and human fallback. It does not establish support for other Indian languages, Hindi mixed with a third language, or untested regions and devices.

That is more credible than “supports Hinglish.”

Use the STT bake-off method when choosing the recognition configuration. Use the voice-agent scorecard to keep the final evaluation tied to task and policy.

Where Voxeval fits

Voxeval is being built to let teams define language lanes, audio conditions, entity sets, tool assertions, reviewer rubrics, and final-state checks without collapsing them into one multilingual score. The framework works without Voxeval too.

The principle matters more than the tool: support claims should be as specific as the evidence behind them.

Common questions

Is Hindi-English the same as Hinglish?

People often use “Hinglish” for Hindi-English mixing, but the label covers different switch patterns, scripts, and speaker communities. Define the exact behavior your product supports instead of relying on the label alone.

Should the agent always mirror the caller’s language?

Not automatically. Mirroring may feel natural in one workflow and confusing in another. Choose a language policy with user research, then test switching, consistency, comprehension, and escalation.

Is bilingual WER enough?

No. Add switch-boundary errors, critical entities, script normalization, tool arguments, pronunciation, repair, caller effort, timing, and final outcome. WER remains a useful diagnostic.

Can synthetic voices cover code-switching tests?

They can add controlled combinations and privacy-safe fixtures. They do not prove performance with spontaneous bilingual speakers, real telephony, local pronunciation, or natural repair behavior.