arXiv · 2025 · Preprint
Dharma Teja Donepudi (Aggregation AI) · → Paper · Demo: ✓ · Code: ✗
Introduces an engine-agnostic orchestration pipeline that segments code-switched text by Unicode script, resolves each segment’s language and locale, and stitches together existing commercial TTS voices via SSML, without training any new speech model.
Problem
Intra-sentence code-switching, where a speaker shifts language mid-utterance, is poorly handled by both conventional monolingual TTS engines and large multilingual TTS models. Monolingual systems render foreign spans with the wrong accent or fail outright, while multilingual models, despite being trained on many languages, are typically trained on monolingual utterances and therefore lack robustness to within-sentence language alternation unless fine-tuned on scarce code-switched corpora. The paper frames the gap as a lack of a practical, deployable solution: existing academic work on code-switched TTS mostly targets either perceptual measurement of the intelligibility drop at switch points, or offline synthetic corpus generation for future model training, rather than a system that produces fluent mixed-language audio today without retraining.
Method
SFMS-ALR is a rule-based, multi-stage pipeline that orchestrates existing commercial TTS engines rather than training a new acoustic or language model. Given multilingual input text, it: (1) segments the text by Unicode script boundary (Latin, Devanagari, Han, etc.), giving each segment a preliminary language tag; (2) refines ambiguous same-script segments (e.g., shared-Latin-script languages) using a lightweight LLM-based language identifier that assigns ISO 639-1 codes, optionally guided by a user-specified language hint; (3) performs adaptive locale and voice planning, selecting a locale-appropriate voice for each segment from a target provider’s catalogue (Google TTS, Amazon Polly, Apple, or Azure) while trying to preserve timbre and gender consistency across languages; (4) applies sentiment-aware prosody normalization, in which a rule-based sentiment analyzer (using punctuation and lexical cues) estimates global and per-segment tone and adjusts SSML prosody attributes (pitch, rate, pauses) so expressive continuity is preserved across a language switch; and (5) assembles a single unified SSML document with per-segment <voice>/<lang> spans, which is submitted to the TTS engine(s) in one synthesis call when all segments share a provider, or as parallel per-engine calls that are concatenated and normalized to 16 kHz mono PCM with brief (~50 ms) boundary pauses otherwise. The prototype implementation draws all voices from Google TTS for consistency and reports typical end-to-end latency of about 0.5-1.2 seconds per utterance. Because no acoustic or language model is trained or fine-tuned, the system can add a new language by registering a new provider voice and updating locale-resolution rules.
Key Results
On a self-constructed evaluation set of code-switched sentences and short paragraphs spanning English-Spanish, English-Chinese, Hindi-English, and French-Arabic pairs, ASR-based transcription of SFMS-ALR output against reference text yielded 0.0 WER across all evaluated languages, and a 76-second multilingual sample showed a mean F0 of 262 Hz with natural pitch variation and boundary pauses around 0.6 s (§7.2). In a subjective listening test with bilingual raters using 5-point MOS, SFMS-ALR scored an average MOS of 4.3, against 3.5 for a single-voice accented baseline and 3.8 for a pretrained multilingual/bilingual voice model baseline synthesizing the same sentences (§7.3); 71% of listeners preferred native-voice switching for full clauses while 29% preferred a single continuous voice for short insertions. A qualitative demonstration further extends the approach to a single utterance spanning nine languages (English, Hindi, Kannada, Telugu, Bengali, Gujarati, German, Mandarin, Japanese), each rendered with its own provider voice (§6). The comparisons are against the authors’ own re-implementations of baseline strategies rather than against a published code-switching TTS system, and sample sizes and rater counts for the subjective test are not reported.
Novelty Assessment
The contribution is an engineering orchestration layer, not a new speech-generation architecture: script segmentation, language identification, sentiment analysis, and SSML-based voice control are all pre-existing techniques, and the paper’s synthesis step is delegated entirely to third-party commercial TTS engines. The genuinely new element is the combination of script-first segmentation with sentiment-aware prosody normalization applied specifically to maintain expressive continuity across a code-switch boundary, packaged as a single deployable, retraining-free pipeline. This is best read as a systems/engineering contribution addressing deployability rather than a research advance in code-switching TTS modeling itself; the underlying per-language speech quality is entirely inherited from whichever commercial voice is selected.
Field Significance
low - this is a practical orchestration recipe for code-switched speech using off-the-shelf commercial TTS voices, evaluated by a single author on a small, self-constructed test set. It demonstrates that script-first segmentation plus sentiment-aware prosody normalization can improve perceived naturalness over naive monolingual or generic multilingual baselines without any model retraining, which is a useful deployment pattern, but the paper does not introduce a new speech-generation architecture, training method, dataset, or evaluation methodology, and its own comparisons are against self-implemented baselines rather than established published systems.
Claims
- supports: Orchestrating per-segment native voices for code-switched text through explicit script- and language-based segmentation can outperform both a single accented monolingual voice and a pretrained multilingual model on naturalness, without retraining any speech model.
Evidence: SFMS-ALR reached MOS 4.3 versus 3.5 for a single-voice accented baseline and 3.8 for a pretrained multilingual/bilingual model baseline on the same code-switched sentences. (§7.3)
- supports: Normalizing prosody parameters (pitch, rate, pauses) according to detected sentiment across a language switch can preserve perceived expressive continuity in code-switched synthesis.
Evidence: The sentiment-aware prosody module maintained a stable mean F0 (~262 Hz) and consistent pitch range across a 76-second multilingual sample and across cross-script transitions such as French-Arabic, with listeners citing “more expressive intonation” as a strength of the proposed system. (§7.2, §7.3)
- complicates: Delegating each code-switched segment to a separate native monolingual voice, rather than a single multilingual model, trades away single-speaker vocal consistency for per-language intelligibility.
Evidence: The paper notes the resulting speech “may sound like multiple speakers” and that this trade-off would need voice conversion post-processing for applications requiring a single consistent persona, such as branded assistants. (§8.2)
- complicates: Word-level language identification for code-switching is unreliable for named entities and transliterated foreign words rendered in a shared script, independent of how well the downstream synthesis is orchestrated.
Evidence: Named entities like “Paris” can trigger unnecessary language switches, and Latin-script transliterations of foreign words (e.g., “shukriya”) can be misidentified by the language-identification stage. (§8.4)
Limitations and Open Questions
The reported 0.0 WER and MOS results come from a single-author evaluation on an unnamed, self-constructed test set with unreported participant counts and no statistical testing, and all comparisons are against the authors' own re-implementations of baseline strategies rather than published code-switching TTS systems, which limits how much the numbers generalize.
Beyond the evaluation scope, the framework’s per-language speech quality is entirely bounded by whichever third-party voice is selected, so it cannot improve on a provider’s native intelligibility or naturalness. Sentiment detection relies on punctuation and lexical cues rather than richer discourse or syntactic modeling, and the authors note this could miss context that signals an upcoming language switch. The approach also does not address single-speaker vocal consistency across languages, and its stated scalability (arbitrary numbers of languages per utterance) is only theoretical, with perceptual coherence expected to decline as more distinct voices are introduced. No code repository is released; only a live hosted demo is provided.
Wiki Connections
- Multilingual TTS — proposes a training-free orchestration alternative to end-to-end multilingual TTS models for handling intra-sentence code-switching across many language pairs.
- Prosody Control — introduces a sentiment-aware prosody normalization mechanism that adjusts pitch, rate, and pauses independently of content or speaker identity to preserve expressive continuity across language switches.
- Subjective Evaluation — validates the proposed pipeline against baseline strategies using a bilingual-listener MOS study alongside ASR-based objective intelligibility measurement.