arXiv · 2025 · Preprint
Heyang Liu et al. (Shanghai Jiao Tong University, Ant Group) · → Paper · Demo: ✗ · Code: ?
Introduces a benchmark and training recipe that expose and partially close a large performance gap in speech-to-speech interaction models when queries mix Mandarin and English.
Problem
Multimodal speech-to-speech interaction systems have reached strong monolingual performance in both Mandarin and English, but real conversational speech frequently mixes languages within a single utterance: foreign named entities, loanwords, and technical terms embedded in an otherwise Mandarin sentence. Prior evaluation work on end-to-end spoken dialogue models has focused on monolingual settings or, where multilingual data exists, on simplistic word-level translation rather than naturalistic code-switching. As a result, it is unknown how badly speech-to-speech models degrade under realistic bilingual input, and no training recipe exists to address the specific failure modes (mishearing embedded foreign terms, mispronouncing them in the response, or reverting entirely to the wrong language).
Method
The paper contributes both an evaluation resource and a training recipe built on top of an existing speech-to-speech architecture.
CS3-Bench is a Mandarin-centric benchmark in which individual instances contain embedded English content. It has two subsets: a knowledge set (362 instances, factual questions where the target answer or a key entity is an English term) and an open-ended set (200 instances simulating naturalistic bilingual conversation). Knowledge instances are categorized as acoustic (phonetically salient proper names), semantic (concept-level alignment), or integrated (both). Queries are synthesized into speech with CosyVoice2, using speaker prompts drawn from a clean subset of seedtts-eval, and transcribed back with Whisper-large-v3 for quality control, with manual correction of any mismatches. Evaluation covers three axes: semantic accuracy (keyword spotting, falling back to LLM-as-a-Judge), pronunciation clarity (a newly defined pronunciation success rate, PSR, the fraction of English words in the response recognized by an ASR system, plus WER on the English segments), and language selection accuracy (LSA, whether Mandarin remains the dominant response language).
Starting from VocalNet-ML, a bilingual speech-to-speech model built on a Qwen2.5-7B-Instruct backbone with extended speech encoding and a multi-token-prediction (MTP) speech generation head, the authors apply two interventions. First, language alignment data construction: Mandarin conversations from moss-003-sft-data are rewritten to embed English terms by a Rewriter LLM and filtered by a Checker LLM for naturalness, then synthesized and re-verified through the same TTS/ASR round-trip used for the benchmark, yielding 19.4k code-switching instances; this is combined with 24k CoVoST-derived speech-translation instances and matched speech-instruction data from VoiceAssistant400K and Belle. Second, two training-time mechanisms operate on top of this data: Chain of Recognition (CoR) inserts a pre-task in which the model first recognizes the English content of the query, chains this recognition (bounded to four words) before the response using a special separator token, and discards it at inference — analogous to chain-of-thought prompting but applied to cross-lingual understanding. Keyword Highlighting (KH) operates on the generation side: the LLM backbone is LoRA fine-tuned to emit indicative tokens around English-content keywords in its text output, and the frozen speech decoder is trained to attend to these highlighted tokens so that the corresponding speech tokens are pronounced correctly in both languages. During training, the speech encoder and speech vocoder remain frozen; only the LLM backbone (via LoRA) and the mechanisms above are updated, on 4 A100 GPUs.
Key Results
On CS3-Bench, all seven evaluated models (GLM-4-Voice, Qwen2.5-Omni, Kimi-Audio, VITA-Audio-Plus-Vanilla, Baichuan-Omni-1.5, MiniCPM-o 2.6, and the VocalNet-ML baseline) perform far better on the monolingual (English) version of the knowledge set than on the code-switched Mandarin version. Six of the seven show an absolute accuracy drop of 30 points or more; VocalNet-ML falls from 75.41% to 25.14% (relative drop up to 66%, the largest of the group), while VITA-Audio-Plus-Vanilla is comparatively more robust at a 12-point absolute drop. Language selection accuracy stays above 95% for every model, indicating the failure is in content and pronunciation rather than language choice. In open-ended conversation, roughly half the models fail to adequately understand more than 20% of queries.
Applying the proposed data and training interventions to VocalNet-ML raises knowledge-set accuracy from 26.24% to 41.99% with combined speech-translation and code-switching data alone, to 43.37% with CoR added, and to 46.13% with KH added on top, alongside reduced English-segment WER. Open-ended understanding rate rises from 64.5% to a peak of 89.5% with code-switching data alone (86.5% with CoR, which also yields the best open-ended pronunciation WER of 15.18% among all reported configurations, including the seven baselines). The enhanced model still trails Qwen2.5-Omni and Kimi-Audio in absolute knowledge accuracy, which the authors attribute to those models’ much larger training-data volume (millions of hours vs. thousands). On monolingual VocalBench and VocalBench-zh, the same interventions also improve scores, indicating the added data does not harm (and modestly helps) non-code-switched performance. Applying CoR and KH simultaneously degrades results relative to either alone, which the authors attribute to inconsistency between the words the model recognizes and the words it ultimately generates.
Novelty Assessment
The benchmark itself is the paper’s clearest contribution: no prior evaluation resource specifically targets naturalistic code-switching in speech-to-speech (rather than speech-to-text) interaction, and the three-axis metric design (semantic accuracy, pronunciation success rate, language selection accuracy) is tailored to that setting. The enhancement methods are comparatively incremental from an architectural standpoint: CoR is a chain-of-thought-style reformulation of the recognition step, and KH is a keyword-conditioning mechanism realized through LoRA fine-tuning and indicative tokens, both applied to an existing speech-to-speech architecture (VocalNet-ML) rather than a new one. The contribution is best characterized as a training-recipe and evaluation-methodology advance rather than an architectural one; the value lies in isolating and partially fixing a specific, previously unmeasured failure mode rather than in introducing new model structure.
Field Significance
Moderate — this paper is the first to systematically quantify code-switching degradation in speech-to-speech interaction models specifically (as opposed to speech recognition or translation), and it demonstrates that the failure is substantial even in models that otherwise perform well monolingually. Its training recipe provides a concrete, replicable path to partially closing that gap, but the underlying architecture and most training techniques (LoRA fine-tuning, chain-of-thought-style auxiliary tasks) are not new, and the recipe alone does not close the gap to models trained on far larger corpora.
Claims
- supports: Speech-to-speech interaction models that perform well on monolingual queries can degrade substantially when the same content is presented in a code-switched (mixed-language) form, even though the underlying semantic content is unchanged.
Evidence: Across 7 mainstream speech-to-speech models, 6 show an absolute knowledge-accuracy drop of 30 percentage points or more moving from the English monolingual version to the Mandarin-English code-switched version of the same instances, with the worst case (VocalNet-ML) dropping from 75.41% to 25.14% (a 66% relative drop). (§2.3, Table 1)
- supports: Fine-tuning a speech-to-speech model on constructed code-switching and speech-translation data measurably improves its ability to understand and correctly pronounce mixed-language content, without degrading monolingual performance.
Evidence: Adding 19.4k code-switching and 24k speech-translation instances raises VocalNet-ML’s CS3-Bench knowledge accuracy from 26.24% to 41.99% and open-ended understanding from 64.5% to 86.0-89.5%, while also improving scores on the monolingual VocalBench and VocalBench-zh test sets. (§3.1, §4.2, §4.3, Table 2, Table 3)
- refines: Explicitly conditioning the speech decoder on recognized or highlighted foreign-language keywords, rather than relying on the language model’s implicit understanding, further improves pronunciation accuracy of code-switched speech output beyond what data augmentation alone achieves.
Evidence: Adding Keyword Highlighting on top of the language-alignment data and Chain of Recognition raises knowledge accuracy from 43.37% to 46.13% and produces the largest gains specifically on semantic and integrated question types, consistent with the mechanism’s role in signalling which keywords the speech decoder must render correctly. (§3.3, §4.2, Table 2)
- complicates: Combining multiple auxiliary training objectives intended to independently improve understanding and generation of mixed-language content can interfere with each other rather than compound their individual gains.
Evidence: Applying Chain of Recognition and Keyword Highlighting simultaneously degrades performance relative to applying either alone, attributed to inconsistency between the English items the model recognizes during the auxiliary task and the items it ultimately generates in its response. (§4.2)
Limitations and Open Questions
Warning
The enhancement methods are validated on a single base architecture (VocalNet-ML, a 7B-parameter model) fine-tuned with comparatively small data volumes (tens of thousands of instances); the paper itself attributes its remaining gap to larger commercial models (Qwen2.5-Omni, Kimi-Audio) primarily to training-data scale, leaving open whether CoR and KH would transfer their benefits, or still leave a gap, at that scale.
The benchmark’s own construction pipeline relies on Whisper-large-v3 for filtering and evaluation, and the authors note that Whisper has inherent limitations recognizing code-switched content, which complicates the precision of the pronunciation-quality metrics derived from it. The combination of CoR and KH is reported to be unstable when applied jointly, and the paper does not resolve this interaction beyond noting it as a source of degradation. CS3-Bench itself covers only Mandarin-English code-switching; whether the deterioration pattern and the proposed fixes generalize to other language pairs is untested.
Wiki Connections
- Spoken Language Model — evaluates and fine-tunes a speech-to-speech LLM (VocalNet-ML) that consumes an external speech query and generates a speech response, and introduces training objectives (CoR, KH) specific to that setting.
- Multilingual TTS — targets a bilingual (Mandarin-English) interaction setting and constructs code-switching training data to improve cross-lingual generation quality within a single model.
- Speech-to-Speech — the benchmark and enhancement methods are defined specifically for end-to-end speech-to-speech interaction, distinguishing degradation in generation and pronunciation from degradation in recognition alone.
- Evaluation Metrics — introduces pronunciation success rate (PSR) and language selection accuracy (LSA) as new metrics tailored to measuring code-switching quality in generated speech.
- GLM-4-Voice — evaluated as one of seven baseline speech-to-speech models on CS3-Bench, showing significant code-switching degradation.
- Qwen2.5-Omni — evaluated as a baseline; achieves the strongest open-ended pronunciation accuracy among the seven baselines and outperforms the enhanced VocalNet-ML on absolute knowledge accuracy.
- Kimi-Audio — evaluated as a baseline; along with Qwen2.5-Omni, cited as evidence that larger training-data volume closes the code-switching gap that CoR and KH only partially address.
- Baichuan-Omni-1.5 — evaluated as one of the seven baseline speech-to-speech models on CS3-Bench.
- CosyVoice 2 — used as the TTS system to synthesize both the CS3-Bench benchmark queries and the constructed code-switching training data.
- Seed-TTS — its seedtts-eval speaker-prompt subset is used as the source of voice prompts for synthesizing CS3-Bench speech queries.
- SLAM-Omni — its VoiceAssistant400K dataset is used as part of the speech-instruction training data for the enhanced model.