Interspeech · 2026 · Conference
Szu-Chi Chen et al. (National Taiwan University, NVIDIA) · → Paper · Demo: ✓ · Code: ?
Fine-tuning a frozen AudioLLM with five emotion-specific LoRA experts and a soft router lets English-Chinese speech-to-speech translation reproduce laughter and crying, which existing S2ST systems almost never preserve.
Problem
Speech-to-speech translation (S2ST) systems keep semantics but drop non-verbal vocalizations (NVs) such as laughter and crying, which carry pragmatic intent. Two obstacles are named: expressive S2ST training data with authentic NVs is scarce and noisy, and a single model has difficulty representing similar or conflicting affective states without interference. The paper also asks how much data a pretrained AudioLLM needs before it can perform S2ST at all.
Method
The work has three parts. First, a data synthesis pipeline builds en-zh parallel expressive speech from GigaSpeech/GigaST text using IndexTTS2 (IndexTTS2). Prompts come from emotional corpora (Happy, Sad, Angry), a laughter detector with manual verification, and crying clips from JVNV. For laughter and crying, the NV prompt supplies the paralinguistics and a separately sampled neutral prompt supplies speaker identity. Outputs are filtered by silence trimming, Whisper-small WER (threshold 0.5), and a pair-level filter requiring both the en and zh utterances to pass, yielding a 1000-hour corpus.
Second, MoVE builds on Kimi-Audio (Kimi-Audio) with the base LLM frozen. Five parallel LoRA experts (Happy, Sad, Angry, Laughing, Crying; rank 256) sit on the attention and gate projections of every transformer layer, in the style of X-LoRA. A linear softmax router assigns a continuous weight to each expert per speech token, so hybrid states such as nervous laughter can be blended instead of hard-routed. The pretrained detokenizer is also fine-tuned on NV-rich synthetic speech, and the Whisper encoder is fine-tuned on the expressive corpus.
Third, training has two stages. Stage 1 trains each expert independently on its own expressive subset (2 epochs). Stage 2 trains only the router on the full data (1 epoch) with the language modeling loss, with no emotion labels.
Key Results
On CVSS-T, MoVE reaches ASR-BLEU 32.5 (en to zh) and 21.4 (zh to en), and Aro-Val similarity of 0.53 on a NonverbalTTS-derived set. It trails SeamlessM4T-Large-v2 on zh to en (23.6) (§3.2, Table 1). In the human evaluation (N = 5 bilingual raters, 30 utterances), MoVE scores Naturalness MOS 3.85 and Emotion SMOS 3.79, the highest among all compared systems including the cascaded reference. NV match accuracy is 76% versus 14% for SeamlessExpressive and 2-4% for the other end-to-end baselines. The cascaded oracle, which uses the source audio as a prompt, reaches 26% (Table 1). In a pairwise test MoVE is preferred over the best single-LoRA model in 60% of cases, with 22.67% ties and 17.33% preference for single-LoRA (Table 2).
Note
Subjective scores rest on only five raters and 30 utterances, and the cascaded oracle is a different system paradigm, so it is a reference rather than a like-for-like baseline.
Data comparisons with a single LoRA show that even 50 h of the synthesized corpus outperforms 100 h SynStard and 67 h SeamlessAlignExpressive on ASR-BLEU and Aro-Val similarity. A scaling sweep (0.1 h to 1000 h) shows a plateau from 0.5 h onward after a sharp drop at 0.1 h. Re-initializing the base model from scratch fails to converge at any data size (§3.3, Figure 2).
Novelty Assessment
The contribution is primarily engineering integration. Mixture-of-LoRA-experts with soft routing is adapted from X-LoRA, and the AudioLLM backbone, TTS engine and detokenizer are existing components. The new elements are the application to NV-preserving S2ST, the two-stage expert-then-router recipe, and a released synthetic corpus whose value rests on the data-comparison experiments. The “first to fine-tune general AudioLLMs for end-to-end S2ST” claim is the authors’ own and is not verified here. The MoVE-versus-single-LoRA gap on objective metrics is small, and the architectural benefit appears mainly in the small human study. Evaluation is limited to one language pair, five expressive categories, and synthetic training data.
Field Significance
Moderate. The paper gives evidence that expressive paralinguistics can be recovered in S2ST by adapting a pretrained AudioLLM with very little data, and that the knowledge is activated from pretraining rather than learned by the adapter. It also shows the training-data quality of synthetic expressive corpora can matter more than volume. The architectural claims rest on a small listening test.
Claims
- supports: Pretrained audio language models can be adapted to speech-to-speech translation with very small amounts of task data because the adapter activates pretrained knowledge rather than creating it.
Evidence: Single-LoRA performance plateaus from 0.5 h through 1000 h of data, while re-initializing the base model from scratch fails to converge and outputs babble at every data size. (§3.3, Figure 2)
- supports: Training data quality, rather than volume, drives expressiveness and fidelity in adapted speech translation models.
Evidence: Under an identical single-LoRA setup, 50 h of the synthesized corpus beats 100 h SynStard and 67 h SeamlessAlignExpressive on ASR-BLEU and Aro-Val similarity. (§3.2, Table 1)
- supports: Separating expressive categories into independent low-rank experts with soft routing preserves non-verbal vocalizations better than a single shared adapter.
Evidence: MoVE obtains 76% NV match against 26% for the single-LoRA model trained on the same 100 h, and is preferred in 60% of A/B comparisons (17.33% for single-LoRA). (§3.2, Table 1, Table 2)
- complicates: Existing speech translation systems, including commercial and expressive ones, largely fail to preserve non-verbal vocalizations even when semantic quality is high.
Evidence: Baseline NV match is 2-14% (SeamlessExpressive 14%) despite ASR-BLEU of 23.8-26.3 on en to zh. (§3.2, Table 1)
- complicates: A dimensional arousal-valence similarity metric did not separate systems that differ sharply in human-judged non-verbal vocalization rendering.
Evidence: On the same 100 h of data, single-LoRA and MoVE score Aro-Val SIM 0.51 and 0.53 (SE 0.03 each) and ASR-BLEU 31.2 and 32.5 (en to zh), while human-judged NV match is 26% (SE 6.0) against 76% (SE 7.48). NV match rests on 5 raters, and the paper does not report how many clips in the 306-sample Aro-Val set contain laughter or crying. (§3.2, §3.3, Table 1)
Limitations and Open Questions
The evaluation covers English-Chinese only, five expressive categories, and a synthetic training corpus produced by one TTS engine, so the generalization to real expressive speech, other NV types, and other languages is untested. The authors acknowledge they do not cover all emotions and NV categories (§4). Subjective evaluation uses five raters on 30 utterances. The cascaded oracle’s low ASR-BLEU (9.7 and 10.6) is not explained, and MoVE’s zh to en BLEU trails SeamlessM4T-Large-v2, which the authors attribute to prioritizing expressive fidelity without an ablation showing this. The router reaches 63.68% alignment with expert categories on a development set, and the paper interprets the off-diagonal confusion as hybrid expression rather than error without independent verification (§3.4). Code availability is not stated; an audio demo and dataset are linked.
Wiki Connections
- Speech-to-Speech — an en-zh S2ST system aimed at preserving laughter and crying across translation.
- Spoken Language Model — adapts a frozen AudioLLM that consumes source speech with LoRA experts for translation.
- Emotion Synthesis — targets emotion and NV transfer with five emotion-specific experts and a soft router.
- Multilingual TTS — adapts the speech generation model across English and Chinese in both directions.
- Subjective Evaluation — uses MOS, SMOS, A/B preference and a perceptual NV match test with five bilingual raters.
- Kimi-Audio — frozen AudioLLM backbone, tokenizer and detokenizer on which MoVE is built.
- IndexTTS2 — the synthesis engine that generates the expressive training corpus.
- Seamless — SeamlessExpressive is the expressive S2ST baseline, with 14% NV match.
- SeamlessM4T — SeamlessM4T-Large-v2 is a baseline that leads MoVE on zh to en ASR-BLEU.
- GPT-4o — gpt-4o-audio-preview is a commercial baseline with 2% NV match.
- CosyVoice 3 — synthesizer in the cascaded oracle reference system.
- Whisper — used for WER filtering of synthetic data, the cascaded oracle’s ASR, and encoder pretraining.
- VALL-E X — earlier cross-lingual codec language modeling cited among S2ST systems that struggle with expressiveness.