arXiv · 2025 · Preprint
Xi Chen et al. (The Chinese University of Hong Kong; Nara Institute of Science and Technology) · → Paper · Demo: ✓ · Code: ?
A cascaded speech-to-speech translation system that predicts and preserves cross-lingual word-level emphasis, using an LLM-generated emphasis-aligned training corpus and a controllable TTS module to render the stress markers as prosodic prominence in the target language.
Problem
Speech-to-speech translation (S2ST) has moved from cascaded ASR+MT pipelines to end-to-end and discrete-unit systems that achieve strong semantic translation quality, and some recent systems (SeamlessExpressive, AudioPaLM, TransVIP) transfer general prosody and speaker timbre. However, the explicit preservation of lexical or sentential emphasis, the stress patterns that convey focus, intent, and nuance, is largely unaddressed. Prior work demonstrating joint translation of words and emphasis levels relied on small-scale, manually annotated datasets, which limits scalability and generalization to new language pairs.
Method
The system, StressTransfer, is a cascaded pipeline with two stages: an emphasis-preserving speech-to-text translation (S2TT) model and a controllable text-to-speech synthesizer. To train the S2TT model, the authors first build EmphST-Instruct, an automated data-construction pipeline that uses multiple LLMs as translation experts to generate target-language text (English to Chinese) with emphasis markers, followed by an additional LLM acting as a selection expert to choose the best candidate translation from the Stress17k and TinyStress source corpora. Human experts spot-check the synthetic data and iteratively refine the selection prompt.
The S2TT model itself consists of a Whisper-large-v3 speech encoder, a 3-layer MLP adaptor that projects and downsamples encoder features into the embedding space of a decoder-only LLM (a LoRA-fine-tuned Qwen2.5-3B), and a translation prompt that instructs the LLM to generate target-language text interleaved with emphasis tags (e.g., **word**). Training minimizes cross-entropy loss over emphasis-tagged target text; LoRA is applied to both the speech encoder and the LLM, while the adaptor is fully fine-tuned. The TTS module, CosyVoice2, a multilingual zero-shot TTS system with fine-grained prosody control (text encoder, semantic-token speech tokenizer, LLM-based token generator, conditional flow-matching token-to-mel model, and HiFi-GAN vocoder), is used off-the-shelf without fine-tuning. The predicted markdown-style emphasis markers are post-processed into CosyVoice2-compatible emphasis tags and passed as a custom prompt so the TTS module modulates pitch, energy, and duration on the marked words at inference. Inference uses beam search (beam size 5) for text generation.
To evaluate the system, the authors also construct EmphST-Bench, a 218-sample English-to-Chinese benchmark built by machine-translating and human-filtering the StressTest corpus’s test set, and adopt an “LLM-as-Judge” (GPT-4.1) protocol for scoring Sentence Stress Reasoning Accuracy (SSR) at scale.
Key Results
On text-based emphasis prediction (SSR on EmphST-Bench), StressTransfer scores 78.0, ahead of a cascaded ASR+stress-detection baseline (WhiStress + Qwen2.5-3B-Instruct, 72.9) and end-to-end proprietary LLMs prompted for translation with stress tags (Gemini-2.5-Pro, 68.4; GPT-4o, 47.4) (§5.1, Table 2). Semantic translation quality on the CoVoST2 English-to-Chinese test set is competitive with recent speech translation systems: BLEU 46.9 and COMET 0.8347, versus SeamlessM4T-Large-V2 (BLEU 35.9), SALMONN (33.1), and Qwen2-Audio-7B (45.2) (§5.1, Table 3).
For the full speech-to-speech output, subjective human evaluation of emphasis preservation and objective ASR-BLEU/UTMOS scores on the StressTest corpus show StressTransfer+TTS achieving ASR-BLEU 50.26, UTMOS 3.86, and subjective SSR 46.0, ahead of GPT-4o-audio (49.38 / 3.46 / 41.3) and Gemini-2.5-Pro+TTS (48.47 / 3.91 / 45.0), and substantially ahead of SeamlessM4T-Large-V2 (41.39 / 3.52) and SeamlessExpressive (35.06 / 2.63 / 11.0) (§5.2, Table 4). An ablation shows that training without the EmphST-Instruct data yields marginally higher BLEU (47.4 vs. 46.9) and COMET (0.8362 vs. 0.8347) but eliminates the model’s stress-aware translation capability (§5.3, Table 5). A separate consistency check finds the GPT-4.1 LLM-as-Judge protocol agrees with a seven-rater human majority-vote consensus at F1 0.9271 across two evaluated systems (§5.3, Table 6).
Novelty Assessment
The contribution is primarily a data-construction pipeline (EmphST-Instruct) and an evaluation benchmark (EmphST-Bench) rather than a new architecture: the S2TT component follows an established speech-encoder-plus-adaptor-plus-LLM recipe (the authors cite their own prior LLaST work), and the TTS component (CosyVoice2) is used entirely off-the-shelf. The novel engineering step is using multiple LLMs as translation and selection experts to synthesize large-scale, emphasis-aligned parallel data at low annotation cost, addressing the data-scarcity bottleneck that limited prior emphasis-preserving S2ST work to small manually annotated corpora. The paper’s own framing, “an initial attempt at stress-aware Speech-to-Speech Translation,” and “a simple yet effective baseline,” is consistent with this assessment.
Field Significance
Moderate — this paper provides a reusable data-generation recipe and the first dedicated benchmark for evaluating cross-lingual emphasis preservation in speech translation, and demonstrates that a cascaded LLM-based S2TT model paired with an unmodified prosody-controllable TTS synthesizer can preserve word-level stress more reliably than prompting proprietary end-to-end audio LLMs. Its contribution is data and evaluation infrastructure rather than a new architecture.
Claims
- supports: Explicitly generating and training on emphasis-aligned parallel translation data enables a cascaded speech translation model to preserve source-language stress more reliably than prompting general-purpose end-to-end audio LLMs.
Evidence: StressTransfer achieves SSR 78.0 on EmphST-Bench, exceeding GPT-4o (47.4) and Gemini-2.5-Pro (68.4) prompted directly for stress-aware translation, and a cascaded ASR+stress-detection baseline (WhiStress+Qwen2.5-3B-Instruct, 72.9). (§5.1, Table 2)
- supports: Pairing an emphasis-aware text-level translation model with an off-the-shelf prosody-controllable TTS synthesizer can transfer prosodic emphasis to synthesized target-language speech without a corresponding loss in perceived audio quality or translation intelligibility.
Evidence: StressTransfer+TTS reaches UTMOS 3.86 and ASR-BLEU 50.26 on the StressTest corpus, comparable to or exceeding Gemini-2.5-Pro+TTS (3.91 / 48.47) and GPT-4o-audio (3.46 / 49.38), while both cascaded systems substantially outperform the direct S2ST baseline SeamlessExpressive (2.63 / 35.06). (§5.2, Table 4)
- complicates: Training data optimized for a secondary prosodic objective (emphasis preservation) can trade off marginally against standard semantic translation metrics.
Evidence: Training without the EmphST-Instruct emphasis-aligned data yields slightly higher BLEU (47.4 vs. 46.9) and COMET (0.8362 vs. 0.8347) on CoVoST2 en-to-zh, though omitting the data removes the model’s stress-aware translation capability entirely. (§5.3, Table 5)
- complicates: Benchmarks for cross-lingual prosodic phenomena such as emphasis remain small and narrow in language coverage, limiting confidence in how well reported gains generalize.
Evidence: EmphST-Bench, the paper’s own evaluation resource, contains only 218 parallel utterances restricted to a single language pair (English source, Chinese target), with average source utterances of just 6.95 words. (§4.1, Table 1)
Limitations and Open Questions
Warning
All quantitative results are reported on a single language pair (English to Chinese) using a 218-sample benchmark built from one source corpus (StressTest); the paper does not evaluate other language pairs or larger, more diverse test sets, so generalization of the emphasis-preservation gains is untested.
The TTS module (CosyVoice2) is used entirely off-the-shelf and is not fine-tuned or evaluated for how faithfully its prosody-control mechanism renders the predicted emphasis tags as acoustic prominence beyond the aggregate subjective and UTMOS scores reported. The LLM-as-Judge reliability check (§5.3) is validated on only two systems and against a majority vote of seven human raters, a relatively small consistency study for a protocol used to score the full benchmark. The subjective speech-based evaluation in §5.2 also relies on four human experts, a modest panel size for perceptual claims. Finally, the emphasis-marker representation (markdown-style **word** tags) is specific to the English-to-Chinese setup used here; the paper does not test whether the same tagging scheme transfers to languages with different stress or tonal systems.
Wiki Connections
- Prosody Control — introduces an explicit cross-lingual emphasis-tag mechanism that conditions a downstream TTS synthesizer to modulate pitch, energy, and duration independently of the translated text content.
- Speech-to-Speech — builds a complete cascaded S2ST system (speech-to-text translation with emphasis markers followed by TTS rendering) and benchmarks it against direct end-to-end S2ST and speech-to-speech baselines.
- Multilingual TTS — trains its own S2TT model specifically on English-to-Chinese cross-lingual emphasis-aligned data, addressing the difficulty of projecting stress across languages without shared prosodic universals.
- Subjective Evaluation — uses human expert raters to score emphasis preservation in synthesized speech and validates an LLM-as-Judge protocol against human majority-vote consensus.
- CosyVoice 2 — adopts this multilingual, prosody-controllable zero-shot TTS system unmodified as the rendering module of the full S2ST pipeline, relying on its emphasis-tag prompting interface to realize predicted stress as acoustic prominence.