arXiv · 2026 · Preprint

Sitong Cheng et al. (Hong Kong University of Science and Technology / Tencent Youtu Lab / Tsinghua University) · → Paper · Demo: ✓ · Code: ?

Introduces a 32.6-hour Chinese-English speech-to-speech translation benchmark that scores emotion, scenario style, and nonverbal vocalization preservation with a reference-free caption-then-summarize LLM judge validated against human listeners, and shows that six current systems transfer content far better than expressiveness.

Problem

Expressive speech-to-speech translation (S2ST) should carry over how an utterance is spoken, not only what is said, which matters for uses such as cross-lingual video dubbing. Existing S2ST benchmarks cover expressiveness only partially: CVSS and FLEURS have no expressive annotations, mExpresso and MELD-ST annotate emotion only, and EmphAssess covers emphasis transfer, so scenario style (news, drama, audiobook narration) and nonverbal vocalizations (NVs: laughter, crying, breathing) go unexamined (§2, Table 1). A second obstacle is that parallel target speech that is both translation-faithful and expressively matched is expensive to collect, so reference-based expressiveness evaluation does not scale; the authors state that no reference-free framework had been validated for S2ST expressiveness (§1).

Method

Data curation. Candidate audio is collected from public web sources across six scenarios (drama, audiobooks, advertisements, interviews, news, commentary). Preprocessing applies BS-Roformer source separation, Silero VAD plus pyannote diarization with CAM++ embeddings to merge same-speaker segments (cosine threshold 0.75), a 3-30 s length filter, a narrowband filter (effective bandwidth below 8 kHz discarded), SNR and C50 thresholds, a DNSMOS cutoff of 3.0, and Whisper-large-v3-turbo language identification keeping only Chinese and English (§3.1, §A.1).

Annotation. Utterances are transcribed with Qwen3-ASR, word-aligned with Qwen3-ForceAlign, and translated (Chinese↔English) with Qwen3-30B-A3B. NVs are detected with BEATs inside the PretrainedSED framework and inserted as bracketed markers into the transcript by temporal overlap. For emotion and scenario style, a two-stage caption-then-summarize pipeline runs: Qwen3-Omni-Captioner writes a detailed caption from the audio, and Qwen3-30B-A3B summarizes it into a short emotion description and a scenario-style description (§3.2, §B).

Quality assurance. From 80 hours of candidates, a rule-based filter prioritizes NV-bearing utterances and balances scenario, language, and gender; a multimodal LLM then scores each annotation on a 1-5 scale and only samples scoring 5 on all dimensions (with at most one 4) are kept. On a 170-sample check, when the automatic score was above 4 the human score was also above 4 in 100% of cases for transcription, 91.3% for translation, 86.7% for emotion, 88.8% for scenario style, and 75.6% for NV annotation. Bilingual human annotators then validate retained utterances, partly via a binary accept/reject protocol (§3.3, §E.1). The final benchmark has a normal subset (20,370 utterances, 32.27 h) and an NV subset (901 utterances, 1.26 h), with some overlap, for 32.6 h unique (§3.3).

Reference-free expressiveness scoring. A system’s translated output is passed through the same captioning, summarization, and NV-detection pipeline, and a text LLM judge (Qwen3-30B-A3B via vLLM) compares source and hypothesis descriptions on a 1-5 rubric per dimension: emotion (category, intensity, attitude), scenario style (scene/genre, with formality and delivery as secondary cues), and NVs (type, count, order, approximate position, penalizing added events) (§3.4, §C). Each sample is judged three times and aggregated with an outlier-aware rule (identical scores used directly; a single score differing by 2 or more is dropped; otherwise mean or median) (§F). The authors argue against classifier label matching because it cannot capture intensity differences or event-level NV comparison (§3.4, §I.1).

Standard dimensions. Translation fidelity is BLEU (SacreBLEU on Qwen3-ASR transcripts of the output speech), COMET (wmt22-comet-da), and XCOMET-XL in reference and QE variants; speaker similarity (SIM) is WavLM-Large + ECAPA-TDNN cosine similarity following the SeedTTS-eval protocol; duration alignment is Speech Length Compliance (SLC), the fraction of utterances whose output/input duration ratio lies within 1±ε for ε of 0.2 and 0.4 (§4.1, §H.1).

Systems evaluated. Six systems: a Three-Stage cascade (Qwen3-ASR with NV detection, Qwen3-235B-A22B translation plus emotion/style instruction generation, VoxCPM2 instruction-controlled voice-cloning TTS); a Two-Stage cascade (Qwen3-Omni-Instruct audio-to-text translation plus instruction, then VoxCPM2); UniSS; SeamlessExpressive; the commercial Seed LiveInterpret 2.0; and the speech LLM Step-Audio 2. Both cascades are also reported without style instructions (w/o Instruct) (§4.1, §G).

Key Results

Translation fidelity is strong, especially for cascades. On Chinese→English, Two-Stage reaches 41.59 BLEU, the highest in Table 2 (its w/o Instruct variant 41.41; the Three-Stage rows about 38.2); end-to-end systems range from 17.83 (SeamlessExpressive) to 30.82 (Step-Audio 2). On English→Chinese the Two-Stage cascades reach 59.66 (with instructions) and 59.73 (w/o Instruct), with end-to-end systems from 36.13 (SeamlessExpressive) to 48.37 (Step-Audio 2) (§4.2, Tables 2-3). Intermediate text outputs score close to the final speech (Two-Stage 43.65 vs. 41.59 BLEU zh→en; 61.51 vs. 59.66 en→zh), so speech generation loses little semantic content (§4.2, Table 4).

Emotion preservation lags, and cascades are weakest on it. Cascaded systems score 1.67-1.73 on emotion across both directions, while end-to-end systems score 3.10-3.82. The highest emotion score is UniSS at 3.82 (en→zh; Step-Audio 2 next at 3.77); on zh→en UniSS also leads at 3.61 (Step-Audio 2 3.57) (§4.2, Tables 2-3). Adding style instructions to the cascade TTS changes emotion scores by at most 0.02 points in either direction, which the authors read as textual instructions being insufficient to recover source expressiveness (§4.2, Tables 2-3). Scenario style scores are higher overall (cascades 3.92-4.03, end-to-end 4.27-4.46), with a smaller cascade/end-to-end gap (Tables 2-3).

NV preservation favours explicit markers but stays low. Cascades that pass NVs as inline text markers score highest: Two-Stage 2.31 on zh→en and Three-Stage 2.21 on en→zh. End-to-end systems score 1.09-1.58, with Step-Audio 2 at 1.58 on zh→en despite being prompted to preserve NVs (§4.2, Tables 2-3).

Duration alignment requires explicit control. UniSS, which models duration explicitly, reaches SLC-0.2 of 0.915 (zh→en) and 0.980 (en→zh); the next system reaches 0.614 and 0.659 respectively (§4.2, Tables 2-3).

Speaker similarity is low throughout (0.254-0.498), which the authors attribute to the cross-lingual comparison between source and target speech (Tables 2-3, §H.2). The cascades using VoxCPM2 voice cloning have the highest SIM in both directions (about 0.50 zh→en, 0.425-0.428 en→zh).

Judge validation. On 160 source-hypothesis pairs, each rated by at least three bilingual annotators (after inter-annotator consistency filtering), the summary-based judge correlates significantly with humans on all three dimensions: emotion ρ = 0.515 (human-human 0.584), style ρ = 0.427 (human-human 0.514), NV ρ = 0.518 (human-human 0.789). Its exact agreement exceeds human-human agreement on emotion (0.567 vs. 0.438) and style (0.640 vs. 0.544) and nearly matches it on NVs (0.644 vs. 0.651), but its MAE is higher than human-human on every dimension. Audio-based direct scoring (a multimodal LLM listening to both clips) and caption-based scoring (full captions) are not significant on any dimension (ρ between -0.009 and 0.193) (§4.3, Table 5).

Comparisons are fair in that all systems are scored on the same data with the same pipeline, but the cascades share components with the benchmark construction (Qwen3-ASR, Qwen-family LLMs, and the same BEATs NV detector in the Three-Stage system), and the expressiveness judge itself uses Qwen3-Omni-Captioner, so some advantage on NV detection and transcription for those systems cannot be ruled out; the paper does not discuss this.

Novelty Assessment

The benchmark’s coverage is new: it is the first S2ST benchmark, by the authors’ account, to jointly score emotion, scenario style, and NV preservation, and its 32.6 hours exceed the test sets of CVSS, FLEURS, mExpresso, MELD-ST, and EmphAssess for the Chinese-English pair (Table 1). The most transferable contribution is the evaluation methodology: a reference-free judge that compares structured text descriptions of source and output, together with a human correlation study showing this decomposed design works where direct audio comparison and long-caption comparison fail. The individual components (source separation, diarization, captioning, LLM-as-judge rubrics) are existing tools assembled into a pipeline. The system comparison itself is an empirical snapshot of six systems on one language pair.

Field Significance

Moderate. The paper provides a bilingual, human-validated benchmark and scoring method for expressiveness preservation in S2ST, an aspect that translation metrics such as BLEU and COMET do not capture. Its results give concrete evidence of a gap between semantic and expressive transfer across cascaded, end-to-end, and speech LLM systems, and its judge comparison is a useful data point for how LLM-based evaluators of paralinguistic attributes should be designed. Coverage is limited to Chinese↔English, and the judge correlations, while significant, remain moderate.

Claims

  • supports: Current speech-to-speech translation systems preserve lexical content much better than expressive attributes such as emotion and nonverbal vocalizations.

    Evidence: Cascades reach up to 41.59 BLEU (zh→en) and 59.73 (en→zh), while the highest emotion score across all systems is 3.82/5 and the highest NV score 2.31/5; text-stage BLEU is close to speech-stage BLEU, locating the loss in expressive rather than semantic transfer. (§4.2, Tables 2-4)

  • complicates: Generating text style instructions for an instruction-controllable TTS stage does not recover source emotion in cascaded speech translation.

    Evidence: With and without instructions, the Three-Stage and Two-Stage cascades score 1.67-1.73 on emotion in both directions, while end-to-end systems that operate on speech score 3.10-3.82. (§4.2, Tables 2-3)

  • refines: Nonverbal vocalizations transfer better when represented as explicit discrete events in an intermediate text than when left implicit in end-to-end speech generation, though absolute preservation remains low.

    Evidence: Cascades that insert NV markers before TTS reach 2.31 (zh→en) and 2.21 (en→zh), while end-to-end systems score at most 1.58, including Step-Audio 2 when prompted to preserve NVs. (§4.2, Tables 2-3)

  • supports: Reference-free LLM judging of paralinguistic preservation aligns with human listeners when the speech is first reduced to short structured attribute descriptions, but not when the judge compares raw audio or long captions directly.

    Evidence: The summary-based judge reaches significant Spearman correlation with human scores on emotion (0.515), style (0.427), and NVs (0.518), while audio-based direct scoring and caption-based scoring show no significant correlation on any dimension (ρ from -0.009 to 0.193). (§4.3, Table 5)

  • supports: Close duration alignment between source and translated speech requires an explicit duration-control mechanism.

    Evidence: UniSS, which models duration explicitly, reaches SLC-0.2 of 0.915 and 0.980, versus at most 0.614 and 0.659 for all other systems. (§4.2, Tables 2-3)

Limitations and Open Questions

Warning

The expressiveness scores are produced by a chain of models (Qwen3-Omni-Captioner, Qwen3-30B-A3B summarizer and judge, BEATs NV detector), and the benchmark’s reference annotations come from the same chain. The judge’s agreement with humans is significant but moderate (ρ 0.43-0.52, with higher MAE than human-human), and systems that share components with the pipeline (the cascades use Qwen3-ASR, Qwen-family LLMs, and in the Three-Stage case the same BEATs NV detector) may be favoured in ways the paper does not test.

  • Coverage is Chinese↔English only; the authors describe the pipeline as language-agnostic but do not demonstrate other pairs (§Limitations, §I.2).
  • Scenario style is the most subjective dimension and has the weakest judge correlation; the authors note room for improvement (§Limitations, Table 5).
  • The human correlation study keeps only pairs with sufficient inter-annotator consistency, which may make the validated correlations optimistic relative to the full score distribution (§4.3, §E.2).
  • Benchmark reference annotations are themselves automatically generated and filtered; human checks show agreement drops to 75.6% for NV annotation (§3.3).
  • Part of the normal-subset human validation uses a binary accept/reject protocol rather than full ratings (§E.1).
  • Prosody beyond emotion and style (for example emphasis or speaking rate) is not scored, and no naturalness or audio-quality metric (MOS, UTMOS) is reported for system outputs.
  • The data, prompts, and scoring scripts are stated as to be released; source audio without redistribution permission will be released as metadata and annotations only (§5, Ethical Considerations).

Wiki Connections

  • Speech-to-Speech — Provides a benchmark and reference-free scoring method for expressiveness preservation in speech-to-speech translation across cascaded, end-to-end, and speech LLM systems.
  • Spoken Language Model — Feeds real source audio into speech LLMs (Step-Audio 2, Qwen3-Omni in the Two-Stage cascade) and measures how much expressiveness their translations carry.
  • Evaluation Metrics — Defines and validates LLM-judged emotion, scenario-style, and NV preservation scores, and shows decomposed summary-based judging outperforms direct audio judging.
  • Subjective Evaluation — Uses a human correlation study with at least three bilingual raters per pair, including a human-human agreement baseline, to validate the automatic judge.
  • Emotion Synthesis — Evaluates how to measure emotion transfer from source to translated speech and finds emotion preservation is the weakest expressive dimension for cascades.
  • UniSS — Evaluated system; obtains the highest emotion scores and by far the best duration alignment, with lower speaker similarity than the cascades.
  • Seamless — SeamlessExpressive is evaluated as an end-to-end expressive S2ST baseline and has the lowest BLEU in both directions.
  • Seed LiveInterpret 2.0 — Evaluated commercial simultaneous S2ST system; strong on scenario style but low on NV preservation.
  • Step-Audio 2 — Evaluated speech LLM; highest NV score among end-to-end systems but still far below the marker-based cascades.
  • Qwen3-Omni — Supplies Qwen3-Omni-Captioner for the annotation and judge pipeline and Qwen3-Omni-Instruct for the Two-Stage cascade.
  • VoxCPM2 — Instruction-controlled voice-cloning TTS used as the synthesis stage of both evaluated cascades.
  • TTS-PRISM — Cited as prior LLM-based multidimensional speech evaluation that STEB contrasts with its S2ST-specific validated judge.
  • EchoMind — Cited as prior LLM-based evaluation of paralinguistic attributes for speech LMs, not specific to S2ST.