arXiv · 2026 · Preprint

Yanjie An et al. (Shanghai Jiao Tong University / Shanghai Innovation Institute / Microsoft / University of the Chinese Academy of Sciences) · → Paper · Demo: ? · Code: ✓

Packages translation, speech-side, and timing metrics behind one sample record and evaluator interface so that speech-to-text and speech-to-speech translation systems, offline or streaming, can be profiled side by side, and shows that their rankings diverge sharply across those dimensions.

Problem

Speech translation evaluation grew out of machine translation practice: BLEU and COMET against reference text. Once systems emit target speech, and do so incrementally, translation adequacy describes only part of their behaviour. Naturalness, intelligibility, speaker and emotion preservation, nonverbal events, duration structure, and latency are all measured in the literature, but under separate, task-specific protocols. SimulEval standardizes quality-latency measurement for simultaneous translation but has no speech-quality dimension. The result is that heterogeneous systems (cascaded or end-to-end, text or speech output, offline or streaming) are hard to compare on the same footing.

Method

OpenSTBench is an evaluation toolkit, not a model. Every system’s output is mapped to a shared sample record (source audio, generated text and/or speech, timing information), and evaluators are invoked according to which modalities and timestamps are present. Metrics fall into three groups:

  • Translation quality (S2TT and S2ST): sacreBLEU, chrF++, COMET (wmt22-comet-da), BLEURT. For speech output, scoring uses the associated generated text or transcription.
  • Speech quality (S2ST only): UTMOS for naturalness; Whisper-based CER (Chinese, Japanese, Korean targets) or WER (otherwise) against the system’s own generated target text, so this measures realization fidelity rather than translation correctness; speaker similarity with both WavLM and Resemblyzer embeddings; emotion preservation via emotion2vec cosine similarity (E2V) plus emotion-classification accuracy; and paralinguistic fidelity via Event Content F1 (event type and count) and Event Timing F1 (relative timing), with events in the generated speech detected by CLAP.
  • Temporal quality: Speech Length Compliant score (SLC_p, the share of samples whose target-to-source duration ratio falls within [1-p, 1+p], reported at p = 0.2 and 0.4, higher is better), following the duration-consistency evaluation of UniSS and VideoDubber; streaming latency through a SimulEval-style interface extended to speech output (Start Offset, Average Token Delay computed on aligned text of the generated speech, and Custom ATD, which subtracts target-audio playback time from the measured delay to isolate generation-side latency; lower is better for all three); and real-time factor (RTF, processing time over source duration) for locally run offline systems.

A design decision with measurement consequences concerns speaker similarity. The authors argue that comparing target-language output against source-language speech conflates speaker identity with language mismatch. They test this with same-speaker pairs: LibriTTS speech against Qwen3-TTS resynthesis in English and in Chinese under the same speaker prompt. They therefore build a 300-utterance, 35-speaker LibriTTS paired set in which Chinese references are synthesized by Qwen3-TTS from LLM-translated transcripts. EN→ZH outputs are compared with these synthetic Chinese references, ZH→EN outputs with the original LibriTTS English speech.

Evaluation data are direction-specific: MSLT dev (capped at 1,000 samples per direction) for translation, speech quality, SLC, and latency; the LibriTTS paired set for speaker preservation; RAVDESS (EN→ZH, 1,440) and MCAE-SPPS (ZH→EN, 1,029 class-balanced) for emotion; NonverbalTTS test (EN→ZH, 359) and SynParaSpeech (ZH→EN, 500) for paralinguistic events, where source event positions are approximated from text-tag positions rather than frame-level annotation.

Key Results

Six systems are evaluated on EN↔ZH: four streaming API systems (Qwen3-LiveTranslate, Doubao AST 2.0 and GPT Realtime Translate with speech output; Baidu Realtime ST, which emits sentence-final text and is treated as an S2TT baseline) and two locally run offline S2ST models (SeamlessM4T-v2-Large, 2.3B; UniSS, 1.5B). All results are automatic; the paper reports no listening test.

  • Translation quality (Table 5): Qwen3-LiveTranslate has the highest score on all eight columns (four metrics, two directions), e.g. EN→ZH BLEU 43.27 vs. 36.78 for Doubao AST 2.0 and 34.10 for UniSS. Among offline systems UniSS is ahead of SeamlessM4T-v2 on every translation metric.
  • Speech quality (Table 6): rankings do not follow translation quality. Doubao AST 2.0 has the lowest UTMOS of the five speech-output systems in both directions (2.87 EN→ZH, 3.40 ZH→EN) yet the lowest CER/WER in both (0.0782, 0.0346). The highest UTMOS goes to SeamlessM4T-v2 in EN→ZH (3.72 vs. 3.60 for Qwen3-LiveTranslate) and to Qwen3-LiveTranslate in ZH→EN (4.11).
  • Speaker preservation (Table 6): UniSS and Doubao AST 2.0 occupy the top two positions on all four speaker-similarity columns (UniSS leads Resemblyzer, Doubao leads WavLM, in both directions). Qwen3-LiveTranslate, the strongest translator, has the lowest WavLM similarity in both directions (0.2255 EN→ZH, 0.0288 ZH→EN).
  • Emotion (Table 6): UniSS leads E2V similarity in both directions, Doubao AST 2.0 leads emotion classification accuracy in both, illustrating that the two preservation evaluators do not agree on ordering.
  • Paralinguistic fidelity (Table 6): low for every system; no system exceeds 0.151 Event Content F1 or 0.085 Event Timing F1 in either direction.
  • Temporal quality (Table 7): in EN→ZH, Doubao AST 2.0 has the lowest Start Offset, ATD, and Custom ATD among streaming systems; in ZH→EN it has the lowest ATD and Custom ATD, while GPT Realtime Translate has the lowest Start Offset (3,019 vs. 3,163 ms). Low delay does not track duration consistency: Doubao’s SLC 0.2 is 0.27 (EN→ZH) and 0.15 (ZH→EN), below Qwen3-LiveTranslate and, in ZH→EN, GPT Realtime Translate. UniSS reaches SLC 0.2 above 0.99 in both directions, whereas SeamlessM4T-v2 drops to 0.024 in ZH→EN; SeamlessM4T-v2 is, however, the faster offline model (RTF 0.30 / 0.16 vs. 1.54 / 1.08 for UniSS, which runs a VAD-chunked pipeline while SeamlessM4T-v2 generates full utterances).

Comparisons between streaming APIs and offline local models are not matched (different input regimes, unknown provider hardware), and the paper says so; RTF is reported only for the two local models.

Novelty Assessment

None of the individual metrics are new: UTMOS, Whisper ASR error, WavLM/Resemblyzer similarity, emotion2vec, CLAP event detection, SLC, and SimulEval latency all exist. The contribution is integration and protocol: a common record and evaluator interface spanning S2TT and S2ST, offline and streaming; a SimulEval-style latency path extended to speech output (Custom ATD); event-level paralinguistic F1 scores applied to cross-lingual S2ST; and the same-language anchor procedure for speaker similarity, supported by a small but informative diagnostic. The empirical study is narrow (one language pair, six systems, no human validation), and several reference resources are themselves synthetic (Qwen3-TTS Chinese anchors) or approximate (text-derived event positions). The SLC metric is taken from the UniSS paper, which is also one of the evaluated systems, so UniSS’s near-ceiling SLC should be read with that provenance in mind.

Field Significance

Moderate — OpenSTBench provides a reusable, open protocol that treats S2ST output as speech to be judged on naturalness, identity, emotion, nonverbal events, and timing, not only as text. Its comparative study demonstrates on current commercial and research systems that translation rankings, speech-side rankings, and timing rankings disagree. Its value depends on the automatic evaluators it bundles, none of which are validated against human judgments here.

Claims

  • complicates: In speech-to-speech translation, text-side translation quality does not predict how well the target speech preserves speaker identity or other speech-side attributes.

    Evidence: Qwen3-LiveTranslate leads all eight translation-quality columns but has the lowest WavLM speaker similarity of the five speech-output systems in both EN→ZH (0.2255) and ZH→EN (0.0288), while UniSS and Doubao AST 2.0 hold the top two speaker-similarity positions. (§5.1, §5.2, Table 5, Table 6)

  • complicates: Automatic naturalness predictors and ASR-based intelligibility can rank speech generation systems in opposite orders, so neither substitutes for the other.

    Evidence: Doubao AST 2.0 has the lowest UTMOS among speech-output systems in both directions (2.87, 3.40) and simultaneously the lowest CER/WER in both (0.0782, 0.0346). (§5.2, Table 6)

  • complicates: Embedding-based speaker similarity across languages is confounded by language mismatch and by whether the reference is natural or synthesized, so the choice of anchor materially changes the score.

    Evidence: For the same speaker, LibriTTS vs. Qwen3-TTS resynthesis scores WavLM 0.7721 in English but 0.4556 in Chinese (Resemblyzer 0.9073 vs. 0.8462); swapping source anchors for target-language anchors moves WavLM similarity for EN→ZH outputs from 0.0195 to 0.2669 (SeamlessM4T-v2) and from 0.4516 to 0.6849 (Doubao AST 2.0). (§4.2, Table 4, Appendix A.1, Table 8)

  • supports: Current speech-to-speech translation systems largely fail to carry nonverbal vocal events across languages.

    Evidence: Across five S2ST systems and both directions, Event Content F1 stays at or below 0.1502 and Event Timing F1 at or below 0.0843, measured with CLAP detection against NonverbalTTS and SynParaSpeech event annotations. (§5.2, Table 6)

  • refines: Streaming responsiveness and output duration consistency are distinct properties of streaming speech generation and need separate measurement.

    Evidence: Doubao AST 2.0 has the lowest ATD and Custom ATD in both directions but a lower SLC 0.2 than Qwen3-LiveTranslate in both (0.2725 vs. 0.4209 EN→ZH; 0.1527 vs. 0.2670 ZH→EN), and the lowest of the three streaming speech-output systems in ZH→EN (GPT Realtime Translate: 0.4521). (§5.3, Table 7)

Limitations and Open Questions

Every speech-side score (UTMOS, CER/WER, speaker, emotion, event F1) comes from an automatic evaluator, and the paper reports no human listening test to check that these evaluators agree with listeners on S2ST output; the authors list this calibration as future work.

  • Only EN↔ZH is evaluated, so the cross-dimensional ranking patterns are not shown to generalise to other language pairs.
  • Different datasets serve different dimensions (MSLT, LibriTTS paired set, RAVDESS / MCAE-SPPS, NonverbalTTS / SynParaSpeech), so per-system profiles combine scores from unrelated test distributions.
  • EN→ZH speaker anchors are Qwen3-TTS syntheses rather than natural speech, which the appendix shows depresses or shifts WavLM scores; the anchor itself introduces a natural-vs-synthetic mismatch.
  • Source event positions are approximated from text-tag positions (NonverbalTTS) or VAD alignment (SynParaSpeech), which limits how much Event Timing F1 can resolve.
  • Streaming APIs and offline local models differ in input regime, hardware, and output form; the paper flags that strictly matched comparisons are not possible on some dimensions.
  • The radar-plot summary (Figure 3) depends on hand-chosen normalization ranges and assigns 0 to missing speech dimensions for the text-only system.
  • The released package is described as forthcoming in the body text, though the abstract gives a public repository URL.

Wiki Connections

  • Evaluation Metrics — bundles naturalness, intelligibility, speaker, emotion, nonverbal-event, duration, and latency metrics into one protocol for speech translation output and documents an anchor confound in cross-lingual speaker similarity.
  • Speech-to-Speech — the framework’s object is how to evaluate S2ST speech output (offline and streaming), and it profiles five S2ST systems along speech-side and timing dimensions.
  • Seamless — SeamlessM4T-v2-Large is one of the two offline S2ST systems evaluated, scoring high on EN→ZH UTMOS and RTF but low on ZH→EN duration consistency.
  • UniSS — evaluated offline S2ST system and the source of the SLC duration-consistency metric; leads speaker (Resemblyzer) and E2V preservation and SLC.
  • UTMOS — used as the speech-naturalness predictor for all S2ST outputs.
  • emotion2vec — provides the embedding for the E2V emotion-preservation similarity.
  • Qwen3-TTS — synthesizes the same-speaker Chinese reference speech used as EN→ZH speaker anchors.
  • LibriTTS — source of the 35-speaker paired set for speaker-preservation evaluation.
  • SynParaSpeech — supplies the ZH→EN paralinguistic-event test data.