arXiv · 2026 · Preprint

Alkis Koudounas et al. (Sony Group Corporation / Carnegie Mellon University) · → Paper · Demo: ✗ · Code: ✗

COMPASS puts 46 metrics for offline speech-to-speech translation (S2ST) into one toolkit, runs them on 1,248 cascaded and end-to-end system-language configurations, cuts them to a 10-metric subset per translation direction, and checks that subset against human ratings in three deployment domains. One finding: automatic MOS predictors do not track what listeners prefer.

Problem

Offline S2ST papers report non-overlapping subsets of metrics (ASR-BLEU, BLASER, AutoPCP, duration metrics, and so on), often with different implementations such as different ASR backends for ASR-BLEU. Results therefore cannot be compared across papers. It is also unclear which quality dimensions matter for which application: dubbing needs timing and lip sync, podcasts need speaker consistency and naturalness, and medical dialogue needs strict accuracy (Table 1). Text and speech-to-text translation have shared metric protocols (e.g. the IWSLT offline tracks); offline S2ST does not.

Method

COMPASS has three parts:

  1. An a priori taxonomy of eight evaluation axes (§3.2): translation quality on the intermediate text, translation quality on ASR transcripts of the output audio, audio naturalness, speaker consistency, prosody and emotion, isochrony, isometry, and lip sync.

  2. A catalogue of 46 metrics (§3.3, Appendix B). It covers:

    • translation: BLEU, ChrF/ChrF++, TER, COMET-DA, COMET-Kiwi, Semantic Score, BLASER, and WER;
    • naturalness: UTMOS and NISQA-MOS;
    • speaker: WavLM-based embedding similarity;
    • prosody and emotion: AutoPCP, F0 and energy contour similarity, speech-rate correlations, emotion match and emotion-embedding similarity, and articulation measures;
    • timing and length: isochrony metrics (delta duration, relative duration error (RDE), speech overlap, speech-length compliance) and isometry metrics (character and word deltas, length ratio, compliance, chars-per-second ratio);
    • lip sync: viseme DTW and co-occurrence scores.

    All metrics use fixed checkpoints. The output audio is transcribed with Whisper-Large-v3 before the ASR-based metrics are computed.

  3. A filtering pipeline that removes redundant metrics (§3.4, Appendix D):

    • Utterance scores are averaged to system level.
    • Metric pairs are compared with Spearman correlation across systems, and Ward hierarchical clustering at distance 0.15 (|ρ| > 0.85) groups redundant metrics.
    • Within each cluster, metrics are ranked by an unweighted mean of four normalized scores: discriminability (coefficient of variation across systems), cross-language ranking stability, independence from other metrics, and a literature-adoption score.
    • Groups are merged iteratively until silhouette ≥ 0.25 and cross-group correlation ≤ 0.70.

The benchmark covers three system families (§4, Appendix C, Table 3):

  • End-to-end: SeamlessM4T-Medium and Large-v2, plus Qwen2.5-Omni and Qwen3-Omni in both standard and chain-of-modality prompting.
  • S2TT+TTS: eight speech-to-text translation front ends (Whisper-Lv2/v3, SeamlessM4T, Gemma-4-E4B, Voxtral-Small, the two Omni models), each paired with CosyVoice3 or Chatterbox.
  • ASR+MT+TTS: Whisper or SeamlessM4T for ASR; NLLB-200, Gemma-3-27B or Qwen3-8B for MT; CosyVoice3 or Chatterbox for TTS.

The cascade TTS stage is prompted with the source utterance to preserve the voice. The evaluation data are FLEURS in both directions over ten languages and CVSS X→EN over eight languages (first 1,000 utterances per pair). That gives 460 + 420 + 368 = 1,248 configurations (Appendix C.5). Decoding is greedy with fixed seeds. As a sanity check, the authors’ SeamlessM4T-Lv2 BLEU and ASR-BLEU scores fall within ±0.8 points of the published numbers.

For the analysis, the authors pick one representative per family by mean rank on the compact axes: Qwen3-Omni, SeamlessM4T-Large-v2, Voxtral-Small + Chatterbox, and Whisper-Lv3 + Gemma-3 + CosyVoice3.

Human study (§6, Appendix F). Three native speakers of the target language rate each clip on 5-point Likert scales covering 7 to 9 domain-specific questions, and also pick the best and second-best system. The three domains are:

  • dubbing: 50 MELD-ST EN↔JA clips;
  • podcasts: 48 European Parliament clips, EN↔{DE, ES, FR, IT};
  • medical: 40 MultiMed-ST EN↔ZH clips.

Ground-truth references are included where available (not in the medical domain). Metric-human agreement is measured with system-level Spearman ρ and 1,000-resample bootstrap confidence intervals.

Key Results

Compact subsets (§5 RQ1-RQ2, Table 2). Filtering reduces the 46 metrics to 10 per direction across six empirical dimensions:

  • Speaker consistency and prosody/emotion merge into one cluster on read speech.
  • Lip sync is absorbed into isochrony.
  • Isochrony and isometry stay separate (RDE vs. character compliance ρ = -0.55 EN→X, 0.26 X→EN).
  • Three axes pick different representatives in each direction:
    • text quality: TER (X→EN) vs. ChrF++ (EN→X);
    • naturalness: UTMOS vs. NISQA-MOS;
    • speaker/prosody: speaker similarity vs. energy contour similarity.
  • The subsets keep the full-suite system rankings (Spearman ρ > 0.80) and cut compute from about 10,845 s to 4,300 s per CVSS configuration on an A6000 (§3.4).
  • On spontaneous MELD-ST speech, the speaker-prosody merge weakens: mean |ρ| drops to about 0.75, from above 0.85 on read speech (Table 5). That analysis uses only 5 systems.

Architecture trade-offs (§5 RQ3, Appendix E.5, Table 7). No family leads on every axis.

Axis (Table 7)LeaderOther three systems
Speaker similarity, X→ENS2TT+TTS cascade: 0.915 (FLEURS), 0.889 (CVSS)FLEURS 0.805, 0.663, 0.606; CVSS 0.779, 0.524, 0.608
UTMOS, X→ENQwen3-Omni: 3.591 (FLEURS), 3.492 (CVSS)FLEURS 3.071, 3.332, 3.060; CVSS 2.882, 2.890, 3.131
NISQA-MOS, EN→XQwen3-Omni: 4.3212.410, 2.661, 3.654
  • Translation quality is close across families. COMET-DA (text) on FLEURS X→EN ranges 0.833 to 0.880.
  • Metric choice flips the EN→X ranking. The three-stage cascade has the highest text ChrF++ (57.342), while the S2TT+TTS cascade has the highest COMET-DA on ASR transcripts (0.804, against 0.716 for the three-stage cascade).
  • The three-stage cascade fails on CVSS. Its COMET-DA (text) on CVSS X→EN falls to 0.646 because of repetitive decoding loops on Spanish, French and Italian.
  • All systems shorten their output (X→EN). RDE is negative for every system; values closer to zero mean a better duration match. On FLEURS the three-stage cascade is closest (-0.243) and SeamlessM4T-Large-v2 is furthest (-0.404).
  • Character-length compliance is low everywhere, from 18.69% to 33.29%.

Metric-human alignment (§6, Appendix F.4, Table 12).

  • Across all questions and domains, COMET-DA on ASR transcripts has the highest average |ρ| (0.82), followed by Semantic Score (ASR) at 0.79. The rest of the top ten are isometry, prosody and translation metrics.
  • No MOS predictor or speaker-similarity metric is in the top ten.
  • NISQA-MOS correlates negatively with dubbing emotional match (ρ = -0.90, CI [-1.00, -0.69]).
  • UTMOS is near zero in podcasts and medical.
  • Timing metrics reach |ρ| = 1.00 with averaged dubbing timing ratings in EN→JA, even though annotators agree least on timing (α = 0.72) and lip sync (α = 0.68) (Table 9).
  • No metric reliably predicts medical terminology accuracy or real-world trust.

Human preference (Tables 13-15).

  • SeamlessM4T-Large-v2 gets zero first-preference votes in every domain and direction.
  • In podcast X→EN, the S2TT+TTS cascade ties the ground truth on first-preference rate (0.50 each) and has a higher average overall score (4.89 vs. 4.72). In EN→X, the ground truth wins 0.89 of clips.
  • In medical, the three-stage cascade is preferred in 0.70 of clips overall (0.60 ZH→EN, 0.80 EN→ZH). Its overall-quality score is 4.27, against 3.64 (Qwen3-Omni), 3.31 (S2TT+TTS) and 1.92 (SeamlessM4T-Large-v2).
  • Official dubs win every dubbing clip.

Novelty Assessment

The paper’s value is that it consolidates and measures; it adds no new technique. Every metric comes from prior work, and correlation-based metric pruning is a standard method. The new parts are:

  • one reproducible implementation covering eight axes;
  • a large, controlled sweep of cascaded and end-to-end S2ST systems under identical settings;
  • the finding that the best compact metric subset differs between the X→EN and EN→X directions;
  • a multi-domain human study that ties metric choice to the deployment setting.

The literature-adoption term in the selection score builds current practice into the recommended subsets. That favours comparability with existing papers, but it is partly circular. The human-correlation analysis is the most informative part, but it is statistically thin (see Limitations).

Field Significance

Moderate. The paper provides a controlled, multi-axis comparison of cascaded and end-to-end S2ST systems. It also gives direct evidence that MOS predictors trained for TTS naturalness do not transfer as indicators of perceived S2ST quality. As a toolkit, its broader impact depends on uptake, and the code was not yet available at the time of writing.

Claims

  • complicates: Automatic MOS predictors are poor proxies for listener-perceived quality of translated speech, and can correlate negatively with emotional fidelity.

    Evidence: UTMOS and NISQA-MOS have near-zero or negative system-level correlations with human ratings in the dubbing, podcast and medical domains. NISQA-MOS reaches ρ = -0.90 against dubbing emotional match, and no MOS predictor appears among the top-10 metrics by average |ρ|. (§6, §F.4, Table 12)

  • supports: In speech-to-speech translation, cascades with a dedicated voice-cloning TTS stage preserve source speaker identity better than current end-to-end models, while end-to-end models produce more natural-sounding speech.

    Evidence: On FLEURS X→EN, the S2TT+TTS cascade reaches speaker similarity 0.915 vs. 0.606 to 0.663 for the end-to-end models. Qwen3-Omni leads on UTMOS (3.591) and EN→X NISQA-MOS (4.321) over both cascades. (§5 RQ3, §E.5, Table 7)

  • complicates: Translation-accuracy metrics alone barely separate S2ST architectures, so single-metric rankings hide large differences in voice, naturalness and timing.

    Evidence: COMET-DA (text) on FLEURS X→EN spans 0.833 to 0.880 across the four representative systems. Speaker similarity spans 0.606 to 0.915, and best-vs-worst gaps exceed 30% on naturalness and speaker preservation. (§5 RQ3, Table 7)

  • refines: The best automatic metric subset for speech generation evaluation depends on the target language and the deployment domain, not only on the task.

    Evidence: Correlation filtering selects UTMOS for English targets but NISQA-MOS for non-English targets, and TER vs. ChrF++ for text quality. In the human study, the strongest predictors are translation metrics for medical and podcasts, timing and prosody metrics for dubbing, and isometry metrics for podcast naturalness. (§5 RQ2, Table 2; §6, §F.5)

  • complicates: On read-speech corpora, speaker and prosody metrics vary together so strongly that they look redundant, and this redundancy partly disappears on spontaneous, emotional speech.

    Evidence: The speaker-prosody cluster has mean |ρ| > 0.85 on FLEURS and CVSS but about 0.75 on MELD-ST. F0 and energy contour similarity correlate -0.70 with speaker similarity in EN→JA. (§5 RQ1, §E.3, Table 5)

Limitations and Open Questions

Warning

Every metric-human correlation is computed over only 4-5 systems per domain and direction. The authors state that these ρ values should be read as indicative rather than statistically confirmed (§F.4). Headline values such as ρ = 1.00 or -1.00 therefore rest on very few points, even with bootstrap confidence intervals.

Other limitations:

  • Read speech only in the large-scale benchmark. FLEURS and CVSS may suppress prosodic and emotional variation (§Limitations).
  • Shared ASR model. The same Whisper-Lv3 model transcribes outputs for the ASR-based metrics and serves as a front end in several cascades. A 50-sample control with a different ASR model kept rankings stable (ρ > 0.82).
  • Noisy source audio. Speaker and prosody metrics compare outputs against the noisy source, which can penalize systems that denoise.
  • Offline only. No latency is measured, so streaming and simultaneous systems are out of scope.
  • Short clips. Human evaluation uses clips of 30 s or less, so long-form drift is not captured.
  • EN→X conclusions rest on FLEURS alone.
  • Default voice for SeamlessM4T. The model is run with its default speaker settings rather than with any voice-preservation option. Its low speaker scores may therefore partly reflect this configuration choice, not only the architecture.
  • Representative selection. Each family is represented by its top system chosen on COMPASS axes, which may favour the selected metrics.
  • A statement in the appendix does not match Table 7. Appendix F.6 says SeamlessM4T-Large-v2’s last-place human ranking mirrors “last-place automatic scores throughout the benchmark”. In Table 7, however, it has the highest word accuracy on FLEURS X→EN (0.977) and the second-highest NISQA-MOS on EN→X (3.654).
  • Code not yet released. The authors plan to release the code under a commercial license upon acceptance (§Ethics Statement). No repository is linked.
  • Open problem. No metric predicts medical terminology accuracy or real-world trust.

Wiki Connections

  • Speech-to-Speech — Provides a unified offline evaluation protocol and a large controlled comparison of cascaded and end-to-end S2ST systems.
  • Spoken Language Model — Benchmarks Qwen2.5-Omni and Qwen3-Omni on real source audio as end-to-end translators; Qwen3-Omni, the family representative, leads on naturalness but trails voice-cloning cascades on speaker similarity (Table 7).
  • Evaluation Metrics — Reduces 46 S2ST metrics to direction-specific compact subsets and shows MOS predictors fail as cross-domain quality proxies.
  • Subjective Evaluation — Runs a native-speaker Likert and preference study across dubbing, podcast and medical domains, reporting per-question Krippendorff’s α.
  • SeamlessM4T — Medium and Large-v2 checkpoints are benchmarked as dedicated end-to-end S2ST systems and as ASR/S2TT front ends; they receive the lowest human preference.
  • Seamless — Source of the SeamlessM4T-Large-v2 checkpoint and the AutoPCP prosody metric used in the catalogue; its reported BLEU and ASR-BLEU are reproduced within ±0.8 points.
  • Qwen3-Omni — Benchmarked as the representative end-to-end speech LLM; it leads on naturalness but trails cascades on speaker similarity.
  • Qwen2.5-Omni — Benchmarked in standard and chain-of-modality end-to-end modes and as an S2TT front end.
  • CosyVoice 3 — One of the two TTS back ends in all cascades, including the representative three-stage cascade preferred in the medical domain.
  • Whisper — Used both as a cascade front end and as the ASR model for all ASR-based metrics, creating the evaluation dependence the authors control for.
  • UTMOS — Selected as the naturalness representative for X→EN, yet shows near-zero or negative correlation with human S2ST judgments.
  • Gemma 3 — Gemma-3-27B is the MT stage of the representative three-stage cascade.