arXiv · 2026 · Preprint

Yu Pan et al. (Kyushu University / Recho Inc. / National Institute of Informatics / University of Luxembourg / Donghua University / The University of Tokyo / University of Alberta) · → Paper · Demo: ? · Code: ?

S2ST-Omni 2 replaces the flat per-language label used to condition a speech-LLM translation frontend with structured typological priors (morphology, reordering profile, language family, plus a residual channel), injected at the representation, acoustic-modulation and prompt levels, and improves many-to-one speech-to-speech translation on CVSS-C, with larger gains as supervised data shrinks.

Problem

Compositional speech-to-speech translation (S2ST) built on speech LLMs pairs a speech-to-text translation (S2TT) frontend with a separate TTS backend. Multilingual many-to-one systems in this family either ignore the source language or encode it as an isolated identifier (a “language-as-label” embedding). The authors’ own prior system, S2ST-Omni (Findings of ACL 2026), follows the label paradigm: each source language gets an independent flat embedding plus a prompt that names the language. Such labels carry no information about what languages share (word order, morphological type, genealogy), so related languages cannot share structure explicitly. The paper argues this matters most when supervised S2ST data are scarce, where a model cannot infer these regularities on its own.

Method

S2ST-Omni 2 keeps the S2ST-Omni skeleton unchanged and redesigns only the language-conditioning pathway. The backbone is a frozen Whisper-Large-V3 encoder, the hybrid speech adapter inherited from S2ST-Omni (input projection to 1024 dims, two depthwise-separable convolution blocks, stride-2 downsampling, two global self-attention blocks, output projection to the LLM), and a Qwen3-4B decoder that generates English text. The text is passed to an off-the-shelf, plug-and-play TTS system that is not trained or fine-tuned. The source-language identity comes from ground-truth labels in training and from a Whisper language prediction at inference.

What S2ST-Omni 2 adds over S2ST-Omni is three conditioning changes:

  • Typology-informed hierarchical language encoding (TI-HLE). Each source language is described by four learnable embeddings: a morphology profile (fusional for French and Spanish, fusional plus compounding for German, agglutinative for Japanese; 64 dims), an English-directed reordering profile (SVO-oriented vs. verb/clause-final; 64 dims), a genealogical family (Romance, Germanic, Japonic; 64 dims), and a language-specific residual channel (128 dims, sized to match S2ST-Omni’s flat embedding). These are concatenated and fused by a linear layer, LayerNorm and GELU into a 256-dim language vector. Because French and Spanish share morphology, reordering and family embeddings, related languages share parameters explicitly.
  • Dynamically-gated language-aware Dual-CTC. The language vector drives a FiLM generator whose bounded scale and shift modulate the downsampled adapter features. Unlike a single static gate, the modulation strength is a per-frame sigmoid gate computed from both the acoustic frame and the language vector, with a learnable temperature and a bias initialised at -2.0 so modulation starts weak. The modulated features feed a source-language CTC head (8k SentencePiece units), while an unmodulated branch feeds an English target CTC head (4k units).
  • Typology-aware LLM prompting. S2ST-Omni’s prompt only names the source language. The new prompt adds a shared translation instruction plus a fixed language-level note on the main translation difficulties (German compound decomposition and clause-final reordering; French and Spanish idioms and lexical choice; Japanese SOV-to-SVO reordering, omitted subjects and honorifics). No utterance-level information is used.

TI-HLE and the Dual-CTC module are used only during training and are discarded at inference along with the CTC heads, so the encoder-adapter-LLM forward path and inference cost are unchanged. The only inference-time change is the prompt. Training follows S2ST-Omni’s two-stage progressive fine-tuning with identical loss weights and hyperparameters. Stage I trains the adapter, TI-HLE and Dual-CTC modules with LLM cross-entropy plus source and target CTC losses (weights 0.1 and 0.2). Stage II adds LoRA (rank 8, alpha 32) on Qwen3’s query and value projections and reduces the CTC weights to 0.01 and 0.05. Whisper and the base Qwen3 weights stay frozen throughout.

Overall architecture and two-stage training pipeline of S2ST-Omni 2. LA denotes language-aware, CE is cross-entropy, and src/tgt denote source/target. TI-HLE and Dynamically-Gated LA-Dual-CTC are training-time auxiliary modules, whereas typology-aware prompting is retained during inference.

Key Results

All main experiments use one many-to-one model trained jointly on CVSS-C French, German and Spanish to English (264 h + 184 h + 113 h = 561 h). The main ASR-BLEU numbers come from the IndexTTS2 backend (Table VII), and ASR-BLEU is computed with a wav2vec 2.0 English ASR model.

  • Against the direct baseline (Table II, Tables IV-V). Average BLEU rises from 35.67 to 37.73 (+5.8% relative) and average ASR-BLEU from 33.45 to 35.00 (+4.6%). Average COMET rises from 82.02 to 83.31 and BLASER 2.0 from 4.14 to 4.24. Every direction improves. The largest BLEU gain is on German to English (33.34 to 35.70), the direction the authors link to compound morphology and clause-final reordering.
  • Against other S2ST systems (Table II). S2ST-Omni 2 has the highest average BLEU and ASR-BLEU of all listed systems, ahead of RosettaSpeech (29.08 / 27.68), StreamSpeech (28.77 / 25.54), SimulS2S-LLM (ASR-BLEU 24.92), ComSpeech and Translatotron 2. Only S2ST-Omni and RosettaSpeech are marked as many-to-one models evaluated across the three directions. The other systems are taken from their own papers and settings, so the comparison is not fully controlled.
  • Against a cascaded text reference (Table II). A Whisper-Large-V3 ASR plus Qwen3-4B MT cascade scores 36.54 average BLEU. S2ST-Omni 2 beats it on French (+2.68) and Spanish (+1.23) but is 0.37 BLEU lower on German (35.70 vs. 36.07).
  • Ablations (Table III). Every variant is worse than the full model. Replacing TI-HLE with a 320-dim flat per-language embedding (matching dimensionality) costs the most: -1.64 BLEU and -1.32 ASR-BLEU. Dropping the typology-aware prompt for S2ST-Omni’s language-naming prompt costs -0.93 / -1.04, and replacing the dynamic frame gate with a static gate costs -0.77 / -0.93. Removing a single TI-HLE group costs 1.13 to 1.52 BLEU, with the residual and morphology channels mattering most.
  • TTS backend (Table VII). With the frontend fixed, average ASR-BLEU across six backends ranges from 33.87 (VoxCPM1.5) to 35.00 (IndexTTS2). CosyVoice3 (34.88), Qwen3-TTS (34.42), FireRedTTS-2 and ZipVoice (both 34.18) fall in between.
  • Data budget (Figure 2). Training both systems on 30 to 561 h, the BLEU advantage over S2ST-Omni grows as data shrinks: +2.06 BLEU (5.8%) at 561 h and +3.93 BLEU (about 15.1%) at 30 h.
  • Japanese with about 3 h of supervision (Table VIII). S2ST-Omni 2 beats S2ST-Omni on all four metrics: BLEU 22.00 vs. 19.61, ASR-BLEU 20.93 vs. 18.59, COMET 80.31 vs. 78.29, BLASER 2.0 3.779 vs. 3.692.

No human listening test, speech naturalness metric or speaker-similarity metric is reported. Speech-side evaluation is limited to ASR-BLEU and reference-based BLASER 2.0.

Novelty Assessment

The contribution is a targeted redesign of language conditioning inside an existing system, and the paper deliberately keeps everything else fixed. That makes the attribution unusually clean: S2ST-Omni is reproduced under identical training, the hyperparameters are shared, and the key ablation replaces TI-HLE with a flat embedding of matched width. The ablation therefore shows that decomposing the language code into typological channels, not simply adding capacity, accounts for part of the gain. Using typological features (lang2vec-style descriptors) for multilingual text MT is established in NLP. What is new here is bringing them into a speech-LLM S2ST frontend through FiLM-gated CTC auxiliaries and language-level prompts. Two of the three components are training-time auxiliaries that are discarded at inference.

The scope is narrow. The typology covers four languages with hand-assigned, coarse categories. Two of them (French and Spanish) have identical typological profiles and differ only in the residual channel, so the claim about cross-language sharing rests on a small and partly degenerate feature space. The S2ST side contributes no new speech generation: the speech output comes from unmodified third-party TTS. Absolute gains over the direct baseline are about 2 BLEU and 1.5 ASR-BLEU, apparently from single runs without significance testing.

Field Significance

Moderate: this paper provides controlled evidence that, in a speech-LLM S2ST frontend, how source-language identity is represented matters beyond simply providing it, and that structured typological priors help most in low-supervision regimes (30 h budgets, a 3 h Japanese setting). It is an incremental extension of S2ST-Omni rather than a new S2ST paradigm. Its main transferable result is the data-budget trend and the matched-capacity flat-embedding ablation.

Claims

  • supports: Structured, linguistically grounded language representations condition multilingual speech translation models better than flat per-language embeddings of equal dimensionality.

    Evidence: Replacing the four-channel typology-informed encoding with a 320-dim flat per-language embedding lowers average BLEU from 37.73 to 36.09 and ASR-BLEU from 35.00 to 33.68 on CVSS-C Fr/De/Es to En, the largest drop among all ablations. (§IV-B, Table III)

  • refines: The benefit of explicit linguistic priors in multilingual speech translation grows as supervised training data shrink, and narrows as data become abundant.

    Evidence: Against the flat-label S2ST-Omni baseline, the BLEU gain grows from +2.06 (5.8%) at 561 h to +3.93 (about 15.1%) at 30 h. In a separate Japanese to English setting with about 3 h of supervision, gains are +2.39 BLEU and +2.34 ASR-BLEU. (§IV-D, Figure 2, Table VIII)

  • supports: In compositional speech-to-speech translation with an explicit text interface, translation quality of the synthesized speech depends mainly on the text frontend and is only modestly affected by the choice of modern TTS backend.

    Evidence: With the S2TT frontend fixed, six publicly available TTS systems (IndexTTS2, CosyVoice3, Qwen3-TTS, FireRedTTS-2, ZipVoice, VoxCPM1.5) give average ASR-BLEU between 33.87 and 35.00, a 1.13-point spread. All six exceed the strongest prior non-Omni system’s ASR-BLEU of 27.68. (§IV-D, Table VII, Table II)

  • supports: Frame-adaptive gating of language-conditioned feature modulation outperforms applying the same modulation strength uniformly across an utterance.

    Evidence: Replacing the per-frame gate, computed from the acoustic frame and the language vector, with a static scalar gate reduces average BLEU by 0.77 and ASR-BLEU by 0.93. (§IV-B, Table III)

  • complicates: Unified speech-LLM translation frontends do not uniformly beat a strong cascaded ASR-plus-LLM text translation pipeline.

    Evidence: S2ST-Omni 2 beats a Whisper-Large-V3 plus Qwen3-4B cascade on average BLEU (37.73 vs. 36.54) but trails it on German to English (35.70 vs. 36.07), the direction where its gain over the flat-label baseline was largest. (§IV-A, Table II)

Limitations and Open Questions

  • The speech output is produced by unmodified off-the-shelf TTS systems. Speech is evaluated only through ASR-BLEU and BLASER 2.0, with no MOS, naturalness, speaker-similarity, latency or human evaluation. The target side is English only, and CVSS-C targets are themselves synthetic single-speaker speech.
  • The typology is hand-assigned for four languages and is coarse by design. French and Spanish share all three typological categories. It is unclear how the scheme scales to many source languages, to languages with mixed or contested profiles, or to non-English targets, where the English-directed reordering profile would need redefining.
  • Most baselines in Table II come from their original papers under differing settings. Only S2ST-Omni (and the Whisper-Qwen reference) are run under the paper’s own protocol. No variance across runs or significance tests are reported, which matters for gains of 1 to 2 BLEU.
  • Language identity at inference comes from Whisper language prediction, and the typology-aware prompt depends on it. Robustness to language-ID errors is not analysed.
  • The Japanese experiment uses about 3 h and compares only against S2ST-Omni, so it cannot show how the system ranks against other low-resource S2ST methods.

Wiki Connections

  • Speech-to-Speech — Contributes a many-to-one compositional speech-to-speech translation system and evidence that structured source-language conditioning improves translation quality of the synthesized speech on CVSS-C.
  • Spoken Language Model — The S2TT frontend is a speech LLM (frozen Whisper encoder, trained adapter, LoRA-adapted Qwen3-4B) consuming source speech, and the paper shows how language-conditioning design inside such a model affects translation.
  • RosettaSpeech — Strongest prior many-to-one baseline in the comparison, outperformed by S2ST-Omni 2 on average BLEU, ASR-BLEU, COMET and BLASER 2.0.
  • SimulS2S-LLM — Speech-LLM S2ST baseline reported from its original paper, below S2ST-Omni 2 in ASR-BLEU on all three CVSS-C directions.
  • Whisper — Whisper-Large-V3 is the frozen speech encoder and supplies the predicted source-language identity at inference.
  • Qwen3 — Qwen3-4B is the LoRA-adapted LLM decoder that generates the English translation text.
  • IndexTTS2 — Default TTS backend for the main ASR-BLEU results and the best of the six backends tested.
  • CosyVoice 3 — Alternative plug-and-play TTS backend, within 0.12 average ASR-BLEU of the best backend.
  • ZipVoice — Flow-matching TTS backend evaluated in the backend-interchangeability study.
  • VoxCPM — VoxCPM1.5 is the lowest-scoring of the six TTS backends, still within 1.13 ASR-BLEU of the best.