arXiv · 2026 · Preprint
Lalaram Arya et al. (IIT Jammu) · → Paper · Demo: ? · Code: ✓
Introduces DS2ST-LM, a single-stage, LLM-based direct speech-to-speech translation framework that integrates a frozen Whisper encoder, a projection module, a fine-tuned Qwen2-0.5B LLM, and a timbre-controllable CosyVoice-style vocoder, trained end-to-end to jointly predict target semantic tokens and text.
Problem
Cascaded speech-to-speech translation (S2ST) pipelines (ASR → MT → TTS) suffer from error propagation across independently trained modules, added latency from multi-stage processing, and loss of speaker and prosodic information at each handoff. Prior end-to-end S2ST work has closed some of this gap but has largely relied on self-supervised discrete units or continuous spectrogram mapping, leaving several questions underexplored: how LLM-based direct S2ST behaves when trained on natural versus synthetic parallel speech, which cross-modal projection design best interfaces a speech encoder with an LLM, whether semantic tokens should be derived from speech or from text, and how well a single trained system generalizes across typologically distant language pairs while preserving speaker timbre.
Method
DS2ST-LM is a single-stage, end-to-end trainable framework adapted from the SLAM-Omni conversational architecture for the direct S2ST task. It chains four components: a frozen Whisper-small encoder that converts source-language speech into frame-level acoustic features; a projection module that maps those features into the Qwen2-0.5B LLM’s embedding space; the LLM itself, which autoregressively predicts semantic speech tokens in the target language interleaved with target-language text; and a timbre-controllable vocoder that renders the predicted semantic tokens into a waveform.
Semantic tokens follow the supervised approach from CosyVoice: a multilingual SenseVoice ASR encoder is split into two halves with a vector-quantization bottleneck inserted between them, producing tokens that are explicitly trained to be predictive of transcribed text rather than merely reconstructive of the acoustic signal. Because target-language speech is not always available (many parallel corpora provide only speech-to-text translation pairs), the paper also trains a text-to-token LLM path that derives the same semantic token vocabulary directly from target text, letting the model be trained on the much larger pool of speech-to-text data when speech-to-speech pairs are scarce.
Three interface designs are compared for the projector: a lightweight Linear module that groups consecutive encoder frames and passes them through a two-layer MLP; a Conv1D-Linear variant that replaces frame-grouping with a learned 1-D convolution; and a Q-Former, which uses learnable query embeddings and cross-attention (as in BLIP-2) to extract content-aware features from the encoder output. To reconcile the LLM’s differing token rates for text (~3 Hz) and semantic speech tokens (50 Hz), the model predicts semantic tokens in groups of 3, following SLAM-Omni’s group-modeling strategy, and is trained with a weighted sum of audio-token and text-token cross-entropy losses (weights fixed to 1 each).
At synthesis time, a conditional flow-matching model (following CosyVoice’s design) generates mel-spectrograms conditioned on the predicted semantic tokens and a speaker embedding extracted from a short reference audio prompt, and a HiFi-GAN vocoder converts the mel-spectrograms to waveform. Because the semantic tokens are trained to exclude acoustic/speaker information, timbre is reintroduced only through this explicit reference-embedding conditioning, giving the system zero-shot control over the output speaker identity independent of the translation path.
To address the scarcity of parallel S2ST training data, the authors construct GigaS2S-1000: they extend the GigaST English-to-Chinese text-translation corpus (itself built on GigaSpeech) by synthesizing 1,000 hours of single-speaker Chinese target speech with XTTS-v2, after filtering English-Chinese sentence pairs by SONAR embedding cosine similarity (> 0.9) to ensure the underlying translations are semantically well aligned. The Whisper encoder is kept frozen throughout training; the LLM is fully fine-tuned. Training uses Adam (lr 1e-4, batch size 8, 1000-step warm-up, decay factor 0.85) for up to 4 epochs with early stopping, in FP16, on a single H100 GPU.
Key Results
On the primary zh-en comparison (Table I), DS2ST-LM outperforms both a matched-component cascaded ASR→MT→TTS baseline and a Qwen-Audio-based ST+TTS baseline across BLEU, METEOR, BLEURT, and COMET on three test sets (Seamless-Align, GigaS2S-1000, FLEURS), with the largest margins on the paper’s own GigaS2S-1000 corpus (BLEU 14.71 vs. 6.84 cascaded / 11.36 ST+TTS). Among three projector designs, the plain Linear projector gives the best BLEU/METEOR/BLEURT/COMET despite the higher-capacity Conv1D-Linear and Q-Former variants converging faster during training. Speech-derived semantic tokens outperform text-derived tokens trained on the same data (BLEU 14.71 vs. 12.58 on GigaS2S-1000, a 17.28% relative drop). Extending to six additional language pairs (fr, de, es via CVSS; hi, ben, urd via Bhasaanuvaad), DS2ST-LM consistently beats both baselines, though absolute performance tracks each language’s representation in the LLM’s pretraining data (BLEU 24.57 fr-en down to 5.41 urd-en). On speaker-timbre preservation (fr-en test set, Table IV), DS2ST-LM reaches a cosine speaker-similarity (SIM) of 0.83 and DNSMOS of 3.54, both exceeding four prior direct S2ST timbre-preserving baselines (TransVIP, DA-Speech, Translatotron, Translatotron 2), and a 15-listener MOS study corroborates this with naturalness 3.55 and speaker-similarity 3.95, approaching ground-truth naturalness (3.86 DNSMOS / 3.81 MOS).
Novelty Assessment
The contribution is primarily an engineering integration: Whisper, Qwen2, a CosyVoice-style supervised semantic tokenizer plus conditional-flow-matching vocoder, and HiFi-GAN are combined into a single-stage trainable pipeline, directly adapting the SLAM-Omni conversational architecture to the S2ST task rather than introducing a new architectural mechanism. The systematic empirical comparisons are the paper’s more durable contribution: the projector-capacity study (showing that higher-capacity Conv1D-Linear/Q-Former projectors converge faster but generalize worse than a simple Linear projector) and the speech-vs-text semantic-token study are genuine ablations with a clear, somewhat counterintuitive finding (more expressive ≠ better for this interface). The GigaS2S-1000 dataset release is a concrete, reusable artifact, though it depends on synthetic (XTTS-v2-generated), single-speaker target speech rather than natural recordings.
Field Significance
moderate — the paper is a solid empirical study extending an existing single-stage LLM architecture (SLAM-Omni) to the S2ST task, with useful negative-result findings on projector capacity and a released training corpus, but it does not introduce a new training objective, tokenization mechanism, or synthesis architecture beyond what CosyVoice and SLAM-Omni already established.
Claims
- supports: Single-stage, end-to-end trained LLM-based direct speech-to-speech translation frameworks can outperform cascaded ASR-MT-TTS pipelines even when both approaches use identical component models and training data.
Evidence: With Whisper-small, Qwen2-0.5B, and CosyVoice held constant across both system types, the end-to-end DS2ST-LM beats the cascaded and ST+TTS baselines on every metric (BLEU, METEOR, BLEURT, COMET) across three zh-en test sets. (§IV-E1, Table 1)
- complicates: Higher-capacity cross-modal projection modules between a speech encoder and an LLM do not necessarily improve downstream translation quality, despite converging faster during training.
Evidence: Conv1D-Linear and Q-Former (2- and 4-layer) projectors converge faster than a simple Linear projector but underperform it on all four metrics (BLEU 13.33/13.10/12.05 vs. 14.71 for Linear), attributed to the higher-capacity projectors disrupting frame-level temporal structure the pretrained encoder had already encoded. (§IV-E2, Table 2)
- supports: Semantic tokens derived directly from target speech carry translation-relevant information that text-derived tokens (generated by a text-to-token LLM from the same target text) do not fully replicate.
Evidence: Replacing S3 speech-derived tokens with text-driven LLM-generated tokens, holding all other training conditions fixed, degrades BLEU by a relative 17.28% on GigaS2S-1000 (14.71 → 12.58), with the same direction of effect replicated on Seamless-Align. (§IV-E3, Table 3)
- supports: Conditioning a vocoder on an explicit, fixed reference-speaker embedding separate from content tokens preserves cross-lingual speaker timbre more reliably than continuous or entangled discrete-unit representations used by prior direct S2ST systems.
Evidence: On the fr-en evaluation set, DS2ST-LM’s speaker-embedding-conditioned vocoder reaches cosine speaker similarity 0.83, well above Translatotron (0.32), Translatotron 2 (0.43), DA-Speech (0.37), and TransVIP (0.40). (§IV-E5, Table 4)
- complicates: A single LLM-based S2ST system’s translation quality varies sharply across language pairs according to that language’s representation in the underlying LLM’s pretraining data, even when trained with comparable in-domain parallel data volumes.
Evidence: The same trained DS2ST-LM system scores BLEU 24.57 on fr-en but only 5.41 on urd-en, with human adequacy dropping correspondingly from 3.38 (fr-en) to 1.62 (urd-en). (§IV-E4, Tables 10–11; Appendix Table VI)
Limitations and Open Questions
The target-language speech in the paper's largest and best-performing training corpus (GigaS2S-1000) is entirely synthetic, generated by XTTS-v2 from a single Chinese speaker's voice, not natural human recordings — the strongest reported results may partly reflect this corpus's unusually clean, single-speaker acoustic-semantic alignment rather than the model's ability to handle real bilingual speech variability.
The paper’s own conclusion acknowledges that performance remains constrained by the language coverage of the Whisper encoder, Qwen2 LLM, and CosyVoice vocoder, and that low-resource language pairs (Urdu, Bengali) continue to lag substantially behind higher-resource pairs. The subjective evaluation is comparatively small in scale: 15 evaluators rating only 20 samples per model per language pair. The projector and semantic-token ablations are conducted only on the GigaS2S-1000/Seamless-Align zh-en setting and are not re-verified across the other six language pairs the system is extended to.
Wiki Connections
- Speech-to-Speech — presents a single-stage LLM-based direct S2ST system directly compared against cascaded and ST+TTS baselines across seven language pairs.
- Spoken Language Model — feeds external source-language speech (via a frozen Whisper encoder and learned projector) into a Qwen2 LLM that autoregressively predicts target semantic tokens and text jointly.
- Neural Audio Codec — relies on a CosyVoice-style supervised semantic tokenizer (VQ bottleneck inside a split ASR encoder) as its target-token vocabulary.
- Zero-Shot TTS — the timbre-controllable synthesis stage conditions on a short reference-speaker prompt to reproduce unseen speaker identities, independent of the translated content.
- Multilingual TTS — the same trained architecture is extended across seven target/source language pairs spanning European and Indic languages.
- CosyVoice — DS2ST-LM’s semantic tokenizer, flow-matching mel decoder, and HiFi-GAN vocoder chain directly follow CosyVoice’s supervised semantic-token design.
- SLAM-Omni — DS2ST-LM’s overall single-stage encoder-projector-LLM-vocoder architecture and semantic-token grouping strategy are adapted from SLAM-Omni’s conversational framework.
- HiFi-GAN — used as the final waveform vocoder converting predicted mel-spectrograms to audio.
- VALL-E X — cited as an earlier cross-lingual neural codec language model preserving speaker identity via zero-shot prompt conditioning, motivating this paper’s own timbre-control design.
- XTTS — used to synthesize the 1,000 hours of Chinese target speech that constitutes the GigaS2S-1000 training corpus.
- AudioPaLM — cited as prior work combining pretrained speech encoders with an LLM decoder for speech tasks including translation.
- Whisper — its encoder (frozen) serves as the source-speech front end for DS2ST-LM.