EMNLP · 2025 · Conference
Jeongsoo Choi et al. (Korea Advanced Institute of Science and Technology) · → Paper · Demo: ? · Code: ?
Proposes a textless speech-to-speech translation framework for cross-lingual dubbing that explicitly controls output duration and speaking speed while preserving the source speaker’s identity, without relying on any text intermediate.
Problem
Automatic dubbing requires more than accurate translation: the output speech must also match the source utterance’s duration, speaking pace, and speaker identity, or the result sounds unnatural even when the words are correct. Cascaded ASR-NMT-TTS pipelines lose speaker identity and prosodic information through the intermediate text representation, and because text carries no precise timing information, cascaded systems typically need post-hoc waveform stretching to match source duration, which degrades quality. Textless speech-to-speech translation (S2ST) systems avoid the text bottleneck but, in their existing form, still lack native duration control, so their outputs also require post-processing for dubbing use. A further complication is that large-scale S2ST training data (e.g., synthesized-target or web-mined corpora) rarely preserves consistent speaker identity or speaking speed between source and target, making it hard for models to learn to retain these attributes even if the architecture supports it.
Method
Dub-S2ST is a two-stage textless S2ST pipeline: a speech-to-unit translation model followed by a unit-to-speech synthesizer. Source and target speech are first converted into discrete units using mHuBERT with k-means quantization, a representation designed to retain linguistic content while suppressing speaker-dependent acoustic variation. Before training, a unit-based speed adaptation step adjusts the repetition rate of target units according to the ratio between source and target speaking speed (estimated from unit-sequence deduplication ratios), so the model learns to produce target sequences whose implicit pace matches the source without any reliance on text or syllable counts.
The speech-to-unit translation model uses a Conformer encoder over source speech features and a discrete-diffusion decoder built from Diffusion Transformer (DiT) layers with cross-attention to the encoder output. Training follows a masked discrete-diffusion objective: units are masked according to a schedule γ(t), and the decoder is trained with cross-entropy loss restricted to masked positions, conditioned on the encoder output and the diffusion timestep. At inference, the decoder is initialized with a fully masked sequence whose length is fixed to the source’s unit-sequence length, which gives the model explicit duration control without any post-hoc stretching. The encoder is initialized from a pretrained autoregressive speech-to-unit translation model and fine-tuned end-to-end.
Translated units are then converted to speech using a unit-to-speech synthesizer based on Optimal-Transport Conditional Flow Matching (OT-CFM), implemented as a U-Net of convolutional and transformer blocks operating on mel-spectrogram latents. The synthesizer is conditioned on the translated unit embeddings, the source speaker embedding (from a pretrained speaker-verification model), and the source mel-spectrogram, concatenated channel-wise to enable in-context conditioning on both speaker identity and source prosody. This module is initialized from a pretrained multilingual zero-shot TTS model and fine-tuned with the semantic-unit input, giving it zero-shot synthesis ability across speakers and languages. A pretrained HiFi-GAN vocoder converts the resulting mel-spectrogram to a waveform.
Key Results
On the CVSS-C French-English test split, the single-speaker configuration (Dub-S2ST-single) reaches 23.88 ASR-BLEU, outperforming other duration-controllable baselines, while the full multi-speaker Dub-S2ST reaches 24.16 ASR-BLEU and the best speaker similarity (SPK-SIM 0.266) among all compared systems, versus 0.036-0.145 for baselines that lack explicit speaker conditioning. On duration and speed control, Dub-S2ST achieves 100% duration compliance at both the 20% and 40% tolerance thresholds, compared to 56-81% for prior duration-controllable baselines (VideoDubber-style and isochrony positional-embedding approaches) under the stricter 20% threshold. Removing the proposed speed adaptation step drops speed compliance and speed correlation substantially (e.g., speed correlation falls from 0.614 to 0.388 for the single-speaker model), confirming its contribution independent of the duration-matching mechanism. In a 15-listener MOS study, Dub-S2ST-single scores 3.37 naturalness versus 2.52-2.60 for cascaded/duration-uncontrolled baselines whose outputs required manual waveform stretching to match source duration for fair comparison. Ablations show that a DiT decoder outperforms a standard Transformer decoder at matched inference budgets, and that computing the training loss only on masked (not all) unit positions improves ASR-BLEU by roughly 0.6-0.9 points.
Novelty Assessment
The contribution is primarily architectural, combining two mechanisms that are individually established (discrete diffusion decoding, OT-CFM speech synthesis) but not previously combined for duration- and speed-controllable dubbing: fixed-length masked-diffusion decoding gives exact duration control without post-hoc stretching, and the unit-based speed adaptation method is a genuinely new data-preprocessing technique that requires no text or syllable-level speed estimate. The unit-to-speech synthesizer itself is an engineering integration, built by fine-tuning a pretrained multilingual TTS backbone on semantic-unit input rather than introducing a new synthesis architecture. The evaluation is conducted on a single language pair (French-English) from one dataset (CVSS-C), which limits evidence about generalization across language pairs with different syllable-rate and duration statistics.
Field Significance
Moderate — this paper demonstrates that explicit, text-free duration control is achievable in textless S2ST through fixed-length masked-diffusion decoding, and that speaking-speed matching can be learned purely from unit-sequence statistics without syllable or phoneme information. It provides a concrete architectural template for duration- and speed-aware S2ST that other dubbing-focused systems can build on, though the evidence is currently confined to one language pair and one training corpus.
Claims
- supports: Fixing the target sequence length at inference time, combined with a masked-diffusion decoder trained on variable-length masking, gives non-autoregressive S2ST models exact duration control without post-hoc time-stretching.
Evidence: Dub-S2ST reaches 100% duration compliance at both the 20% and 40% tolerance thresholds by initializing decoding with a fully masked sequence whose length matches the source unit sequence, compared to 56-81% for prior duration-controllable baselines under the 20% threshold. (§5.1, Table 2)
- supports: Speaking-speed adaptation for cross-lingual translation can be learned directly from discrete unit-sequence deduplication statistics, without relying on text, phonemes, or syllable counts.
Evidence: The unit-based speed adaptation ratio (reduced-sequence length over original length) correlates strongly with ground-truth syllable speed (correlation 0.606), and training on speed-adapted sequences raises the correlation between generated and source syllable speed from 0.235 to 0.519. (§5.3, Table 4)
- complicates: Conditioning a speech synthesizer on the source speaker embedding and source mel-spectrogram to preserve speaker identity in cross-lingual translation trades off against acoustic naturalness relative to a single-speaker-only configuration.
Evidence: The multi-speaker Dub-S2ST achieves the best speaker similarity (SPK-SIM 0.266) but a lower DNSMOS (3.693) than the single-speaker Dub-S2ST-single (DNSMOS 3.945), which does not condition on source speaker identity. (§5.1, Table 1)
- complicates: Applying a separately trained voice-conversion model to a translation system’s output, rather than conditioning the synthesizer jointly on speaker identity during training, can degrade translation intelligibility.
Evidence: Applying a cross-lingual voice-conversion model to the single-speaker Dub-S2ST-single output raises speaker similarity (SIM 0.315) but lowers ASR-BLEU from 23.88 to 23.09, which the authors attribute to oversmoothing of pronunciation introduced by the voice-conversion process. (§5.4, Table 8)
Limitations and Open Questions
The system is trained and evaluated on a single language pair (French-English) from one dataset (CVSS-C), which is recorded in controlled, single-speaker-target conditions; the paper does not report results on other language pairs or in-the-wild acoustic conditions.
The model also operates at sentence level and requires an external segmentation step before it could be applied to longer-form video or dialogue content, which the authors note as a limitation for real-world dubbing pipelines. Training data quality (studio-recorded rather than in-the-wild speech) may limit robustness to noisy or reverberant source audio encountered in practical dubbing applications.
Wiki Connections
- Speech-to-Speech — Dub-S2ST is a direct, textless speech-to-speech translation system built specifically for the dubbing use case, contributing explicit duration and speed control to this sub-paradigm.
- Flow Matching — the unit-to-speech synthesizer is trained with an Optimal-Transport Conditional Flow Matching objective conditioned on translated units, source speaker embedding, and source mel-spectrogram.
- Diffusion TTS — the speech-to-unit translation model uses a discrete-diffusion Diffusion Transformer decoder to predict target speech units, applying diffusion-style masked prediction to a translation task rather than direct speech generation.
- Zero-Shot TTS — the unit-to-speech synthesizer is initialized from a pretrained multilingual zero-shot TTS model and retains zero-shot synthesis capability across speakers and languages after fine-tuning.
- Neural Audio Codec — the framework represents speech as discrete self-supervised units (mHuBERT plus k-means) rather than a neural codec, illustrating an alternative discretization route for S2ST that this paper builds its duration-control mechanism around.
- CosyVoice — the unit-to-speech synthesizer is initialized from CosyVoice-300M and fine-tuned on semantic-unit input; CosyVoice’s voice-conversion variant is also used as an ablation baseline.
- Flow Matching for Generative Modeling — the unit-to-speech synthesizer’s OT-CFM training objective is a direct application of this flow-matching formulation.
- HiFi-GAN — a pretrained HiFi-GAN vocoder converts the synthesizer’s output mel-spectrogram into the final audio waveform.