arXiv · 2025 · Preprint
Sina Rashidi et al. (Sharif University of Technology) · → Paper · Demo: ? · Code: ?
Presents a direct, textless Persian-English speech-to-speech translation system for audio dubbing and a synthetic-data pipeline that roughly sextuples the amount of available Persian-English parallel speech using LLM translation and zero-shot TTS synthesis.
Problem
Direct (textless) S2ST models learn a single mapping from source speech to target speech, avoiding the error propagation and latency of cascaded ASR-MT-TTS pipelines, which is attractive for audio dubbing where many utterances must be processed with limited delay. However, direct S2ST is data-hungry, and existing parallel speech corpora are concentrated in a small set of high-resource language pairs. Persian-English has almost no parallel speech data, so training a direct S2ST system for this pair means confronting data scarcity directly rather than as a secondary concern, and any dubbing-oriented system must additionally produce natural-sounding target speech, not just accurate content.
Method
The system follows the discrete-unit direct S2ST paradigm and has three components. First, a Conformer-based speech encoder is initialized from self-supervised pretraining: the encoder architecture is wav2vec 2.0 with its Transformer layers replaced by Conformer layers, and it is pretrained on unlabeled Persian speech from Common Voice using a wav2vec-2.0-style contrastive objective (masking random spans and distinguishing the true latent representation of the masked region from negative examples). The pretrained encoder is then fine-tuned jointly with the rest of the S2ST model. Second, target English speech is represented as discrete units: frame-level latents are learned by HuBERT (a self-supervised speech model) and clustered into a finite codebook, so each target utterance becomes a sequence of discrete indices. A causal Transformer decoder, using relative-position multi-head attention rather than absolute positional encodings, autoregressively predicts this target unit sequence conditioned on the encoder output, trained with a cross-entropy objective (optionally with label smoothing) over unit indices. A convolutional length adapter bridges the differing temporal resolutions of Persian and English speech before decoding. Third, a HiFi-GAN unit vocoder, trained separately on English speech and unit sequences derived from LJSpeech and then frozen, converts the predicted unit sequence into a waveform. SpecAugment (time and frequency masking) is applied to encoder inputs during training for robustness.
To address data scarcity, the authors construct a synthetic Persian-English parallel speech corpus in three stages: Persian speech and transcriptions are drawn from Common Voice; each Persian transcription is translated into English using GPT-4o; and the translated English text is synthesized into speech using VoiceCraft, a zero-shot TTS system, yielding an English utterance aligned at the utterance level with the original Persian recording. This process is repeated at scale, producing a corpus roughly six times larger than the existing CVSS Fa-En parallel set. Two versions of the S2ST model are trained: one on CVSS Fa-En only, and one on CVSS Fa-En combined with the synthetic corpus.
Key Results
On the CVSS Fa-En evaluation set (ASR BLEU, Table 1), the proposed model trained on CVSS-only data reaches 4.1 BLEU, ahead of the strongest baseline, Translatotron 2 with pretraining, at 3.8 BLEU, and clearly ahead of Translatotron (1.4) and a speech-to-unit baseline with pretraining (2.8), all trained on the same data. Adding the synthetic corpus raises the proposed model to 17.8 BLEU, a gain of 13.7 BLEU from data alone, and the CVSS+Synthetic proposed model outperforms every other CVSS+Synthetic baseline reported, including the pretrained speech-to-unit baseline (13.2) and Translatotron (6.9); Translatotron 2 with pretraining is not reported in the CVSS+Synthetic condition. The paper frames its overall gain as 4.6 BLEU relative to direct baselines when combining the architecture with synthetic data, though the largest single jump in the table (13.7 BLEU for the proposed model, CVSS-only to CVSS+Synthetic) comes from adding data rather than from the architectural changes alone (0.3 BLEU over the best CVSS-only baseline). All baselines are the authors’ own implementations rather than the original released systems, since neither Translatotron nor the speech-to-unit baseline is natively trained for Persian-English translation. Beyond BLEU, the authors report only qualitative listening observations, noting that synthetic-data-trained models produce more fluent, complete translations on longer utterances; no naturalness or speaker-similarity metric (MOS, SPK-SIM, etc.) is reported.
Novelty Assessment
The architecture is an assembly of established components: a Conformer encoder with wav2vec-2.0-style self-supervised pretraining, HuBERT-derived discrete target units, a causal Transformer decoder with relative-position attention (an existing technique applied to this decoder), and a HiFi-GAN unit vocoder, following the general discrete-unit S2ST recipe established by prior work (direct S2ST with discrete units, UnitY). No new training objective, decoding procedure, or model component is introduced, and the architectural gain over the strongest pretrained baseline under matched training data is small (0.3 BLEU). The more substantive contribution is the synthetic-data pipeline itself: using an LLM for text translation and a zero-shot TTS system for target-speech synthesis to sextuple the available Persian-English parallel data, and the empirical demonstration that this synthetic data drives most of the reported improvement. This is a useful engineering and data-construction result for a specific, previously underserved low-resource language pair, but it is not a new data-generation technique in itself; synthesizing parallel speech from LLM-translated text is a direct application of existing MT and TTS tools rather than a novel method.
Field Significance
Low. The paper’s contribution is narrowly scoped: a single low-resource language pair (Persian-English), evaluated on one benchmark (CVSS Fa-En), using a system assembled from existing self-supervised pretraining, discrete-unit, and vocoder components. It demonstrates that combining encoder pretraining with a synthetic parallel corpus built from off-the-shelf LLM translation and zero-shot TTS is an effective recipe for scaling direct S2ST when real parallel data is scarce, but the paper does not release the synthetic corpus, code, or a demo, and reports no naturalness or speaker-similarity evaluation for the generated dubbed speech.
Claims
- supports: Self-supervised pretraining of the source-language speech encoder improves direct speech-to-speech translation quality in low-resource settings, independent of architectural changes on the target side.
Evidence: Adding self-supervised pretraining to the speech-to-unit baseline architecture, with no other change, raises ASR BLEU from 1.6 to 2.8 on CVSS-only training, and the proposed pretrained encoder outperforms Translatotron 2 with pretraining (4.1 vs. 3.8 BLEU) under identical training data. (§5.3, Table 1)
- supports: Constructing synthetic parallel speech data by machine-translating existing transcriptions with a large language model and synthesizing the translated text with a zero-shot TTS system can substantially improve direct speech-to-speech translation for a low-resource language pair, even though none of the synthetic target audio is real recorded speech.
Evidence: Training the proposed model on CVSS Fa-En augmented with the synthetic corpus (roughly six times the parallel data of CVSS alone) raises ASR BLEU from 4.1 to 17.8, and produces consistent gains for every baseline architecture tested (e.g. Translatotron 1.4 to 6.9, pretrained speech-to-unit 2.8 to 13.2). (§5.3, Table 1)
- complicates: In low-resource direct speech-to-speech translation, gains from encoder and decoder architectural refinements are small relative to gains available from increasing parallel training data volume.
Evidence: Under matched CVSS-only training data, the proposed architecture (Conformer encoder with relative-position decoder) outperforms the strongest pretrained baseline by only 0.3 BLEU (4.1 vs. 3.8), while adding the synthetic corpus to the same proposed architecture, with no architectural change, produces a 13.7 BLEU improvement (4.1 to 17.8). (§5.3, Table 1)
- supports: Representing target speech as discrete self-supervised units, rather than continuous spectrograms, lets a direct speech-to-speech translation system match or exceed spectrogram-based approaches once combined with source-encoder pretraining.
Evidence: With pretraining and CVSS-only training data, the discrete-unit speech-to-unit baseline (2.8 BLEU) and the proposed unit-based model (4.1 BLEU) both exceed the spectrogram-based Translatotron (1.4 BLEU), and the proposed model also exceeds pretrained Translatotron 2 (3.8 BLEU). (§6, Table 1)
Limitations and Open Questions
The system and synthetic-data pipeline are evaluated on a single low-resource language pair (Persian-English) and a single benchmark (CVSS Fa-En); no results are reported for other language pairs, so it is untested whether the synthetic-data gains generalize beyond this specific pair.
No naturalness, speaker-similarity, or other subjective/perceptual metric is reported for the dubbed output; evaluation relies entirely on ASR BLEU, which measures translation content but not speech quality or speaker preservation, both stated goals of dubbing. The synthetic corpus itself inherits translation errors from the LLM and voice characteristics of a single zero-shot TTS system (VoiceCraft), which could bias the trained model toward that system’s acoustic properties in ways not analyzed in the paper. Neither the synthetic corpus, code, nor a demo appears to be released, limiting reproducibility and reuse by other low-resource language pairs.
Wiki Connections
- Speech-to-Speech — the paper is a direct, textless S2ST system for Persian-English audio dubbing, combining an established discrete-unit S2ST architecture with a synthetic-data augmentation pipeline.
- Self-Supervised Speech — the source-language Conformer encoder is initialized from wav2vec-2.0-style contrastive self-supervised pretraining on unlabeled Persian speech, and target units are derived from HuBERT.
- Neural Audio Codec — target English speech is represented as discrete HuBERT-derived units obtained via clustering, the same discretization strategy used by codec- and unit-based S2ST systems more broadly.
- Multilingual TTS — the system is trained end-to-end to map Persian source speech to English target speech, a cross-lingual generation setting for a previously underserved language pair.
- VoiceCraft — VoiceCraft’s zero-shot TTS capability is used to synthesize the English half of the paper’s synthetic Persian-English parallel corpus from LLM-translated text.
- Common Voice — the Persian portion of Common Voice supplies both the unlabeled speech for self-supervised encoder pretraining and the source utterances used to build the synthetic parallel corpus.