arXiv · 2026 · Preprint
Jing Xu et al. (The Chinese University of Hong Kong / Huawei Artificial Intelligence Laboratory (Leibniz)) · → Paper · Demo: ? · Code: ?
Introduces PROST-LLM, which progressively boosts an LLM’s speech-to-speech translation ability through tri-task/chain-of-modality supervised fine-tuning followed by preference optimization on preference pairs constructed automatically via back-translation, without any human-labeled preference data.
Problem
LLM-based direct speech-to-speech translation (S2ST) offers a simpler architecture than cascaded ASR-MT-TTS pipelines or discrete-unit sequence-to-sequence models, but paired S2ST training data remains scarce, limiting how well LLMs can learn the task. Separately, preference optimization (RLHF-style alignment, or its offline variants DPO and SimPO) has proven effective for aligning LLM outputs with human preferences in NLP, but applying it to S2ST is blocked by the same data problem: constructing preference pairs conventionally requires costly human evaluation of candidate translations, and no prior work had investigated how to build S2ST preference data automatically.
Method
PROST-LLM proceeds in three stages. First, supervised fine-tuning (SFT) equips a pretrained LLM (LLaMA 3.2-3B) with initial S2ST ability using the CVSS corpus. Speech is represented as discrete units extracted via mHuBERT and K-means clustering, added to the LLM’s vocabulary and embedding matrix so the model can be fine-tuned end-to-end to translate speech units across languages; a separately trained unit HiFi-GAN vocoder reconstructs waveforms from the generated target-language units. To strengthen this initial stage under limited paired S2ST data, the paper introduces two training strategies: tri-task learning, which converts each S2ST triple into parallel ASR and speech-to-text (S2T) supervision so the model jointly learns ASR, S2T, and S2ST with task-specific instructions; and chain of modality, where the model is instructed to generate the target-language text before the target-language speech units within a single forward pass, giving the model an explicit textual bridge between the two modalities.
Second, the SFT model is used to construct preference data without human labeling. For each source-language utterance, the fine-tuned model generates two candidate target-language translations, which are then back-translated into the source language using the same model. Because supervised fine-tuning already ensures the candidates are of reasonable quality, comparing each back-translated candidate against the original source utterance provides an automatic proxy for translation quality. The paper evaluates four scoring signals for this comparison: an acoustic measure (Mel-Cepstral Distortion), a text-based measure (Word Error Rate on Whisper-large-v3 transcriptions), and two translation measures (BLEU and METEOR against the source text). A candidate pair is only kept as a valid preference pair if the score gap between the two candidates exceeds a margin hyperparameter δ, discarding ambiguous comparisons.
Third, the SFT model is further fine-tuned with Direct Preference Optimization (DPO) using the constructed preference pairs, maximizing the model’s probability margin between the preferred and rejected candidate given the source speech. SFT uses full fine-tuning (4 epochs, batch size 64, lr 1e-4); the preference-optimization stage applies LoRA (rank 8) to all linear layers except the LM head, for 2 epochs at batch size 32 and lr 2e-5, with margin δ = 0.1 and 5,000 preference samples per translation direction by default. Because back-translation only requires source-language speech (not target-language speech), the same procedure also works with monolingual speech corpora rather than paired S2ST data, which the paper tests directly.
Key Results
All experiments target English-French translation on the CVSS corpus (CVSS-C: single-speaker target speech; CVSS-T: multi-speaker/voice-cloned target speech). Before preference optimization, chain-of-modality SFT outperforms both tri-task learning and a vanilla LLM-S2ST baseline on S2ST BLEU across both corpora and both translation directions, and produces more natural speech (higher UTMOS) than the cascaded ASR-MT-TTS baseline. Preference optimization then provides a further, consistent boost on top of every SFT variant: on CVSS-C, the BLEU gap between the end-to-end system and the cascaded baseline narrows from 14.38 to 3.15 (en2fra) and from 8.83 to 1.04 (fra2en) after DPO. Substituting preference pairs built from monolingual Common Voice speech (rather than the paired CVSS corpus) for the chain-of-modality model narrows this gap further still, to 1.65 (en2fra) and 1.2 (fra2en). Results are robust to which automatic scoring metric is used to build the preference pairs (WER, MCD, BLEU, or METEOR all yield comparable downstream gains, with MCD showing a small edge for en2fra) and to which preference-optimization algorithm is used (SimPO performs comparably to DPO). An ablation over preference-pair sample size and iteration count shows diminishing returns once a training strategy already achieves strong SFT performance: for chain-of-modality, doubling the DPO training budget from one 5,000-sample iteration to two 2,500-sample iterations gives no further gain, while the weaker vanilla and tri-task-learning variants continue to improve with more iterations.
Novelty Assessment
The core novelty is procedural rather than architectural: this is presented as the first framework to apply preference optimization to LLM-based S2ST, and the first to construct the required preference pairs automatically via back-translation instead of human evaluation, which directly targets the field’s persistent S2ST data-scarcity problem. The tri-task and chain-of-modality SFT designs are themselves incremental (task-multiplexing and modality-bridging are established ideas from LLM/speech-LM literature), and the underlying speech representation (mHuBERT units + separately trained HiFi-GAN vocoders) and preference-optimization algorithms (DPO, SimPO) are unmodified off-the-shelf components. The genuinely new contribution is the back-translation-based, human-annotation-free preference-pair construction pipeline and the demonstration that it works even with unpaired monolingual speech.
Field Significance
high — this paper establishes that preference optimization, a technique previously validated mainly for text-only LLM alignment, transfers effectively to speech-to-speech translation once the human-labeling bottleneck for preference data is removed via automatic back-translation scoring, and that the resulting recipe further reduces (rather than increases) the field's dependence on scarce paired S2ST data by working with monolingual corpora.
Claims
- supports: Preference optimization can substantially improve LLM-based direct speech-to-speech translation quality even when the preference pairs used for training are constructed entirely automatically, without any human evaluation.
Evidence: Applying DPO to back-translation-derived preference pairs narrows the BLEU gap between the end-to-end LLM system and a matched cascaded ASR-MT-TTS baseline from 14.38 to 3.15 (en2fra) and from 8.83 to 1.04 (fra2en) on CVSS-C. (§4.2, Table 1)
- supports: Explicitly generating intermediate target-language text before target-language speech tokens within a single autoregressive pass more effectively bridges the speech-text modality gap for direct S2ST than multi-task training on correlated tasks alone.
Evidence: Chain-of-modality SFT outperforms tri-task-learning SFT on S2ST BLEU across both CVSS-C and CVSS-T, in both translation directions, before any preference optimization is applied (e.g., CVSS-C en2fra 24.20 vs. 18.68). (§4.2, Table 1)
- supports: Preference-pair construction via back-translation scoring generalizes beyond paired parallel corpora, allowing monolingual speech data to substitute for scarce paired speech-to-speech translation data during preference optimization.
Evidence: Preference pairs built from separate monolingual Common Voice English and French speech achieve BLEU scores matching or exceeding those built from the paired CVSS-T corpus, further narrowing the cascaded-system gap to 1.65 (en2fra) and 1.2 (fra2en). (§4.3, Table 2)
- complicates: The specific automatic metric used to score and select back-translation preference pairs has only a secondary effect on downstream translation quality once preference optimization is applied, undercutting the idea that translation-specific metrics (BLEU, METEOR) are necessary for constructing useful preference signals over acoustic or ASR-based ones.
Evidence: Substituting WER or Mel-Cepstral Distortion for BLEU or METEOR as the back-translation scoring metric yields broadly comparable post-DPO S2ST BLEU across settings, with MCD showing only a small edge specific to English-to-French translation. (§4.4, Table 3)
- complicates: Additional preference-optimization training samples and iterations yield diminishing or no further gains once a model’s supervised fine-tuning stage already achieves strong initial task performance, making the benefit of more preference data conditional on how much headroom the SFT baseline leaves.
Evidence: For the chain-of-modality variant, increasing from one 5,000-sample DPO iteration to two 2,500-sample iterations produces no further BLEU improvement, while the weaker vanilla and tri-task-learning variants continue to benefit from the additional iteration. (§4.5, Table 5)
Limitations and Open Questions
All experiments are restricted to a single language pair (English-French) using the CVSS corpus; the paper's own conclusion explicitly defers extension to more languages and LLM backbones to future work, so the generality of the back-translation preference-construction approach to typologically distant or lower-resource language pairs is untested.
The back-translation preference signal depends on Whisper-large-v3 transcription accuracy for both the WER-based and translation-metric-based scoring variants; systematic ASR errors on either the original or back-translated speech could bias which candidate is labeled “preferred” without the paper measuring this failure mode directly. The monolingual-preference-data comparison (§4.3) and the metric-robustness and PO-algorithm-robustness ablations (§4.4, §4.5) are each conducted only on the chain-of-modality SFT variant, so it is not established whether these findings hold equally for the tri-task-learning or vanilla variants.
Wiki Connections
- Spoken Language Model — fine-tunes a pretrained LLaMA-3.2 text LLM end-to-end on discretized mHuBERT speech units to perform direct speech-to-speech translation.
- Speech-to-Speech — proposes an LLM-based direct S2ST training recipe evaluated against a matched cascaded ASR-MT-TTS baseline on the CVSS corpus.
- RLHF Speech — applies DPO and SimPO preference optimization to a speech generation task using preference pairs constructed automatically via back-translation, without human feedback.
- Evaluation Metrics — systematically compares four automatic scoring signals (WER, MCD, BLEU, METEOR) for constructing preference-pair supervision and analyzes their downstream effect on translation quality.
- Whisper — used to transcribe both source-language and back-translated speech for the WER-based and translation-metric-based preference scoring signals.
- Common Voice — supplies the monolingual English and French speech used to construct preference pairs independent of the paired CVSS corpus.
- CoVoST 2 — one of the two corpora (with Common Voice) that CVSS itself is derived from.
- Llama 3 Herd of Models — provides the LLaMA 3.2-3B backbone that PROST-LLM fine-tunes for speech unit generation.