arXiv · 2025 · Preprint
Chary et al. (Universidade de São Paulo) · → Paper · Demo: ? · Code: ?
LatinX applies Direct Preference Optimization to a three-stage multilingual autoregressive TTS system, improving cross-lingual intelligibility and objective speaker similarity while uncovering a significant divergence between automated similarity metrics and human perceptual judgments.
Problem
Zero-shot voice cloning in multilingual or cross-lingual speech-to-speech translation is hampered by two interconnected problems: models trained with standard supervised objectives often generate outputs with accumulated intelligibility errors across languages, and speaker identity preservation is difficult to verify because automated speaker similarity metrics do not reliably track what human listeners perceive. Prior autoregressive TTS systems lack a principled way to jointly optimize for intelligibility and perceptual voice fidelity, relying instead on supervised fine-tuning that gives no explicit signal about preference between competing outputs.
Method
LatinX is a 12-layer, 210M parameter decoder-only Transformer modeled after the LLaMA 2 architecture, operating on discrete audio tokens produced by a custom VQ-VAE codec (Spectrogram Patch Codec, 4096-entry codebook with 4x downsampling of mel-spectrograms). A HiFi-GAN vocoder converts reconstructed mel-spectrograms back to waveforms. Phoneme input is handled by LatPhon, a lightweight Transformer-based G2P model designed for Romance languages and English. The context window is 8,192 tokens, partitioned into 256 tokens for phoneme input, 3,968 tokens for the acoustic prompt (a 3-second voice reference clip), and 3,968 tokens for autoregressive generation. Training uses RoPE positional embeddings and FlashAttention for efficiency.
The model is trained across three sequential stages. Stage 1 is standard cross-entropy pre-training on a filtered 9,736-hour corpus spanning English, Portuguese, French, Spanish, Italian, and Romanian (roughly 40% Portuguese by design). Stage 2 fine-tunes on speaker-conditioned triplets where audio prompt, text, and target audio share the same speaker, enforcing a cosine similarity threshold of 0.6 between utterances to ensure acoustic consistency. Stage 3 applies DPO using an automatically constructed preference dataset: five candidate outputs are generated per text, filtered to remove samples with WER above 20% or speaker similarity below 0.5, and Pareto-dominance labeling selects pairs where the preferred output wins on both WER (measured by Whisper) and speaker similarity (measured by TitaNet embeddings). Training on the resulting balanced preference pairs for 4,000 steps uses the standard DPO loss with inverse temperature beta=0.3. At inference the model achieves a real-time factor of approximately 4.85 on an NVIDIA RTX 4090 in float16 precision.
Key Results
DPO alignment produces consistent WER reductions across nearly all cross-lingual pairs. LatinX (DPO) achieves an average WER of 8.30% for English as the source language, compared to 14.10% for the fine-tuned baseline and 10.93% for XTTSv2. For the low-resource Romanian source, LatinX (DPO) achieves 0.45% WER on the ro-to-es pair and 1.74% on ro-to-pt (Table 2). XTTSv2 maintains an advantage for Portuguese-origin pairs such as pt-to-en and pt-to-fr.
On objective speaker similarity (TitaNet cosine), XTTSv2 achieves the highest Sim-O (original audio) score, but LatinX (DPO) matches XTTSv2 on Sim-E (similarity between codec-reconstructed reference and generated audio), the metric that directly reflects the model’s training target (Table 4).
Human evaluations reveal a different picture for perceptual similarity. On the SMOS listening test, both LatinX models substantially outperform XTTSv2, with LatinX (DPO) averaging 3.54 and LatinX (fine-tuned) averaging 3.63 across the five main languages, compared to XTTSv2 at 3.24 (Table 5). On naturalness MOS, however, XTTSv2 retains an advantage at 3.45 versus 3.35 for LatinX (DPO) and 3.41 for the fine-tuned version (Table 6). The SMOS results for Romanian (4.47 for DPO model, exceeding the real audio baseline at 3.98) should be treated cautiously given the large confidence intervals and the predominantly non-native Romanian rater pool.
Novelty Assessment
The system assembles well-established components: a decoder-only Transformer trained autoregressively on codec tokens (the VALL-E paradigm), DPO (Rafailov et al., 2024), and HiFi-GAN as a vocoder. The primary novelty is the automatic preference pair construction methodology for TTS, specifically the Pareto-dominance labeling criterion that selects preference pairs where the winner beats the loser on both WER and speaker similarity simultaneously, avoiding ambiguous single-metric labels. The multilingual scope across six Romance languages and the cross-lingual evaluation design also add value. The codec (Spectrogram Patch Codec) is the authors’ own, though it is a variant of VQ-VAE applied to mel-spectrograms rather than a fundamentally new approach. The contribution is methodological and application-focused rather than architectural; the paper is best understood as demonstrating that preference alignment transfers effectively to multilingual voice-cloning TTS and surfacing the discrepancy between objective and subjective similarity measures.
Field Significance
Moderate - This paper provides early evidence that DPO can serve as an effective post-training alignment stage for autoregressive codec-based TTS, extending preference optimization beyond text LLMs into the multilingual speech generation domain. The careful cross-lingual evaluation design and the well-documented divergence between objective TitaNet-based similarity and human SMOS judgments offer a useful methodological reference point for future work designing preference datasets and evaluation protocols in multilingual TTS.
Claims
-
supports: Preference-based alignment (DPO) improves intelligibility in multilingual autoregressive TTS systems beyond what supervised fine-tuning achieves.
Evidence: LatinX (DPO) reduces WER across nearly all 30 cross-lingual language pairs compared to the supervised fine-tuned baseline, and outperforms XTTSv2 in most pairs; Romanian-source conditions show particularly large gains (e.g., ro-to-es at 0.45% WER). (§4.3, §5.2, Table 2, Table 3)
-
complicates: Automated speaker similarity metrics based on speaker encoder embeddings do not reliably reflect human perceptual judgments of voice identity in zero-shot TTS.
Evidence: XTTSv2 achieves higher Sim-O scores than both LatinX models, yet human evaluators strongly prefer LatinX speaker similarity (SMOS 3.63/3.54 vs. 3.24); the paper explicitly flags this as a divergence between objective and subjective evaluation. (§5.2, §6, Table 4, Table 5)
-
complicates: DPO alignment in TTS involves a trade-off: optimizing for intelligibility and objective similarity can reduce naturalness MOS and, in some language conditions, perceptual similarity relative to the fine-tuned baseline.
Evidence: LatinX (DPO) improves WER and Sim-E over the fine-tuned model but achieves lower average MOS (3.35 vs. 3.41) and lower SMOS in several cross-lingual conditions; the paper attributes this partly to the codec introducing artifacts that cap perceptual quality. (§5.2, §6, Table 5, Table 6)
-
complicates: Lossy neural audio codecs set a perceptual quality ceiling in codec-based TTS that preference alignment cannot overcome, because the model learns to replicate codec artifacts introduced during reference encoding.
Evidence: The paper notes that the VQ-VAE codec is lossy and the model learns to reproduce its artifacts, limiting the maximum perceptual quality achievable regardless of post-training alignment method. (§6)
Limitations and Open Questions
Warning
The evaluation uses an internal test set of unseen speakers with no publicly named benchmark, and the human rating pool is predominantly English and Portuguese native speakers. Conclusions about multilingual naturalness and similarity, especially for French, Italian, and Romanian, should be treated with caution.
The DPO preference signal is constructed solely from WER and speaker similarity; no prosody, naturalness, or rhythm metric is incorporated, which likely explains the MOS regression relative to the fine-tuned baseline. The preference labeling is fully automated with no human verification of winner/loser assignments. The real-time factor of 4.85 makes the system unsuitable for real-time applications, and the authors note that non-autoregressive architectures are a necessary direction. The Romanian evaluation suffers from very small rater counts and predominantly non-native listeners, undermining the interpretation of the unusually high SMOS scores that exceed real audio.
Wiki Connections
- Multilingual TTS - this paper presents a six-language TTS system with emphasis on Portuguese and provides detailed cross-lingual WER and similarity analyses.
- Zero-Shot TTS - LatinX performs zero-shot voice cloning from a 3-second audio prompt across source and target language pairs.
- Autoregressive Codec TTS - the model follows the VALL-E paradigm of autoregressive decoding over discrete codec tokens from a custom VQ-VAE codec.
- RLHF Speech - the paper applies DPO with automatically constructed WER and speaker-similarity preference pairs, extending preference alignment to multilingual TTS.
- Neural Audio Codec - the system uses a custom Spectrogram Patch Codec (mel-based VQ-VAE + HiFi-GAN vocoder) and identifies codec lossy reconstruction as a quality ceiling.
- Subjective Evaluation - the paper conducts ITU-T P.808 crowdsourced MOS and SMOS evaluations and documents a pronounced divergence between automated speaker similarity metrics and human perceptual judgments.