arXiv · 2026 · Preprint
Chihiro Arata et al. · → Paper · Demo: ? · Code: ✓
T5Gemma-TTS is an encoder-decoder codec language model that solves the text-conditioning dilution problem of decoder-only autoregressive TTS by routing bidirectional text representations through cross-attention at every decoder layer, and couples this with Progress-Monitoring RoPE (PM-RoPE) to provide reliable duration control in multilingual zero-shot voice cloning.
Problem
Decoder-only neural codec language models (e.g. VALL-E, Llasa) prepend input text as a prefix to the growing audio token sequence. As generation proceeds, the audio sequence length grows while the text prefix length stays fixed; text tokens consequently occupy an ever-smaller fraction of each attention window, and their influence weakens as output length increases. This dilution is most harmful for long-utterance TTS (audiobooks, document reading). Additionally, without explicit positional grounding, autoregressive models have no signal for how far through target-length speech they have progressed, making duration control inconsistent.
Method
T5Gemma-TTS is an autoregressive encoder-decoder sequence-to-sequence model built on the T5Gemma pretrained backbone (2B encoder + 2B decoder = 4B parameters total), which was originally pretrained on large-scale multilingual text via the Unifying Language Learning (UL2) objective combining span corruption, prefix-LM, and extreme masking.
Audio tokenization. The model uses XCodec2, a single-codebook neural codec operating at 50 Hz with vocabulary size 65,536. XCodec2’s single-codebook avoids the interleaved multi-codebook prediction required by EnCodec’s residual vector quantization and produces shorter token sequences than DAC (86 Hz), reducing memory cost. The T5Gemma-TTS codebase uses a Japanese fine-tuned variant of XCodec2 as its default (encoder identical to the base, decoder fine-tuned on Japanese speech); this is consistent across all evaluated languages in the metrics.
Text encoder. The T5Gemma encoder has 26 transformer layers with model dimension d=2304. It operates directly on SentencePiece subword tokens (vocabulary 256,000), covering English, Chinese, and Japanese without any language-specific phoneme conversion. This preserves the pretrained multilingual embedding weights at the cost of losing the monotonic phoneme-audio correspondence.
Audio decoder. The decoder mirrors the encoder depth (26 layers, d=2304) and autoregressively predicts the next audio token. The reference speech prompt waveform is prepended to the generation target in the token sequence, separated by a prompt-separator token, enabling zero-shot voice cloning. A reference prompt of approximately 3 seconds is used.
Progress-Monitoring RoPE (PM-RoPE). Adopted verbatim from VoiceStar, PM-RoPE is applied to all 26 cross-attention layers. It encodes normalized generation progress into cross-attention queries (decoder side) and keys (encoder side). Given target token count S and encoder length T, progress position IDs are p_j^dec = (j/(S-1)) * 2000 and p_i^enc = (i/(T-1)) * 2000, with separate RoPE modules applied to each. This allows the decoder to attend to encoder positions proportional to its generation stage, providing continuous duration signals without requiring explicit duration labels during training.
Duration estimation at inference. Target speech duration is estimated from the phoneme count ratio between reference and target text: D_hat = (D_ref / N_ref) * N_tgt. Language-specific tools supply phoneme counts (espeak-ng for English, pyopenjtalk for Japanese, character count for Chinese). Out-of-training languages fall back to the English estimator. Once the target token count S_hat is computed (floor of D_hat * 50), the decoder progress ID at each step is set accordingly.
Training. The model is trained with next-token cross-entropy loss over audio tokens. AdamW optimizer with peak learning rate 1e-4, weight decay 1e-2, 2% linear warmup followed by linear decay over ~143,000 steps. Dynamic batching at ~240,000 tokens per parameter update across 8 AMD MI300X GPUs, training for approximately two weeks in bfloat16 mixed precision. Corpus: ~100k hours of English (LibriHeavy), ~50k hours of Chinese (Emilia), ~20k hours of Japanese (Emilia and other sources), totaling ~170k hours.
Inference sampling uses top-k=30, top-p=0.9, temperature=0.8.
Key Results
Evaluation uses 100 randomly sampled utterances per test set across six languages. All systems evaluated in zero-shot voice cloning mode using ground-truth reference audio. Metrics: CER/WER via Whisper large-v3, SPK-SIM via ECAPA-TDNN, UTMOS via UTMOS22 strong predictor. Baselines: F5-TTS, XTTSv2, CosyVoice 2, Kokoro (not a zero-shot system).
Japanese (JSUT, in-training): T5Gemma-TTS achieves the numerically lowest CER (0.126 ± 0.018) and the highest SIM (0.677 ± 0.016) among zero-shot systems. The SIM advantage over XTTSv2 (0.622 ± 0.017) is statistically significant with non-overlapping 95% CIs. The CER advantage over Kokoro (0.139 ± 0.016) should be interpreted cautiously due to partial CI overlap.
Mandarin Chinese (AISHELL-1, in-training): T5Gemma-TTS achieves the second-highest SIM among zero-shot systems (0.722 ± 0.017); F5-TTS scores higher (0.864) but at the cost of severely degraded intelligibility (CER 0.155).
English (LibriSpeech, in-training, upper bound): Results must be treated as an upper-bound estimate because LibriHeavy (a training source) is a superset of LibriSpeech test-clean. UTMOS 4.01 ± 0.11 compared to CosyVoice 2’s best-in-class 4.32 and Kokoro’s 4.51.
Korean (FLEURS, out-of-training): T5Gemma-TTS achieves the numerically highest SIM (0.747 ± 0.029) despite Korean being absent from training, though the CI overlaps with XTTSv2 (0.741 ± 0.010) and the advantage is not statistically conclusive. Korean CER (0.082) is competitive with CosyVoice 2 (0.090).
French and German (FLEURS, out-of-training): T5Gemma-TTS shows elevated WER (French 0.475, German 0.453), substantially worse than XTTSv2 (0.080 and 0.060), which was explicitly trained on these languages.
PM-RoPE configuration analysis (Table 3): Disabling PM-RoPE at inference on the same checkpoint causes near-complete synthesis failure: CER degrades from 0.129 to 0.982 (non-overlapping CIs), SIM drops from 0.666 to 0.109, UTMOS from 3.85 to 2.25, and Duration Accuracy drops from 79% to 46%. This is a flag-switch experiment rather than a controlled from-scratch ablation; the result reflects PM-RoPE’s inference-time effect on a checkpoint trained with PM-RoPE active throughout.
Real-time factor on AMD MI300X: 0.8–2.0 (varies by utterance length), higher than non-autoregressive baselines.
Novelty Assessment
The primary architectural contribution is the combination of a large pretrained encoder-decoder LLM (T5Gemma, 4B parameters) with PM-RoPE cross-attention for multilingual codec TTS. Neither component is new individually: encoder-decoder TTS dates back to Tacotron, and PM-RoPE was introduced by VoiceStar for English-only TTS trained from scratch. The genuine novelty lies in:
- Demonstrating that PM-RoPE generalizes beyond English monolingual training when integrated into a large multilingual pretrained backbone and scaled to ~170k hours across three typologically diverse languages.
- Showing that subword-level text input (rather than phonemes) to PM-RoPE cross-attention preserves duration control across languages without language-specific phonemizers.
- The surprising Korean cross-lingual generalization despite Korean being entirely absent from training, attributed to the T5Gemma backbone’s multilingual representations and typological proximity to East Asian training languages.
The contribution is primarily a training-recipe and architecture-combination paper rather than a fundamental methodological advance. The PM-RoPE analysis is a flag-switch experiment rather than a controlled ablation, which limits causal claims about the training-time contribution of PM-RoPE specifically. The paper is also transparent about the English evaluation being an upper-bound estimate due to LibriHeavy/LibriSpeech overlap.
Field Significance
Moderate — T5Gemma-TTS demonstrates that encoder-decoder LLM backbones can provide persistent, dilution-free text conditioning for autoregressive codec TTS, offering a concrete alternative to the dominant decoder-only paradigm at the cost of higher inference latency. The paper’s primary value is empirical: it provides the first multi-language validation of PM-RoPE duration control at scale and contributes evidence on cross-lingual generalization from typologically adjacent East Asian training languages to unseen Korean.
Claims
- Encoder-decoder architectures avoid the text-conditioning dilution present in decoder-only codec language models by routing bidirectional text representations through dedicated cross-attention at every decoder layer. (§1, §2.1)
- Progress-Monitoring RoPE duration control, originally validated for English-only TTS, generalizes to multilingual autoregressive synthesis when integrated into a large pretrained encoder-decoder backbone. (§3.2, §5.3, Table 3)
- Subword tokenization can substitute for language-specific phoneme conversion in autoregressive encoder-decoder TTS while preserving intelligibility across typologically diverse training languages. (§2.2, §5.1)
- Cross-lingual speaker similarity generalization is stronger for languages typologically adjacent to the training distribution than for phonologically distant European languages unseen during training. (§5.2)
- Explicit duration control mechanisms improve autoregressive TTS robustness substantially, but removing them at inference from a checkpoint trained with them causes near-complete synthesis failure rather than graceful degradation. (§4.2, §5.3, Table 3)
Limitations and Open Questions
- Higher WER on unseen European languages (French, German) compared to systems explicitly trained on those languages indicates limited generalization to typologically distant out-of-training languages.
- RTF of 0.8–2.0 on AMD MI300X makes the system unsuitable for real-time applications; the authors note it is positioned for offline batch TTS (audiobooks, document synthesis).
- Lower UTMOS naturalness compared to XTTSv2 and Kokoro, attributed to XCodec2’s quantization ceiling and the absence of a diffusion or flow-matching refinement stage.
- The PM-RoPE configuration analysis uses the same trained checkpoint in two modes rather than two separately trained models, so the training-time contribution of PM-RoPE cannot be disentangled from its inference-time effect.
- The effect of phoneme vs. subword input on PM-RoPE duration control effectiveness is not ablated.
- Duration estimation for out-of-training languages uses a language-agnostic English fallback, introducing estimation error.
- The Japanese fine-tuned XCodec2 decoder may slightly disadvantage non-Japanese reconstruction quality, though the evaluation metrics are computed consistently across languages.
- Future directions mentioned: distillation and speculative decoding for faster inference, latent diffusion or flow-matching post-processing for improved naturalness, continual pretraining on additional languages.
Wiki Connections
This paper advances autoregressive-codec-tts by demonstrating that encoder-decoder architectures solve the text-conditioning dilution problem inherent in decoder-only codec LMs, at the cost of higher inference latency. It contributes to zero-shot-tts with a 4B-parameter voice cloning system requiring ~3s reference audio, and to multilingual-tts through training on three typologically diverse languages and evaluation on six. The PM-RoPE mechanism is a prosody-control technique (specifically, explicit duration control in autoregressive generation) originally proposed in VoiceStar. XCodec2 represents a neural-codec design choice favoring single-codebook simplicity over residual vector quantization. The encoder-decoder architecture places this work in the transformer-enc-dec-tts lineage alongside FastSpeech 2 and Tacotron, though with autoregressive decoding rather than the non-autoregressive parallel synthesis of classical enc-dec systems.
In-corpus papers cited: VALL-E, Seed-TTS, F5-TTS, CosyVoice 2.