arXiv · 2025 · Preprint
Hyeongju Kim et al. (Supertone, Inc.) · → Paper · Demo: ? · Code: ?
Proposes LARoPE, a length-normalized extension of rotary position embedding for cross-attention TTS that induces a diagonal inductive bias aligned with the monotonic structure of text-speech alignment, yielding consistent gains in pronunciation accuracy, speaker similarity, and stability under long-form synthesis up to 30 seconds.
Problem
Standard RoPE encodes positional information using absolute token indices, which creates a mismatch when applied to cross-attention layers where query (speech) and key (text) sequences differ substantially in length. The resulting attention score map does not align along a diagonal, conflicting with the monotonic nature of text-speech correspondence. Existing RoPE extensions such as position interpolation, YaRN, and LongRoPE target self-attention extrapolation and do not address this cross-attention alignment problem. Attention-based TTS models trained with standard RoPE therefore suffer from mispronunciations, skips, and degraded performance on long utterances without relying on auxiliary alignment tools.
Method
LARoPE modifies the RoPE rotation matrix by normalizing each token’s positional index by its sequence length before computing the rotary transformation. In cross-attention, the speech query at position m in a sequence of length L_q and the text key at position n in a sequence of length L_k are each embedded with indices m/L_q and n/L_k, scaled by a hyperparameter γ (set to 10 in experiments). The inner product then depends on the difference of length-normalized indices rather than absolute indices, and the resulting upper bound on the attention score preserves a diagonal structure regardless of the ratio L_q/L_k. This directly mirrors the monotonic alignment expected between speech frames and phoneme tokens.
The method is evaluated within the SupertonicTTS framework, which uses flow matching with cross-attention text conditioning. Four configurations are compared by crossing the positional embedding (RoPE vs. LARoPE) with a batch expansion factor (K_e = 1 or K_e = 4). Training uses a composite dataset of LJSpeech, VCTK, Hi-Fi TTS, and LibriTTS, resampled to 44.1 kHz, with AdamW optimization over 700k iterations at a batch size of 64 on four RTX 4090 GPUs. The model contains 19M parameters excluding the vocoder. LARoPE requires no external aligners, auxiliary losses, or changes to model size or inference cost.
Key Results
On the tc-short subset of LibriSpeech test-clean (4–10 seconds), LARoPE with K_e = 4 achieves 2.25% WER, 0.50 SIM, and 3.21 UTMOS compared to RoPE’s 2.41% WER, 0.48 SIM, and 3.23 UTMOS (Table 1). The advantage is substantially larger on tc-long (10–30 seconds): LARoPE K_e = 4 reaches 2.16% WER versus RoPE’s 4.98%, nearly halving pronunciation errors on longer utterances (Table 1). Under duration scaling, LARoPE consistently outperforms RoPE across factors of 0.7x to 1.4x; at the extremes (0.7x and 1.4x), WER for RoPE rises to 6.03% and 7.39%, while LARoPE holds at 4.77% and 5.40% (Table 2).
In head-to-head comparison against other zero-shot TTS models using RoPE, the 19M LARoPE model (WER 2.25%, RTF 0.05) outperforms E2 TTS (333M, 2.95%), F5-TTS (336M, 2.42%), DiTTo-TTS (740M, 2.56%), and the SupertonicTTS RoPE baseline (19M, 2.41%), all at the same or higher computational cost (Table 3). Training convergence is also accelerated: at 200k iterations without batch expansion, LARoPE reduces CER from 2.00% to 1.23%, a gap that persists throughout training (§4.3, Figure 2).
Novelty Assessment
The core contribution is a simple one-equation change to the RoPE positional encoding: divide the absolute position index by the sequence length before computing the rotation angle. The idea is motivated by a clean theoretical argument (diagonal bias preservation in the attention upper bound) and supported by consistent empirical gains across short and long utterances, duration variations, and training speed. The novelty is narrow but well-targeted: the modification is specific to the cross-attention alignment problem in TTS and does not transfer directly to self-attention or other sequence modeling tasks. The evaluation uses established benchmarks (LibriSpeech tc-short/tc-long) with fair comparisons to reproduced baselines under identical conditions. The paper does not compare LARoPE against other alignment strategies (e.g., guided attention loss or diagonal priors in MAS), which limits understanding of how it ranks among the broader space of alignment improvement techniques.
Field Significance
Moderate — LARoPE provides a lightweight and principled fix to a real but underappreciated problem in attention-based TTS: the diagonal misalignment introduced by applying standard RoPE in cross-attention with mismatched sequence lengths. The contribution is narrow (positional embedding modification only) but the results are consistent and the theoretical motivation is clear. The approach can serve as a drop-in replacement for RoPE in any cross-attention TTS system without added parameters or inference cost.
Claims
-
supports: Length-normalized positional encoding in cross-attention TTS substantially improves alignment accuracy for long-form speech synthesis, where standard absolute-index RoPE degrades.
Evidence: LARoPE with K_e=4 reduces WER on tc-long (10–30s) from 4.98% to 2.16%, while RoPE’s WER on tc-long is nearly double that of tc-short (2.41%), indicating progressive degradation with utterance length. (§4.2, Table 1)
-
supports: A diagonal inductive bias in cross-attention positional embeddings accelerates text-speech alignment learning during training.
Evidence: At 200k training iterations without batch expansion, LARoPE reduces CER from 2.00% to 1.23% versus RoPE, with the performance gap maintained throughout 700k iterations. (§4.3, Figure 2)
-
complicates: Standard RoPE is suboptimal in cross-attention TTS because relative positional distances based on absolute indices do not align diagonally when query and key sequence lengths differ, violating the monotonic text-speech alignment structure.
Evidence: Analysis of relative upper bounds shows that RoPE’s attention score map loses diagonal structure whenever L_q ≠ L_k (e.g., L_q=64, L_k=256), whereas LARoPE preserves the diagonal regardless of length ratio. (§3, Figure 1)
-
supports: Compact attention-based TTS systems can achieve pronunciation accuracy competitive with models more than an order of magnitude larger through improved positional embedding design rather than parameter scaling.
Evidence: The 19M-parameter LARoPE model achieves 2.25% WER on LibriSpeech test-clean, outperforming E2 TTS (333M params, 2.95% WER), F5-TTS (336M, 2.42%), and DiTTo-TTS (740M, 2.56%) at RTF 0.05. (§4.5, Table 3)
-
refines: Existing RoPE extensions (position interpolation, YaRN, LongRoPE) address self-attention length extrapolation but leave the cross-attention alignment problem in TTS unresolved.
Evidence: The paper identifies that all prior RoPE extensions are designed for self-attention and do not modify the relative distance formulation for cross-attention with differing query and key lengths; LARoPE specifically targets this gap. (§5.1)
Limitations and Open Questions
The evaluation is conducted entirely within the SupertonicTTS framework; generalisability to other cross-attention TTS architectures (such as masked diffusion or AR codec models with cross-attention) is asserted but not tested. The paper does not benchmark LARoPE against alternative alignment improvement strategies such as guided attention loss or diagonal priors in monotonic alignment search, leaving its relative advantage over the broader toolkit unclear. The scaling hyperparameter γ is set to 10 without ablation across values. All experiments use English-language data, and performance on morphologically diverse or tonal languages with different text-to-speech length ratios is unknown.
Wiki Connections
- Zero-Shot TTS — evaluated on the standard zero-shot TTS tc-short benchmark and compared against F5-TTS, E2 TTS, and DiTTo-TTS; achieves the lowest WER (2.25%) among attention-based zero-shot TTS systems in Table 3.
- Flow Matching — LARoPE is integrated into the SupertonicTTS flow-matching framework; the positional embedding improvement is orthogonal to the generative objective.
- Transformer Encoder-Decoder TTS — the paper targets cross-attention architectures where speech features serve as queries and text embeddings as keys, the structure used by encoder-decoder TTS systems.
- Evaluation Metrics — uses WER (CTC-based HuBERT-large ASR), SPK-SIM (WavLM-TDNN cosine distance), and UTMOSv2 as objective proxies; evaluates on tc-short and tc-long splits derived from LibriSpeech test-clean.