arXiv · 2025 · Preprint
Yifan Yang et al. (Microsoft / Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✗
PALLE introduces pseudo-autoregressive (PAR) codec language modeling — a new paradigm combining AR temporal ordering with NAR parallel generation — achieving state-of-the-art zero-shot TTS quality on LibriTTS-trained models while enabling up to 10x faster inference than AR systems, and outperforming large-scale-trained systems including F5-TTS, E2-TTS, and MaskGCT.
Problem
Autoregressive TTS models have strong temporal modeling but suffer O(T) inference latency, exposure bias, and limited bidirectional context. Non-autoregressive models like MaskGCT and E2-TTS generate tokens in a temporally unordered manner, causing alignment inaccuracies, robustness issues in high-entropy regions (fast transitions, expressive prosody), and poor convergence. Grouped AR methods (VALL-E 2) improve efficiency but still scale linearly with speech duration. No prior work had unified AR temporal grounding with NAR parallel decoding while achieving constant-step inference independent of target speech duration.
Method
PAR Formulation. The pseudo-autoregressive (PAR) paradigm is built on a bidirectional masked generative transformer. During training, a contiguous suffix of speech tokens is masked from a random start position. The model is trained to predict only the leftmost span of size k=⌊rT⌋ of the masked region, where r is a fixed ratio. The mask start position is dynamically sampled, enabling the model to predict the next span given varying amounts of left context. Full attention is used across all positions — this is the key departure from standard AR, which restricts attention to left context only.
During inference, the model generates all token positions in parallel at each step, but commits only to the leftmost k’ tokens. The committed tokens are treated as given context in subsequent steps. This creates span-level causal ordering (earlier spans committed before later ones) with O(1) inference steps regardless of target duration, since the number of steps equals 1/r’ (a constant ratio of the target length).
PALLE Architecture. Both stages share a bidirectional masked generative transformer backbone: 12 decoder layers, 16 attention heads, 1024-dim embeddings, 4096-dim FFN (177M params). A one-layer ConvNeXt V2 block (1024-dim, 2048-dim FFN) processes text embeddings for strong temporal alignment with the speech sequence. Sinusoidal positional encoding with learnable scaling plus convolutional positional embedding encode temporal structure. Text and speech sequences are fused in the feature dimension (as in E2-TTS) after padding text with [PAD] filler tokens.
Speech Tokenizer. The S3Tokenizer v2 from CosyVoice 2 is used (25 Hz supervised semantic tokens from a fine-tuned SenseVoice-Large ASR model). Reconstruction uses the CosyVoice 2 CFM model with HiFi-GAN vocoder.
Stage One (PAR Generation). The first stage implements PAR with a span ratio r’=0.01 at inference (100 steps total). Each step predicts all positions in parallel and commits the leftmost 1% of remaining tokens.
Stage Two (NAR Refinement). The second stage refines low-confidence tokens from stage one. A confidence score (negative min-entropy of predicted distributions) ranks all positions; those below a predefined quantile γ=0.05 are re-masked and re-predicted in parallel, using global context. Seven refinement steps are used. Stage two is fine-tuned from stage one’s checkpoint.
Training details: Stage one trained for 179k steps, stage two for 87k steps; both on 8x NVIDIA V100 32GB GPUs; ScaledAdam optimizer with Eden scheduler; peak lr 0.045 (stage 1) and 0.005 (stage 2).
Key Results
PALLE two-stage achieves WER-H 2.83 and SIM-o 0.716 on the cross-sentence task of LibriSpeech test-clean — the best WER among all systems evaluated, outperforming F5-TTS (WER-H 2.77, SIM-o 0.705), E2-TTS (WER-H 2.92, SIM-o 0.756), and MaskGCT (WER-H 4.22, SIM-o 0.756), all of which are trained on 100k+ hours of data. PALLE achieves this with only LibriTTS (580h). Real-Time Factor (RTF) is 0.06 — over 10x faster than F5-TTS (RTF 0.15) and 10x faster than MaskGCT (RTF 0.65).
With GT duration, PALLE achieves WER-H 2.62 and SIM-o 0.776 on the continuation task. Subjective evaluation (CMOS): PALLE two-stage (GT duration) scores CMOS −0.09 relative to ground truth, outperforming MaskGCT (−0.25), E2-TTS (−0.41), and F5-TTS (−0.20). SMOS 4.09 vs. F5-TTS 4.04.
Controlled comparison (Table 3) confirms the PAR paradigm itself: PAR (PALLE stage-one) achieves 36% relative WER-W reduction and 4x speedup vs. AR (VALL-E stage-one) under identical conditions, with comparable speaker similarity. PAR also yields 43% WER-W reduction and 2x speedup vs. NAR (MaskGCT stage-one) under the same conditions.
Novelty Assessment
The PAR paradigm is a genuinely new modeling abstraction with a clean theoretical formulation. The key insight is that span-level causal commitment can be implemented entirely within a bidirectional masked generative model without any modification to its architecture — the soft inductive bias is enforced only through the training masking strategy and the inference commitment rule. The result is a model that achieves O(1) inference steps regardless of target duration while maintaining temporal coherence. The ablation study isolating AR vs. NAR vs. PAR under controlled conditions is particularly rigorous. The paper is accepted at ACM MM 2025 (proceedings version arXiv 2504.10352).
Field Significance
Tip
High — PALLE introduces the pseudo-autoregressive paradigm as a principled third option between AR and NAR codec language modeling, demonstrating that span-level causal commitment within a bidirectional masked transformer achieves O(1) inference complexity while preserving temporal coherence. The controlled three-way comparison (AR vs. NAR vs. PAR under identical conditions) provides unusually clean evidence for the efficiency-quality trade-off analysis, and the result that a small-scale-trained system can outperform 100k-hour-trained NAR systems on intelligibility challenges assumptions about the data requirements of non-sequential generation.
Claims
- Pseudo-autoregressive generation, which commits spans left-to-right within a bidirectional masked transformer, achieves constant inference steps regardless of target speech duration while maintaining temporal coherence. (§3, §5.4)
- A model trained on 580 hours of English speech can match or exceed the intelligibility of NAR flow-matching systems trained on 100,000+ hours when temporal ordering is explicitly enforced during generation. (§5.3, Table 1)
- Confidence-guided iterative NAR refinement of an initial PAR generation substantially reduces word error rate with only a small number of additional inference steps. (§5.5, Figure 4)
- Temporally unordered NAR generation produces higher alignment errors than span-level causal generation across both continuation and cross-sentence evaluation tasks. (§5.4, Table 3)
- Separate model capacity for each generation stage is necessary; unifying PAR and NAR refinement into a single multitask model degrades cross-sentence intelligibility by approximately 20%. (§5.5)
Limitations and Open Questions
PALLE is evaluated only on English (LibriTTS). The 100-step inference (with 7 refinement steps) may still be too slow for the most latency-sensitive streaming applications despite the 10x speedup. Duration estimation for the cross-sentence task relies on a simple linear heuristic; errors in duration estimation lead to modest quality degradation (WER-H 2.83 vs. 2.62 with GT duration). The shared architecture between stage one and stage two (joint multitask fine-tuning) causes stage two loss to degrade stage one performance, suggesting that separate model capacity is required.
Wiki Connections
PALLE establishes PAR as a third paradigm alongside AR and NAR for autoregressive-codec-tts and zero-shot-tts. The use of the S3Tokenizer from CosyVoice 2 (supervised semantic tokens) links to neural-codec. The confidence-guided refinement stage is conceptually related to iterative decoding in MaskGCT (NAR masked generative models), which is a pattern also discussed in the autoregressive-codec-tts concept page. The RTF of 0.06 positions PALLE as among the fastest TTS systems for streaming-tts applications that do not require true token-by-token streaming. Back-linked by VoiceCraft-X, which cites PALLE in related work on efficient AR TTS. Also cited by Dragon-FM, which compares the PAR paradigm with the hybrid chunk-AR+FM approach as competing strategies for reducing sequential decoding bottlenecks. Also cited by TaDiCodec, which builds on codec-LM foundations and proposes text-conditioned diffusion decoding as an alternative. IndexTTS2 cites PALLE in its discussion of efficient autoregressive zero-shot TTS approaches.