ACL 2026 (Findings) · 2026 · Conference

Puyuan Peng et al. (The University of Texas at Austin / Rembrand) · → Paper · → Preprint · Demo: ✓ · Code: ✓

VoiceStar replaces absolute positions in an autoregressive encoder-decoder codec language model with progress fractions (Progress-Monitoring RoPE), which gives the model a user-specified output duration and lets it generate speech longer than any training utterance while keeping zero-shot voice cloning.

Problem

Neural codec language models for zero-shot TTS, such as VALL-E, VoiceCraft, CosyVoice and Llasa, concatenate text and speech tokens in one decoder and leave the text-speech alignment to standard positional encodings. This has three consequences the paper targets: robustness errors (skipped, repeated or inserted words, long silences), no way to specify the length of the output, and failure when the requested sequence is longer than the maximum context seen in training. Non-autoregressive models such as F5-TTS and MaskGCT accept a target duration, but in the paper’s tests they also degrade beyond their training length. A second issue is a train/inference mismatch: continuation-style training always pairs a prompt with the rest of the same utterance, so prosody and emotion of the prompt carry over into the target, while at inference the prompt and target text can differ in content and affect.

Method

VoiceStar is an encoder-decoder Transformer (author term: neural codec language model). The encoder reads IPA phonemes (espeak-ng) of the prompt transcript and the target text, separated by a learnable separator token. The decoder autoregressively predicts tokens of the 4-codebook, 50 Hz EnCodec model released with VoiceCraft, with a delay pattern across codebooks; prompt speech tokens and target speech tokens are also separated by a separator token, and the loss is computed on target tokens only. The main model has 840M parameters (12 encoder and 40 decoder layers, hidden size 1024, 16 heads); ablation models have 230M parameters.

Progress-Monitoring RoPE (PM-RoPE). Standard RoPE rotates queries and keys by angles proportional to absolute positions. PM-RoPE instead rotates by the fraction of the sequence covered, scaled to a fixed pseudo length N (2000 in all experiments): a source token at position s of S is rotated by (s/S)Nθ and a target token at position t of T by (t/T)Nθ. The attention score then depends on the difference in relative progress, t/T minus s/S. In decoder self-attention this tells each speech token how far it is through a target length T chosen by the user, and the model learns to emit end-of-generation when progress reaches 1, which is the duration control mechanism. In cross-attention it gives a flat-start alignment prior: a speech token attends most strongly to phonemes at the same relative position. Applying PM-RoPE in the encoder as well puts both sequences in the same interval [0, N], so a longer sequence at inference is a denser sampling of that interval rather than unseen positions (interpolation rather than extrapolation).

Figure 2: Left: The architecture of VOICESTAR. Right: the common general architecture for zero-shot TTS models, such as VALL-E (AR part), VoiceCraft, CosyVoice, Llasa etc. VOICESTAR differs from them in three aspects: 1) it uses an encoder-decoder architecture with PM-RoPE to provide text-speech alignment, duration control, and extrapolation capability; 2) it uses prompt-continuation mixed training to mitigate the training/inference mismatch and enhance robustness. Our speech tokenizer based on Encodec (Defossez et al., 2022) uses multiple codebooks and we apply the delay pattern (Copet et al., 2023) to them, which is not depicted in this figure for simplicity.

Figure 3: An example on how PM-ROPE turns extrapolation into interpolation: during training, the maximal training sequence length is 4, and during inference the target length is 7. The positional encodings for both can be expressed as sampling points inside the same interval [0, N].

Continuation-Prompt Mixed (CPM) training. With probability p, a training example pairs two different utterances of the same speaker (one as prompt, one as target); otherwise it uses ordinary continuation of a single masked utterance. Because the prompt is a separate utterance, speed perturbation (factor within ±0.25) can be applied to the prompt alone. Speaker labels come from Emilia’s automatic diarization and from Libriheavy. CPM also enables prompt repetition at inference: repeating the reference audio and transcript several times raises speaker similarity, which the authors attribute to the reference tokens receiving more attention weight.

Training and inference. The main model is trained on about 65K hours (the English part of Emilia, about 40K hours after filtering, plus a 25K-hour sample of Libriheavy and Libriheavy-long), using only utterances of at most 30 s. It is trained for 50K steps plus 18K steps with re-weighted codebook losses, on 8 L40 and 16 GH200 GPUs for 8 days, with ScaledAdam and the Eden scheduler. Inference uses top-k sampling (k = 10). For short-form test sets the target duration is estimated from the prompt’s speaking rate times the target’s character count; long-form and ablation runs use the ground-truth duration.

Key Results

Ablations (230M models, Libriheavy validation, Table 2). Moving from a decoder-only model with sinusoidal positions to an encoder-decoder with RoPE lowers WER from 11.04 to 8.82 and raises UTMOS from 2.767 to 3.301, but duration error (DurDiff, absolute difference between target and generated duration, in seconds) rises from 1.245 to 1.812. Replacing RoPE with PM-RoPE cuts DurDiff to 0.009, which the authors note is the floor given the 0.02 s token resolution, and slightly improves WER (8.49), UTMOS (3.337) and SpkSim (0.601). CPM training lowers WER to 6.42, and adding prompt speed augmentation to 5.66, with SpkSim falling slightly from 0.601 to 0.587 and then 0.583. In the extrapolation ablation (trained up to 20 s, tested at 20-30 s, Table 3), PM-RoPE on both encoder and decoder gives WER 6.75 against 11.46 for RoPE on both; PM-RoPE on the decoder only gives 28.67, so the encoder side is required for extrapolation. The decoder-only ablation model is not size-matched to the encoder-decoder ones (16 layers, hidden size 1024, against 10 encoder and 16 decoder layers at hidden size 768).

Prompt repetition (Figure 5, Figure 7). For VoiceStar, SpkSim rises from about 0.63 to 0.66 as repetitions go from 0 to 3 with little change in WER, and both degrade with more repetitions. Applied to F5-TTS, repeating the prompt up to the training length raises WER from 3.1 to 75.4.

Short-form zero-shot TTS (Table 4, baselines reproduced by the authors). On LibriSpeech-PC, VoiceStar has the highest S-CMOS (0.60 ± 0.19; next CosyVoice 2 0.40 ± 0.19) and the highest N-CMOS (0.18 ± 0.21; next Llasa 1B 0.13 ± 0.20, with overlapping intervals), both measured against ground truth. Its automatic scores are not the best: WER 2.64 against 2.57 for F5-TTS, and SpkSim 0.63 against 0.66 for CosyVoice, CosyVoice 2 and MaskGCT and 0.65 for F5-TTS. On Seed-TTS eval (en), baselines beat VoiceStar on most columns: WER 2.15 against 1.78 for F5-TTS; SpkSim 0.63 against 0.71 (MaskGCT), 0.66 (CosyVoice 2, F5-TTS) and 0.64 (CosyVoice); S-CMOS -0.32 ± 0.15 against -0.11 (CosyVoice 2), -0.13 (MaskGCT), -0.18 (F5-TTS) and -0.23 (Llasa 1B); N-CMOS -0.18 ± 0.14 against 0.05 (Llasa 1B) and -0.14 (MaskGCT). The abstract’s “outperforms or is on par” on short-form therefore holds for the LibriSpeech-PC listening tests but not for Seed-TTS (en), where VoiceStar is mid-table. The reproduced numbers for FireRedTTS and VoiceCraft differ from their published ones (FireRedTTS LibriSpeech-PC WER 7.73 reproduced against 3.82 reported; Table 6).

Long-form zero-shot TTS (Table 5, ground-truth durations, only duration-controllable baselines). VoiceStar has the lowest WER and the highest intelligibility MOS (I-MOS) and naturalness MOS (N-MOS) in every range:

RangeWER (VoiceStar / MaskGCT / F5-TTS)I-MOSN-MOS
20-30 s (within training length)3.53 / 3.91 / 5.313.75 / 3.55 / 3.473.45 / 3.07 / 2.97
30-40 s (extrapolation)7.27 / 13.81 / 34.154.16 / 3.06 / 2.213.69 / 2.79 / 2.15
40-50 s (extrapolation)11.91 / 82.29 / 52.443.23 / 1.42 / 1.943.23 / 1.61 / 2.35

Speaker similarity is weaker. MaskGCT has higher automatic SpkSim at 20-30 s (0.76 vs 0.72) and 30-40 s (0.75 vs 0.70, F5-TTS tied with VoiceStar at 0.70), and higher S-MOS at 20-30 s (4.29 vs 4.13) and 30-40 s (4.27 vs 4.11). At 40-50 s VoiceStar ties F5-TTS on SpkSim (0.70 vs MaskGCT 0.65) but has the lowest S-MOS of the three (3.94 vs 4.04 for F5-TTS and 3.95 for MaskGCT), so the paper’s description of VoiceStar as “a close second on every range” for speaker similarity does not hold for S-MOS in that range. With durations estimated from the prompt instead of ground truth (Table 8), VoiceStar’s WER rises to 4.60, 8.29 and 17.33 for the three ranges but remains below both baselines (MaskGCT 4.74, 26.39, 77.24; F5-TTS 7.94, 33.34, 50.28).

Novelty Assessment

The architectural contribution is PM-RoPE: a small change to RoPE (normalizing positions by sequence length and rescaling to a fixed pseudo length) that, applied to both encoder and decoder of an autoregressive codec model, yields duration control, an alignment prior and length extrapolation together. The ablations isolate each effect cleanly (Tables 2 and 3). Each capability on its own has precedents the paper acknowledges: non-autoregressive models such as F5-TTS and MaskGCT already take a target duration, and Very Attentive Tacotron extrapolates to long inputs but without zero-shot cloning or duration control. The authors’ description of VoiceStar as the first zero-shot TTS model with both duration control and extrapolation is their own positioning (Table 1). CPM training is a modest extension of different-utterance prompting used in T5-TTS, adding the prompt transcript and stochastic mixing with continuation; prompt repetition is an inference trick that CPM makes usable.

The comparisons have caveats. Short-form gains appear only in the LibriSpeech-PC listening tests; on Seed-TTS (en) VoiceStar trails several baselines on every metric. The long-form test sets come from Libriheavy, the same audiobook source as 25K hours of VoiceStar’s training data, while F5-TTS and MaskGCT were trained on their own corpora, so part of the long-form margin may reflect domain match. The long-form comparison includes only two non-autoregressive baselines, since autoregressive baselines cannot be forced to a target duration. All baselines were rerun by the authors, and two (FireRedTTS, VoiceCraft) reproduce differently from their published numbers.

Field Significance

Moderate — The paper demonstrates that a positional-encoding change can give an autoregressive codec TTS model explicit output-duration control and generation well beyond its training length, a pair of properties its own baselines lack, with ablations showing which part of the design is responsible. Its short-form quality is comparable to contemporary zero-shot systems rather than ahead of them, so the contribution is a mechanism for length and duration behaviour more than a quality advance.

Claims

  • supports: Normalizing positional encodings by total sequence length lets an autoregressive speech-token decoder hit a user-specified output duration almost exactly.

    Evidence: Replacing RoPE with PM-RoPE in an encoder-decoder codec model reduces the absolute duration error from 1.812 s to 0.009 s, the floor set by the 0.02 s token resolution, while WER, UTMOS and SpkSim also improve slightly. (§4.2, Table 2)

  • supports: Mapping text and speech positions to a shared fixed interval allows codec-token TTS to generate utterances longer than any seen in training without collapsing intelligibility.

    Evidence: Trained on at most 30 s, VoiceStar keeps WER at 7.27 (30-40 s) and 11.91 (40-50 s) while F5-TTS reaches 34.15 and 52.44 and MaskGCT 13.81 and 82.29; in the ablation, PM-RoPE on the decoder alone gives WER 28.67 at 20-30 s versus 6.75 when applied to both encoder and decoder. (§4.2, Table 3; §4.4, Table 5)

  • supports: Mixing same-speaker different-utterance prompts with continuation prompts in training improves the intelligibility of zero-shot codec language model TTS, at a small cost in automatic speaker similarity.

    Evidence: CPM training lowers WER from 8.49 to 6.42 and prompt speed augmentation to 5.66, while SpkSim moves from 0.601 to 0.587 and 0.583. (§4.2, Table 2)

  • complicates: Repeating the reference prompt can raise speaker similarity in prompt-based TTS, but whether it is safe depends on how the model was trained.

    Evidence: For VoiceStar (CPM-trained), up to three repetitions raise SpkSim from about 0.63 to 0.66 with little WER change; for F5-TTS (continuation-trained), repeating the prompt to the training length raises WER from 3.1 to 75.4. (§4.2, Figure 5; §Appendix B.1, Figure 7)

  • complicates: Automatic speaker-similarity scores and human speaker-similarity judgements of zero-shot TTS can rank systems differently.

    Evidence: On Seed-TTS (en), Llasa 1B has lower SpkSim than VoiceStar (0.58 vs 0.63) but higher S-CMOS (-0.23 vs -0.32); on LibriSpeech-PC, VoiceStar has lower SpkSim than CosyVoice 2 and MaskGCT (0.63 vs 0.66) but the highest S-CMOS (0.60). (§4.3, Table 4)

Limitations and Open Questions

  • Speaker similarity on long-form speech. Prompt repetition cannot be used when the context already reaches the training length, and VoiceStar trails MaskGCT on automatic SpkSim and S-MOS in the 20-40 s ranges and has the lowest S-MOS at 40-50 s. The authors argue that a better codec or multi-stage modelling could fix this, but do not test it (§Limitations).
  • Generation speed. The 840M model runs slower than real time (real-time factor above 1); quantization, grouped prediction and speculative decoding are suggested but not evaluated (§Limitations).
  • Duration source. Precise duration control needs a target duration. The short-form results use a simple speaking-rate estimate, and with that estimate long-form WER rises (for example from 11.91 to 17.33 at 40-50 s). A learned duration predictor is proposed but not built (§Appendix B.3, Table 8).
  • Evaluation coverage. All evaluation is English. The long-form sets share the Libriheavy domain with the training data, and the long-form comparison covers only two non-autoregressive baselines.
  • Listening tests. Amazon Mechanical Turk raters were filtered by country of residence as a proxy for native English, which the authors acknowledge may bias judgements (§Appendix C).
  • Disentangling prompt prosody. CPM training is motivated as a way to separate speaker identity from prompt prosody and emotion, but no experiment measures prosody or emotion transfer directly.

Wiki Connections

  • Autoregressive Codec TTS — Proposes an encoder-decoder codec language model with PM-RoPE that adds duration control and length extrapolation to autoregressive codec-token TTS.
  • Zero-Shot TTS — Evaluates prompt-based voice cloning on LibriSpeech-PC, Seed-TTS (en) and long-form Libriheavy sets, and introduces CPM training and prompt repetition for cloning.
  • Transformer Encoder-Decoder TTS — Shows in ablation that an encoder-decoder with cross-attention alignment outperforms a decoder-only codec model on WER, UTMOS and SpkSim.
  • Prosody Control — Provides explicit total-duration control, evaluated as the gap between target and generated duration (0.009 s with PM-RoPE).
  • Subjective Evaluation — Runs crowd-sourced CMOS tests for short-form and S-MOS, I-MOS and N-MOS tests for long-form speech, and reports disagreements between human and automatic speaker-similarity rankings.
  • VoiceCraft — Same group’s earlier decoder-only codec model; VoiceStar reuses its EnCodec tokenizer and codebook-weighting recipe and compares against it as a baseline.
  • F5-TTS — Duration-controllable flow-matching baseline on short- and long-form sets; it degrades sharply beyond 30 s and under prompt repetition.
  • MaskGCT — Duration-controllable baseline with the highest long-form speaker similarity at 20-40 s (SpkSim 0.76 and 0.75) but the lowest SpkSim (0.65) and a WER of 82.29 at 40-50 s.
  • CosyVoice 2 — Short-form baseline with higher SpkSim and Seed-TTS S-CMOS than VoiceStar.
  • Llasa — Short-form baseline used to show automatic SpkSim and human S-CMOS ranking systems differently.
  • EnCodec — Codec architecture behind the 4-codebook 50 Hz tokenizer VoiceStar models.
  • Emilia — Supplies the English in-the-wild training data and the speaker labels used to form CPM prompt pairs.
  • Seed-TTS — Source of the Seed-TTS (en) short-form evaluation set; its reported numbers appear in the appendix comparison.