arXiv · 2025 · Preprint
Luo et al. (Honor Device Co., Ltd. / Shanghai Jiao Tong University) · → Paper · → Peer-reviewed version · Demo: ✓ · Code: ✗
Replaces the frame-by-frame length regulator of non-autoregressive TTS with a sparse temporal embedding that marks only the central frame of each phone, and pairs it with a codec whose first RVQ channel is distilled from a self-supervised speech model, so that an 83M masked generative transformer can predict all codec tokens in a single stage.
A peer-reviewed version appeared at ASRU 2025. This page, including its metrics and claims, summarizes the arXiv preprint.
Problem
Zero-shot TTS faces a three-way trade-off among robustness, latency and prosody. Autoregressive codec language models are slow and can accumulate content errors, which makes them hard to deploy on mobile devices. Classic non-autoregressive systems in the FastSpeech style use a duration predictor and a length regulator (LR) that expands each phone over its frames. This gives low WER but rigid, flat prosody. Masked generative systems such as MaskGCT drop phone-level alignment and sound more natural, but the authors report that their WER is too high for commercial use, especially on hard text. Most codec-based systems also predict semantic and acoustic tokens in two separate stages, which adds latency.
Method
StellarTTS has three components.
Semantic-aware codec. An RVQ codec quantizes speech into six channels: one semantic channel (channel 0) and five acoustic channels. Channel 0 is trained with a KL-divergence distillation loss toward features from a frozen w2v-BERT encoder, in addition to the usual reconstruction, codebook and commitment losses (§III.A). Because semantic and acoustic information live in one token stack, the TTS model predicts all channels in one stage instead of running a separate semantic model. The authors credit SoundStorm for the inspiration. Codec frame rate and sample rate are not reported.
Sparse phone-level temporal embedding. This replaces the LR (§III.B). For each phone of duration t_i frames, only the central frame (j = ⌊t_i/2⌋) receives the phone embedding. All other frames receive a shared padding-phone embedding. The central token anchors the linguistic content, and the padding-filled span leaves the model free to learn transitions and rhythm. During training, ground-truth durations place the anchors. At inference, a duration predictor trained with MSE places them and sets the output length. Gradients from the duration loss are stopped at the phone embeddings so the duration predictor does not distort the phone representation.
Masked generative transformer. An 83M LLaMA-based decoder (§III.C) takes a condition prefix of prompt phones, target phones and prompt codec tokens, separated by delimiter tokens. It is followed by target positions whose input sums three terms: the masked target codec-token embeddings, a projected speaker embedding from a pre-trained 3D-Speaker voice-print model, and the sparse temporal embedding. Training uses cross-entropy on masked positions with a cosine masking schedule, plus the weighted duration loss. Inference uses iterative parallel decoding with [8, 4, 1, 1, 1, 1] steps across the six RVQ channels (16 in total). Channels 0 and 1 use top-10% sampling with temperature annealing, and the remaining channels use greedy decoding. This is a non-autoregressive model in the authors’ terms; the autoregressive-LM tag follows the wiki’s convention for SoundStorm-style masked codec-token transformers.

The model is trained on Emilia using 8 A100 GPUs.
Key Results
All evaluation is on the Mandarin Seed-TTS test-zh set and the harder test-hard set. Baselines are FireRedTTS, CosyVoice, MaskGCT and F5-TTS (Table I).
- Robustness (WER). StellarTTS has the lowest WER of the synthesized systems on both sets. On test-zh it scores 1.44, against 1.51 (FireRedTTS), 1.56 (F5-TTS), 2.27 (MaskGCT) and 3.63 (CosyVoice); ground truth is 1.26. On test-hard it scores 7.47, against 8.67 (F5-TTS), 10.27 (MaskGCT), 11.75 (CosyVoice) and 17.45 (FireRedTTS).
- Speed (RTF on one A100, 10 s utterances). StellarTTS has the lowest RTF of the systems with a reported value: 0.08, against 0.31 (F5-TTS), 0.67 (CosyVoice) and 0.71 (MaskGCT). This is the 4× to 9× speed-up the authors cite. FireRedTTS has no RTF.
- Speaker similarity (SIM-o). StellarTTS ranks fourth of five on both sets: 0.712 on test-zh (MaskGCT 0.774, F5-TTS 0.741, CosyVoice 0.723) and 0.697 on test-hard (MaskGCT 0.748, F5-TTS 0.713, CosyVoice 0.709). Only FireRedTTS is lower. The authors attribute the gap to the codec: re-synthesis of real speech through it reaches only 0.668 SIM-o, which they link to the semantic distillation (§IV.A).
- Subjective scores (Table II, 20 native listeners). CMOS is measured relative to ground truth (ground truth = 0). StellarTTS has the highest CMOS on both sets: 0.06 on test-zh (F5-TTS 0.02, MaskGCT −0.08) and 0.05 on test-hard (F5-TTS −0.01, MaskGCT −0.12). MaskGCT has the highest SMOS on both sets: 4.09 against 3.96 on test-zh and 3.86 against 3.78 on test-hard. No confidence intervals or significance tests are reported, so the CMOS margins of a few hundredths over F5-TTS cannot be separated from noise.
- Ablation (Table III). Removing the temporal embedding has the largest effect on WER: 4.1 on test-zh and 18.12 on test-hard, against 1.44 and 7.47. The text describes the row labeled “w/o central strategy” as a random anchor position inside each phone. It raises WER to 2.34 and 9.15. Removing the speaker embedding mainly lowers SIM-o (0.654 and 0.626), while test-hard WER is slightly lower without it (7.31 against 7.47).
- Duration control (§IV.D, Figure 2). Scaling every predicted phone duration by 0.7 to 1.3 keeps test-zh WER at or below 2.18, its worst value (reached at 0.7×). That is below MaskGCT’s best value of 2.27. The comparison is not like-for-like: the MaskGCT curve is copied from the MaskGCT paper and scales the total ground-truth duration rather than per-phone durations.
Novelty Assessment
The main idea, sparse central-frame phone anchors in place of a dense length regulator inside a masked generative codec model, is a genuine new conditioning design. It sits between FastSpeech-style explicit alignment and alignment-free masked generation, and the ablation shows that it drives the WER gains. The semantic-aware codec is less new. Distilling a self-supervised teacher into the first RVQ channel follows earlier semantic-distilled codecs, and single-stage prediction over such a stack follows SoundStorm-style decoding. The mobile claim rests on model size (83M) and A100 RTF only. The paper reports no on-device latency on the Qualcomm SM8650 it names. The evaluation is narrow: one language, two test sets and four baselines. All results are reported once, with no variance.
Field Significance
Moderate — The paper shows that a phone-level anchor which carries duration information without dictating frame-level content can recover much of the robustness of explicit-duration NAR TTS inside a masked generative model, at a fraction of the inference cost. It also gives a measured cost of single-stage semantic-acoustic codecs: the distilled codec caps speaker similarity, and that cap carries through to the TTS output.
Claims
- supports: Sparse duration anchors in non-autoregressive masked generative TTS reduce content errors relative to alignment-free generation while keeping explicit per-phone duration available.
Evidence: Removing the sparse temporal embedding raises WER from 1.44 to 4.1 on Seed-TTS test-zh and from 7.47 to 18.12 on test-hard. Replacing the central anchor with a random in-phone anchor gives 2.34 and 9.15. (§IV.C, Table III)
- supports: Explicit per-phone duration control in a parallel codec-token generator allows speaking-rate changes with limited loss of intelligibility.
Evidence: Scaling predicted phone durations from 0.7× to 1.3× keeps test-zh WER at or below 2.18, below MaskGCT’s best value of 2.27. However, the MaskGCT curve is taken from its own paper and uses total-duration scaling. (§IV.D, Figure 2)
- complicates: Distilling semantic information into a codec’s first quantizer channel, so that TTS can run as a single stage, can limit the speaker similarity that downstream synthesis can reach.
Evidence: Re-synthesis of real speech through the semantic-aware codec reaches only 0.668 SIM-o. StellarTTS’s SIM-o (0.712 test-zh, 0.697 test-hard) trails MaskGCT, F5-TTS and CosyVoice on both sets, even though its WER is the lowest. (§IV.A, Table I)
- supports: Small non-autoregressive codec-token models with few decoding steps can reach real-time factors well below those of larger autoregressive and flow-matching zero-shot TTS systems without losing robustness.
Evidence: The 83M model with 16 total decoding steps runs at 0.08 RTF on an A100, against 0.31 to 0.71 for F5-TTS, CosyVoice and MaskGCT, and has the lowest WER on both test sets. (§IV, Table I)
Limitations and Open Questions
Warning
All evaluation is Mandarin-only (Seed-TTS test-zh and test-hard). The mobile-deployment claim has no on-device latency or memory measurement, since RTF is measured on an A100. No code or checkpoints are released.
- The codec is described only through its losses and channel layout. Frame rate, sample rate, codebook sizes and reconstruction quality beyond one SIM-o figure are not reported.
- The duration-control experiment measures only WER. It does not check whether the realized speaking rate or prosody follows the scaling. The MaskGCT comparison uses numbers from a different control method and setup.
- The subjective study has 20 raters and reports no confidence intervals. StellarTTS’s CMOS lead over F5-TTS is 0.04 on test-zh and 0.06 on test-hard.
- The authors argue that the sparse anchors improve prosodic naturalness. Apart from CMOS, the paper has no prosody-specific measurement such as F0 or duration variance.
- The paper reports no model sizes for the baselines and does not compare against CosyVoice 2, so the speed comparison mixes systems of different scales.
Wiki Connections
- Autoregressive Codec TTS — Generates codec tokens with a masked generative transformer over RVQ channels, the MaskGCT/SoundStorm lineage that this wiki files under codec-token language modelling, while replacing iterative multi-stage decoding with a single stage.
- Zero-Shot TTS — StellarTTS clones speakers from a prompt (prompt codec tokens plus a speaker embedding) and is evaluated on the Seed-TTS zero-shot test sets.
- Neural Audio Codec — Proposes an RVQ codec with a semantically distilled first channel so that TTS can run as a single stage, and reports the speaker-similarity cost of that design.
- Self-Supervised Speech — The codec’s semantic channel is distilled from w2v-BERT features, which makes self-supervised pre-training part of the token design.
- Subjective Evaluation — Reports CMOS and SMOS from 20 native listeners against four zero-shot TTS baselines.
- MaskGCT — The closest masked generative baseline. StellarTTS adds sparse duration anchors and single-stage decoding, giving lower WER but lower SIM-o and SMOS.
- F5-TTS — Flow-matching baseline. StellarTTS beats it on WER and RTF but trails it on SIM-o.
- CosyVoice — Autoregressive baseline, with higher WER and RTF than StellarTTS.
- FireRedTTS — Autoregressive baseline, with the highest test-hard WER in Table I.
- SoundStorm — The stated inspiration for the semantic-aware codec and the masked parallel decoding over RVQ channels.
- Seed-TTS — Source of the test-zh and test-hard evaluation sets.
- Emilia — The in-the-wild multilingual training corpus.
- FastSpeech 2 — The length-regulator design that the sparse temporal embedding replaces.