arXiv · 2026 · Preprint
Yitian Gong et al. (Shanghai Innovation Institute / Fudan University) · → Paper · Demo: ✓ · Code: ✓
MOSS-TTS demonstrates that a high-quality causal Transformer audio tokenizer combined with large-scale autoregressive pretraining on millions of hours of multilingual speech provides a practical, open-source speech generation foundation model with strong zero-shot voice cloning, token-level duration control, phoneme/pinyin pronunciation control, and stable hour-scale generation.
Problem
Large-scale speech generation increasingly resembles LLM scaling, yet most competitive systems introduce multiple intermediate modules — external semantic teachers, refinement stages, cascaded supervision, or post-hoc alignment — that complicate scaling because each additional component adds new supervision contracts and failure modes. The challenge is to reconcile three competing requirements simultaneously: tokens that are compact enough for efficient autoregressive modeling yet semantically rich enough for downstream quality; a generative model stable over long sequences and compatible with streaming; and a training signal that scales over diverse, noisy real-world data. Prior tokenizers such as EnCodec and DAC relied on CNN-based architectures with explicit inductive biases, while semantic richness required separate SSL teachers (HuBERT, Whisper). No open system had demonstrated a single, pure Transformer tokenizer trained end-to-end at 1.6B parameters together with an AR speech model trained at this data scale.
Method
MOSS-TTS is built around three co-designed components.
MOSS-Audio-Tokenizer is a 1.6B-parameter causal Transformer codec (encoder and decoder each with ~0.8B parameters in 68-layer architectures) that compresses 24 kHz audio to 12.5 fps using 32-layer RVQ with codebook size 1024. The architecture uses patchify downsampling at four stages rather than CNN striding, with RoPE positional embeddings throughout. Semantic supervision is provided by a jointly trained 0.5B decoder-only LLM head that autoregressively predicts text (ASR, multi-speaker ASR, audio captioning) conditioned on quantizer outputs. A multi-period discriminator and complex STFT discriminator provide adversarial perceptual supervision. All components are trained end-to-end for ~1M steps in two stages (non-adversarial pretraining then adversarial fine-tuning), with no external pretrained audio encoders. Variable bitrate is supported through random quantizer dropout, ranging from 0.125 to 4 kbps.
Two AR generator architectures are trained under the same tokenizer and pretraining recipe to expose the tradeoff between simplicity and efficiency:
-
MOSS-TTS (Delay Pattern): a single Transformer backbone with 33 prediction heads (one per RVQ layer plus a text/pad channel). RVQ layers are time-shifted by their layer index, so the backbone processes a sequence of length T + Nq - 1 and all heads project from the same backbone hidden state. This design is simpler, more scalable to large model sizes (8B), and more natural for long-context and control-oriented tasks.
-
MOSS-TTS-Local-Transformer: the backbone produces one global latent per aligned frame, and a lightweight autoregressive Local Transformer module expands that latent into the 33-channel per-step token block sequentially. This mirrors the RQ-Transformer design in Moshi. The inner loop adds compute but improves frame-level modeling efficiency and speaker preservation, and yields a shorter time-to-first-audio since no delay offsets must materialize. The released model uses 1.7B parameters.
Pretraining data pipeline converts raw open-domain recordings through a five-stage pipeline: (1) noise suppression with MossFormer2-SE-48K, volume normalisation; (2) neural speaker diarization with DiariZen, segment consolidation; (3) multilingual ASR with MOSS-Transcribe-Diarize, LLM-based transcript quality control; (4) joint acoustic/transcript filtering using DNSMOS and language consistency; (5) data synthesis for timbre-cloning pairs (using WavLM-Large speaker embeddings to select the best 30s prompt crop from same-speaker segments), noisy-text augmentation, phonetic script input, and dictionary-style short forms. The resulting corpus spans millions of hours across domains (podcasts, audiobooks, broadcast, film/drama, commentary, online content).
Training follows a four-phase curriculum: Phase 1 bootstraps basic multilingual TTS alignment on clean text-speech pairs; Phase 2 introduces the full data universe with heavy upsampling of timbre-cloning pairs; Phase 3 restores normal data proportions while linearly decaying learning rate from 2×10⁻⁴ to 2×10⁻⁶; Phase 4 extends the context window from 32k to 64k tokens with long-form upsampling.
Duration control is built natively: every training example is stored in two variants — one with an explicit token-count target and one without — so the model learns to honour explicit duration requests without a separate fine-tuning stage. Phoneme/pinyin control is enabled by training on IPA (for English) and tone-marked pinyin (for Chinese) transcript variants with unmodified audio.
Key Results
Tokenizer (Table 2): At 1000 bps (8 VQ layers), MOSS-Audio-Tokenizer achieves SIM 0.88/0.81 (EN/ZH), STOI 0.94/0.91, PESQ-WB 2.87/2.43 on LibriSpeech/AISHELL-2, outperforming XY-Tokenizer, Mimi, and other open codecs at the same bitrate. At 4000 bps, it reaches PESQ-WB 3.69/3.30 and STOI 0.97/0.96 — competitive with Qwen3-TTS-Tokenizer (3.19/2.62 PESQ-WB at 2200 bps vs MOSS at 3.69/3.30 at 4000 bps), with comparable performance at lower bitrate.
Zero-shot voice cloning (Table 3, Seed-TTS-eval): Among open-source models, MOSS-TTS-Local-Transformer (1.7B, Continuation) achieves EN WER 1.93%, EN SIM 73.28%, ZH CER 1.44%, ZH SIM 79.62% — the strongest ZH SIM among open-source systems, and competitive with closed systems like CosyVoice3 (78.1 ZH SIM at 1.5B). The 8B MOSS-TTS achieves EN SIM 70.86 and ZH SIM 76.98 in Continuation mode. Continuation inference consistently outperforms Clone-mode inference, indicating native speech continuation is an effective speaker anchor.
Multilingual (Table 4, CV3-Eval): Without multilingual fine-tuning, MOSS-TTS-Local-Transformer achieves ZH CER 3.68%, EN WER 4.35%, and competitive error rates on de/es/it/ru. Gaps remain on ja/ko, consistent with the difficulty of those language pairs.
Duration control (Table 5): Token-level duration control without dedicated fine-tuning achieves overall AbsErr Mean ~0.7% for both Chinese and English, indicating the behaviour emerges naturally from the paired pretraining format.
Ultra-long generation (Table 6): Chinese content fidelity (CER) remains below 2% through the 500–2500 token bucket and degrades mainly at 10000+ tokens. Speaker similarity drift is the dominant bottleneck, particularly in English. Continuation mode provides measurably better long-horizon speaker anchoring than Clone mode.
Pronunciation control (Table 7): Span-only CER/WER for partial-replace is 1.0% (ZH) and 4.32% (EN); for full-sentence phonetic replacement, 1.65% (ZH) and 5.84% (EN), demonstrating practical controllability.
Novelty Assessment
The core claim — that discrete tokens + AR modeling + large-scale pretraining suffice without elaborate cascades — is not new (VALL-E and Seed-TTS make related arguments), but MOSS-TTS delivers it as a fully open, reproducible release with unusually detailed pipeline documentation. The MOSS-Audio-Tokenizer’s pure-Transformer, end-to-end design at 1.6B scale, achieving competitive reconstruction without any external SSL teachers, is a meaningful contribution to the codec literature. The two-architecture ablation (Delay Pattern vs. Local Transformer) under identical tokenizer and data is genuinely informative and uncommon in technical reports. The token-level duration control emerging from pretraining format alone (no dedicated stage) is a practically useful design insight. The data pipeline description is unusually detailed. Overall, this is primarily an engineering and scaling contribution rather than an architectural breakthrough — the novelty lies in integrating and open-sourcing a complete, coherent system at this scale.
Field Significance
Moderate — MOSS-TTS provides a fully open, reproducible demonstration that a pure-Transformer audio tokenizer and a straightforward discrete AR backbone, trained at scale on millions of hours, can match or exceed closed systems on speaker similarity benchmarks without cascaded supervision. Its primary value to the field is as a clean open baseline: the unusually detailed data pipeline, dual-architecture ablation, and open model release enable direct comparison for future work on long-context and controllable speech generation.
Claims
- A pure-Transformer audio tokenizer trained end-to-end without external SSL teachers can achieve reconstruction quality competitive with CNN-based codecs at equivalent bitrate regimes. (§3, Table 2)
- Native speech continuation (prepending reference audio to the assistant speech prefix) provides stronger speaker identity anchoring than explicit clone-mode conditioning when using autoregressive codec models. (§6.2, Table 3)
- Token-level duration control can emerge from paired pretraining format alone, without a dedicated fine-tuning stage. (§6.4, Table 5)
- Cumulative speaker drift over elapsed time, rather than immediate lexical degradation, is the dominant failure mode in ultra-long autoregressive speech generation. (§6.5, Table 6, Figure 6)
- A hierarchical local-transformer decoding pattern yields stronger speaker preservation than a flat delay-pattern backbone at smaller model scale, at the cost of greater architectural complexity. (§4, §6.2, Table 3)
Limitations and Open Questions
Speaker drift in ultra-long English generation remains the primary bottleneck; the report explicitly identifies this as a failure mode rather than solving it. Japanese and Korean multilingual performance lags behind other languages, suggesting insufficient low-resource training coverage. The 8B MOSS-TTS backbone underperforms the 1.7B Local Transformer on speaker similarity, suggesting the architectural tradeoff is not purely a size effect. The report does not address latency or real-time factor, leaving streaming viability for the 8B model unclear. Objective metrics only — no subjective listening tests (MOS/MUSHRA) are reported, making absolute quality comparisons against perceptually evaluated baselines difficult. Whether the data pipeline scales further (e.g., to 100M+ hours) and whether model size beyond 8B continues to yield gains are left open.
Wiki Connections
This paper advances autoregressive-codec-tts through a clean discrete-tokens-only scaling recipe and comparison of two AR modeling patterns. The MOSS-Audio-Tokenizer is directly relevant to neural-codec — its pure-Transformer, end-to-end design at 1.6B scale is a significant point of comparison with EnCodec and DAC. Zero-shot voice cloning capabilities connect to zero-shot-tts. The multilingual pretraining and code-switching support are relevant to multilingual-tts. Token-level duration control and phoneme/pinyin pronunciation control are instances of prosody-control. The 64k-token context extension for hour-scale generation connects to streaming-tts.
In-corpus papers cited by this work: Qwen3-TTS, Seed-TTS, FireRedTTS-2, CosyVoice, VALL-E.