MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

arXiv · 2026 · Preprint

Yitian Gong et al. (MOSI Intelligence, Shanghai Innovation Institute, Fudan University) · → Paper · Demo: ? · Code: ?

Introduces CAT, a fully Transformer-based (CNN-free) discrete audio tokenizer jointly optimized end-to-end, and scales it to a 1.6-billion-parameter model trained on 3 million hours of speech, sound, and music, achieving state-of-the-art reconstruction across bitrates and enabling a purely autoregressive TTS system to outperform prior non-autoregressive and cascaded baselines.

Problem

Discrete audio tokenizers are the native interface that lets large language models operate on audio, analogous to text tokenizers for text. Existing neural audio codecs typically rely on pretrained encoders, semantic distillation from self-supervised teachers, or heterogeneous CNN/hybrid architectures to inject semantic structure into the discrete representation. The paper argues these design choices introduce fixed inductive biases and architectural dependencies that limit reconstruction fidelity and hinder scaling model capacity, training data, and quantization depth together. The authors position this as analogous to the shift from hand-engineered NLP pipelines toward simple, homogeneous, end-to-end-trained Transformer architectures that scale predictably with data and compute in large language models.

Method

CAT (Causal Audio Tokenizer with Transformer) is a CNN-free codec built entirely from causal Transformer blocks. The encoder operates directly on raw 24 kHz waveforms (no mel-spectrogram intermediate), patchifying the input and progressively downsampling between Transformer stages until it reaches a frame rate of 12.5 Hz; the decoder mirrors this process in reverse. In the full-scale MOSS-Audio-Tokenizer, the encoder and decoder each contain approximately 0.8B parameters across four stages (hidden sizes 768/768/768/1280; 12/12/12/32 blocks) using 10-second sliding-window causal attention and rotary position embeddings, giving roughly 1.6B parameters total (§A.1).

Discretization uses a 32-layer residual vector quantizer (RVQ) with factorized codebooks (size 1024, latent dimension 8, L2-normalized) and quantizer dropout, giving native support for variable bitrates from 0.125 to 4 kbps. To inject semantic structure without a pretrained teacher, a 0.5B decoder-only causal LLM is attached to the quantizer output and trained with a cross-entropy audio-to-text objective covering ASR, multi-speaker ASR, and audio captioning tasks (§3.2, Eq. 1). Acoustic fidelity is driven by a multi-scale mel-spectrogram reconstruction loss plus adversarial training (multi-period and complex-STFT discriminators, following XY-Tokenizer’s objective). All modules, including the discriminator and the semantic LLM head, are optimized jointly rather than in separate stages, though for compute efficiency training itself proceeds in two phases: 520k non-adversarial steps followed by 500k adversarial fine-tuning steps (§A.3).

Building on CAT’s tokens, the authors construct CAT-TTS, a purely autoregressive TTS system using a Temporal Transformer (initialized from Qwen3-1.7B) plus a 4-block Depth Transformer to model RVQ tokens along the temporal and depth axes respectively, following the Moshi-style multi-stream AR pattern. To let a single AR model generate at variable bitrates, the paper introduces Progressive Sequence Dropout: during training, with probability p a random RVQ prefix length K is sampled and only the first K quantizer layers are retained as both input and training target, exposing the model to truncated RVQ depths so it learns to generate coherently at any inference bitrate (§3.3). At inference, the depth K_infer is chosen explicitly to trade off bitrate against quality. The same tokenizer is also evaluated for ASR (CAT-ASR) by feeding summed RVQ embeddings directly into a Qwen3-1.7B LLM trained to predict transcripts, without any dedicated audio encoder.

Key Results

On reconstruction (Table 2), MOSS-Audio-Tokenizer outperforms prior open-source codecs (Encodec, DAC, SpeechTokenizer, Mimi, BigCodec, StableCodec, XCodec2.0, XY-Tokenizer, DualCodec, Higgs-Audio-Tokenizer, MiMo-Audio-Tokenizer, Qwen3-TTS-Tokenizer) at matched or lower bitrates across low, medium, and high bitrate regimes on LibriSpeech test-clean (English) and AISHELL-2 (Chinese) for speech, and on AudioSet/MUSDB for sound and music. For example, at 2000 bps it reaches SIM 0.95/0.89 and STOI 0.96/0.94 (English/Chinese), exceeding Mimi at a comparable 2475 bps (SIM 0.89/0.76) and matching or beating Qwen3-TTS-Tokenizer at 2200 bps.

On zero-shot TTS (Table 3, Seed-TTS-Eval), CAT-TTS achieves the highest speaker similarity among the compared open-source systems for both English (SIM 73.1) and Chinese (SIM 78.5), while keeping WER (1.89) and CER (1.23) below the 2% threshold met by most competitors, outperforming cascaded AR+NAR systems (MaskGCT, CosyVoice2/3, IndexTTS2, GLM-TTS), the non-autoregressive F5-TTS, and prior discrete-AR systems (Llasa, SparkTTS, HiggsAudio-v2, FireRedTTS2). An ablation over Progressive Sequence Dropout probability (p ∈ {0, 0.25, 0.5, 1.0}) shows that without dropout, WER and SIM degrade sharply as inference bitrate decreases, whereas any nonzero dropout probability yields stable performance across bitrates; the paper adopts p=1.0 for memory efficiency.

An ablation comparing full end-to-end optimization against a partial protocol (frozen encoder/quantizer, only decoder+discriminator trained, as used in some prior codecs) shows the partial protocol plateaus early while full joint optimization keeps improving with training compute (§5.1, Figure 4). A parameter-scaling study (hidden sizes 256/384/512/768, giving 319M/505M/710M/1169M combined encoder-decoder parameters) shows the largest model benefits most at high bitrates but can underperform smaller models at low bitrates, indicating bitrate and parameter count are co-dependent bottlenecks (§5.2). CAT-ASR, using the codec’s own tokens directly as LLM input with no auxiliary audio encoder, reaches WER 2.96 (English, LibriSpeech test-clean) and CER 3.44 (Chinese, AISHELL-2), competitive with but not surpassing dedicated encoder-based systems such as Kimi-Audio (WER 1.28) and Qwen2.5-Omni (WER 2.37).

Novelty Assessment

The core architectural contribution is genuine: CAT removes convolutional components entirely, uses a single homogeneous stack of causal Transformer blocks for both encoder and decoder, and trains the encoder, quantizer, decoder, discriminator, and an auxiliary semantic LLM head jointly end-to-end rather than through pretrained-encoder initialization or staged/frozen-component training, which prior large-scale tokenizers (MiMo-Audio-Tokenizer, XY-Tokenizer, Qwen3-TTS-Tokenizer, Baichuan Audio Tokenizer) do not do in combination. The paper backs this design choice with a direct ablation isolating the effect of joint optimization versus the partial-freezing protocol used elsewhere. Progressive Sequence Dropout is a genuinely new, parameter-free training strategy for bitrate-flexible autoregressive generation, distinct from architectural approaches to variable bitrate.

The TTS and ASR systems built on top of CAT (CAT-TTS, CAT-ASR) are largely engineering integrations that reuse an established Temporal+Depth Transformer AR pattern (introduced for Moshi) and a standard LLM backbone (Qwen3-1.7B); their contribution is demonstrating what a stronger, jointly-optimized codec enables, not a new generation architecture. The “first purely autoregressive discrete TTS system to outperform non-autoregressive and cascaded systems” claim is scoped to the specific set of open-source systems compared in Table 3, not a universal claim.

Field Significance

High significance

high — CAT demonstrates that a homogeneous, CNN-free, fully end-to-end-trained Transformer architecture can outperform hybrid and pretrained-encoder-dependent codec designs at scale (1.6B parameters, 3M hours), directly validated with a joint-vs-partial-optimization ablation. It further shows that a sufficiently strong, jointly-optimized codec combined with a simple training-time RVQ dropout strategy lets a purely autoregressive discrete-token TTS system match or exceed non-autoregressive and cascaded systems on zero-shot speaker similarity, providing a concrete data point for the design-space debate between architectural specialization and end-to-end scale in audio tokenizers.

Claims

• supports: A homogeneous, fully Transformer-based (CNN-free) codec encoder-decoder, jointly trained end-to-end with its discriminator and without a pretrained encoder or semantic teacher, can match or exceed hybrid and CNN-based codec designs in reconstruction fidelity at comparable or lower bitrates.

  Evidence: MOSS-Audio-Tokenizer at 2000 bps reaches SIM 0.95/0.89 and STOI 0.96/0.94 (English/Chinese), exceeding Mimi at a higher 2475 bps (SIM 0.89/0.76) and matching or beating the hybrid-architecture Qwen3-TTS-Tokenizer at 2200 bps. (§4.2, Table 2)

• supports: Freezing the encoder and quantizer while training only the decoder and discriminator caps a codec’s achievable reconstruction quality, while joint end-to-end optimization of all components sustains improvement with additional training compute.

  Evidence: An ablation comparing full end-to-end training against a partial protocol (frozen encoder/quantizer) used in several prior codecs shows the partial protocol plateaus early, while the fully end-to-end variant continues improving across all metrics without early saturation. (§5.1, Figure 4)

• supports: Randomly truncating the RVQ token depth during autoregressive speech-generation training (a parameter-free prefix-dropout strategy) lets a single AR model generate coherent speech across a wide range of inference bitrates without architectural modification.

  Evidence: Progressive Sequence Dropout keeps CAT-TTS’s WER and speaker similarity stable across dropout probabilities p=0.25, 0.5, and 1.0 at reduced bitrates, whereas the model trained without dropout (p=0) shows steep WER and similarity degradation as bitrate decreases. (§4.3, Figure 3)

• complicates: Increasing codec parameter count alone does not guarantee better reconstruction quality; quantization depth (bitrate) and model capacity are co-dependent bottlenecks that must be scaled together.

  Evidence: In a parameter-scaling sweep (319M-1169M combined encoder-decoder parameters), the largest 1169M model benefits most at high bitrates but can underperform smaller models at low bitrates, indicating bitrate becomes the binding constraint independent of parameter count. (§5.2, Figure 5)

• supports: A purely autoregressive discrete-token TTS architecture can achieve zero-shot speaker similarity competitive with, or exceeding, non-autoregressive and cascaded AR+NAR systems when built on a sufficiently strong, jointly-trained tokenizer.

  Evidence: CAT-TTS attains the highest speaker similarity among compared open-source systems on Seed-TTS-Eval for both English (SIM 73.1) and Chinese (SIM 78.5), with WER 1.89 and CER 1.23, outperforming cascaded systems (CosyVoice3-1.5B, IndexTTS2, GLM-TTS), non-autoregressive F5-TTS, and prior discrete-AR systems (HiggsAudio-v2, FireRedTTS2). (§4.3, Table 3)

Limitations and Open Questions

Reproducibility depends on undisclosed internal data

CAT-TTS is trained on VoxBox plus “an internal dataset” (~200k hours combined) and CAT-ASR on an “internal dataset” of ~2M hours of paired audio-text data; the composition and provenance of these internal corpora are not disclosed, and the ~3M-hour tokenizer pretraining corpus is likewise described only generically as “diverse, general audio data.” This limits independent reproduction of the reported speech generation and understanding results even though the codec architecture itself is fully specified.

The paper does not report whether MOSS-Audio-Tokenizer’s weights, code, or a public demo will be released, and none of the abstract, introduction, or conclusion state a release plan. CAT-ASR is evaluated only against systems with dedicated audio encoders and does not surpass the strongest of them (Kimi-Audio, Qwen2.5-Omni), so the “competitive ASR without an auxiliary encoder” claim is a parity result rather than a state-of-the-art one. The TTS comparison in Table 3 is restricted to open-source systems; no comparison against closed proprietary TTS systems is provided. The paper does not ablate the contribution of the auxiliary 0.5B semantic LLM head in isolation from the reconstruction and adversarial losses, so it is not established how much of the tokenizer’s downstream TTS/ASR performance depends specifically on that objective versus the architecture and scale alone.

Wiki Connections

• Neural Audio Codec — proposes CAT, a CNN-free, fully Transformer-based, end-to-end-optimized neural audio codec and scales it to 1.6B parameters and 3M training hours, directly targeting the architectural-dependency limitations of prior hybrid and pretrained-encoder-based codecs.
• Autoregressive Codec TTS — builds CAT-TTS, a purely autoregressive discrete-token TTS system, and introduces Progressive Sequence Dropout to make a single AR model robust across variable inference bitrates.
• Zero-Shot TTS — evaluates CAT-TTS on Seed-TTS-Eval zero-shot cloning, reporting the highest speaker similarity among compared open-source systems for English and Chinese.
• Spoken Language Model — frames the tokenizer explicitly as a native discrete interface for autoregressive audio foundation models, extending the text-tokenizer analogy to audio, and evaluates it for both generation (CAT-TTS) and understanding (CAT-ASR).
• Evaluation Metrics — reports objective reconstruction metrics (SIM, STOI, PESQ, mel/STFT distance) across matched bitrate bands plus a MUSHRA subjective listening test, providing a systematic bitrate-controlled comparison protocol against twelve prior codecs.
• MiMo-Audio — MiMo-Audio-Tokenizer is used as a direct reconstruction-quality baseline in the codec comparison and is discussed in related work as a scale-driven semantic tokenizer approach.
• Qwen3-TTS — Qwen3-TTS-Tokenizer is used as a hybrid-architecture reconstruction baseline that MOSS-Audio-Tokenizer matches or exceeds at comparable bitrates.
• IndexTTS2 — used as a cascaded AR+NAR baseline TTS system on Seed-TTS-Eval that CAT-TTS is compared against and outperforms on speaker similarity.
• VoxCPM — cited as a recent state-of-the-art non-autoregressive/continuous-AR TTS system that CAT-TTS is benchmarked against on Seed-TTS-Eval.
• Seed-TTS — the Seed-TTS-Eval benchmark introduced by this paper is used as the primary zero-shot TTS evaluation protocol for CAT-TTS.
• VALL-E — cited as the foundational neural-codec-language-model TTS paradigm that established scaling codec-token generation as a viable zero-shot TTS approach, which CAT-TTS extends with variable-bitrate generation.
• FlexiCodec — FlexiCodec-TTS is used as a baseline TTS system in the Seed-TTS-Eval comparison and is the only other compared system offering explicit inference-time bitrate control.
• CosyVoice — CosyVoice2/CosyVoice3 are used as cascaded AR+NAR baseline TTS systems in the Seed-TTS-Eval comparison against CAT-TTS.