Speech Generation Research Wiki

Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations

8 min read

arXiv · 2025 · Preprint

Han et al. (AMAP Speech) · → Paper · Demo: ? · Code: ?

Introduces QTTS, a TTS framework that scales residual vector quantization to multiple codebooks using a novel ASR-guided codec (QDAC) for semantic disentanglement, paired with two complementary autoregressive decoding architectures that trade synthesis quality against inference speed.

Problem

Autoregressive TTS systems have largely relied on single-codebook discrete representations, which compress audio aggressively and discard fine-grained detail such as prosodic nuance, speaker-specific timbre, and the high-frequency content required for expressive or singing voice synthesis. Post-hoc refinement via flow matching can recover some information, but cannot restore detail that was lost at the quantization stage. Multi-codebook RVQ codecs can in principle preserve more detail, but naive application entangles semantic content (phonemes) with acoustic attributes (timbre, prosody) across codebook layers, limiting the model’s ability to generate efficiently from learned discrete representations.

Method

QTTS is built on QDAC (Quantization-Decoupled Audio Codec), a GAN-based encoder-decoder that extends the DAC architecture with an end-to-end semantic disentanglement mechanism. The key innovation is that an autoregressive ASR module is trained jointly with the codec, but conditioned exclusively on the first codebook’s tokens. The cross-entropy loss from ASR transcription is backpropagated only through that primary codebook, forcing it to encode linguistic content. The remaining codebooks are supervised only by the waveform reconstruction and adversarial losses, and are thus free to capture residual acoustic detail such as speaker timbre and fine prosody. This explicit separation distinguishes QDAC from approaches that apply SSL distillation (e.g. WavLM) to the first RVQ layer, which do not prevent semantic-acoustic entanglement in downstream codebooks.

Architecture of the audio tokenizer model. It adopts a GAN-based framework consisting of an encoder-decoder generator and a discriminator, trained with semantic guidance from a AR based ASR model. During training, only the first codebook is explicitly supervised by semantic loss, while the remaining codebooks are left to learn freely, allowing the model to flexibly capture additional residual details.

QTTS models the resulting discrete tokens using two decoding architectures that span a quality-speed trade-off. The Hierarchy Parallel (200M parameters) approach uses a dual-AR structure: an outer autoregressive loop proceeds frame by frame over time, and an inner autoregressive loop over codebook indices generates each codebook conditioned on all previously decoded layers within the same frame. This exhaustive inter-codebook conditioning maximises reconstruction fidelity but is inherently sequential in two dimensions. The Multihead Delay (120M parameters) approach assigns a dedicated prediction head to each codebook, enabling parallel decoding with a fixed-step temporal offset between heads. Each head’s output is causally conditioned on the previous head with a delay of k steps, preserving local inter-codebook context without requiring full hierarchical conditioning. The codec operates at 25 Hz with 8 codebooks (2048 entries each, 1024-dimensional embeddings) for the primary TTS experiments. Inference acceleration exploits vLLM’s PagedAttention for the backbone AR and a statically compiled CUDA graph for the decoder AR, enabling efficient first-chunk latency.

Comparison of two multi-codebook prediction strategies for TTS: (Left) The Parallel Hierarchy structure decodes tokens autoregressively across both time and codebook dimensions, allowing full contextual conditioning at the cost of slower sequential inference. (Right) The MultiheadDelay architecture predicts each codebook stream with a dedicated head and fixed delay offset, enabling parallel inference but with limited cross-codebook context. Each approach represents a trade-off between modeling capacity and runtime efficiency.

Key Results

On codec reconstruction (Table 5), QDAC improves monotonically with codebook count: 8 codebooks at 25Hz yield PESQ 2.98 and Spk Sim 0.90, while 16 codebooks at 50Hz reach PESQ 3.83 and Spk Sim 0.93, approaching (but not reaching) the DAC 32-codebook reference (PESQ 4.35). Compared to SpeechTokenizer at 8 codebooks, QDAC shows substantially higher reconstruction quality across all reported metrics.

On zero-shot TTS synthesis (Table 6), QTTS (8 codebooks, 0.2B) achieves WER 1.66% on SeedTTS-Easy versus CosyVoice v1’s 3.05%, and PGC-hard WER 6.89 versus CosyVoice v1’s 7.86%. MOS is marginally higher at 3.03 versus 3.01 and 3.02 for the two CosyVoice baselines. However, speaker similarity is weaker at 0.75 versus 0.82 and 0.81 for CosyVoice v1 and v2. Notably, CosyVoice 2 (0.5B, single codebook) achieves lower WER (1.32%) and higher speaker similarity (0.81) than QTTS on the same test set, indicating that QTTS is competitive with but not uniformly superior to the more recent single-codebook baseline at current scales.

Inference latency (Tables 1 and 2) is measured on a single H20 GPU. For a 512-token backbone input, TTFT is 26ms (Hierarchy, 200M) and 24ms (Multihead, 120M). The Multihead Delay model achieves significantly higher decode throughput due to its parallelised prediction.

Note

Ground truth speech achieves MOS 2.7 in Table 6 while all systems score above 3.0, suggesting the test set, rating scale, or crowd-sourcing setup produces non-standard reference scores. Absolute MOS values should be interpreted cautiously.

Novelty Assessment

The QDAC codec design is genuinely novel in its use of an end-to-end AR-ASR module to enforce semantic isolation in the first RVQ codebook, rather than relying on frozen SSL distillation targets. The dual-AR Hierarchy Parallel decoder is a principled extension of RQ-Transformer-style generation that explicitly models both temporal and codebook dimensions autoregressively. The Multihead Delay architecture echoes prior work in music generation (Copet et al. [MusicGen]) but applies the pattern to RVQ-based TTS with a specific trade-off analysis.

The evaluation is limited to comparison against the CosyVoice family, with no comparison to other strong multi-codebook systems or flow-matching baselines such as F5-TTS or Voicebox-style architectures. The PGC-hard test set is proprietary, limiting reproducibility. The MOS gains over CosyVoice baselines are small (0.01-0.02) and the speaker similarity metric is lower than both baselines, suggesting that the codec’s better reconstruction does not straightforwardly translate to better zero-shot speaker transfer in the current system. The training data is not disclosed.

Field Significance

Moderate — QTTS makes a technically concrete case for scaling codebook depth as a direction for expressive TTS, and the QDAC disentanglement mechanism offers a principled alternative to SSL distillation for RVQ codec design. The paper’s primary contribution is demonstrating that AR-ASR-guided quantization enables cleaner multi-codebook representations, though the downstream TTS gains are modest and the evaluation scope is narrow. The architecture provides a reference design for researchers exploring multi-codebook generation strategies, particularly for challenging content types like singing where single-codebook systems show known limitations.

Claims

  • supports: Scaling the number of RVQ codebooks in a discrete speech codec reduces information loss and improves reconstruction quality for expressive and challenging vocal content.

    Evidence: QDAC reconstruction improves monotonically from 1 to 16 codebooks across PESQ, STOI, and SI-SDR; 16-codebook QDAC at 50Hz achieves PESQ 3.83 versus PESQ 2.98 for 8 codebooks, with similar gains on Mel distance and speaker similarity. (§3.2, Table 5)

  • complicates: Higher multi-codebook reconstruction fidelity in a codec does not necessarily translate into better speaker identity preservation in downstream zero-shot TTS.

    Evidence: QTTS achieves Spk Sim 0.75 on SeedTTS-Easy compared to 0.82 and 0.81 for single-codebook CosyVoice v1 and v2, despite QDAC’s superior reconstruction metrics in Table 5. (§3.2, Table 6)

  • supports: Semantic disentanglement in audio codecs can be achieved by backpropagating ASR loss exclusively through the first RVQ codebook, enforcing content isolation without relying on general-purpose self-supervised representations.

    Evidence: QDAC trains an AR-ASR module conditioned only on first-codebook tokens; the resulting WER at reconstruction (6.42 for 8cb/25Hz) is close to ground truth (6.01), indicating the first codebook encodes phoneme-level content while residual codebooks capture acoustic detail. (§2.1.2, Table 5)

  • refines: Multi-codebook autoregressive TTS admits a principled speed-quality trade-off by choosing between strict hierarchical inter-codebook conditioning and a delayed multi-head parallel prediction scheme.

    Evidence: Hierarchy Parallel (200M, dual-AR) and Multihead Delay (120M, parallel with fixed delay) achieve comparable TTFT at 512 tokens (26ms vs 24ms) but differ substantially in decode throughput; the Multihead variant reaches over 196K codebook tokens/s at short output lengths versus 105K for Hierarchy. (§2.2, §2.3, Tables 1, 3, 4)

  • complicates: MOS evaluations in zero-shot TTS can yield above-reference scores for synthesised speech, undermining direct absolute comparisons across studies.

    Evidence: Ground truth speech achieves MOS 2.7 while QTTS, CosyVoice, and CosyVoice2 all score between 3.01 and 3.03 on the same test set, producing a ranking inconsistent with naturalness expectations. (§3.2, Table 6)

Limitations and Open Questions

Warning

Training data is not disclosed, the PGC-hard benchmark is proprietary, and no code or demo is available, making results difficult to reproduce or build upon.

The evaluation compares only against the CosyVoice v1/v2 family, leaving open how QTTS performs relative to flow-matching systems, other multi-codebook approaches, or stronger autoregressive baselines. Speaker similarity is lower than both single-codebook baselines (0.75 vs 0.81-0.82), suggesting the multi-codebook generation pipeline needs further work to fully leverage the improved codec for speaker transfer. The paper positions singing and music synthesis as motivating use cases but does not evaluate on these tasks. Only the 8-codebook QTTS model is evaluated for TTS synthesis, leaving open whether 16-codebook generation would further improve or introduce new training challenges.

Wiki Connections

  • Autoregressive Codec TTS — QTTS extends this paradigm from single-codebook to multi-codebook RVQ generation, proposing two novel decoding architectures for handling inter-codebook dependencies.
  • Neural Audio Codec — QDAC introduces AR-ASR-guided semantic disentanglement as an alternative to SSL distillation for achieving clean multi-codebook RVQ representations.
  • Disentanglement — QDAC explicitly separates semantic content into the first codebook through end-to-end ASR supervision, with subsequent codebooks free to model residual acoustic attributes.
  • Zero-Shot TTS — QTTS is evaluated on zero-shot speaker transfer (SeedTTS-Easy, PGC-hard), where it achieves competitive intelligibility but weaker speaker similarity than single-codebook baselines.
  • Streaming TTS — the Multihead Delay architecture minimises first-chunk latency (TTFT 24ms at 512-token input) through parallelised codebook prediction with fixed delay offsets, with inference benchmarked on a single H20 GPU.
  • CosyVoice (CosyVoice) — this paper uses CosyVoice v1 as the primary single-codebook baseline, comparing against its VQ encoder with 4096 codes and noting its stated limitation for singing voice synthesis.
  • CosyVoice2 (CosyVoice 2) — serves as the stronger single-codebook baseline; CosyVoice 2 matches or exceeds QTTS on WER and speaker similarity despite using one codebook, contextualising the gains from multi-codebook scaling.
  • Seed-TTS — the SeedTTS-Easy evaluation set is drawn from this paper’s benchmark; QTTS uses it as the primary intelligibility and speaker similarity test.
  • F5-TTS — cited as a representative diffusion/flow-matching approach with coarse duration modeling, which QTTS argues lacks fine-grained prosody control compared to explicit multi-codebook AR generation.

Backlinks

Graph View