arXiv · 2026 · Preprint

Huimeng Wang et al. (The Chinese University of Hong Kong / Tsinghua University / SenseTime Research) · → Paper · Demo: ? · Code: ?

Aligns a highly compressed continuous speech VAE with WavLM features during VAE training, then runs a DiTAR-style continuous autoregressive TTS (LLM backbone plus patch-wise diffusion head) on the aligned latents, and shows that the alignment matters more as the latent sequence gets longer.

Problem

Continuous autoregressive TTS models avoid the reconstruction ceiling of discrete tokens by predicting VAE latents directly, usually with a diffusion or flow head to avoid the over-smoothing of plain regression (as in mel-regression models such as MELLE). The authors argue that these VAE latents are trained purely for reconstruction, so they carry little explicit text-semantic structure. The AR model then spends capacity on low-level acoustic texture rather than semantic and prosodic planning, which worsens error accumulation during sequential generation. Semantic alignment regularization for VAE latents had been explored for non-autoregressive TTS (Semantic-VAE, the speech-VAE distillation study), but not for continuous AR generation, where the authors expect the effect to be larger.

Method

Representation. Following LatentLM, a σ-VAE (a VAE variant whose posterior variance is sampled from a fixed distribution rather than learned, giving a non-vanishing variance that is easier for downstream AR modelling) compresses 24 kHz audio to 15 Hz continuous latents, a 1600x temporal reduction. The encoder and decoder are mirror-symmetric stacks of seven stages of Transformer-like blocks with self-attention replaced by 1D depth-wise convolutions. The VAE is trained with a DAC-style objective: multi-resolution mel loss, adversarial and feature-matching losses from a discriminator, and a small KL term.

SFM-guided alignment. During VAE training, features from a frozen WavLM-large encoder (the “speech foundation model”, SFM) are projected to the latent dimension and linearly interpolated to the latent frame rate. Two losses, adopted from an image-domain latent-diffusion method, align the VAE latents to these features: a frame-wise cosine loss for local semantic consistency, and a pair-wise loss that matches the T x T self-similarity matrices of the latents and the projected WavLM features to preserve global temporal structure. The alignment weight is set adaptively from the ratio of the mel-loss and alignment-loss gradient norms on shared encoder parameters (base scale 0.5). The downstream TTS architecture and training are unchanged by this step; only the latent space differs.

Continuous AR TTS. Following the next-token diffusion formulation from autoregressive image generation and DiTAR, latents are grouped into patches of two frames. A causal LLM initialised from Qwen2.5-1.5B consumes text tokens and previous patches and emits a hidden state per step. A lightweight bidirectional Diffusion Transformer head (LocDiT) generates each next patch with a DDPM noise-prediction objective, conditioned on the LLM hidden state and on the previous patch, which is concatenated with the noisy target so the task resembles outpainting. The LLM also predicts an end-of-utterance flag. Classifier-free guidance drops the LLM hidden state during training and uses a guidance scale of 2.5 at inference, needing only one LLM forward pass per step.

Training. The full system is trained on 150K hours (100K hours Emilia plus 50K hours of internal Chinese and English speech); a second configuration uses Emilia only. VAEs are trained on a 20K-hour bilingual Emilia subset. Ablations use the roughly 50K-hour English subset of Emilia with a VAE trained on 10K hours.

Key Results

Seed-TTS objective benchmark (§5.1, Table 1). SemaVoice (150K hours) reaches the lowest English WER in the table, 1.71% against 1.85% for VoxCPM (1.8M hours), 1.94% for OpenAudio-s1, 2.00% for F5-TTS and 2.23% for IndexTTS 2. The other columns are mixed:

  • Chinese CER: 1.18%, behind VoxCPM (0.93%), IndexTTS 2 (1.03%), VoxCPM-Emilia (1.11%) and FireRedTTS-2 (1.14%), level with OpenAudio-s1.
  • Hard-subset CER: 8.09%, behind CosyVoice 2 (6.83%), IndexTTS 2 (7.12%) and Qwen2.5-Omni (7.97%), ahead of F5-TTS (8.67%) and VoxCPM (8.87%).
  • Speaker similarity: 0.694 (EN), 0.754 (ZH) and 0.711 (Hard). These are below VoxCPM, MaskGCT and IndexTTS 2 in English (0.729, 0.717, 0.706), below several systems in Chinese (MaskGCT 0.774, VoxCPM 0.772, IndexTTS 2 0.765, CosyVoice 2 0.757), and below Qwen2.5-Omni (0.747), IndexTTS 2 (0.755) and VoxCPM (0.730) on Hard.

The authors describe SemaVoice as consistently outperforming the open-source baselines, which holds for English WER but not for the other five columns.

Data-matched comparison. SemaVoice-Emilia (100K hours) gets 1.91% English WER, lower than F5-TTS, MaskGCT, SparkTTS and VoxCPM-Emilia at a similar data scale. Its Chinese CER (1.32%) is behind VoxCPM-Emilia (1.11%) and SparkTTS (1.20%), and its Hard CER (9.37%) is better than VoxCPM-Emilia (12.46%) but worse than F5-TTS (8.67%) (Table 1). Baseline numbers are taken from official implementations or reported results, so evaluation conditions may not be fully matched.

Subjective evaluation (§5.2, Table 2). 12 raters scored 20 utterances against CosyVoice 2 and IndexTTS 2. In English, SemaVoice N-MOS is 3.98 ± 0.12 (CosyVoice 2 3.96, IndexTTS 2 3.75, ground truth 4.02) and S-MOS is 3.89 ± 0.14 (IndexTTS 2 3.93, CosyVoice 2 3.78). In Chinese, N-MOS is 4.07 ± 0.13, the highest including ground truth (3.94), and S-MOS is 4.03 ± 0.11, between CosyVoice 2 (4.01) and IndexTTS 2 (4.07). Most differences fall within the 95% confidence intervals given the small test.

Ablations (§5.3, Table 3; Emilia-EN, Seed-TTS English). Removing SFM alignment raises WER from 2.97% to 3.40% and lowers SIM from 0.635 to 0.625. Removing previous-patch conditioning in LocDiT is far more damaging (WER 8.46%, SIM 0.587).

Granularity under fixed information rate (§5.4, Table 4). With frame rate and latent dimension traded off (15 Hz x 32, 30 Hz x 16, 60 Hz x 8), reconstruction on LibriTTS is roughly unchanged by alignment. PESQ moves by under 0.03 and UTMOS at 15 Hz is slightly lower with alignment (4.030 vs 4.066). Generation WER, however, diverges as sequences lengthen: 2.97 vs 3.40 at 15 Hz, 3.24 vs 5.20 at 30 Hz, and 14.71 vs 28.06 at 60 Hz (with vs without alignment).

Novelty Assessment

The individual components are borrowed. The σ-VAE comes from LatentLM, the alignment losses are adopted directly from an image latent-diffusion method, the patch-wise LLM plus LocDiT with history conditioning follows DiTAR, and semantic alignment of speech VAEs with SSL features was already applied to non-autoregressive TTS by Semantic-VAE. The contribution is putting these together for continuous AR TTS and showing, through the fixed-information-rate granularity study, that the benefit of SFM alignment on generation grows sharply as the latent sequence becomes longer, while reconstruction quality is unaffected. That analysis is the most distinctive and reusable part of the paper. The headline system is an engineering integration whose benchmark results are competitive rather than clearly leading.

Field Significance

Moderate. The paper adds evidence that the semantic structure of a continuous latent space, not only its reconstruction fidelity, governs how well an autoregressive model can generate it, and isolates this with a controlled frame-rate versus dimension sweep. It also provides a further data point that a 15 Hz continuous latent with patch-wise diffusion decoding can reach competitive Seed-TTS intelligibility from a 1.5B LLM at 100K to 150K hours of data.

Claims

  • supports: Aligning a reconstruction-trained continuous speech latent space with self-supervised speech representations improves the intelligibility of autoregressive generation from that space without measurably changing reconstruction quality.

    Evidence: At 15 Hz, WavLM-guided alignment of the σ-VAE lowers Seed-TTS English WER from 3.40% to 2.97% and raises SIM from 0.625 to 0.635, while LibriTTS PESQ (3.175 vs 3.179) and STOI (0.950 vs 0.949) are essentially unchanged. (§5.3, Table 3; §5.4, Table 4)

  • refines: The benefit of semantic structure in a continuous latent space grows with the sequence length the autoregressive model must handle, so high-frame-rate latents depend on it most.

    Evidence: Under a fixed information rate, the WER gap between unaligned and aligned latents widens from 0.43 points at 15 Hz (3.40 vs 2.97) to 1.96 at 30 Hz (5.20 vs 3.24) and 13.35 at 60 Hz (28.06 vs 14.71), even though reconstruction metrics stay comparable across all three settings. (§5.4, Table 4)

  • complicates: Reconstruction quality of a continuous speech latent is a poor predictor of its suitability for autoregressive generation.

    Evidence: The 15 Hz, 30 Hz and 60 Hz VAEs have similar PESQ (2.88 to 3.18), STOI (0.944 to 0.953) and UTMOS (3.97 to 4.07), yet aligned generation WER ranges from 2.97% to 14.71%. (§5.4, Table 4)

  • supports: In patch-wise continuous autoregressive TTS with a diffusion head, conditioning the head on the previously generated patch is critical for stable generation.

    Evidence: Removing previous-patch conditioning from LocDiT raises Seed-TTS English WER from 2.97% to 8.46% and lowers SIM from 0.635 to 0.587, a much larger drop than removing SFM alignment. (§5.3, Table 3)

  • complicates: Gains in intelligibility from continuous autoregressive TTS do not automatically carry over to speaker similarity or to hard, complex sentences.

    Evidence: SemaVoice has the lowest English WER on Seed-TTS (1.71%) but trails VoxCPM, MaskGCT and IndexTTS 2 on English SIM, and trails CosyVoice 2, IndexTTS 2 and Qwen2.5-Omni on Hard-subset CER (8.09% vs 6.83%, 7.12%, 7.97%). (§5.1, Table 1)

Limitations and Open Questions

  • Thin evidence for the headline mechanism. The full-scale gain from SFM alignment is never isolated: the alignment ablation exists only at the Emilia-EN scale, where the 15 Hz improvement is modest (0.43 WER points, 0.010 SIM). The large effects appear only at 30 and 60 Hz, which are not the operating point of the final system.
  • Baseline fairness. Baseline numbers are drawn from official reports or implementations “under comparable evaluation settings”, and the full system uses 50K hours of internal data, so the Table 1 comparisons are not strictly controlled.
  • Small subjective test. The MOS study uses 12 raters on 20 utterances and two baselines; most differences overlap within the reported confidence intervals.
  • Reproducibility. No code, checkpoints or demo page are mentioned in the paper, and the VAE and LocDiT parameter counts are not reported.
  • Scope. The authors note evaluation is limited to Chinese and English, and that the continuous AR design retains sequential inference latency and potential error accumulation on long sequences (§Limitations). No latency or real-time-factor measurements are reported.
  • Open question. Whether other self-supervised encoders, or alignment to text-derived targets, would serve better than WavLM-large is not explored.

Wiki Connections

  • Diffusion TTS — Generates each continuous latent patch with a DDPM-trained Diffusion Transformer head conditioned on an autoregressive LLM state and the previous patch.
  • Zero-Shot TTS — Evaluated as a zero-shot voice-cloning system on the Seed-TTS English, Chinese and Hard sets with WER/CER, SIM and MOS.
  • Neural Audio Codec — Proposes a WavLM-aligned σ-VAE continuous speech tokenizer at 15 Hz and evaluates its reconstruction with PESQ, STOI and UTMOS.
  • Self-Supervised Speech — Uses frozen WavLM-large features as the alignment target that shapes the VAE latent space.
  • Multilingual TTS — Trains a single model on Chinese and English data and evaluates synthesis in both languages.
  • Subjective Evaluation — Reports N-MOS and S-MOS from 12 human raters in English and Chinese.
  • DiTAR — Directly adopts DiTAR’s patch-wise LLM plus LocDiT design, adding previous-patch conditioning and an aligned latent space.
  • Semantic-VAE — Closest prior work, applying SSL-based semantic alignment of speech VAE latents to non-autoregressive TTS; SemaVoice extends the idea to continuous AR generation.
  • Speech VAE Distillation Losses — Related study of distillation losses for speech VAEs that SemaVoice cites as a semantic-alignment precedent.
  • LatentLM — Source of the σ-VAE formulation that SemaVoice adopts for its high-compression continuous latents.
  • VoxCPM — Strongest continuous AR baseline, ahead of SemaVoice on Chinese CER and English SIM but behind on English WER.
  • IndexTTS2 — Baseline in both objective and subjective comparisons, ahead of SemaVoice on Hard-subset CER and speaker similarity.
  • CosyVoice 2 — Baseline in objective and MOS comparisons, with the best Hard-subset CER in Table 1.
  • Seed-TTS — Provides the Seed-TTS evaluation sets used for all generation results.