arXiv · 2025 · Preprint
Jingyuan Xing et al. (South China University of Technology) · → Paper · Demo: ✓ · Code: ?
Parallel GPT introduces a hybrid AR+NAR architecture with a Parallel Tokenizer that simultaneously extracts independent semantic and acoustic tokens, then couples them in a non-autoregressive refinement stage, improving zero-shot TTS naturalness and intelligibility over cascaded semantic-first approaches.
Problem
Prior zero-shot TTS systems fall into two main camps: those that predict overall speech tokens (treating speech as monolithic, risking information loss), and those that cascade semantic tokens before acoustic tokens. The cascaded approach imposes a semantic-first bias: acoustic features that are largely independent of semantics — such as formant frequencies, voice onset time, and speaker-specific articulatory patterns — can be suppressed during the first stage and poorly recovered in the second. Neither paradigm directly addresses the dual nature of speech, in which semantic and acoustic content are simultaneously independent (they can vary separately) and interdependent (they must ultimately cohere).
Method
Parallel GPT consists of three main components trained sequentially.
Parallel Tokenizer. A custom tokenizer combines three frozen SSL models: Wav2Vec 2.0 (semantic features), BEATs (acoustic features), and CAM++ / Campplus (speaker features). Each of Wav2Vec 2.0 and BEATs projects 16 kHz speech into 768-dimensional embeddings at 50 Hz; these are independently quantized with 3-layer RVQ encoders (3 bits per layer), yielding semantic and acoustic token streams that are temporally aligned but content-disentangled. A speaker encoder following the ECAPA-TDNN architecture produces a 512-dimensional time-varying embedding; cosine-distance distillation aligns it to CampPlus speaker representations. Reconstruction uses flow matching to generate a Mel spectrogram from the concatenated semantic+acoustic features conditioned on the speaker embedding, followed by a HiFi-GAN vocoder. The tokenizer is trained for 450k steps with combined MSE, speaker cross-entropy, Mel-spectrogram MSE, and adversarial losses.
Parallel Autoregressive Language Model (Parallel AR). Built on a GPT-2 decoder-only transformer with 1024-dimensional embeddings. The first quantizer layer (“top tokens”) of both semantic and acoustic RVQ streams are concatenated with text embeddings and processed through a stack of causal transformer layers. At each step the 1024-dimensional output is split into two 512-dimensional vectors: each is passed to an independent linear predictor for semantic or acoustic top tokens respectively, with top-k sampling applied separately. A joint stop-check module uses both vectors to determine sequence termination. Crucially, both token types are generated simultaneously in a single forward pass, maintaining temporal alignment while preserving content independence. The model is trained for 800k steps using ScaledAdam.
Coupled Non-Autoregressive Transformer (Coupled NAR). Two prediction stages refine the second and third RVQ layers sequentially. Unlike the AR stage, semantic and acoustic tokens are concatenated (not split) before processing in 3-layer transformer decoder blocks — capturing their interdependence at the detail level. The model predicts finer-grained token layers conditioned on the AR output and reference speech tokens. It is trained separately for 200k steps with a fixed learning rate.
At inference: the Parallel Tokenizer encodes reference speech into top tokens; the Parallel AR generates target top tokens from text and reference top tokens; the Coupled NAR adds layers 2 and 3; the Parallel Tokenizer decoder reconstructs speech.
Key Results
On LibriTTS (English), Parallel GPT achieves MOS 4.11 (dev) / 4.08 (test), outperforming MaskGCT (4.01/4.02) and CosyVoice (4.05/4.01) on naturalness (Table I). WER is 0.211 (dev) / 0.241 (test), lower than all baselines except MaskGCT’s dev WER (0.216), and UTMOS 3.632/3.642 exceeds MaskGCT (3.60/3.52). On speaker similarity (SMOS), Parallel GPT (4.08/3.92) is closely competitive with MaskGCT (4.05/3.91) but trails CosyVoice (4.23/4.11) — a gap the authors attribute to CosyVoice’s specialized speaker embedding refinement.
On Chinese (Table II), Parallel GPT leads all reported baselines in MOS (4.23 dev / 4.19 test) and achieves the lowest WER (0.185 dev / 0.193 test), outperforming CosyVoice, E2-TTS, and MaskGCT across metrics.
Ablation (Tables III, IV) confirms the importance of each component: removing parallel tokenization (using only Wav2Vec or only BEATs) sharply degrades MOS and WER; removing parallel AR prediction (-w/o parallel) causes the largest WER increase (0.505 dev English); using only AR (no NAR stage) degrades SMOS and UTMOS. All comparisons use models retrained on identical datasets.
Novelty Assessment
The key contribution is architectural: treating semantic and acoustic tokens as parallel, disentangled streams in a single AR forward pass rather than as a cascade. This is a principled response to the identified failure mode of semantic-first systems. The Parallel Tokenizer’s combination of three specialist SSL models (Wav2Vec 2.0 for semantics, BEATs for acoustics, CampPlus for speaker) for simultaneous RVQ encoding is novel in its explicit separation of information streams. The AR+NAR split (independence at the top token level, coupling at the detail token level) elegantly mirrors the theoretical framing of the problem. The contribution is primarily architectural; training data and model scales are relatively modest compared to frontier systems like CosyVoice or Seed-TTS.
Field Significance
Moderate — Parallel GPT provides a principled architectural solution to the semantic-first bias in cascaded codec TTS systems, demonstrating that parallel rather than sequential token generation can improve naturalness and intelligibility on English and Chinese benchmarks. The disentangled SSL feature extraction design (separate Wav2Vec 2.0 and BEATs encoders feeding independent RVQ streams) introduces a reusable pattern for token-level speech disentanglement. The approach does not reach frontier speaker similarity scores, and the training data scale is modest, limiting its immediate practical impact.
Claims
- Generating semantic and acoustic tokens simultaneously in a single autoregressive forward pass, rather than cascading semantic prediction before acoustic prediction, reduces word error rate and improves naturalness in zero-shot TTS. (§V.A, Table I)
- Combining specialist SSL models for distinct speech attributes (semantic content, acoustic texture, speaker identity) as frozen feature extractors enables more effective token-level disentanglement than using a single encoder for all attributes. (§III.A, Tables III–IV)
- A hybrid AR+NAR design that enforces independence at the coarse token level and interdependence at the fine-grained level distributes modeling complexity more effectively than an all-AR or all-NAR approach. (§V.B, Tables III–IV)
- Parallel semantic-acoustic modeling improves naturalness and intelligibility without fully closing the speaker similarity gap relative to systems with dedicated speaker embedding refinement. (§V.A, Table I)
Limitations and Open Questions
Speaker similarity lags behind CosyVoice (SMOS gap ~0.15–0.2 on English), suggesting the parallel architecture does not yet fully leverage speaker conditioning. UTMOS scores, while competitive, do not reach ground-truth levels. Model size is not reported, making compute comparisons difficult. The internal Chinese dataset and preprocessing pipeline (Emilia + NCSSD) are not publicly released, limiting reproducibility on that front. Extending the framework to prosody control, emotion conditioning, or cross-lingual voice conversion is not explored. The subjective decoupling evaluation (Section V.C) relies on 90% evaluator agreement rather than a standardized metric, leaving quantitative disentanglement assessment as an open question.
Wiki Connections
Parallel GPT extends the autoregressive-codec-tts paradigm by parallelizing semantic and acoustic token prediction within a single LM pass, directly addressing the semantic-first bias seen in VALL-E and related cascade systems. The Parallel Tokenizer approach is related to neural-codec design and disentanglement of speech representations. The use of Wav2Vec 2.0, BEATs, and CampPlus as frozen backbone encoders places this work squarely in the self-supervised-speech tradition. Zero-shot generalization to unseen speakers via reference audio conditioning connects to zero-shot-tts and speaker-adaptation. CosyVoice, CosyVoice 2, and F5-TTS serve as strong baselines; Seed-TTS represents the frontier of large-scale zero-shot TTS this work competes against.