arXiv · 2025 · Preprint
Xinsheng Wang et al. (Hong Kong University of Science and Technology) · → Paper · Demo: ✓ · Code: ✓
Spark-TTS introduces BiCodec, a single-stream speech codec that disentangles speech into semantic tokens (linguistic content) and fixed-length global tokens (speaker attributes), enabling a single autoregressive LLM to perform both zero-shot voice cloning and attribute-controlled voice creation without multi-stage processing.
Problem
Codec-based LLM approaches to zero-shot TTS require either dual generative models (one for semantic prediction, one for acoustic reconstruction) or complex multi-stream codebook architectures that diverge structurally from standard text LLMs. Existing semantic-only tokens lack the capacity to control speaker timbre within the LM itself, forcing separate acoustic models. Meanwhile, acoustic tokens rely on hierarchical residual quantization that is cumbersome to integrate into a single-pass LLM pipeline. The field also lacked open, comprehensively annotated training data for attribute-controllable TTS research.
Method
BiCodec encodes speech into two complementary token streams. The semantic tokenizer extracts 50 tokens-per-second (TPS) discrete codes using a ConvNeXt-based encoder conditioned on wav2vec 2.0 features (specifically layers 11, 14, and 16 of XLSR-53, averaged). These tokens capture linguistic content with strong phonemic alignment. The global tokenizer processes the Mel spectrogram through an ECAPA-TDNN encoder followed by a cross-attention module with learnable queries, producing a fixed-length sequence of 32 tokens that represent time-invariant speaker attributes. The global tokenizer uses Finite Scalar Quantization (FSQ) with 6 dimensions of 4 levels each (codebook size 4,096), specifically chosen over group-VQ to avoid codebook collapse. Both token streams are concatenated for reconstruction by a shared ConvNeXt decoder. BiCodec operates at 50 TPS with a bitrate of 0.65 kbps, trained end-to-end with GAN losses (multi-period and multi-band multi-scale STFT discriminators), multi-scale mel-spectrogram reconstruction, codebook commitment loss, and a wav2vec 2.0 feature reconstruction loss to maintain semantic alignment.
The Spark-TTS language model fine-tunes Qwen2.5-0.5B in a decoder-only configuration, directly predicting the concatenated token sequence: for zero-shot TTS, the model conditions on text and global tokens extracted from a reference audio, then predicts the semantic token sequence. For attribute-controlled voice creation, a chain-of-thought (CoT) mechanism is employed: the model first predicts fine-grained attribute values (pitch in Hz rounded to integer, speaking rate in syllables-per-second) from coarse-grained labels (gender, 5-level pitch, 5-level speed), then generates global tokens, then semantic tokens. Both training objectives are mixed, with each training example constructed as two samples corresponding to the zero-shot and attribute-control formulations.
VoxBox, the accompanying dataset, aggregates 29 open-source corpora totaling 102.5k hours of English and Chinese speech. Each utterance is annotated for gender (using a fine-tuned WavLM-large classifier, 99.4% accuracy on AISHELL-3), pitch (PyWorld extraction, 5 Mel-scale levels separately calibrated for male and female speakers), and speaking rate (syllable-based, 5 levels per language). Additional annotations include age and emotion labels.
Key Results
On the Seed-TTS-eval benchmark, Spark-TTS achieves WER of 1.98% (English) and CER of 1.20% (Chinese) for zero-shot TTS, outperforming all single-stage AR methods including Llasa-8B (WER 2.97%, CER 1.59%) despite using only 0.5B parameters versus Llasa’s 8B and less training data (100k vs. 250k hours). On Chinese intelligibility, Spark-TTS ranks second only to the closed-source Seed-TTS (CER 1.12%); on English it ranks second to F5-TTS (WER 1.83%), which is a flow-matching NAR method (Table 4).
Speaker similarity in zero-shot TTS is Spark-TTS’s relative weakness: it scores 0.672 SIM on test-zh and 0.584 SIM on test-en, outperforming Llasa (0.684, 0.574) but falling behind multi-stage models such as MaskGCT (0.774, 0.714) and Seed-TTS (0.796, 0.762). The gap against multi-stage methods is attributed to the inherent variability of AR generation without disentanglement constraints on global and semantic tokens.
On the LibriSpeech test-clean quality benchmark, Spark-TTS achieves UTMOS 4.35, surpassing ground truth (4.08) and CosyVoice2 (4.23), the leading open-source multi-stage model (Table 5).
For attribute control, Spark-TTS reaches 99.77% gender accuracy, exceeding Parler-TTS (98.12%) and VoxInstruct (82.99%) (Table 3). Pitch and speed confusion matrices show strong coarse-grained control, and fine-grained numerical control produces monotonically ordered outputs across the full pitch and speed ranges.
BiCodec reconstruction at 0.65 kbps achieves STOI 0.92, PESQ-WB 2.51, UTMOS 4.18, and SIM 0.80, outperforming all sub-1 kbps codecs on most metrics, ranking second on UTMOS (behind StableCodec at 4.23) and second on SIM (behind X-codec2 at 0.82) (Table 1).
Novelty Assessment
The primary novelty is BiCodec’s two-stream decomposition strategy: using SSL-derived semantic tokens for linguistic content and FSQ-quantized global tokens for speaker attributes within a single-codebook architecture. This directly addresses the gap between semantic-only tokens (which need acoustic models) and acoustic tokens (which need multi-codebook prediction). The use of learnable queries with cross-attention for fixed-length global token extraction is a specific design choice that outperforms the prior GVQ-based TiCodec approach (Table 2). Framing the LLM training as a standard next-token prediction task with no flow-matching stage is genuinely simpler than CosyVoice2 or Seed-TTS.
The CoT attribute-control approach is a reasonable extension but not deeply novel: it reuses the LLM’s token prediction in sequence, which is a natural consequence of the architecture. The approach resembles instruction-conditioned TTS (Parler-TTS, VoxInstruct) but is integrated at the codec level rather than via natural language descriptions.
VoxBox is a substantive dataset contribution: 102.5k hours with per-utterance pitch, speed, and gender annotations across two languages, assembled entirely from open sources. Its annotation methodology (syllable-based speed, Mel-scale pitch bucketing with gender-stratified thresholds) is carefully designed.
The model achieves competitive zero-shot TTS with an unusually small backbone (0.5B), suggesting that BiCodec’s disentanglement reduces the LM’s modeling burden.
Field Significance
Tip
High — Spark-TTS demonstrates that a single small LLM (0.5B parameters) can match or exceed much larger single-stage AR systems for zero-shot TTS intelligibility when equipped with a well-designed disentangled codec. The BiCodec design, combining SSL semantic tokens with FSQ global tokens in a single codebook, offers a practical simplification over dual-model and multi-stream pipelines, and the open-source VoxBox dataset fills a genuine gap for annotated controllable TTS research in both English and Chinese.
Claims
- Disentangling speech tokens into linguistic content and speaker attributes within a single-stream codec enables a standard LLM to perform zero-shot TTS without a multi-stage pipeline. (§3, §4.1)
- Small LLM backbones can achieve competitive zero-shot TTS intelligibility when the codec reduces per-token modeling complexity through semantic alignment. (§6.4, Table 4)
- Single-stage autoregressive TTS consistently trails multi-stage or non-autoregressive methods on speaker similarity metrics, even when intelligibility is comparable. (§6.4, Table 4, Limitation)
- FSQ-based global token quantization with learnable cross-attention queries produces better speaker attribute representation than group-VQ at equivalent token lengths. (§6.2, Table 2)
- Attribute-controllable TTS benefits from hierarchical coarse-to-fine prediction within the LM inference loop rather than requiring separate conditioning modules. (§4.1, §6.3)
Limitations and Open Questions
Warning
Speaker similarity in zero-shot cloning is meaningfully lower than multi-stage methods (SIM 0.672 vs. 0.774 for MaskGCT on test-zh). The paper attributes this to AR variability without explicit disentanglement constraints between semantic and global tokens, and no solution is evaluated in this work.
The VoxBox training data and BiCodec codec are trained on separate, relatively limited datasets (3k hours for BiCodec; 102.5k hours for the LM). The BiCodec training data is English-only (LibriSpeech + Emilia EN/CN), which may limit acoustic reconstruction quality for languages outside this distribution.
The attribution control is evaluated primarily for gender, pitch, and speed; there is no evaluation of emotion control despite VoxBox containing emotion annotations for many source datasets.
Fine-grained numerical pitch control is evaluated on figures (Figs. 4–5) but no quantitative accuracy metric against target pitch values is reported, making it difficult to assess precision.
Wiki Connections
BiCodec’s design is directly related to neural-codec and disentanglement, using SSL representations (wav2vec 2.0) central to self-supervised-speech. The system targets zero-shot-tts and autoregressive-codec-tts paradigms, with its attribute control mechanism relevant to prosody-control.
Key in-corpus papers cited: 2301.02111 (VALL-E, the foundational codec LM approach), 2308.16692 (SpeechTokenizer, the semantic distillation codec baseline), 2407.05407 and 2412.10117 (CosyVoice / CosyVoice2, the primary two-stage comparison), 2406.02430 (Seed-TTS, the closed-source upper bound), 2210.13438 (EnCodec, the standard codec baseline), 2412.15115 (Qwen2.5, the LLM backbone), 2410.00037 (Moshi), 2204.02152 (UTMOS, the automatic MOS predictor used for evaluation), 1904.02882 (LibriTTS, used in evaluation).
Cited by in-corpus papers: 2505.17589 (CosyVoice 3 benchmarks against Spark-TTS as a recent autoregressive zero-shot TTS baseline)
2507.16632 — Step-Audio 2 Technical Report