ACL · 2025 · Conference
Shengpeng Ji et al. (Zhejiang University / Alibaba Tongyi Speech Lab) · → Paper · Demo: ? · Code: ✓
ControlSpeech is the first TTS system that simultaneously performs zero-shot speaker cloning from a reference audio clip and zero-shot style control from a natural language description, using a disentangled discrete codec space and a novel Style Mixture Semantic Density (SMSD) module to resolve the many-to-many ambiguity between textual style descriptions and audio.
Problem
Prior zero-shot TTS systems (e.g., VALL-E, NaturalSpeech 3) clone any unseen speaker’s voice but fix the speaking style — the synthesized speech inherits whatever style is implicit in the reference audio and the model cannot adjust prosody, pace, emotion, or volume independently. Conversely, prior style-controllable TTS systems (PromptTTS, InstructTTS, PromptStyle) accept natural language style descriptions but require speaker IDs from a closed training-set vocabulary, offering no zero-shot speaker generalization.
A deeper problem also exists specifically in the style-control task: there is a many-to-many mapping between textual style descriptions and audio. Different phrasings (“speaks rapidly” and “articulates at considerable speed”) map to the same acoustic style (many-to-one), while a single description (“fast speech”) corresponds to a range of differing degrees of that attribute in different utterances (one-to-many). Prior work (PromptTTS 2) attributed one-to-many to missing timbre information; ControlSpeech identifies it as an inherent ambiguity of natural language descriptions themselves.
Finally, no large-scale dataset combines both style description labels and multi-speaker data suitable for zero-shot training, so data scarcity compounds the architectural challenge.
Method
Disentangled codec space. ControlSpeech builds on FACodec (from NaturalSpeech 3), a pre-trained factorized codec that separates the speech signal into content codes (Y_c), prosody codes (Y_p), acoustic detail codes (Y_a), and a global timbre embedding (Y_t). The codec encoder is frozen. Style codes Y_s are constructed by concatenating prosody and acoustic codes: Y_s = concat(Y_p, Y_a). This pre-trained disentanglement enables independent conditioning on timbre and style without interference.
Architecture. ControlSpeech is an encoder-decoder model with three input branches: a phoneme-level text encoder for content X_c, a BERT-based word-level style encoder for the style description X_s (using the [CLS] embedding as global style), and a speech prompt branch processed by the frozen FACodec encoder plus a conformer-based timbre extractor that produces a global timbre vector Y_t. The style and text representations are fused via cross-attention. A duration predictor (MFA-supervised, MSE loss) expands frame lengths. The codec generator is a non-autoregressive conformer decoder that iteratively generates discrete codec tokens using mask-based parallel decoding (cosine masking schedule, confidence-based iterative refinement, following MaskGIT and SoundStorm). The timbre vector is injected into this generation stage via conditional layer normalization (AdaSpeech-style), then the frozen FACodec decoder synthesizes the final waveform.
Style Mixture Semantic Density (SMSD) module. To resolve the many-to-many issue, SMSD models the conditional distribution P(Y_s’ | X_s’) as a mixture of K Gaussian distributions using a mixture density network (MDN). The BERT-encoded style prompt X_s’ serves as input; the MDN outputs mixture weights π_k, means μ^(k), and variances σ^2(k). During training, KL divergence between the predicted mixture and the actual style representation Y_s’ (obtained from the style extractor on target audio) is minimized via negative log-likelihood. During inference, sampling from the mixture produces style representations with diversity reflecting the one-to-many range. An additional noise perturbation module controls the isotropy of noise across dimensions; the best-performing mode is “isotropic across clusters” (shared σ² across dimensions and mixture components), which balances accuracy and diversity. The SMSD, codec, and duration losses are summed for joint training.
VccmDataset. The authors build and release VccmDataset based on TextrolSpeech (LibriTTS + emotional speech), with refined attribute label boundaries for gender, volume, speed, pitch, and emotion. Test sets A–D cover in-domain, out-of-domain speakers, out-of-domain style descriptions, and many-to-many style control scenarios respectively.
Baselines. Style-controllable baselines: Salle, PromptStyle, InstructTTS, PromptTTS 2. Zero-shot TTS baselines: VALL-E, MobileSpeech. All baselines were re-implemented on VccmDataset by the authors for fairness.
Key Results
On VccmDataset test set A (style controllability, using GT timbre as prompt), ControlSpeech achieves the best speed accuracy (0.829 vs. 0.786 for PromptStyle), volume accuracy (0.894 vs. 0.831 for Salle), and emotion accuracy (0.557 vs. 0.412 for InstructTTS). Pitch accuracy (0.833) is slightly below InstructTTS (0.849) and PromptTTS 2 (0.867), attributed to the difficulty of simultaneously controlling different timbres and styles. ControlSpeech achieves the best speaker similarity (Spk-sv 0.89), WER (2.9%), and MOS-Q (3.91 ± 0.09), outperforming all baselines on audio quality while uniquely providing zero-shot timbre cloning (all style-only baselines lack this capability).
On test set B (zero-shot speaker cloning, out-of-domain speakers), ControlSpeech achieves WER 3.3% vs. VALL-E at 6.7% and MobileSpeech at 4.1%, with MOS-Q 3.95 and MOS-S 3.96 — comparable to MobileSpeech (MOS-Q 3.94, MOS-S 4.01) despite additionally providing style control.
On test set D (many-to-many style control), ControlSpeech scores MOS-TS 4.01, MOS-SA 3.84, MOS-SD 4.05, substantially outperforming PromptStyle (MOS-SA 3.45, MOS-SD 3.53) and InstructTTS (MOS-SA 3.57, MOS-SD 3.48). The ablation without SMSD drops MOS-SA to 3.59 and MOS-SD to 3.66, confirming SMSD’s contribution.
On test set C (out-of-domain style descriptions), ControlSpeech degrades much less than baselines across all metrics, indicating substantially better generalization from the SMSD’s mixture density mechanism.
The codec decoupling ablation (test set A) is striking: without decoupling, controllability collapses (pitch 0.492, speed 0.517, emotion 0.237), confirming that independently disentangling the speech prompt and style prompt is necessary to prevent cross-channel interference.
Novelty Assessment
The primary contribution is the integration of two independently established capabilities — zero-shot voice cloning and natural language style control — into a single jointly-trained system. No prior work had achieved both simultaneously in a zero-shot setting. The approach is architecturally pragmatic: it composites existing strong components (frozen FACodec disentanglement, BERT, MaskGIT-style codec generator, AdaSpeech conditional normalization) rather than proposing a fundamentally new architecture.
The SMSD module is a genuine conceptual contribution in identifying and formally framing the many-to-many problem as a mixture density estimation task. The noise perturbation mechanism providing isotropy choices is a useful engineering contribution for diversity control. However, mixture density networks and the MDN training objective are well-established; the novelty is in applying them specifically to style semantic conditioning in TTS.
The VccmDataset release (built on TextrolSpeech with more precise attribute labeling) is a practical contribution to the field.
Overall, the contribution is primarily at the system integration level, with a well-motivated and effective module design for the style ambiguity problem. It is not a fundamental algorithmic breakthrough but represents a meaningful advance for controllable personalized TTS.
Field Significance
Moderate — ControlSpeech introduces a new capability boundary for personalized TTS by enabling simultaneous zero-shot speaker cloning and natural language style control in a single system, a combination not demonstrated in prior work. The SMSD module provides a concrete probabilistic framing for the many-to-many ambiguity in textual style conditioning, which can serve as a design reference for future style-controllable systems. The VccmDataset release provides a reusable benchmark for this joint task.
Claims
- Disentangling speaker timbre and speaking style into separate codec representations is a necessary condition for simultaneous zero-shot speaker cloning and style control; without explicit decoupling, the two conditioning signals interfere and controllability collapses. (§4.3, Table 4)
- Natural language style descriptions have an inherent many-to-many relationship with audio that cannot be resolved by timbre conditioning alone, requiring a probabilistic model of style variation such as a mixture density network. (§3.3)
- Zero-shot speaker cloning capability in style-controllable TTS can be achieved by building on a large-scale pre-trained disentangled codec without sacrificing audio quality relative to dedicated zero-shot TTS systems. (§4.2, Table 2)
- Probabilistic sampling from a mixture density model of style representations improves both style diversity and generalization to out-of-domain style descriptions compared to deterministic style encoding. (§4.2, §4.3, Table 3)
- Pitch control is measurably harder to preserve when timbre and style are controlled simultaneously, suggesting that pitch conditioning interacts with speaker identity in ways that speed, volume, and emotion do not. (§4.2, Table 1)
Limitations and Open Questions
The paper explicitly notes two limitations: (1) The training dataset is still limited in scale for style-controllable TTS; significantly larger datasets (tens of thousands of hours with style annotations) may be needed to achieve more advanced controllability. (2) The exploration of generative model architectures is narrow — only non-autoregressive parallel decoding is tried. Diffusion or flow-matching generators operating in the disentangled codec space might offer better quality or diversity.
Beyond the paper’s self-assessment: pitch accuracy is the only metric where ControlSpeech underperforms style-only baselines on both in-domain and out-of-domain test sets. The authors attribute this to simultaneous timbre-style control, but the mechanism is unexplained and unresolved. The paper does not evaluate on standard TTS benchmarks (LibriSpeech, VCTK), relying entirely on VccmDataset, which makes external comparison difficult. The demo availability is not confirmed in the paper. Ethical risks from zero-shot voice cloning are acknowledged but only partially addressed (watermarking is proposed as future work).
Wiki Connections
This paper informs zero-shot-tts by demonstrating that zero-shot timbre cloning can be combined with instruction-based style control without sacrificing either capability — a direct extension beyond the VALL-E family which only clones voice without style control. It is a companion to instruction-conditioned-tts, providing the first zero-shot-capable instance of that paradigm.
The system’s core relies on disentanglement via FACodec (NaturalSpeech 3), building directly on the factorized codec disentanglement approach noted in that concept page. The SMSD module connects to prosody-control by explicitly addressing the diversity and accuracy trade-off in style conditioning via probabilistic modeling.
The paper’s use of FACodec as a frozen disentangled tokenizer connects it to neural-codec, representing a distinctive application pattern: using a codec not just as a compression target but as a disentanglement backbone whose factored representations are individually controlled.