arXiv · 2025 · Preprint
Ohmura et al. (Sony Group Corporation) · → Paper · Demo: ? · Code: ?
Introduces LibriTTS-VI, the first public voice impression corpus, and two training strategies that substantially reduce impression leakage in numerical voice impression control for TTS.
Problem
Voice impression control (VIC) allows TTS systems to adjust perceptual voice characteristics along 11 dimensions (such as pitch, masculinity, brightness, and speaking rate) via numerical target vectors. Prior work demonstrated the concept but depended on a private corpus, making replication and follow-up research impossible. A second structural problem, impression leakage, persisted even in prior published methods: when the same reference utterance is used for both speaker identity conditioning and VI conditioning during training, the synthesized voice is biased toward the reference’s inherent impression rather than the numerically specified target. Standard mitigation techniques (gradient reversal, high-rate dropout) were insufficient to eliminate this entanglement.
Method
The system builds on a VITS backbone augmented with a Conformer-based text encoder, a connectionist temporal classification loss, and an adaptive layer-norm Transformer. A speaker encoder consisting of a pre-trained HuBERT model followed by a BiLSTM with attention encodes a reference utterance into an intermediate vector. A control module (CM) then modulates this vector given the 11-dimensional target VI vector (with HuBERT and VI projected into 32-dimensional spaces), and a style token layer produces the final speaker embedding.
A VI estimator (VIE) sharing the HuBERT-BiLSTM architecture but with independent parameters predicts VI from audio; during training the CM is fine-tuned for 60k steps (after 600k steps of backbone pre-training) using the VIE to supply target VI labels.
The two proposed methods address leakage at different levels:
VIC-dis (disentanglement via different utterances) uses a second utterance from the same speaker as the reference for speaker conditioning while keeping the original utterance as the synthesis target and VI source. The speaker encoder thus sees a reference with a different inherent VI, forcing decoupling without any architectural changes.
VIC-srf (speaker-reference-free) eliminates the speaker reference entirely by replacing the speaker encoding input with Gaussian noise in the CM concatenation. Inspired by pseudo-speaker embedding generation, synthesis is conditioned solely on the target VI vector. The stochastic duration predictor is additionally fine-tuned to stabilise speaking rate under this setup.
The LibriTTS-VI corpus was built by having four in-house annotators rate 130 utterances sampled from distinct LibriTTS-R speakers across 10 VI dimensions on a 7-point Likert scale. Speaking rate was computed automatically via timestamped ASR. To scale annotation to the full LibriTTS-R corpus, the VIE was trained using a similarity-based data augmentation strategy: the 100 most acoustically similar utterances per speaker (by pitch, energy, and WavLM cosine similarity) inherited each speaker’s manual label, yielding 100-fold augmentation.
Key Results
VIC-srf achieves the best VI controllability across both objective and subjective evaluations. Objectively, the random-target VI MSE (RVI-MSE) drops from 0.61 (VIC-base) to 0.51 (VIC-dis) and 0.41 (VIC-srf), with impression leakage (∆V = RVI-MSE minus VI-MSE) falling from 0.22 (VIC-base, statistically significant) to 0.14 (VIC-dis) and 0.05 (VIC-srf, not significantly different from zero). Synthesis quality is maintained: UTMOS is 4.26 for VIC-srf vs. 4.23 for VIC-base, and MOS on the multiple-VI condition for speaker 8555 is 3.60 vs. 3.71, with no statistically significant degradation. Subjective VI-MSE reduces from 1.15 to 0.92 in the multiple-VI setting. In the VI modulation experiment, control slopes average 0.199 (VIC-srf), 0.159 (VIC-dis), and 0.121 (VIC-base), with VIC-srf yielding positive slopes across all 11 dimensions (Table 3).
Qwen3-TTS with natural language prompts performs poorly on numerical VI control: VI-MSE of 0.82-0.87 versus 0.36-0.39 for VITS-based systems. Fine-tuning Qwen3-TTS on LibriTTS-R further worsens VI-MSE, suggesting that adapting large pre-trained LLM TTS systems to fine-grained numerical control is non-trivial. Text-VI entanglement was confirmed empirically: punctuation (e.g., exclamation marks) shifts predicted VI toward restlessness dimensions, and a Ridge regression proxy showed that text embeddings predict VI for five of the eleven dimensions at correlation above 0.2.
Novelty Assessment
The LibriTTS-VI corpus is the primary genuine contribution. There was no public VI corpus prior to this work, making the sub-field effectively unreproducible. The annotation methodology is standard (human raters, Likert scales) but the acoustic similarity-based augmentation strategy for scaling annotation is practical and methodologically appropriate given the strict speaker-consistency assumption would fail for expressive narration.
The training strategies (VIC-dis, VIC-srf) are incremental. VIC-dis is a training recipe change requiring no new modules; VIC-srf replaces the speaker reference with noise and is directly inspired by prior pseudo-speaker-embedding work. Neither method introduces fundamentally new architecture. The improvements in controllability are real and statistically significant, and the methods are clean enough to be adopted in follow-up work. The comparison against Qwen3-TTS is a useful empirical finding, though it is evaluated on one pre-release system.
Field Significance
Moderate — This paper fills a concrete reproducibility gap in the voice impression control sub-field by releasing the first public VI corpus. The training methodology improvements are principled and well-validated, providing evidence that impression leakage in reference-conditioned controllable TTS can be substantially reduced through data-decoupling strategies. The negative finding regarding LLM-TTS and fine-grained numerical control is an empirically useful data point for practitioners considering natural language versus numerical conditioning interfaces.
Claims
-
supports: Impression leakage in controllable TTS arises structurally when a single reference utterance is used for both speaker identity and style conditioning during training.
Evidence: VIC-base achieves ∆V = 0.22 (significantly different from zero), confirming that the reference audio biases synthesized VI toward the reference’s inherent voice impression even when GRL and high-rate dropout are applied. (§2, §5.2, Table 2)
-
supports: Decoupled training using distinct utterances for speaker and style conditioning reduces style leakage without changing the underlying TTS architecture.
Evidence: VIC-dis uses a separate utterance from the same speaker for reference conditioning during training, reducing ∆V from 0.22 to 0.14 with a statistically significant improvement in RVI-MSE (0.61 to 0.51). (§4.1, §5.2, Table 2)
-
supports: Eliminating the speaker reference and conditioning solely on a style vector achieves the strongest leakage reduction, at a moderate cost in speaker fidelity.
Evidence: VIC-srf reduces ∆V to 0.05 (not significantly different from zero), while SECS drops to 0.72, which remains above the cross-speaker bound of 0.63 but below the same-speaker bound of 0.81. (§4.2, §5.2, Table 2)
-
complicates: LLM-based TTS systems conditioned on natural language prompts are inadequate for fine-grained numerical voice impression control.
Evidence: Qwen3-TTS (zero-shot) achieves VI-MSE of 0.82 vs. 0.39 for the VITS-based baseline; fine-tuning worsens controllability further (VI-MSE 0.87), and text-VI entanglement is confirmed empirically by punctuation biasing predicted impressions. (§5.1, §5.2, Table 2, Table 3)
-
supports: Perceptual voice impression annotation can achieve inter-annotator agreement comparable to other subjective speech tasks, supporting the validity of VI corpora.
Evidence: Krippendorff’s alpha averages 0.470 across 10 VI dimensions, exceeding reported agreement for speech emotion recognition (0.442) and singing voice preference (0.153). (§3, Table 1)
Limitations and Open Questions
Warning
The LibriTTS-VI corpus contains only 130 manually annotated utterances from 130 speakers; the remaining LibriTTS-R annotations are estimated by the VIE trained with data augmentation. Several VI dimensions fall below the reliable inter-annotator agreement threshold (alpha below 0.667), particularly J) Cold-Warm (0.197) and D) Calm-Restless (0.295), limiting annotation reliability for these dimensions.
Evaluation is confined to the LibriTTS-R audiobook domain, which is relatively clean and controlled. Generalisation of both the VIE and the VIC systems to spontaneous or noisy speech is untested. The subjective MOS evaluation uses only two speakers and the authors note potential interface bias due to rating-scale anchoring. The comparison with Qwen3-TTS relies on a single external system evaluated in a zero-shot configuration for which the model was not explicitly designed.
Wiki Connections
- Disentanglement — this paper contributes evidence that training-time decoupling of speaker identity and style conditioning (via separate reference utterances) reduces style leakage without requiring a disentanglement-specific architecture.
- Prosody Control — voice impression control spans pitch, speaking rate, and expressiveness dimensions, providing a concrete 11-dimensional numerical conditioning interface for style attributes beyond standard prosody predictors.
- Evaluation Metrics — introduces VI-MSE and RVI-MSE as domain-specific controllability metrics, and ∆V as a leakage quantification measure for reference-conditioned style control systems.
- Subjective Evaluation — the paper conducts listener tests for both single-VI and multi-VI modulation conditions with 30 native-level English raters and four in-house expert annotators, providing subjective validation of controllability alongside objective metrics.
- Self-Supervised Speech — HuBERT forms the core of both the speaker reference encoder and the VI estimator; WavLM embeddings are also used for acoustic similarity in the corpus annotation augmentation strategy.
- Emotion Synthesis — voice impression control is an adjacent paradigm to emotion synthesis, targeting perceptual voice characteristics (brightness, calmness, masculinity) rather than discrete emotion categories, with the same annotation and conditioning challenges.