Interspeech · 2025 · Conference
Frederik Rautenberg et al. · → Paper · Demo: ? · Code: ?
Shows that a TTS system manipulating speaker embeddings through a global normalizing-flow-based attribute block can correctly and selectively control creaky voice, a non-persistent, positionally localized voice quality, without a frame-wise creak predictor.
Problem
Controlling specific perceptual voice qualities (PVQs) in synthesized speech is useful for applications such as phonetics training, where probes with and without a target quality illustrate concepts that are otherwise hard to convey. Creaky voice is a natural test case: it is a phonation type with real communicative and sociolinguistic functions, but unlike persistent voice qualities it is non-persistent and localized (often occurring utterance-finally), which makes it harder to control. Prior systems that manipulate creak condition on word-level or frame-wise creak probabilities, with the most recent approach relying on frame-wise probabilities extracted by an automatic classifier. Frame-wise creak estimation is comparatively unreliable, and prior work has shown that global creak probability estimates agree with human annotations more consistently than frame-wise estimates. This raises the question of whether a coarser, global conditioning signal, without any localized annotation, is sufficient to control a voice quality that itself is localized and intermittent.
Method
The system adapts YourTTS, itself an extension of VITS, which is trained to maximize an ELBO over a latent embedding conditioned on text and a speaker embedding, using a normalizing-flow-enhanced prior encoder and a HiFi-GAN decoder. Three changes are made to the base YourTTS design: the decoder’s direct conditioning on the speaker embedding is removed so that the speaker embedding’s influence is isolated to a single point in the model; the original speaker encoder is replaced with a d-vector model; and a speaker manipulation block is inserted between the speaker encoder and the rest of the model. The duration predictor remains conditioned on the unmanipulated speaker embedding, so speaking rate is unaffected by attribute manipulation.
The manipulation block applies a Conditional Continuous Normalizing Flow (CCNF), following an approach the same authors previously used for persistent, global perceptual voice qualities, here applied to a non-persistent, positionally localized quality. The CCNF is trained independently of the TTS model to make the speaker embedding normally distributed after transformation, conditioned on a scalar attribute (global creak probability, extracted with CreaPy plus an added energy-based voice-activity step to suppress noise in silent segments). Training maximizes a log-likelihood that requires solving two ODE problems (via a black-box ODE solver, with Hutchinson’s trace estimator for the trace term); the flow itself is a single CCNF block with hidden size 512. At inference, the original speaker embedding and its attribute are extracted from a reference utterance, transformed to a latent variable by solving one ODE, and then re-integrated with the attribute value shifted by a manipulation offset to obtain a modified speaker embedding, which replaces the original as TTS input.
The system is trained on LibriTTS-R (585 hours, 2,456 speakers), using the train-test split from the corpus’s original release, which supports zero-shot synthesis and manipulation for speakers unseen during training.
Key Results
A temporal analysis checks whether the global manipulation lands on the correct part of the signal: mapping phonemes to voiced, unvoiced, and silence classes and measuring the mean absolute difference between manipulated and unmanipulated latent embeddings shows that voiced segments are far more affected than unvoiced ones at every manipulation strength, for both seen and unseen speakers (e.g., at the largest manipulation strength, mean difference ×10² of 22-23 for voiced vs. 7-8 for unvoiced) (§3.1, Table 1). Since creak is a phonation type limited to voiced signal, this is consistent with (though does not directly prove) the manipulation correctly targeting creak.
A subjective listening test with 12 phonetics experts (384 ratings, 65 unique speakers) rated perceived creak (0-100 scale) and MOS (1-5) for suppressed, unmanipulated, and amplified conditions, comparing the proposed system (evaluated on LibriTTS-R speakers) against CreakVC, a voice-conversion baseline that relies on frame-wise creak probabilities (evaluated on VCTK speakers, its fine-tuning domain). Both systems produced statistically significant, monotonic creak-rating differences across suppressed/unmanipulated/amplified conditions (Bonferroni-corrected Wilcoxon, mostly p < 0.001). The proposed system’s unmanipulated synthesis scored MOS 3.8 ± 1.2 versus 4.2 ± 1.0 for the original recording, and no significant MOS or creak difference was found between unmanipulated synthesis and the natural recording. Across manipulation strengths, the proposed system tracked CreakVC closely (e.g., amplified creak rating 74.2 ± 16.9 vs. 85.3 ± 12.5 for CreakVC, with proposed MOS holding up better under amplification: 3.8 ± 1.3 vs. 3.3 ± 1.1) (§3.2, Table 2).
An objective acoustic analysis correlates the manipulation strength with established acoustic correlates of creak (pitch, Harmonic-to-Noise Ratio, and H1-H2 amplitude difference), extracted with Praat and CreaPy. Pearson correlations between the manipulation strength and creak probability were strong and positive (r = 0.81-0.82 for seen/unseen speakers), with negative correlations for pitch (r = -0.78 to -0.80) and HNR (r = -0.90 to -0.91), and a weaker negative correlation for H1-H2 (r = -0.59 to -0.67), consistent with the established acoustic-phonetic profile of creaky voice (§3.3, Table 3).
Novelty Assessment
The contribution is primarily an engineering application and validation, not a new architecture. The underlying global speaker-attribute manipulation mechanism (CCNF-based) is carried over from the same authors’ earlier work on persistent perceptual voice qualities; the novel step here is applying it to a non-persistent, positionally localized voice quality and testing whether a purely global, non-local conditioning signal is sufficient in that harder setting. The temporal (embedding-space) localization analysis, which checks that manipulation effects concentrate in voiced phoneme segments despite the model never seeing frame-level labels, is a genuine and useful validation methodology for this class of system. The comparison against a frame-wise-conditioned voice-conversion baseline (CreakVC) provides direct, if narrow, evidence that global conditioning is a viable alternative to local conditioning for at least one localized voice quality.
Field Significance
moderate — This paper provides a focused case study showing that global, utterance-level attribute conditioning can control a non-persistent, positionally localized voice quality without a frame-wise predictor, using both a temporal localization analysis and a listening test against a frame-wise-conditioned baseline. Its scope is narrow: a single voice quality (creak), a small expert-listener panel, and a training corpus limited to read speech, so the paper itself does not demonstrate generalization beyond this case.
Claims
- supports: A single global (utterance-level) conditioning attribute can drive a synthesis system to apply a targeted modification in the correct, phonetically-restricted regions of an utterance, even when the attribute itself carries no positional information.
Evidence: Embedding-space MAE between manipulated and unmanipulated representations grows with manipulation strength and is consistently larger for voiced phonemes than unvoiced phonemes, for both seen and unseen speakers (§3.1, Table 1).
- refines: Frame-wise or word-level conditioning signals are not strictly necessary for controlling voice qualities that are themselves localized and intermittent; a coarser global attribute can produce comparable perceptual control.
Evidence: The proposed global-conditioning system tracks the perceptual creak and MOS trends of CreakVC, a baseline that relies on frame-wise creak probabilities, across suppressed, unmanipulated, and amplified conditions in the listening test (§3.2, Table 2).
- supports: Manipulating a learned global attribute dimension co-varies with the established acoustic correlates of the target voice quality, indicating the manipulation captures genuine phonetic structure rather than an arbitrary embedding-space direction.
Evidence: Pearson correlations between manipulation strength and independently measured pitch, HNR, and H1-H2 are strong and in the direction predicted by the acoustic-phonetic definition of creaky voice (§3.3, Table 3).
- complicates: Comparisons between a global-conditioning system and a local-conditioning baseline for voice-quality control can be confounded by unequal cross-corpus transfer, since the systems may not perform comparably on the same evaluation corpus.
Evidence: Both systems were fine-tuned on VCTK, but the proposed system showed reduced performance there, so the listening test evaluated the proposed system on LibriTTS-R speakers and CreakVC on VCTK speakers rather than on a shared test set (§3.2).
Limitations and Open Questions
The core listening-test comparison evaluates the proposed system and the CreakVC baseline on different speaker sets from different corpora (LibriTTS-R vs. VCTK) because the proposed system underperformed on VCTK. This weakens the strength of the head-to-head comparison between global and local conditioning approaches.
The study is restricted to a single voice quality (creaky voice) and a single read-speech training corpus (LibriTTS-R), which contains no spontaneous speech or dialogue; the paper explicitly sets aside creak’s conversational and sociolinguistic functions and focuses only on its prototypical acoustic form. The subjective evaluation used 12 expert phonetician listeners rather than a larger or non-expert pool, reflecting the specialized nature of creak perception but limiting statistical power. The paper’s closing claim that the approach generalizes to “a wide range of perceptual voice qualities with little or no adjustment” is stated as a conclusion but is not itself tested on any voice quality other than creak in this paper.
Wiki Connections
- Zero-Shot TTS — the manipulation system is built to generalize to speakers unseen during training, extracting and manipulating d-vector embeddings without speaker-specific fine-tuning.
- GAN Vocoder — the TTS backbone uses a HiFi-GAN decoder to synthesize waveforms from the (possibly manipulated) latent embedding.
- Voice Conversion — the paper directly benchmarks its TTS-based, globally-conditioned approach against CreakVC, a voice-conversion system that relies on frame-wise creak conditioning, as its primary comparison point.
- Subjective Evaluation — validation relies on a listening test with 12 expert raters producing MOS and creak-strength ratings across manipulation conditions.
- HiFi-GAN — used as the decoder in the adapted YourTTS/VITS backbone to synthesize the final waveform from latent embeddings.