arXiv · 2026 · Preprint
Frederik Rautenberg et al. (Paderborn University / Bielefeld University) · → Paper · Demo: ? · Code: ?
Shows that a spurious pitch-creak correlation in training data undermines speaker identity preservation during voice-quality manipulation, and fixes it with a pitch-shift data augmentation that decorrelates the two attributes.
Problem
Perceptual voice quality (PVQ) manipulation systems built on conditional continuous normalizing flows (CCNFs) can alter a target voice-quality attribute, such as creaky voice, in synthesized speech by shifting a speaker embedding along that attribute’s conditioning dimension. Prior work in this line trained the CCNF on pseudo-labeled data (LibriTTS-R labeled with a creak detector) where each speaker’s utterances carry a roughly constant creak probability. Because creaky voice is population-level correlated with lower fundamental frequency, this training data encodes an “inter-speaker” creak-to-pitch correlation: speakers with higher creak probability tend to have lower pitch. A model trained on this data learns to shift pitch whenever it is asked to shift creak, so increasing a given speaker’s creak probability at inference also drags their pitch toward the population trend, which distorts their perceived speaker identity. No dataset exists in which the same speaker produces the same content at multiple creak levels, so the “true” intra-speaker relationship between pitch and creak is unknown and cannot simply be learned from more data of the same kind.
Method
The system builds directly on a published pipeline (YourTTS as the TTS backbone, trained with an ELBO-style objective, augmented with a CCNF-based speaker-embedding manipulation block conditioned on a six-dimensional PVQ attribute vector: breathiness, roughness, resonance, weight, mean pitch, and creak probability). The CCNF is trained separately from the TTS model by maximizing an ODE-based log-likelihood objective; at inference, a speaker embedding is mapped to a latent code, the attribute vector is edited (here, only the creak dimension is shifted by a factor beta in [-1.25, 1.25]), and the inverse flow produces a new speaker embedding that is fed to the TTS model for synthesis.
The paper’s contribution is a training-data intervention that targets the inter-speaker correlation directly, rather than a new architecture. Utterances in LibriTTS-R are relabeled with creak probability (CreaPy) and mean pitch, and each utterance’s pitch contour is shifted with TD-PSOLA toward the speaker’s gender-mean pitch (with added Gaussian noise for variability), which removes most of the within-gender correlation between an utterance’s pitch and its creak probability (Pearson R drops from -0.7 to -0.0 for male speakers and from -0.4 to 0.0 for female speakers; §3, Table 1). Three CCNF variants are then trained and compared: a base-flow on the unmodified dataset, an adapted-flow on the pitch-augmented dataset, and a combined-flow on both.
Key Results
Across creak-manipulation strengths, the adapted-flow and combined-flow models achieve substantially more stable speaker verification (lower equal error rate degradation) than the base-flow model, which shows much larger EER increases as the creak conditioning is pushed further from the original value (§4.2, Figure 3). Acoustic correlates of creak (pitch, H1-H2, harmonic-to-noise ratio, cepstral peak prominence) remain appropriately linked to the creak conditioning after adaptation, though the pitch-creak slope shrinks roughly 18-fold relative to the base-flow (from -47.04 to -2.63 Hz per unit creak shift) and the creak-conditioning effect itself weakens somewhat (slope drops from 0.25 to 0.14, partially recovered to 0.19 by the combined-flow; §4.1, Table 2). No baselines external to this line of work are compared; all three flow variants share the same TTS backbone and differ only in training data.
Novelty Assessment
The contribution is a data-centric fix rather than an architectural one: the TTS backbone, the CCNF manipulation block, and the six-dimensional PVQ conditioning scheme are all carried over from the authors’ own prior publications. The genuinely new element is the pitch-shift augmentation procedure that breaks a specific spurious correlation identified in the training corpus, together with the systematic comparison of three training-data configurations that isolates the augmentation’s effect. This is an incremental, well-scoped follow-up that diagnoses and repairs a specific failure mode in an existing system rather than proposing a new capability.
Field Significance
low — This paper documents a narrow but concrete failure mode (spurious inter-speaker attribute correlations degrading identity preservation during voice-quality editing) and demonstrates a straightforward data augmentation fix within one specific pipeline. It provides a useful cautionary data point for anyone training attribute-conditioned voice manipulation models on pseudo-labeled, speaker-consistent corpora, but the evaluation is limited to one architecture, one voice quality, and objective speaker-verification metrics.
Claims
- supports: Training an attribute-conditioned speaker-embedding manipulation model on data where a target attribute is confounded with another correlated attribute at the population level causes the model to entangle the two, degrading speaker identity preservation when only the target attribute is edited at inference.
Evidence: The base-flow, trained on unmodified LibriTTS-R with pseudo-labeled creak probabilities that are negatively correlated with pitch, shows much larger equal-error-rate increases under creak manipulation than the pitch-decorrelated variants. (§4.2, Figure 3)
- supports: A targeted training-data augmentation that explicitly removes a known spurious correlation between two attributes can disentangle them for a downstream conditional generative model without changing its architecture.
Evidence: Shifting each utterance’s pitch contour toward the speaker’s gender-mean via TD-PSOLA reduces the within-gender pitch-creak Pearson correlation from -0.7 (male) / -0.4 (female) to near zero, and the resulting adapted-flow and combined-flow models preserve speaker identity far better than the base-flow across all tested manipulation strengths. (§3, Table 1; §4.2, Figure 3)
- complicates: Removing a spurious inter-attribute correlation in training data does not eliminate all correlation, because some residual correlation can reflect a genuine population-level difference rather than a training artifact.
Evidence: After pitch augmentation, the within-gender pitch-creak correlation is nearly removed, but an overall correlation of R = -0.6 remains, attributed to female speakers generally exhibiting lower creak probability than male speakers regardless of pitch. (§3, Table 1)
- complicates: Decorrelating a conditioning attribute from a confounding attribute can partially weaken the conditioning model’s sensitivity to the target attribute itself, trading off manipulation strength against identity preservation.
Evidence: The creak-conditioning slope drops from 0.25 (base-flow) to 0.14 (adapted-flow), only partially recovered to 0.19 by training on both the original and augmented data (combined-flow). (§4.1, Table 2)
Limitations and Open Questions
The evaluation uses only objective speaker verification (EER) and acoustic correlate slopes; no listening tests are reported in this paper to confirm that the perceived creak manipulation itself remains as convincing as in the prior work this system builds on. The true “intra-speaker” pitch-creak relationship is acknowledged as unknown, since no dataset exists where the same speaker varies creak at multiple levels; the pitch-shift augmentation is a heuristic proxy (shifting toward the gender mean) rather than a validated model of how an individual speaker’s pitch would actually change with creak. The approach is demonstrated for a single voice-quality dimension (creak) and a single TTS backbone (YourTTS), and generalization to other PVQ attributes or architectures is not tested.
Wiki Connections
- Disentanglement — introduces a training-data augmentation that decorrelates a target voice-quality attribute (creak) from a confounding attribute (pitch) so a shared conditioning mechanism can edit one without disturbing the other, validated through a three-way ablation across training-data configurations.
- Prosody Control — manipulates pitch-linked voice-quality conditioning independently of speaker identity via an explicit attribute-conditioned speaker-embedding transformation.
- Voice Conversion — extends a line of identity-preserving voice-quality editing work (following global speaker-attribute manipulation and creak-specific editing systems) though without a dedicated voice-conversion evaluation of its own.