arXiv · 2025 · Preprint
Sigurgeirsson and King · → Paper · Demo: ✓ · Code: ?
RepeaTTS proposes iterative re-labelling and fine-tuning of a prompt-based TTS model, using PCA of Wav2Vec2 embeddings from thousands of generated samples to discover latent controllable features not present in the original training annotations.
Problem
Prompt-based TTS models such as ParlerTTS let users control acoustic features through natural language description prompts, but are constrained to whichever features were explicitly annotated during training. At the same time, these models generate substantial uncontrolled variation for the same inputs, producing a spectrum of prosodic renditions the user cannot direct. Prior work addresses controllability either through manual feature annotation (time-consuming and requiring domain expertise) or through reference-based conditioning (which fails to generalise across text and speaker conditions). No method has exploited the model’s own output variance as a signal for discovering new features worth adding to the controllable set.
Method
RepeaTTS operates as an iterative three-phase loop applied to a pre-trained prompt-based TTS model. The starting point is a multilingual ParlerTTS checkpoint (an encoder-decoder architecture predicting DAC tokens, with a frozen T5 encoder for description prompts) fine-tuned on the Icelandic Talromur-3 emotive speech corpus. Two baseline models are prepared: T3-emotion, trained with explicit emotion category and intensity labels in its description prompts, and T3, trained without any emotion information.
In the analysis phase, the model generates 1,000 or more utterances with fixed inputs (identical target text, target speaker, and description prompt), keeping all conditioning constant so that observed variation reflects only the model’s internal uncertainty. Wav2Vec2 summary embeddings (mean of the 4th layer, using a 300M-parameter multilingual model) are extracted for each generated utterance. PCA projects these embeddings into three dimensions. The authors inspect the PCA projection manually: distinct clusters indicate discrete features (handled by K-means), while continuous gradients indicate continuous features (handled by binning along the principal axis).
Cluster or bin means serve as reference embeddings. Training utterances are re-labelled by minimum cosine distance to these means, and new natural-language labels describing each cluster’s acoustic character are appended to the corresponding description prompts. The model then undergoes a further fine-tuning pass on the re-labelled corpus. The procedure repeats, with each round targeting variance not yet explained by previously enrolled features.
Key Results
For T3 (no emotion labels), PCA of the initial analysis set shows that the first two principal components account for 42.7% of total variance (total embedding variance 0.513). Manual inspection identifies emotional intensity as the primary variance axis, which is discretised into three bins (low, medium, high intensity). After re-labelling and fine-tuning, a second analysis round reveals a binary neutral/emotive split. The cluster assignment confusion between neutral and high-intensity emotion drops substantially across rounds: neutral utterances assigned to the neutral cluster improve from 59.3% to 89.3% (Tables 2 and 3). Mean total embedding variance decreases from 0.513 after the first round to 0.353 after the third, indicating that enrolled features account for progressively more of the initial variation.
For T3-emotion, the neutral-emotion fixed-input analysis set produces highly consistent F0 contours across all 1,000 samples (Figure 6), with too little variance to support useful feature discovery. Attempting a more diverse analysis set instead reveals a principal component correlating with GeMaps-v01b loudness features tied to recording-condition variation in the Talromur-3 corpus, not to prosody. The authors stop enrolment for this model after the first iteration.
Baseline quality for T3-emotion (Table 1, all speakers): synthetic speaker similarity matches real speech (0.74 vs. 0.74), but synthetic WER is higher (0.13 vs. 0.04) and output diversity is substantially greater (0.21 vs. 0.13 diversity score).
Novelty Assessment
The core idea is genuinely new: exploiting a TTS model’s own uncontrolled output variation as a signal for discovering new controllable features, rather than pre-specifying features for manual annotation or relying on reference conditioning. Every component is existing work (ParlerTTS, Wav2Vec2, PCA, K-means, repeated fine-tuning), making this an engineering-integration contribution rather than an architectural one. The method is framed as a proof-of-concept: it succeeds for one of two tested models, evaluation is limited to a single speaker (Ingrid) in one Icelandic corpus, and quality is measured through ASR-based WER and speaker similarity rather than listening tests. The paper does not resolve why the method fails for T3-emotion or propose a way to avoid capturing recording-condition artefacts instead of prosodic features.
Field Significance
Low — RepeaTTS introduces an interesting methodology for feature discovery in prompt-based TTS, but results are preliminary: the approach works only for a model trained without emotion labels, evaluation is single-speaker and single-language, and no perceptual quality assessment is reported. The paper contributes a concrete workflow demonstrating that model variance can be mined for new control labels, but the boundary conditions under which this approach is reliable remain unresolved.
Claims
-
supports: Latent controllable features absent from a TTS model’s training annotations can be discovered by applying PCA to embeddings of fixed-input generated samples and enrolling the identified dimensions as new description-prompt labels.
Evidence: For T3 (no emotion labels), PCA of 1,000 Wav2Vec2 embeddings revealed emotional intensity as the primary variance axis; iterative re-labelling and fine-tuning progressively separated neutral from emotive utterances, improving neutral cluster assignment from 59.3% to 89.3% across rounds. (§4.3, Tables 2–3)
-
complicates: Variance-based feature discovery in prompt-based TTS is unreliable when the model’s output distribution is highly constrained by existing conditioning labels.
Evidence: T3-emotion, already trained with explicit emotion and intensity labels, generated highly consistent F0 contours under neutral-emotion prompts (Figure 6), leaving insufficient variance for meaningful feature discovery in the fixed-input analysis set. (§4.4, Figure 6)
-
complicates: Embedding-based clustering of TTS output variance can surface acoustic artefacts of the training corpus (such as recording-condition variation) rather than prosodic features of interest.
Evidence: For T3-emotion with a diverse analysis set, the principal component correlated strongly with GeMaps-v01b loudness features attributable to microphone distance differences in the Talromur-3 corpus, not to prosody. (§4.4, Figure 8)
-
complicates: Self-supervised speech representations suitable for prosody analysis are inherently entangled with speaker identity and linguistic content, requiring fixed-input generation sets to isolate prosodic variation.
Evidence: Wav2Vec2 summary embeddings cluster by target text and speaker identity regardless of which network layer is used (Figure 1), motivating the fixed-input analysis design where text, speaker, and prompt are held constant. (§3.2, Figure 1)
Limitations and Open Questions
Warning
No subjective listening tests are reported. All quality and controllability assessments use automatic metrics (ASR-based WER, speaker embedding cosine similarity, diversity score). Whether the discovered features correspond to perceptually meaningful and user-controllable dimensions is not established.
The method is evaluated on a single Icelandic speaker (Ingrid) for the fine-tuning stages, with no cross-speaker or cross-language generalisation experiments. The paper does not address why T3-emotion fails and T3 succeeds beyond noting the former’s consistent output distribution; whether adding a more diverse analysis set would help, or whether an alternative embedding choice (not Wav2Vec2) would be less sensitive to recording conditions, remains open. The use of Icelandic as the sole test language, while noted as language-agnostic in principle, is unverified on any other language.
Wiki Connections
- Instruction-Conditioned TTS — ParlerTTS, the base model, accepts natural language description prompts as conditioning; this paper proposes a method for extending the set of features those prompts can control.
- Prosody Control — the paper’s stated goal is discovering and enrolling new prosodic control dimensions (emotional intensity, neutral vs. emotive rendering) that were absent from the model’s original training labels.
- Emotion Synthesis — emotional intensity and the neutral/emotive distinction are the primary features discovered and enrolled through the iterative fine-tuning procedure.
- Self-Supervised Speech — Wav2Vec2 embeddings (4th layer, 300M multilingual model) serve as the core utterance representation for PCA-based variance analysis and cosine-distance re-labelling throughout the feature discovery pipeline.