arXiv · 2026 · Preprint
Kirill Borodin et al. (MTUCI, Moscow, Russia) · → Paper · Demo: ? · Code: ?
Trains a speaker-conditioned prosody text encoder through a multi-stage curriculum of masked language modelling and SigLIP-style cross-modal contrastive learning, then shows that an additional same-phoneme contrastive refinement stage improves intrinsic prosodic retrieval yet degrades downstream diffusion-TTS synthesis quality.
Problem
Prosody cannot be uniquely recovered from text alone: the same phoneme sequence can be realised with markedly different intonation depending on discourse context, speaker identity, and intent. Standard TTS text encoders capture only linguistic content, leaving the model to infer all suprasegmental variation from paired data alone, which tends to produce oversmoothed intonation. Two families of pretraining strategies have been explored to enrich text representations with prosodic knowledge before end-to-end synthesis training: masked reconstruction (e.g. BERT-style MLM on phonemes) and contrastive text-audio alignment (e.g. CLAPSpeech). The paper identifies two open gaps: these two directions have only been studied in isolation, with no evidence on whether or how they should be combined; and CLAPSpeech’s same-token contrastive training never exposes the encoder to diverse phoneme types, leaving open whether a preceding mixed-phoneme contrastive stage would build a stronger phoneme-discriminative foundation before same-token prosodic refinement.
Method
The core contribution is a speaker-conditioned dual-stream text encoder that jointly processes stressed phoneme sequences and BPE text tokens. Each stream passes through four transformer blocks with relative positional encoding and 1D convolutional projections. Speaker conditioning is injected at every block via AdaLN-Zero, using a SimAM-ResNet34 speaker embedding from the WeSpeaker framework (pretrained on VoxBlink2); at inference this embedding is obtained by averaging representations from a reference utterance distinct from the synthesis input, giving a zero-shot speaker conditioning signal. Word-level pooling and word-to-phoneme expansion align the BPE stream to phoneme resolution before the two streams are summed and passed through four more transformer blocks, layer normalisation, and a convolutional projection to produce per-phoneme prosodic embeddings.
Training follows a three-stage curriculum. Stage 1 pretrains the phoneme and BPE encoders independently with a BERT-style MLM objective (mask probability 0.15) on a Russian dialogue corpus, instilling syntactic and semantic structure. Stage 2 jointly trains the full dual-stream encoder with an ECAPA-TDNN acoustic branch using a SigLIP-style pairwise sigmoid contrastive loss over mixed-phoneme batches (each utterance in a batch may carry a different phoneme type), teaching the encoder to discriminate among phoneme categories while absorbing acoustic information. Stage 3, studied as an optional ablation rather than part of the core pipeline, applies the same SigLIP objective but restricts batches to utterances sharing one target phoneme type, mirroring CLAPSpeech’s same-token contrastive regime and gradients computed only at the shared phoneme position.
The resulting encoder is evaluated as a drop-in text/prosody front end for two downstream diffusion acoustic models: Grad-TTS, used for fast controlled ablations across curriculum variants, and a DiTTo-TTS-style latent-diffusion transformer operating on a deep-spectrogram latent space, used to test whether encoder-level conclusions transfer to a larger, more expensive system. All experiments use the Yandex Podcasts split of the Balalaika Russian TTS corpus, chosen for its context-dependent lexical stress, homographs, and rich morphology.
Key Results
Across curriculum variants, the two-stage MLM + mixed-phoneme contrastive curriculum (Stage 1+2) is the strongest configuration on every downstream metric measured in Grad-TTS: highest MOS (1.980 ± 0.172) and IntMOS (2.185 ± 0.255), lowest WER (0.176) and CER (0.067), highest speaker similarity (SIM-o = 0.862), and the only variant with a positive spoofing-robustness score (FKE = 3.30). Standalone Stage 2 drives the strongest intrinsic phoneme discrimination (R@1-diff = 0.933) and best NISQA sub-scores, while Stage 1 adds training stability.
Adding the same-phoneme Stage 3 refinement on top of Stage 1+2 improves prosodic retrieval (R@1-sim rises from 0.746 to 0.770) but the full three-stage curriculum (1+2+3) produces the worst downstream synthesis of any configuration tested: MOS drops to 1.540 and WER rises to 0.429, despite near-best intrinsic retrieval scores. Architectural ablations within the Stage 2 setting show that a SigLIP pairwise-sigmoid loss outperforms a CLIP-style symmetric-softmax loss (R@1-sim 0.690 vs. 0.650) and that removing AdaLN speaker conditioning degrades both discrimination and prosodic sensitivity substantially (R@1-diff 0.871 vs. 0.933; R@1-sim 0.529 vs. 0.690).
In the full-scale comparison, the best encoder (Stage 1+2) integrated into DiTTo-TTS achieves the strongest NISQA noise/coloration scores (NOI = 4.24, COL = 4.23) and competitive UTMOS (2.81) and WER (0.125) among open-source baselines evaluated on the same Russian test set, but trails F5-TTS on subjective MOS (2.521 vs. 2.938) and speaker similarity (0.838 vs. 0.935). The paper notes that all baselines were run with publicly released checkpoints without retraining on the Balalaika corpus, so differences in training data scale and language coverage confound this comparison.
Novelty Assessment
The contribution is primarily architectural and empirical rather than a wholly new modelling paradigm: the individual ingredients (MLM pretraining, SigLIP-style contrastive alignment, AdaLN-Zero speaker conditioning, ECAPA-TDNN acoustic embeddings) are all established techniques from prior TTS and speaker-verification literature. The novel elements are (1) combining MLM and cross-modal contrastive pretraining into a single curriculum for a prosody encoder, where prior work studied each in isolation, and (2) inserting an explicit speaker-conditioning path and a mixed-phoneme contrastive stage before any same-token refinement, directly testing an open question about CLAPSpeech-style training. The most valuable finding is a genuine negative result: the added same-phoneme contrastive stage, despite improving intrinsic prosodic retrieval, actively harms downstream synthesis through what the paper characterises as catastrophic forgetting of phoneme-discriminative features. This decouples “better retrieval metric” from “better generative conditioning” in a controlled, ablated setting, which is a useful and honestly reported caution against optimising representation-learning objectives in isolation from the downstream generative task.
Field Significance
moderate — this is a single-institution ablation study evaluated on one language (Russian) and one dataset, with the strongest evidence being a negative result about curriculum design rather than a new state-of-the-art system. Its value lies in demonstrating, with a controlled ablation, that intrinsic embedding-space retrieval metrics for prosody encoders can diverge from and even inversely predict downstream generative synthesis quality, and in identifying a training-order interaction (mixed-phoneme discrimination before same-token refinement) that other prosody-pretraining work can build on or avoid.
Claims
- supports: Combining self-supervised masked-language pretraining with cross-modal contrastive alignment produces a more effective prosody-conditioning representation for TTS than either objective applied alone.
Evidence: The Stage 1+2 (MLM + mixed-phoneme SigLIP contrastive) curriculum achieves the best MOS, IntMOS, WER, CER, and speaker similarity of all six curriculum variants tested in Grad-TTS. (§4.2, Tables 1–2)
- complicates: Intrinsic embedding-space retrieval metrics for a prosody encoder do not reliably predict downstream generative synthesis quality, and can move in the opposite direction.
Evidence: The Stage 1+2+3 curriculum attains near-best prosodic retrieval (R@1-sim = 0.770) but the worst MOS (1.540) and highest WER (0.429) of any configuration. (§4.2, Tables 1–3)
- complicates: Same-token contrastive refinement designed to isolate prosodic variation from phoneme identity can induce catastrophic forgetting of phoneme-discriminative representations that were established in a prior training stage, degrading intelligibility.
Evidence: Adding Stage 3 to Stage 1+2 drops R@1-diff from 0.926 to 0.893 while WER rises from 0.176 to 0.429, attributed to gradients computed only at a single shared-phoneme position over 10^4 steps. (§4.3)
- supports: Explicit speaker conditioning in a text-audio contrastive prosody encoder prevents the contrastive objective from collapsing speaker-dependent prosodic variation into the learned representation.
Evidence: Removing AdaLN-Zero speaker conditioning from the Stage 2 encoder drops R@1-sim from 0.690 to 0.529 and R@1-diff from 0.933 to 0.871. (§4.1, Table 3)
- refines: A pairwise sigmoid (SigLIP-style) contrastive loss captures cross-modal prosodic variation better than a symmetric softmax (CLIP-style) contrastive loss for text-audio prosody alignment.
Evidence: Replacing SigLIP with a CLIP-style symmetric-softmax loss under otherwise identical Stage 2 conditions reduces R@1-sim from 0.690 to 0.650 and marginally lowers R@1-diff from 0.933 to 0.926. (§4.1, Table 3)
Limitations and Open Questions
The comparison against contemporary open-source TTS systems (F5-TTS, XTTS, Tortoise) in the full-scale evaluation is not a controlled comparison: those baselines were run with their public checkpoints as-is, without retraining on the Balalaika corpus, so differences in training data scale, language coverage, and model capacity confound the results the paper itself reports in this table.
All experiments evaluate a single language (Russian) and a single dataset (Balalaika), so it is unclear whether the curriculum-order findings, especially the Stage 3 catastrophic-forgetting result, generalise to other languages or corpora. The controlled curriculum ablations (Tables 1–3) use only the lightweight Grad-TTS backbone; the larger DiTTo-TTS system is trained with only the single best-performing encoder configuration, so the curriculum comparison itself is not repeated at full scale. Neither code nor a public demo is reported as released alongside the paper.
Wiki Connections
- Diffusion TTS — integrates the proposed prosody encoder into Grad-TTS and a DiTTo-TTS-style latent-diffusion transformer as the downstream backbones used to evaluate prosodic conditioning quality.
- Zero-Shot TTS — conditions synthesis on an averaged reference-utterance speaker embedding at inference time and benchmarks the resulting system against zero-shot baselines F5-TTS, XTTS, and Tortoise.
- Disentanglement — designs a training curriculum that contrasts mixed-phoneme against same-phoneme batches specifically to separate phoneme-discriminative representation from prosodic variation, with paired retrieval metrics (R@k-diff, R@k-sim) that isolate each axis and an ablation showing they trade off against each other.
- Subjective Evaluation — collects MOS and intonation-MOS ratings from at least seven native Russian raters per clip across all curriculum variants and the full downstream TTS comparison.
- F5-TTS — used as a zero-shot flow-matching TTS baseline in the full-scale comparison, outperforming the paper’s DiTTo-TTS system on subjective MOS and speaker similarity despite the latter’s stronger spectral-fidelity (NISQA) scores.