arXiv · 2025 · Preprint
Haoyu Li et al. · → Paper · Demo: ✓ · Code: ?
Introduces Time-Layer Adaptive Speaker Alignment (TLA-SA), a training-time auxiliary loss that dynamically supervises intermediate flow-matching representations with a pre-trained speaker embedding, weighted jointly by denoising timestep and network depth, to improve speaker similarity in zero-shot TTS.
Problem
Flow-matching (FM) based zero-shot TTS systems synthesize high-quality speech and generalize well to unseen speakers, but the standard conditional flow-matching (CFM) training objective only implicitly models the data distribution: it does not explicitly enforce perceptual attributes such as speaker identity. As a result, speaker similarity in zero-shot generation remains sub-optimal unless the system adds extra supervision, typically a dedicated speaker-focused supervised fine-tuning (SFT) stage. Prior representation-alignment work outside and inside TTS aligns intermediate diffusion or FM features to external representations to speed convergence, but applies this alignment uniformly across the network, without asking whether speaker information is actually distributed evenly across denoising steps and layers. This paper first characterizes how speaker information is distributed inside an FM-based TTS model and then asks how alignment supervision should be shaped to match that distribution.
Method
The paper works within the standard FM zero-shot TTS formulation, where a network learns a vector field that transports Gaussian noise to a masked Mel-spectrogram target, conditioned on the unmasked context and a speaker embedding c (§2.1, Eq. 1-2). To probe how speaker information flows through such a model, the authors adopt Centered Kernel Nearest-Neighbor Alignment (CKNNA), a training-free representation-similarity criterion (borrowed from the “platonic representation hypothesis” line of work), and validate that it correlates strongly with measured speaker similarity across training checkpoints (§2.2, Figure 2). Applying CKNNA to a converged baseline FM checkpoint reveals that speaker information is concentrated in the earliest (highest-noise) denoising steps and in shallower-to-middle network layers, with the final layers carrying comparatively little speaker signal (§2.2, Figure 3).
Motivated by this non-uniformity, the paper proposes Time-Layer Adaptive Speaker Alignment (TLA-SA). For each of the N transformer blocks in the FM decoder, a layer-specific MLP adapter maps the block’s temporally pooled intermediate representation to the embedding space of a pre-trained, speaker-verification-fine-tuned WavLM encoder, and a cosine-similarity alignment loss is computed against that reference speaker embedding (§2.3, Eq. 3-4). To account for the observed dependence on denoising timestep, a separate time adapter conditions on the current timestep embedding and predicts a softmax weighting over the N per-layer alignment losses, so that supervision strength is redistributed dynamically across layers as a function of the denoising step (§2.3, Eq. 5). An entropy-penalty regularization term stabilizes the learned weights, and the resulting TLA-SA loss is added to the standard CFM loss with a fixed trade-off coefficient (λ = 0.5) (§2.3, Eq. 6).
TLA-SA is evaluated as a drop-in training-time addition on two distinct FM decoder architectures: an LM-based FM TTS system following CosyVoice 2 (a speech-text LM feeding a Transformer U-Net CFM decoder with 2 down-sampling, 12 middle, and 2 up-sampling layers, with TLA-SA applied to the middle layers), and an LM-free FM TTS system with a transformer text encoder and 9 MMDiT decoder layers in the style of F5-TTS. Models are trained from scratch at 24kHz with AdamW, a cosine learning-rate schedule, and a 0.25-epoch warm-up, at two data scales: a small-scale LibriTTS setup (585 hours) and an industrial-scale TLASA-100k corpus (100k hours, 60k Mandarin / 40k English, sourced from Emilia, WenetSpeech, and WenetSpeech4TTS with speaker clustering and de-duplication) (§3.1, §3.2). No codec is used; the FM decoder targets Mel spectrograms directly.
Key Results
On the industrial-scale TLASA-100k corpus, TLA-SA applied to a CosyVoice-2-style backbone yields average absolute gains of 3.0% in Sim-WavLM and 2.1% in Sim-ERes2Net over the same-data baseline, with comparable CER/WER, and also produces a 2.9x convergence speedup with a 2.2% Sim-WavLM gain at matched training steps (§4.1, Table 1, Figure 4). Without any speaker-specific SFT stage, the TLA-SA-enhanced 100k-hour model outperforms FireRedTTS (150k hours, with SFT), closely approaches FireRedTTS-2 (500k hours, with SFT), and narrows the gap to the official 150k-hour CosyVoice 2 (2x the data, with SFT) to a 2.1% average difference (§4.1). Subjective MOS evaluation by twelve speech experts shows TLA-SA raising speaker-similarity MOS from 4.16 to 4.23 on Seed-TTS test-zh and from 3.98 to 4.10 on test-en (§4.4, Table 4).
On the smaller LibriTTS setting, TLA-SA outperforms two ablated alignment baselines that use only temporal adaptivity (TRA) or only layer adaptivity (Speech Align), improving Sim-WavLM from 0.510 (no alignment) / 0.513 (TRA) / 0.524 (Speech Align) to 0.538, and Sim-ERes2Net from 0.571 / 0.573 / 0.583 to 0.596 (§4.2, Table 2). Structural generalization is tested by applying TLA-SA to both the LM-based and LM-free decoder architectures on LibriTTS: Sim-WavLM improves from 0.510 to 0.538 for the LM-based decoder and from 0.398 to 0.458 for the LM-free decoder, with corresponding Sim-ERes2Net gains from 0.571 to 0.596 and from 0.500 to 0.571 respectively (§4.3, Table 3).
Novelty Assessment
The contribution is primarily a new training objective rather than a new backbone architecture: TLA-SA is a lightweight alignment head and timestep-conditioned weighting mechanism added on top of existing FM TTS decoders. Its novelty rests on two pieces working together: an empirical characterization of how speaker information is unevenly distributed across denoising timesteps and network depth in FM-based TTS (an analysis not previously reported for this setting), and a mechanism that turns that observation into dynamic, jointly time- and layer-adaptive supervision, rather than the static, uniform alignment used in prior representation-alignment work. The ablations against time-only and layer-only alignment variants (Table 2) give direct evidence that the joint adaptivity, not just the presence of alignment supervision, drives the improvement. The demonstrated portability across an LM-based and an LM-free FM decoder is a genuine generalization result, though both are still flow-matching mel-spectrogram decoders from a narrow architectural family, so broader claims of architecture-agnosticism remain untested.
Field Significance
moderate — This paper provides a targeted, empirically grounded mechanism for a well-known weakness of flow-matching zero-shot TTS (implicit, unsupervised speaker modeling) and demonstrates that adaptive, dynamically weighted representation alignment outperforms static alignment strategies. It offers a diagnostic method (CKNNA-based layer/timestep speaker probing) that could be reused by others analyzing internal representations of FM speech models, and shows that auxiliary alignment supervision can substitute for a dedicated SFT stage at reduced data cost. Its scope is a specific training-time addition to existing FM decoders rather than a new generative paradigm.
Claims
- supports: Auxiliary alignment of intermediate flow-matching representations with a pre-trained speaker embedding improves speaker similarity in zero-shot TTS without requiring a dedicated supervised fine-tuning stage.
Evidence: At matched 100k-hour training data, the proposed alignment strategy yields average absolute gains of 3.0% in Sim-WavLM and 2.1% in Sim-ERes2Net over the same-data baseline with comparable CER/WER, and the SFT-free model outperforms a baseline trained on 150k hours with a dedicated speaker SFT stage. (§4.1, Table 1)
- supports: Speaker identity information in flow-matching-based speech generation is not uniformly distributed across denoising timesteps or network depth, concentrating in early (high-noise) denoising steps and in shallower-to-middle layers while weakening toward the final layers.
Evidence: CKNNA-based probing of a converged baseline FM checkpoint’s intermediate representations shows a clear, non-uniform pattern across both the timestep and layer axes, with the final layers carrying comparatively little speaker signal. (§2.2, Figure 3)
- supports: A training-time speaker-alignment auxiliary loss can generalize across structurally distinct flow-matching TTS decoder families, including both LM-conditioned and LM-free variants.
Evidence: On LibriTTS, the alignment strategy improves Sim-WavLM from 0.510 to 0.538 on an LM-based CosyVoice-2-style decoder and from 0.398 to 0.458 on an LM-free MMDiT-based decoder, with matching gains in Sim-ERes2Net. (§4.3, Table 3)
- complicates: Representation-alignment supervision that is uniform across timesteps or layers under-exploits the non-uniform structure of speaker information in flow-matching models, leaving speaker-similarity gains on the table relative to jointly adaptive supervision.
Evidence: Ablated alignment variants using only temporal adaptivity or only layer adaptivity underperform the jointly time- and layer-adaptive strategy on LibriTTS (Sim-WavLM 0.513 and 0.524 respectively vs. 0.538 for the joint strategy). (§4.2, Table 2)
- refines: External representation-alignment supervision for generative speech models is more effective when its supervision strength is dynamically reweighted according to where in the network and denoising trajectory the target attribute is actually encoded, rather than applied at fixed strength throughout.
Evidence: Visualizing the learned adaptive weights shows supervision concentrated in shallower layers and later denoising stages, which the paper interprets as the regions most critical for speaker identity, consistent with the earlier CKNNA-based distribution analysis. (§4.2, Figure 5)
Limitations and Open Questions
The evaluation covers only two FM decoder families (an LM-conditioned CosyVoice-2-style decoder and an LM-free MMDiT-based decoder in the style of F5-TTS), both operating on Mel-spectrogram targets; generalization to codec-token-based or purely autoregressive TTS systems is untested. The method’s supervision signal is entirely defined by a single pre-trained, speaker-verification-fine-tuned WavLM encoder, and the paper does not ablate alternative speaker encoders, so the ceiling of achievable speaker similarity is implicitly bounded by that one supervisor’s own discriminability. Training and evaluation data are restricted to Mandarin and English; broader multilingual generalization is not tested. The CKNNA-based distribution analysis that motivates the method is itself conducted only on the small-scale LibriTTS setting (§2.2), and the paper does not report whether the same timestep/layer pattern holds at the industrial 100k-hour scale.
Wiki Connections
- Flow Matching — proposes an auxiliary training-time loss specifically for FM-based TTS decoders and analyzes how speaker information is distributed across their denoising and depth dimensions.
- Zero-Shot TTS — targets the specific weakness of implicit, unsupervised speaker modeling in zero-shot FM TTS systems, improving speaker fidelity to unseen prompt speakers.
- Speaker Adaptation — introduces a mechanism for adapting internal FM representations toward a target speaker’s identity via dynamic alignment supervision rather than model fine-tuning.
- Self-Supervised Speech — relies on a WavLM-based (self-supervised pre-trained) speaker encoder as the core supervisory signal for its alignment loss.
- CosyVoice 2 — used as the LM-based FM decoder backbone for the industrial-scale experiments and as the primary baseline in Table 1 and Table 3.
- F5-TTS — its LM-free MMDiT-based decoder design is used as the template for the LM-free architecture variant tested for structural generalization.
- Seed-TTS — supplies the Seed-TTS test-zh/test-en benchmark sets and ASR models used for the paper’s WER/CER and speaker-similarity evaluation.
- Spark-TTS — serves as a baseline compared against the TLA-SA-enhanced model at equivalent training data scale in Table 1.
- FireRedTTS — serves as a baseline in Table 1; the TLA-SA-enhanced model is shown to outperform it despite using less training data and no speaker SFT stage.
- FireRedTTS-2 — serves as a baseline in Table 1; the TLA-SA-enhanced model is shown to approach its speaker-similarity performance despite using a fraction of its training data.
- MiniMax-Speech — discussed in related work as another approach using a learnable speaker encoder to strengthen zero-shot speaker generalization in FM-based TTS.
- CosyVoice — cited as part of the CosyVoice architecture family whose FM decoder design underlies the paper’s LM-based backbone.