ACL · 2026 · Conference

Yoonhyung Lee et al. (Qualcomm AI Research) · → Paper · → Preprint · Demo: ✓ · Code: ?

FC-TTS is a flow-matching zero-shot TTS model that takes speaker timbre and speaking style from two different reference utterances. It builds on frozen FACodec factors and adds a timbre-then-style two-stage decoder, a bottlenecked style encoder, and a cross-conditioned attribute-consistency loss so that mismatched timbre/style pairs still synthesize cleanly.

Problem

Reference-based zero-shot TTS systems usually take timbre and style from one prompt, so the two cannot be set independently. Factorized speech representations such as FACodec (from NaturalSpeech 3) separate prosody, content, acoustic detail, and a global speaker embedding, which in principle allows mixing timbre and style from different references. In practice the factorization is imperfect, the jointly trained codec decoders bound synthesis quality, and nothing guarantees robust output for timbre/style combinations that never co-occur in training. The authors report that reusing the FACodec decoder as NaturalSpeech 3 does was insufficient for independent timbre/prosody control in their preliminary experiments (§3.2.1). In-context-learning (ICL) zero-shot models add a further problem: they assume the prompt and the continuation share one style, which fails when style varies inside an utterance (§3.2.2, Figure 2).

Method

FC-TTS generates log-mel spectrograms from phonemes and converts them to 22 kHz audio with a pre-trained HiFi-GAN. A frozen FACodec encoder supplies only two conditions: the continuous speaker embedding z_spk (timbre) and the prosody token stream c_p (style). FACodec content and acoustic-detail tokens are deliberately excluded to prevent information leakage between the two pathways (§3.1.1). During training both conditions come from the target utterance, with z_spk swapped for another utterance from the same long recording; at inference they come from two different references.

Generation is hierarchical (§3.2.1). In the first stage, a transformer phoneme encoder, a RAD-TTS-style aligner, and a timbre adapter (transformer with adaptive layer normalization driven by z_spk) produce a blurry spectrogram h trained with an L1 (MAE) loss, which the authors use precisely because it produces over-smoothed output. In the second stage, a style adapter (non-causal transformer decoder with cross-attention to style embeddings) and a DiT-based conditional flow-matching (CFM) log-mel decoder refine h into the final spectrogram. The intent is that timbre anchors the acoustic subspace first and style is imprinted second, each through its own pathway. A separate CFM duration predictor (following MaskGCT) predicts durations at inference.

Figure 1: FC-TTS architecture. First stage: a phoneme sequence y is conditioned on timbre embedding z_spk via the timbre adapter to generate a blurry log-mel spectrogram h, anchoring timbre characteristics. Second stage: h is refined into a clean spectrogram x̂ by a flow-matching decoder conditioned on style embedding z_sty, obtained from prosody tokens c_p through hierarchical TCF modules, imprinting prosodic characteristics. During training, target speech provides both timbre and style references; at inference, separate references enable disentangled attribute control. Blue modules indicate trainable components.

The style encoder (§3.2.2, Appendix A) stacks two “TCF” modules (Transformer encoder, Cross-attention with a fixed set of learned global queries in Q-Former fashion, and Finite Scalar Quantization) at phoneme and frame level. The fixed-query cross-attention and the FSQ layer act as information bottlenecks so the style code captures prosodic patterns rather than copying acoustic detail from the reference. An auxiliary ResNet spectrogram-reconstruction branch is trained alongside to stop the FSQ codes collapsing to a single code. The authors call this component “VQ-VAE style encoding”.

The conditional consistency loss (CCL, §3.2.3) re-parameterizes the CFM objective so the decoder predicts log-mel directly (following WaveFM). Two predictors, initialized from FACodec encoder transformers, then recover the conditions from the generated spectrogram: a prosody predictor with a cross-entropy loss on c_p and a timbre predictor with a negative-cosine loss on z_spk. Each predictor also receives the other attribute as input (z_spk for prosody, c_p for timbre), which the authors argue sharpens the gradient toward the intended mode when several conditions interact (Figure 3).

The model has 204M parameters and is trained on LibriHeavy (all subsets, about 50,000 hours) for 200k iterations, batch 64, on 8 V100 GPUs for 116 hours. Inference uses 8 NFE for duration and 32 NFE with classifier-free guidance scale 4.0 for the spectrogram (§4.1.3, Appendix C).

Key Results

Standard zero-shot TTS (LibriSpeech test-clean, 4-10 s, Table 1). FC-TTS reaches UTMOS 4.22, WER 1.88, and speaker similarity 0.60. Among systems with UTMOS reported, NaturalSpeech 3 is higher (4.30) and the LibriHeavy-retrained F5-TTS is lower (4.03). On WER, NaturalSpeech 3 is lower (1.81) and FC-TTS is below F5-TTS (2.42 reported; 3.30 retrained), DiTTo-TTS (2.69), and CLaM-TTS (5.11). On speaker similarity FC-TTS (0.60) ties DiTTo-TTS, is above CLaM-TTS (0.50), and is below NaturalSpeech 3 (0.67), both F5-TTS rows (0.66 and 0.67), and the HiFi-GAN resynthesis row (0.64). FC-TTS has the fewest parameters in the table (204M vs. 205M for retrained F5-TTS and 336-584M for the others). Only F5-TTS was retrained for this comparison; the other baseline numbers are reported from their original papers, so tooling and test subsets may differ. The authors attribute the speaker-similarity gap to excluding FACodec content and detail tokens and to the timbre-only first stage (§4.2.1).

Timbre control with expressive prosody references (RAVDESS, Table 2). RAVDESS utterances provide prosody, and one male and one female LibriSpeech speaker provide timbre. The comparison system is FACodec-VC: ground-truth FACodec tokens decoded with an unmatched speaker embedding, which the authors treat as an upper bound for FACodec-based TTS because NaturalSpeech 3 has no public checkpoint. FC-TTS scores UTMOS 4.03, speaker similarity 0.48, and WER 0.18 against 3.19, 0.27, and 8.40 for FACodec-VC. In a human ABX timbre test FC-TTS is preferred in 66.1% of trials and FACodec-VC in 10.7%. RAVDESS has only two short sentences, which the authors note explains the very low WER values (footnote 5).

Style control (RAVDESS, Tables 3 and 4). FC-TTS takes style from an expressive RAVDESS utterance and timbre from a neutral utterance of the same speaker. Retrained F5-TTS gets the expressive utterance as its single prompt. FC-TTS is better on UTMOS (3.95 vs. 3.40), WER (0.30 vs. 4.39), MCD to the expressive reference (3.21 vs. 3.43), and human ABX prosody preference (65.5% vs. 8.9%). It is worse on speaker similarity (0.47 vs. 0.57). An AudioLLM-as-a-judge evaluation with Gemini 2.5 Pro gives FC-TTS a style Win Ratio of 91.7% (F5-TTS 8.3%) and a Style-MOS of 3.92 (F5-TTS 1.50) on a 1-5 scale. These judge scores are automatic and are not a substitute for the human ABX results.

Ablations (Table 5). Removing the entire consistency loss causes the largest degradation on every Table 5 metric except RAVDESS UTMOS (3.70, where removing the two-stage design is lower at 3.57): WER rises from 1.88 to 5.88 on LibriSpeech and from 0.30 to 9.36 on RAVDESS, LibriSpeech speaker similarity drops from 0.60 to 0.48, and RAVDESS speaker similarity from 0.37 to 0.21. Removing only the cross-conditioning inside the loss has small effects (LibriSpeech WER 1.92 vs. 1.88; RAVDESS MCD 3.36 vs. 3.33; RAVDESS UTMOS 3.79 vs. 3.91). Removing the two-stage design lowers RAVDESS UTMOS (3.57 vs. 3.91) and raises LibriSpeech MCD (5.83 vs. 5.60), but slightly improves RAVDESS MCD (3.26 vs. 3.33). Removing the bottlenecked style encoder worsens RAVDESS MCD (3.47 vs. 3.33) and speaker similarity on both sets (0.57 vs. 0.60; 0.34 vs. 0.37), while slightly improving UTMOS (4.25 vs. 4.22 LibriSpeech; 3.99 vs. 3.91 RAVDESS) and RAVDESS WER (0.25 vs. 0.30).

Novelty Assessment

The reference representation is not new: FC-TTS uses the FACodec encoder frozen, and dual-reference timbre/style control has been attempted before (IndexTTS2, EmoSphere++). The new parts are in how the factors are consumed. The timbre-first, style-second generation with an intentionally blurry intermediate is a new decoder structure. The cross-conditioned attribute-consistency loss extends single-condition speaker-consistency losses to multiple interacting conditions. The Q-Former-plus-FSQ style bottleneck combines known components (Q-Former queries, FSQ) for this purpose. The ablations support the consistency loss most clearly. Evidence for the cross-conditioning detail and the two-stage design is weaker: the per-metric effects are small and partly mixed. The headline control comparisons have caveats. The timbre comparison uses an oracle-token FACodec VC proxy rather than a TTS system. The style comparison pits a two-reference model against a single-prompt baseline that was never designed for separate control.

Field Significance

Moderate — The paper shows that frozen factorized codec features are not enough for dual-reference TTS by themselves, and its ablations point to an output-side attribute-consistency loss as the component that keeps mismatched timbre/style pairs intelligible, with smaller and partly mixed gains from the two-stage decoder. It provides a concrete, ablated recipe for separate style and timbre references at a 204M-parameter scale. The control comes at a measurable speaker-similarity cost relative to single-prompt zero-shot systems.

Claims

  • supports: Pre-factorized speech representations decoded with their original codec decoder degrade sharply when timbre and prosody come from mismatched sources, so independent attribute control needs additional generation structure.

    Evidence: FACodec ground-truth tokens decoded with an unmatched speaker embedding reach UTMOS 3.19, speaker similarity 0.27, and WER 8.40 on RAVDESS prosody with LibriSpeech timbres, against 4.03, 0.48, and 0.18 for FC-TTS; human ABX timbre preference is 66.1% vs. 10.7%. (§4.2.2, Table 2)

  • supports: Losses that re-predict the conditioning attributes from generated speech are critical for intelligibility and attribute adherence when a TTS model is conditioned on several separately sourced attributes.

    Evidence: Removing the conditional consistency loss raises WER from 1.88 to 5.88 (LibriSpeech) and from 0.30 to 9.36 (RAVDESS) and lowers speaker similarity from 0.60 to 0.48 and from 0.37 to 0.21; removing only the cross-conditioning of the predictors changes metrics by small amounts (e.g., RAVDESS MCD 3.36 vs. 3.33). (§4.2.4, Table 5)

  • complicates: Taking speaking style and timbre from separate references improves style transfer but lowers speaker similarity relative to single-prompt zero-shot TTS.

    Evidence: On RAVDESS style control, FC-TTS beats retrained F5-TTS on MCD (3.21 vs. 3.43) and ABX prosody preference (65.5% vs. 8.9%) but has lower speaker similarity (0.47 vs. 0.57); on LibriSpeech its speaker similarity is 0.60 against 0.67 for NaturalSpeech 3 and retrained F5-TTS, which the authors attribute to excluding FACodec content/detail tokens and the timbre-only first stage. (§4.2.1, §4.2.3, Tables 1 and 3)

  • refines: Information bottlenecks on a style encoder improve prosody following from an explicit style reference, with a small cost on automatic naturalness predictors.

    Evidence: Removing the Q-Former-plus-FSQ style encoder (falling back to in-context prompting) raises RAVDESS MCD from 3.33 to 3.47 and lowers speaker similarity on both test sets, while UTMOS rises slightly (4.25 vs. 4.22 LibriSpeech; 3.99 vs. 3.91 RAVDESS). (§3.2.2, §4.2.4, Table 5)

Limitations and Open Questions

The human ABX tests are thinly documented. Participants are internal company employees, and the paper does not report the number of listeners, the number of trials, or significance tests (Appendix D.3). The appendix describes the question as which sample “more closely matched the reference in terms of naturalness and intelligibility”, while §4.2.2 and §4.2.3 describe timbre and prosody questions (the instruction screens in Figures 7-8 show separate timbre and style experiments). The ABX preference rates in Tables 2 and 3 do not sum to 100%; the style-test interface includes a “Not sure” option (Figure 8).

The comparisons are uneven. The timbre experiment compares against an oracle-token FACodec VC proxy rather than any dual-reference TTS model. The style experiment compares a two-reference model against single-prompt F5-TTS. Dual-reference systems discussed in related work (IndexTTS2, EmoSphere++) are not evaluated. Most Table 1 baselines are reported values from original papers, not re-run under the same protocol.

FC-TTS numbers for RAVDESS style control differ between Table 3 (UTMOS 3.95, speaker similarity 0.47, MCD 3.21) and the full-model row of Table 5 (UTMOS 3.91, speaker similarity 0.37, MCD 3.33), although both are described with neutral-timbre and emotional-style references. The paper does not explain the difference.

Other limits the authors name (§Limitations): training and evaluation are English only. Quality remains bounded by FACodec’s disentanglement, and residual timbre cues may sit in the excluded tokens. The boundary between timbre and style (for example, a husky voice) is not well defined, and no metric isolates either attribute cleanly. MCD is used as a prosody-similarity proxy despite also reflecting spectral differences. Code availability is not stated; a demo page is provided.

Wiki Connections

  • Flow Matching — The second-stage log-mel decoder and the duration predictor are trained with conditional flow matching, with the decoder re-parameterized to predict spectrograms directly so attribute predictors can score its output.
  • Zero-Shot TTS — Evaluated on LibriSpeech test-clean speakers unseen in LibriHeavy training, with timbre and style taken from separate reference clips.
  • Prosody Control — Style comes from a reference utterance through a bottlenecked prosody encoder separate from timbre, and prosody transfer is evaluated with MCD, human ABX, and an LLM judge on RAVDESS.
  • Disentanglement — Adds a cross-conditioned attribute-consistency loss, a timbre-then-style decoder, and a quantized style bottleneck on top of frozen FACodec factors, with ablations for each.
  • Subjective Evaluation — Uses human ABX preference tests for timbre and prosody similarity, alongside a Gemini-based automatic judge.
  • NaturalSpeech 3 — Source of the frozen FACodec encoder; Table 1 baseline, and its codec decoder (as FACodec-VC) is the comparison system for timbre control.
  • F5-TTS — Retrained on LibriHeavy at matched size as the single-reference baseline for zero-shot quality and for style control.
  • DiTTo-TTS — Zero-shot TTS baseline with reported LibriSpeech results in Table 1.
  • IndexTTS2 — Prior dual-reference timbre/style control approach discussed in related work but not evaluated.
  • LSCodec — Speaker-decoupled codec cited as an alternative disentangled representation.
  • HiFi-GAN — Pre-trained vocoder that converts FC-TTS log-mel output to waveform, also used as a resynthesis reference row.
  • UTMOS — Automatic naturalness predictor used across all objective evaluations.
  • Gemini 2.5 — Used as the AudioLLM judge for style Win Ratio and Style-MOS.