arXiv · 2026 · Preprint
Vikentii Pankov et al. (Rask AI) · → Paper · Demo: ✓ · Code: ✗
PFluxTTS fuses a duration-guided and an alignment-free flow-matching decoder at inference time to combine stable alignment with fluent, natural-sounding cross-lingual voice cloning, paired with a modified 48 kHz super-resolution vocoder.
Problem
Flow-matching TTS systems face three unresolved gaps. First, duration-guided decoders (explicit duration predictors) produce stable but over-smoothed, low-variance timing that limits prosodic naturalness, while alignment-free decoders (implicit text-speech alignment, as in F5-TTS) improve fluency but remain prone to instability such as word skipping, especially under noisy cross-lingual prompts. Second, zero-shot voice cloning conditioned on a single fixed-dimensional speaker embedding discards time-varying timbre information, which is particularly damaging for cross-lingual prompts where speaker identity must be preserved without a matching prompt transcript. Third, most TTS pipelines emit low-rate mel features (e.g. 24 kHz, hop 512) and pair them with vocoders that were not designed to recover full-band 48 kHz audio, leaving the mel-to-waveform quality gap under-addressed. The paper further notes that most prior TTS evaluation is confined to clean, monolingual benchmarks (LibriSpeech, VCTK), which does not stress-test robustness for cross-lingual, in-the-wild dubbing-style use cases.
Method
PFluxTTS synthesizes a mel-spectrogram from phonemes and an acoustic prompt using two independently trained conditional-flow-matching (CFM) TTS models with no weight sharing: a duration-guided (DG) path and an alignment-free (AF) path. Both paths share phoneme inputs (extracted via espeak-ng) and are additionally conditioned on a language ID and a pretrained ECAPA-TDNN speaker embedding via AdaLN in their text encoders.
The DG path follows a FLUX-style decoder with eight DoubleStream blocks (separate parameters for prompt and content tokens interacting via self-attention over the concatenated sequence) followed by 16 SingleStream blocks that jointly refine the merged representation before content tokens are retained and passed through a length regulator and CFM decoder. The AF path follows F5-TTS: a DiT-style conditional decoder that expands the phoneme sequence with learned filler tokens to match the acoustic length and predicts the mel with CFM without an explicit duration module, reusing the duration predicted by the DG path so both fields operate on the same (F, T) grid.
For voice cloning, each path has its own SpeechPromptEncoder. In the DG path, an 8-layer Transformer encodes the prompt mel-spectrogram and a learnable query-pooling mechanism reduces it to a sequence of K=16 embeddings that are attended jointly with content tokens inside the FLUX decoder. In the AF path, the same backbone with self-attention pooling instead produces a single fixed 1024-d prompt embedding, because sequence-level prompt conditioning was found to destabilize the AF path. Training crops a random 1-6s span of reference audio as the prompt and masks the same span in the target mel to avoid content leakage, following F5-TTS’s convention.
At inference, the two independently trained vector fields are fused within a single ODE integration using a piecewise-constant mixing coefficient α(t): the DG field is used for the first N1 of N solver steps and the AF field for the remainder, so generation finishes purely under the AF field. This lets the DG field stabilize alignment early while preserving the AF field’s fluency in later steps. Both models apply joint classifier-free guidance that nulls text and prompt conditioning together, with conditional dropout (p=0.1) applied independently to each pathway during training.
The generated mel is converted to 48 kHz audio by a PeriodWave-based vocoder retrained from scratch on multilingual 48 kHz audio with two modifications: (i) an additional upsampling/downsampling block pair inserted into the Period-Aware Estimator so the vocoder consumes low-frame-rate mel input (hop 512 at 24 kHz) and reconstructs full-band 48 kHz output, and (ii) a global speaker-prompt embedding, extracted from 48 kHz audio via a ConvNeXt V2 encoder with attention pooling, injected into the PeriodWave Mel Encoder to compensate for high-frequency speaker detail lost at the low mel rate.
Training used four NVIDIA A100 GPUs for 1.5M iterations at batch size 128 with AdamW (initial LR 1e-4, decayed to 1e-6 over the final 100k steps), logits softcapping (threshold 70, following Gemma) to stabilize training, and gradient clipping at 5. Training data is multilingual conversational speech (English, Spanish, German, French, Italian, Portuguese, Russian) drawn from Yodas and other sources, filtered through a pipeline of diarization, language ID, transcription, forced-alignment boundary refinement, silence normalization, and multi-stage QA (sample-rate, LID, extraneous-sound, single-speaker checks); roughly 28% of candidates pass, yielding about 50k hours used for training. Total model parameter count is not reported.
Key Results
On subjective evaluation (40 mTEDx-test utterances synthesized in English from Spanish, French, Italian, and Portuguese prompts, rated by at least seven native-English annotators per sample on Prolific), PFluxTTS reaches 4.11 naturalness MOS and 3.51 SMOS, statistically better than FishSpeech in naturalness and better than ElevenLabs Multilingual v2 in SMOS (paired t-test, p<0.05), and statistically indistinguishable from ChatterBox on both metrics (Table 1). On objective cross-lingual evaluation (VoxLingua-dev, 397 utterances across 33 source languages, English target synthesis), PFluxTTS is statistically better than all compared systems (ChatterBox, FishSpeech, F5-TTS, SparkTTS) on WER, CER, and SPK-SIM (Wilcoxon signed-rank, Holm-adjusted p<0.05), achieving 6.9% WER versus ChatterBox’s 9.0% (a 23% relative reduction) and 0.68 SPK-SIM versus ChatterBox’s 0.61 (Table 2). FishSpeech, SparkTTS, and F5-TTS show substantially degraded intelligibility (WER 45.4%, 82.5%, and 60.2% respectively) under noisy cross-lingual prompts, which the paper attributes to frequent word skipping.
Ablations on VoxLingua-dev paired with ELLA-V-hard text show that fusing the two decoders (α=0.75) reduces CER to 8.6%, versus 14.1% for the AF path alone (α=0) and 10.6% for the DG path alone (α=1) (Fig. 2). A CMOS listening test (24 mTEDx-based samples, 10 raters each) found the fused model preferred over the DG-only model with mean ΔCMOS=0.33 (Wilcoxon signed-rank, p<0.012), winning in 79% of comparisons. Isolating the sequence-based prompt conditioning against a fixed-embedding baseline shows a large CMOS improvement (ΔCMOS=1.19, p<0.05) and raises SPK-SIM from 0.47 to 0.57. The modified PeriodWave+SR vocoder achieves the best log-spectral distance among compared systems on both VCTK-test (0.66 vs. 0.70 for NVSR and 0.99 for BigVGAN+AudioSR) and mTEDx (1.01 vs. 1.63 and 1.39 respectively), with the largest margin on the out-of-domain mTEDx set (Table 3).
Novelty Assessment
The genuinely new contribution is the inference-time fusion of two independently trained flow-matching decoders through a piecewise-constant mixing schedule over the shared ODE solver trajectory, which lets a duration-guided and an alignment-free model contribute at different stages of generation without joint training or weight sharing. The voice-cloning mechanism (a sequence of query-pooled prompt embeddings attended inside a FLUX-style decoder, rather than a single fixed embedding) is an incremental but well-validated refinement over prior sequence-prompt approaches such as XTTS and MegaTTS-2, applied specifically to strengthen cross-lingual robustness. The vocoder contribution is an engineering adaptation of an existing architecture (PeriodWave) to a mismatched mel frame rate and 48 kHz target with added speaker conditioning, rather than a new vocoder design. Individually, the AF path directly follows F5-TTS’s alignment-free formulation and the DG path follows FLUX’s block design, so the system’s components are largely assembled from existing building blocks; the fusion mechanism and the systematic cross-lingual, in-the-wild evaluation protocol are the paper’s most distinctive contributions.
Field Significance
moderate — This paper demonstrates that two independently trained flow-matching TTS decoders can be combined at inference time to trade off the stability/naturalness balance without retraining either model, and that sequence-based prompt conditioning materially improves cross-lingual speaker similarity over fixed embeddings. Its cross-lingual, in-the-wild evaluation protocol (VoxLingua-dev and mTEDx dubbing-style tests) also demonstrates that several strong open-source baselines degrade sharply outside clean, monolingual conditions.
Claims
- supports: Fusing the vector fields of two independently trained flow-matching decoders during ODE integration can combine the stability of duration-guided alignment with the fluency of alignment-free decoding, without joint training.
Evidence: On VoxLingua-dev paired with ELLA-V-hard text, CER falls from 14.1% (alignment-free only, α=0) and 10.6% (duration-guided only, α=1) to 8.6% under the fused schedule (α=0.75). (§3.2, Fig. 2)
- supports: Conditioning zero-shot voice cloning on a sequence of pooled prompt embeddings, rather than a single fixed speaker vector, improves cross-lingual speaker-similarity preservation.
Evidence: Replacing a fixed prompt embedding with query-pooled sequence conditioning (K=16) raises objective SPK-SIM from 0.47 to 0.57 and wins a CMOS listening test by ΔCMOS=1.19 (p<0.05). (§3.2)
- complicates: Sequence-level prompt conditioning that stabilizes voice cloning in a duration-guided decoder can destabilize an alignment-free decoder’s implicit alignment.
Evidence: Applying sequence-level prompt conditioning to the alignment-free path caused frequent word skipping, which the authors mitigate by retaining a fixed prompt embedding for that path instead. (§2.2)
- supports: A flow-matching vocoder can be adapted to reconstruct full-band audio from a lower mel frame rate than it was originally designed for by adding resampling blocks and prompt-aware speaker conditioning, without introducing a new vocoder architecture.
Evidence: The modified PeriodWave+SR vocoder achieves the best log-spectral distance on VCTK-test (0.66) and mTEDx (1.01), outperforming NVSR (0.70 / 1.63) and BigVGAN+AudioSR (0.99 / 1.39), with the largest margin on the out-of-domain mTEDx set. (§3.2, Table 3)
Limitations and Open Questions
The subjective and cross-lingual evaluation is restricted to English as the sole synthesis target language, with prompts drawn from only four source languages (Spanish, French, Italian, Portuguese) for the subjective test; the paper does not report results for non-English target synthesis, so cross-lingual robustness claims are demonstrated only in one target-language direction.
The paper also notes that all open-source baselines were trained on larger multilingual datasets than were evaluated here (the comparison is restricted to English-only synthesis “where systems are robust”), which may understate baseline non-English performance while not affecting the reported comparison itself. Total model parameter count and the vocoder’s specific training corpus are not reported. The authors state that future work includes scaling to larger training data, exploring alternative fusion schedules beyond the piecewise-constant one used here, and developing explicit control over the balance between prompt-driven and text-driven prosody.
Wiki Connections
- Flow Matching — proposes fusing the vector fields of two independently trained conditional-flow-matching decoders at inference time within a single ODE integration.
- Zero-Shot TTS — performs voice cloning from short reference audio with no additional training or prompt transcript required.
- Multilingual TTS — trains on multilingual conversational data spanning seven languages and specifically targets cross-lingual voice-prompt robustness.
- Speaker Adaptation — introduces a sequence-based speech-prompt encoder as an alternative to fixed speaker embeddings for adapting synthesis to a target voice.
- Subjective Evaluation — runs MOS naturalness, SMOS similarity, and CMOS ablation listening tests with native-English raters on the Prolific platform.
- F5-TTS — the alignment-free decoder path directly follows F5-TTS’s DiT-style implicit-alignment formulation and F5-TTS is used as a baseline in the cross-lingual evaluation.
- Fish-Speech — used as a baseline in both subjective and objective evaluations, where it shows substantially higher WER/CER under cross-lingual noisy prompts.
- Spark-TTS — used as a baseline in the objective evaluation, showing the weakest intelligibility and speaker similarity among compared systems.
- PeriodWave — its vocoder architecture is retrained from scratch and modified with additional resampling blocks and prompt conditioning for 48 kHz super-resolution synthesis.
- XTTS — its approach of conditioning on a set of prompt embeddings rather than a single vector is the precedent this paper extends with query pooling and cross-lingual robustness.
- E2-TTS — discussed as an alignment-free predecessor that removes the duration predictor but suffers stability and convergence issues that this paper’s fusion mechanism aims to address.
- MegaTTS 3 — discussed as a related approach bridging alignment-free and duration-guided modeling via sparse alignment guiding a latent DiT.
- NaturalSpeech 3 — discussed as a prior approach that disentangles timbre, prosody, and acoustic content to stabilize timbre while controlling prosody separately.