Interspeech · 2026 · Conference
Jinhyeok Yang et al. (Supertone Inc / Independent Researcher) · → Paper · Demo: ✓ · Code: ?
Turns the two most common alignment failures of flow-matching TTS, skipped and repeated spans, into synthetic hard negatives in latent space and trains with a contrastive flow-matching objective that pushes the predicted velocity away from them, reducing content errors in a 0.06B-parameter model without an aligner, ASR model, or preference data.
Problem
Non-autoregressive flow-matching TTS that learns text-speech alignment implicitly (for example through cross-attention, without explicit durations) still produces skip and repeat errors, and these get worse when model capacity is small or the number of function evaluations (NFE) is reduced at inference. Existing remedies each carry a cost: architectural guidance changes the model, preference-based post-training (DPO on curated intelligibility datasets) requires building failure-case data, and ASR- or CTC-derived auxiliary objectives add external models to training. For compact, deployment-oriented systems the authors want a fix that touches only the training objective.
Method
The starting point is contrastive flow matching (CFM, from image generation), which adds to the standard conditional flow-matching loss a term that is subtracted: the squared distance between the predicted velocity and a velocity pointing toward a negative sample. In the plain version (here called ContrastiveFM), the negative is another utterance from the same batch, which is semantically unrelated to the conditioning text and so does not specifically target alignment errors.
RobustSpeechFlow adds a second negative built from the same utterance’s ground-truth latent, corrupted to imitate an alignment failure while keeping the sequence length fixed (variable-length corruption would complicate batching in the fixed-length latent pipeline). Per utterance, repeat or skip is chosen with equal probability, a coverage budget is sampled (20 to 40% of frames for repeat, 40 to 80% for skip), and spans of 0.1 to 5.0 seconds are edited until the budget is reached:
- Repeat: a source span is copied over a different target span. Because the target content is overwritten, this also produces a simultaneous skip, imitating combined repeat-and-skip failures.
- Skip: a span is deleted by shifting the rest of the sequence forward, and the freed tail frames are filled with a precomputed silence latent (made by encoding a zero waveform).
These negatives keep the speaker identity and acoustic texture of the target but break local text-speech correspondence, so they are harder than random in-batch negatives. The total loss is L_pos minus λ_rand·L_rand minus λ_aug·L_aug, with λ_rand = λ_aug = 0.2 (ContrastiveFM uses λ_aug = 0). Inference is unchanged: an Euler solver with classifier-free guidance weight 3.0 at NFE 12 or 24.
The method is applied to SupertonicTTS, a compact (0.06B) flow-matching text-to-latent model operating on continuous latents from the Supertonic speech autoencoder, with raw text input (no grapheme-to-phoneme conversion), Length-Aware RoPE, and context-sharing batch expansion. The architecture is held fixed across the three compared objectives; an utterance-level duration predictor is trained separately and the same pretrained text-to-latent checkpoint is shared across methods. Training uses roughly 10k hours, 5M utterances, and 80k speakers per language of internal English and Korean data (mixed human and ASR transcripts), 44.1 kHz audio, 500k steps on 8 H100 GPUs, and a 3 to 10 second same-speaker reference segment.
Key Results
Seed-TTS-eval (Table 1). Within the compact family, WER drops from 1.44 (Baseline SupertonicTTS) to 1.41 (ContrastiveFM) to 1.38 (RobustSpeechFlow), relative reductions of 4.2% and 2.1%. Speaker similarity (SIM) is 0.60 for all three. 1.38 is the lowest WER in the table; the lowest external value is MiniMax-Speech at 1.65, followed by DiTAR (1.69) and Seed-TTS DiT (1.73), with the rest between 1.85 (VoxCPM) and 3.14 (Spark-TTS). On SIM the picture reverses: 0.60 is below ten of the twelve external systems (range 0.67 to 0.79, highest Seed-TTS DiT at 0.79) and above only Spark-TTS (0.57) and OpenAudio S1-Mini (0.55). The table does not name the Seed-TTS-eval subset, and the paper does not say whether external numbers were re-measured with its own Whisper-large-v3 scoring and light text normalization or copied from prior reports. The external systems also differ in training data, so the cross-system WER comparison is indicative only; the fair comparison is the three-way in-family ablation.
ZERO500 (Table 2). ZERO500 is a new in-house benchmark with 50 diverse reference voices (game, news, conversational, and others) per language, each paired with 10 texts, for 500 pairs each in English and Korean; each pair is synthesized with two seeds and averaged.
| System | NFE | En CER | En WER | Ko CER | Ko WER |
|---|---|---|---|---|---|
| Baseline | 12 | 0.55 | 1.25 | 0.93 | 8.46 |
| Baseline | 24 | 0.48 | 1.18 | 0.81 | 8.40 |
| ContrastiveFM | 12 | 0.41 | 1.10 | 0.77 | 7.92 |
| ContrastiveFM | 24 | 0.39 | 1.06 | 0.65 | 7.72 |
| RobustSpeechFlow | 12 | 0.43 | 1.14 | 0.57 | 7.59 |
| RobustSpeechFlow | 24 | 0.35 | 1.03 | 0.57 | 7.45 |
At NFE=24 RobustSpeechFlow is lowest in all four columns. At NFE=12 it is lowest on both Korean metrics, but ContrastiveFM is better on English CER (0.41 vs 0.43) and English WER (1.10 vs 1.14); both contrastive variants beat the baseline everywhere. The largest margin over ContrastiveFM is on Korean (CER 0.57 vs 0.77 at NFE=12, 0.57 vs 0.65 at NFE=24).
Training dynamics (Figure 1). CER tracked at 100k-step intervals shows RobustSpeechFlow below both other systems at every checkpoint on Korean for both NFE settings. On English the curves cross repeatedly and ContrastiveFM is volatile; at NFE=12 the baseline is lowest at 200k, and RobustSpeechFlow is lowest from 300k onward at NFE=24.
All results are automatic ASR-based metrics; no MOS, listening test, or significance test is reported, and absolute differences in English CER are a few hundredths of a percentage point.
Novelty Assessment
The contribution is a training-objective change: contrastive flow matching already exists, and the new part is the speech-specific, length-preserving negative construction that mimics skip and repeat failures. It is simple, adds no models at training or inference, and is cleanly isolated by holding architecture, duration predictor, and initial checkpoint fixed across the three objectives. The evidence base is narrow, though: one compact model, internal training data, Seed-TTS-eval gains of 0.03 WER points over ContrastiveFM, and a non-public in-house benchmark (no release is stated) where the advantage over ContrastiveFM is clear on Korean but mixed on English at low NFE. Generality to larger flow-matching or autoregressive-diffusion systems is left to future work.
Field Significance
Moderate. The paper shows that alignment failures in flow-matching TTS can be targeted directly through the choice of contrastive negatives, giving a lightweight alternative to preference-based post-training and ASR-guided objectives. It is an incremental, objective-level contribution evaluated only on automatic intelligibility metrics in a 0.06B model, so it is best read as early evidence that failure-mode hard negatives help, rather than as a demonstrated general recipe.
Claims
- supports: Contrastive flow-matching regularization that pushes the predicted velocity away from negative targets reduces content errors in flow-matching TTS without changing the inference procedure.
Evidence: Adding random in-batch negatives (ContrastiveFM) to the same compact SupertonicTTS model lowers Seed-TTS-eval WER from 1.44 to 1.41 and lowers CER and WER on both ZERO500 languages at both NFE settings, with SIM unchanged at 0.60. (§5.1, Table 1; §5.2, Table 2)
- supports: Hard negatives that imitate realistic alignment failures provide a stronger anti-misalignment training signal for flow-matching TTS than randomly mismatched negatives.
Evidence: Length-preserving repeat and skip corruptions of the target latent reduce Seed-TTS-eval WER further to 1.38, and on ZERO500 Korean reduce CER to 0.57 at both NFE=12 and NFE=24 versus 0.77 and 0.65 for random-negative ContrastiveFM. (§3.3, §5.1, Table 1; §5.2, Table 2)
- complicates: The benefit of failure-mode negatives over generic contrastive negatives is not uniform across languages and inference budgets.
Evidence: On ZERO500 English at NFE=12, random-negative ContrastiveFM beats RobustSpeechFlow on CER (0.41 vs 0.43) and WER (1.10 vs 1.14), and English CER curves over training cross repeatedly between the three objectives. (§5.2, Table 2; §5.3, Figure 1)
- complicates: Intelligibility improvements in compact flow-matching TTS do not close the speaker-similarity gap to larger zero-shot systems.
Evidence: The 0.06B system reaches the lowest WER in the Seed-TTS-eval comparison but its SIM of 0.60 is below ten of twelve larger external systems (0.67 to 0.79); the authors attribute this to the compact codec and backbone rather than the objective, which is untested. (§5.1, Table 1; §6)
Limitations and Open Questions
- Evaluation is entirely objective and ASR-based (Whisper-large-v3 with light normalization); the authors acknowledge recognition and normalization bias and leave MOS and other subjective tests to future work, so naturalness effects of the subtracted contrastive terms are unmeasured.
- Only one compact model (0.06B SupertonicTTS) is tested; whether the gains carry over to larger flow-matching or autoregressive-diffusion systems is not examined.
- Training data is internal, and ZERO500 is an in-house benchmark with no stated release, limiting reproducibility; ZERO500 reports no speaker-similarity metric.
- Cross-system comparisons on Seed-TTS-eval mix systems with different data and possibly different scoring; the Seed-TTS-eval subset is not named.
- Gains are small in absolute terms (WER 1.44 to 1.38 on Seed-TTS-eval), results average only two seeds, and no significance testing is reported.
- The negatives cover only skip and repeat; other failure types (mispronunciation, hallucinated content, truncation) are left to future work, as is sensitivity to λ_aug and the coverage budgets, which are not ablated.
Wiki Connections
- Flow Matching — Proposes a speech-specific contrastive flow-matching objective whose negatives are latent corruptions mimicking skip and repeat errors.
- Zero-Shot TTS — Evaluates a compact zero-shot model prompted with reference speech on Seed-TTS-eval and a diverse-voice benchmark, targeting content fidelity rather than speaker similarity.
- F5-TTS — Cited as the alignment-free flow-matching TTS design whose skip/repeat issues motivate the work, and compared against on Seed-TTS-eval.
- DiTTo-TTS — Cited as evidence that strong pretrained representations improve but do not eliminate alignment failures in latent diffusion TTS.
- Intelligibility Preference Alignment — Represents the preference-based post-training alternative that RobustSpeechFlow avoids by needing no curated failure dataset.
- LARoPE — Length-Aware RoPE is applied in all compared systems and positioned as orthogonal to the contrastive objective.
- Seed-TTS — Supplies the Seed-TTS-eval benchmark and the Seed-TTS DiT comparison row.
- DiTAR — Compared against on Seed-TTS-eval and cited for low-NFE alignment degradation in autoregressive-diffusion systems.
- MiniMax-Speech — Strongest external WER baseline on Seed-TTS-eval in the comparison table.
- Whisper — Whisper-large-v3 is the ASR model used to compute all CER and WER scores.