arXiv · 2026 · Preprint
Wei Fan et al. (Anhui Province Key Laboratory of Digital Security / Alibaba Group) · → Paper · Demo: ✓ · Code: ✗
Trains a single flow-matching DiT to generate raw waveform patches directly from characters and a prompt waveform, with no mel or codec interface, vocoder, speaker encoder or pretrained component at inference, and shows that the training recipe (WavLM representation alignment, a two-stage noise schedule, and a velocity-weighted spectral loss) lifts it to roughly the level of a same-data mel-space F5-TTS on objective metrics.
Problem
Strong zero-shot TTS systems still route generation through an intermediate representation (mel-spectrogram, codec tokens or latents) and then render audio with a separately trained vocoder or decoder; even simplified flow-matching systems such as F5-TTS and E2 TTS operate in mel space. Earlier direct text-to-waveform systems (Wave-Tacotron, FastSpeech 2s, DiffAR, E3-TTS) each kept some inference-time dependency, such as phoneme or prosodic inputs, a duration predictor, a speaker embedding or a pretrained text encoder, and most did not report voice cloning (§1, Table 1). The authors argue that once those components are removed, training becomes the bottleneck: raw waveform lacks a pretrained representational prior, no single noise-level distribution suits both early and late training, and data-space perceptual losses are implicitly down-weighted near clean timesteps relative to a velocity-space flow loss.
Method
The generator is a waveform-patch DiT trained with conditional flow matching as a speech-infilling task, following F5-TTS: a random span covering 70 to 100% of the utterance is masked and generated from the remaining waveform plus the full transcript (§3.2.1, §A.3). The noisy 24 kHz waveform is patchified by a 1-D convolution with non-overlapping 768-sample patches (32 ms per token). The prompt audio enters through two parallel streams, a convolutional frontend that produces coarse downsampled features at the patch rate and the raw prompt waveform patches, concatenated into the main stream (§3.3). Characters are refined by four ConvNeXt blocks and prepended as in-context tokens. The backbone has 32 DiT blocks, hidden size 1280 and 16 heads, 983.6M parameters in total, trained from scratch (§4.1.4, Table 5). The network predicts the clean waveform (x-prediction) while the loss is computed in velocity space, which amounts to an x-loss reweighted by 1/(1-t)² with the denominator clipped at 0.01 (§3.2.3).

Three training-only additions carry the method (§3.4). Representation alignment (REPA) projects the 18th DiT block’s hidden states through a small Conv1d head and maximises cosine similarity with frozen WavLM-base+ features of the target waveform, following the dual-modality alignment design of A-DMA (Table 6). Staged noise scheduling samples timesteps from a logit-normal distribution (μ = -0.4, σ = 0.8) for the first 240k equivalent updates and from a uniform distribution afterwards, which places more mass on cleaner states. Velocity-Aware Perceptual Alignment (VAPA) scales a WaveFM-style multi-resolution STFT distance (phase, log-magnitude, spectral-gradient and Laplacian terms) by (1-t)^-γ with γ = 1, so that its weight relative to the flow loss no longer shrinks near the clean endpoint; it is enabled only in the uniform stage. At inference the WavLM teacher and projection head are dropped and sampling uses a 50-step Heun solver, sway sampling (s = -1.0) and CFG scale 3.5 (§4.1.4). Total duration is estimated from the prompt duration and text-length ratio (§A.2).
Key Results
All controlled systems train on the same 19.4k-hour Emilia English subset; evaluation follows the F5-TTS cross-sentence protocol with Whisper-large-v3 WER, WavLM-based SIM-o and UTMOS (§4.1, Table 2).
| System | Seed-TTS WER | Seed-TTS SIM-o | Seed-TTS UTMOS | LS-PC WER | LS-PC SIM-o | LS-PC UTMOS |
|---|---|---|---|---|---|---|
| Ground truth | 1.86 | 0.734 | 3.53 | 2.48 | 0.695 | 4.10 |
| CosyVoice 2 (167K h, multilingual) | 2.51 | 0.659 | 4.15 | 2.05 | 0.659 | 4.38 |
| FireRedTTS (248K h, reported) | 3.82 | 0.46 | not reported | 2.69 | 0.47 | not reported |
| F5-TTS Base + Vocos (same data) | 2.09 | 0.573 | 3.83 | 3.17 | 0.597 | 4.10 |
| E2-TTS Base + Vocos (same data) | 3.50 | 0.582 | 3.41 | 4.32 | 0.632 | 3.84 |
| Simple direct-wave baseline | 2.42 | 0.424 | 3.35 | 3.45 | 0.416 | 3.60 |
| BareWave, basic training | 2.34 | 0.478 | 3.43 | 3.32 | 0.471 | 3.69 |
| BareWave, full recipe | 1.75 | 0.602 | 3.72 | 2.88 | 0.614 | 4.01 |
Among same-data systems, BareWave has the lowest WER on both test sets and the highest SIM-o on Seed-TTS test-en. On LibriSpeech-PC it trails E2-TTS Base on SIM-o (0.614 vs 0.632), and on both sets it trails F5-TTS Base on UTMOS (3.72 vs 3.83; 4.01 vs 4.10). The larger-data CosyVoice 2 remains ahead on SIM-o and UTMOS on both sets and on LibriSpeech-PC WER; the FireRedTTS numbers are transcribed from the F5-TTS paper. BareWave’s Seed-TTS WER is below the ground-truth recording’s 1.86. The comparison is fair on data but not on size: BareWave has 983.6M parameters against roughly 349M for F5-TTS Base plus Vocos.
The training recipe accounts for most of the gain over the waveform baselines. In the matched-budget ablation near 600k equivalent updates on LibriSpeech-PC (§4.3, Table 3), REPA moves the base model from 3.32 / 0.471 / 3.69 (WER / SIM-o / UTMOS) to 2.86 / 0.522 / 3.70; late uniform continuation gives 2.93 / 0.543 / 3.82; adding the plain refined STFT loss gives 3.38 / 0.585 / 3.88; applying VAPA to it gives 3.52 / 0.610 / 3.97. Speaker similarity and UTMOS rise at every step, but WER rises from 2.93 to 3.52 once the spectral loss and VAPA are added, which the authors describe as staying “in the same operating range”. The dual-stream prompt frontend alone lowers WER from 3.45 to 3.32 and raises SIM-o from 0.416 to 0.471 (Table 4). Training curves show REPA improves SIM-o throughout training (Figure 4), the logit-normal schedule converges faster than uniform early on (Figure 5), and REPA teacher layer and logit-normal mean each trade WER against SIM-o (Figures 6, 7). Higher CFG lowers WER, while SIM-o peaks at 3.5 (Figure 8). No human listening test is reported.
Novelty Assessment
Direct text-to-waveform TTS is not new (Wave-Tacotron, DiffAR, E3-TTS), and the backbone is the F5-TTS infilling recipe moved from mel frames to 768-sample waveform patches. Representation alignment comes from image-domain REPA and the earlier A-DMA speech work, the spectral distance comes from WaveFM, and two-stage noise schedules are an engineering choice. The new pieces are VAPA, which reweights a data-space perceptual loss to match the implicit 1/(1-t) scaling of an x-prediction, v-loss objective, and the hierarchical prompt frontend. Both are ablated (Tables 3, 4), though the VAPA ablation also shows a WER cost. The main contribution is showing that this combination, with all auxiliary components dropped at inference, brings a vocoder-free waveform model close to a same-data mel-space flow-matching baseline. Evidence is objective only, English only, and from a single run per configuration.
Field Significance
Moderate. The paper gives same-data evidence that a vocoder-free, fully waveform-native flow-matching model can match or beat a mel-space F5-TTS on WER and roughly match it on speaker similarity, while trailing it slightly on UTMOS. That result depends on a training recipe whose parts are mostly borrowed, needs about three times the parameters, and is not backed by any human evaluation, so it is a useful data point for waveform-domain generation rather than a change of direction.
Claims
- supports: Zero-shot TTS can reach intelligibility and speaker similarity comparable to mel-space flow-matching systems while generating raw waveform directly, with no vocoder or other pretrained component at inference.
Evidence: Trained on the same 19.4k-hour Emilia English subset, the 983.6M waveform-patch DiT reaches 1.75% WER and 0.602 SIM-o on Seed-TTS test-en against 2.09% and 0.573 for F5-TTS Base with Vocos, and 2.88% / 0.614 against 3.17% / 0.597 on LibriSpeech-PC, though E2-TTS Base has higher LibriSpeech-PC SIM-o (0.632). (§4.2.1, Table 2)
- complicates: Single-stage waveform generation without a dedicated vocoder still trails vocoder-based pipelines on predicted naturalness and needs considerably more parameters.
Evidence: UTMOS is 3.72 vs 3.83 (Seed-TTS test-en) and 4.01 vs 4.10 (LibriSpeech-PC) against F5-TTS Base, which the authors attribute to single-stage waveform rendering; the model has 983.6M parameters against 335.8M + 13.5M for F5-TTS Base plus Vocos. (§4.2.1, Table 2; §Limitations)
- supports: Aligning a generator’s hidden states with frozen self-supervised speech features during training improves data efficiency and speaker similarity in raw-waveform TTS without adding inference cost.
Evidence: Adding WavLM-base+ alignment to the base waveform model moves LibriSpeech-PC WER / SIM-o from 3.32 / 0.471 to 2.86 / 0.522 at a matched budget, and training curves show a SIM-o advantage throughout training; the teacher is discarded at inference. (§4.3.1, Table 3, Figure 4)
- refines: In x-prediction flow-matching models trained with a velocity-space loss, data-space perceptual losses need timestep-dependent upweighting near clean states to be effective.
Evidence: Applying a (1-t)^-1 weight to the same multi-resolution STFT distance raises LibriSpeech-PC SIM-o from 0.585 to 0.610 and UTMOS from 3.88 to 3.97 relative to the unweighted loss, while WER moves from 3.38 to 3.52. (§3.4.3, §4.3.3, Table 3)
- complicates: Training choices for waveform-domain flow-matching TTS (noise-level distribution, alignment target, guidance strength, perceptual losses) trade intelligibility against speaker similarity and naturalness rather than improving all metrics together.
Evidence: Lower logit-normal means favour WER while higher means favour SIM-o and UTMOS; deeper WavLM target layers lower WER while layer 10 gives the strongest SIM-o; stronger CFG lowers WER but SIM-o peaks at 3.5; adding the spectral loss and VAPA raises SIM-o and UTMOS while WER climbs from 2.93 to 3.52. (§4.4.3 to §4.4.5, Figures 6 to 8; Table 3)
Limitations and Open Questions
Warning
All results are objective (Whisper WER, WavLM-based SIM-o and the UTMOS predictor) with no human listening test, so the naturalness and speaker-similarity claims for vocoder-free waveform generation rest on automatic proxies only.
- The model is about three times larger than the mel-space baselines it is compared with, and the paper reports no inference speed or real-time factor, so the deployment simplicity of a single waveform model is not weighed against its compute cost (§Limitations).
- Training and evaluation are English only, on one 19.4k-hour dataset; larger-data scaling and other languages are untested.
- The ablation in Table 3 is at roughly 600k equivalent updates and the main results at 640k, so the two tables’ final rows differ; each configuration appears to be a single run with no variance reported.
- Adding the perceptual losses raises WER in the ablation (2.93 to 3.52), a trade-off the paper underplays.
- Total duration comes from a prompt-to-text length ratio rather than a learned predictor, so robustness to unusual speaking rates is unknown.
- Code and checkpoints are announced but not yet released.
Wiki Connections
- Flow Matching — Uses conditional flow matching with x-prediction and a velocity-space loss directly on raw waveform patches, and proposes a velocity-aware weighting for auxiliary perceptual losses.
- Zero-Shot TTS — Evaluated on cross-sentence prompt-based voice cloning of unseen speakers on Seed-TTS test-en and LibriSpeech-PC test-clean.
- Self-Supervised Speech — Uses frozen WavLM-base+ features as a training-only representation-alignment target for the generator’s hidden states.
- F5-TTS — Supplies the infilling formulation, evaluation protocol and sway sampling; retrained on the same data as the main mel-space baseline, which BareWave beats on WER but trails on UTMOS.
- E2 TTS — Retrained same-data mel-space baseline; ahead of BareWave on LibriSpeech-PC SIM-o, behind on WER and UTMOS.
- CosyVoice 2 — Larger-data reference system run from its public checkpoint; still ahead of BareWave on SIM-o and UTMOS.
- WaveFM — Source of the multi-resolution refined STFT distance that VAPA reweights.
- A-DMA — Earlier training-time representation alignment for TTS whose alignment-head design BareWave follows.
- Vocos — The vocoder used by the mel-space baselines, the stage BareWave removes.
- Emilia — Source of the 19.4k-hour English training subset shared by all controlled systems.
- WavTTS — Concurrent raw-waveform flow-matching zero-shot TTS that this paper does not cite or compare against; both target vocoder-free generation and both rely on objective evaluation.