arXiv · 2026 · Preprint
Guanrou Yang et al. (Shanghai Jiao Tong University) · → Paper · Demo: ? · Code: ✓
Turns frozen WavLM-Large features into a compact 128-dim, 50Hz continuous latent through a two-stage “compress-then-enrich” recipe. The same latent supports SUPERB probing, waveform reconstruction, and F5-TTS-style flow-matching TTS, where it beats Mel, VAE, and Semantic-VAE latents in a controlled substitution.
Problem
Speech understanding and speech generation run on different continuous representations. SSL features (wav2vec 2.0, HuBERT, WavLM) carry semantic structure but drop the fine acoustic detail a vocoder needs. Reconstruction-trained latents (Mel-spectrograms, speech VAEs) keep that detail but have little semantic structure, and they suffer a reconstruction-generation dilemma: widening the latent helps reconstruction and hurts diffusion. Prior unification attempts each leave a gap, by the authors’ account. Semantic-VAE and JMAS-VAE remain reconstruction-dominated or depend on fragile loss weighting. DashengTokenizer and SemanticVocoder keep the full encoder dimensionality, which is hard to diffuse. Ming-UniAudio needs an extra semantic module to lift its low-dimensional latent for understanding. The paper also diagnoses two obstacles in raw WavLM features (§5). A DiT trained directly on 1024-dim WavLM-Large features collapses (WER 110.28%). Decoding WavLM features yields poor reconstruction (SIM 0.67, STOI 0.85).
Method
WavCube is built on WavLM-Large, using its last (24th) hidden layer as the source feature at 50Hz.
Stage 1 (semantic feature compression). With WavLM frozen, a symmetric adapter auto-encoder is trained. The compressor is 3 Transformer layers, initialised from WavLM’s first three layers, followed by a 2-layer MLP (intermediate width 576, GELU) projecting to 128 dims, an 8x channel compression. The restorer mirrors it and lifts the latent back to 1024 dims. The semantic loss is MSE plus cosine distance between restored and original WavLM features. At the same time, an acoustic decoder is warmed up on the detached latent, so its gradient never shapes the latent. The decoder uses the MiMo-Audio tokenizer’s Transformer audio decoder and vocoder design (MiMo-Audio): 317M parameters, a 24-layer AudioDecoder plus a 16-layer TransformerVocos with an ISTFT head producing 16kHz audio. It is trained with Mel, adversarial, and feature-matching losses from multi-period and multi-resolution discriminators, following Vocos (λ_mel:λ_adv ratio 45:1). The bottleneck is a plain auto-encoder, not a VAE. An appendix ablation (Table 7, run with a lightweight 19.3M decoder) compares it with a KL-VAE and the σ-VAE of LatentLM. The AE gives WER 2.09 / SIM-o 0.660, the VAE 2.36 / 0.667, and the σ-VAE 4.49 / 0.658.
Stage 2 (joint semantic-acoustic enrichment). WavLM is unfrozen and the whole pipeline (encoder, compressor, decoder) is trained end-to-end on the acoustic reconstruction loss. A semantic anchoring regulariser (λ_sem = 1.0) ties two things to a frozen copy of WavLM: the fine-tuned encoder’s features (feature-level) and the restorer’s output from the compressed latent (reconstruction-level). Two variants are pre-trained: WavCube on LibriSpeech 960h and WavCube-Pro on 6,000h (LibriSpeech plus Libriheavy small/medium).

Downstream TTS. To compare representations, the F5-TTS v1 Base DiT is retrained on each latent (hidden 1024, 22 layers, 337.2M parameters), changing only the target representation (F5-TTS). It uses the standard text-conditioned flow-matching setup with audio-prompt infilling for zero-shot synthesis. Understanding and the SUPERB-SG generative tasks (SE, SS, VC) are evaluated with frozen upstream features and lightweight task heads.
Key Results
Reconstruction (Table 1, LibriSpeech test-clean). WavCube reaches STOI 0.97, UTMOS 4.04, SIM 0.94, and WER 4.20%. WavCube-Pro reaches 0.97 / 4.00 / 0.95 / 4.12%. Both trail the VAE and Semantic-VAE baselines on every column (0.98 / 4.13 / 0.97 / 4.07%). Against Mel-spectrograms (0.98 / 3.63 / 0.93 / 3.86%), WavCube is better on UTMOS and SIM and worse on STOI and WER. “On par” is fair as a description of a small gap, but no column in Table 1 has WavCube ahead of the VAE baselines.
Understanding (Table 2, SUPERB). WavCube beats the Fbank, VAE, and Semantic-VAE latents by wide margins on every task. For example, PR PER is 9.91 against 83.71 to 88.53, and IC accuracy is 90.41 against roughly 10. How close it gets to WavLM-Large depends on the task. It is close on KS (97.42 vs 98.12), ASV EER (5.86 vs 4.93), and ER (63.41 / 66.27 for Pro vs 70.05). It is far behind on SID (42.36 vs 93.78), PR (9.91 vs 3.23), and ASR WER (9.36 vs 3.70). The authors call the change from the semantic-only Stage 1 to Stage 2 “negligible”. Stage 2 loses ground on PR, IC, ER, ASR (6.91 → 9.36 WER), QbE, SF, and SD, and gains on KS (96.73 → 97.42), SID (38.20 → 42.36), and ASV (7.35 → 5.86). The 960h to 6,000h scaling is also mixed. WavCube-Pro improves PR, ER, ASR, QbE, and SD but is slightly worse on KS, IC, SID, SF, and ASV, which does not support “improvements across most tasks”.
Zero-shot TTS (Table 3, LibriSpeech-PC test-clean). With the DiT held fixed, WavCube gives the best WER and SIM-o in both training regimes:
- LibriTTS: WER 1.86 / SIM-o 0.678, against Mel 2.02 / 0.598, VAE 2.10 / 0.593, and Semantic-VAE 2.25 / 0.626.
- Emilia ZH-EN: WavCube-Pro gets 2.20 / 0.709, against Mel 2.29 / 0.628, VAE 2.47 / 0.673, and Semantic-VAE 2.35 / 0.706. The SIM-o margin over Semantic-VAE here is 0.003.
In the system-level comparison (Table 4), WavCube-Pro (2.20 / 0.709) beats the reported numbers for F5-TTS (2.42 / 0.66), E2 TTS (2.95 / 0.69), CosyVoice (3.59 / 0.66), and FireRedTTS (2.69 / 0.47). All baseline numbers are copied from the F5-TTS paper. The authors’ own Mel reproduction of F5-TTS is stronger (2.29 / 0.628), so the like-for-like WER gain is 0.09 rather than 0.22. Training curves (Figure 2) show faster convergence of WER and SIM-o for WavCube, and faster convergence for both semantic latents (WavCube, Semantic-VAE) than for Mel and VAE. This is shown only as curves.
SUPERB-SG (Table 5). On SE and SS, Fbank is slightly ahead of WavCube (PESQ 2.11 vs 2.08, SI-SDRi 9.75 vs 9.20). WavCube-Pro ties Fbank on SE STOI (86.2), and both WavCube variants clearly beat VAE and Semantic-VAE. On VC, WavCube-Pro has the best MCD among the acoustic latents and final WavCube variants (8.43) and a much lower WER than the acoustic latents (18.7 vs 32.6 to 40.1), while Fbank has a slightly higher ASV score (72 vs 71). WavLM-Large (MCD 7.65, WER 9.8, ASV 96) is better on all three VC metrics, and so is the semantic-only Stage 1 latent (MCD 7.26, WER 11.0, ASV 100). So the VC advantage comes from the WavLM semantics, and Stage 2 enrichment partly erodes it.
Ablation of the two stages (Table 6). Raw WavLM-Large under a 338.7M DiT collapses (WER 110.28%, SIM-o 0.09). Widening the DiT to 753.5M recovers intelligibility (WER 3.38%) but not speaker identity (SIM-o 0.27). Stage 1 compression alone gives WER 2.24% / SIM-o 0.32, even though its reconstruction is worse than raw WavLM (SIM 0.54 vs 0.67). The full two-stage WavCube reaches 1.86% / 0.68.
Novelty Assessment
The contribution is a training recipe for a representation, not a new generator. The TTS model is an unmodified F5-TTS DiT, and the decoder follows MiMo-Audio’s tokenizer design. The recipe has two parts. The first compresses SSL features through a learned bottleneck before any acoustic enrichment. The second fine-tunes the SSL encoder end-to-end on reconstruction while anchoring both its features and the restored bottleneck features to a frozen copy. Each part has precedents: the paper cites PS-VAE, RePack, and FAE as vision versions of the bottleneck idea, and the anchoring loss resembles Semantic-VAE’s alignment. The speech-specific combination and the diagnosis in §5 are the new parts. That diagnosis shows that compression alone fixes diffusability (but not speaker fidelity), and enrichment alone is not tested. The comparisons against acoustic latents in Table 3 are well controlled. They omit the closest unified-latent competitors the paper discusses (JMAS-VAE, DashengTokenizer, Ming-UniAudio), so the claim of advantage over the unified-representation line rests on Semantic-VAE alone.
Field Significance
Moderate. This work gives speech-domain evidence that an SSL-derived continuous latent can be made diffusion-friendly by compressing it and then re-injecting acoustic detail under a semantic anchor. In the TTS backbone tested, the resulting latent beats Mel and VAE latents on WER and speaker similarity while keeping much of WavLM’s SUPERB discriminability. The Table 6 ablation is a useful data point on its own: it separates two failure modes of raw SSL features as generative targets, dimensional redundancy and missing acoustic detail.
Claims
- supports: Semantically structured continuous latents are easier targets for diffusion and flow-matching TTS than purely reconstruction-trained acoustic latents, producing better intelligibility and speaker similarity and faster convergence at fixed model size.
Evidence: With the same 337M F5-TTS DiT, the WavCube latent gives WER 1.86 / SIM-o 0.678 on LibriTTS, against 2.02 / 0.598 for Mel and 2.10 / 0.593 for a VAE latent. Convergence curves show both semantic latents (WavCube, Semantic-VAE) converging faster than Mel and VAE. (§4.3.2, Table 3, Figure 2)
- supports: High-dimensional raw SSL features are poor direct targets for diffusion-based speech generation, and compressing them through a learned low-dimensional bottleneck restores generative tractability.
Evidence: A 338.7M DiT on 1024-dim WavLM-Large features collapses (WER 110.28%, SIM-o 0.09), and a 753.5M DiT only reaches WER 3.38% / SIM-o 0.27. A 128-dim Stage 1 bottleneck with a 335.9M DiT reaches WER 2.24% while reconstructing worse than raw WavLM. (§5, Table 6)
- refines: For SSL-derived generative latents, intelligibility and speaker fidelity depend on different properties: compression of the semantic space fixes intelligibility, while speaker similarity requires explicit acoustic-detail injection through end-to-end reconstruction.
Evidence: Stage 1 compression alone gives TTS SIM-o 0.32 and reconstruction SIM 0.54. Adding Stage 2 encoder fine-tuning with reconstruction loss and semantic anchoring raises these to 0.68 and 0.94 and lowers TTS WER from 2.24% to 1.86%. (§5, Table 6)
- complicates: Injecting acoustic detail into an SSL-derived latent trades away some content and semantic discriminability, so a single latent does not fully reach SSL-level understanding performance.
Evidence: Compared with the Stage 1 latent, the final WavCube has worse SUPERB ASR WER (6.91 → 9.36), PR PER (8.68 → 9.91), and IC accuracy (91.58 → 90.41), and worse SUPERB-SG VC WER (11.0 → 24.9) and ASV (100 → 67). It remains far below WavLM-Large on SID (42.36 vs 93.78). (§4.2.2, Table 2; §4.4.2, Table 5)
- complicates: Choosing a speech-latent dimension trades intelligibility against speaker similarity in generation.
Evidence: Reducing the WavCube latent from 128 to 64 dims lowers TTS WER (2.09 → 1.98) but reduces SIM-o (0.660 → 0.581), and halving the frame rate to 25Hz worsens both (2.36 / 0.638). (§Appendix B, Table 7)
Limitations and Open Questions
Warning
No human listening tests are reported anywhere, and generated TTS speech is scored only by WER and speaker similarity on one English test set (LibriSpeech-PC test-clean). There is no UTMOS or naturalness measure for synthesised speech. This leaves the “state-of-the-art zero-shot TTS” claim resting on two automatic metrics, with system-level baselines copied from the F5-TTS paper.
- Data exposure in the comparisons. The Table 3 ”# Recon. Data” column (960h for WavCube vs 6,000h for VAEs) understates WavCube’s data exposure, because WavCube starts from a pre-trained WavLM-Large whose own pre-training data is not counted. The Emilia comparison uses WavCube-Pro at 6,000h, which equalises the reconstruction data.
- Missing comparisons. None of the unified-latent competitors discussed in §2.1 (JMAS-VAE, DashengTokenizer, SemanticVocoder, Ming-UniAudio) appears in any table.
- Narrow language and rate coverage. Output is 16kHz only. Reconstruction and understanding are evaluated on English only, and although the large-scale TTS model trains on Chinese and English, Chinese output is never evaluated.
- Unmeasured streaming behaviour. The acoustic decoder uses causal Transformer layers, but streaming behaviour and latency are not measured.
- Unaddressed degradations. The Stage 2 losses on SUPERB content tasks and on SUPERB-SG VC (see Key Results) are not discussed, even though they bear directly on how unified the latent is.
- Unverified claims. The appendix ablations use a 19.3M decoder, not the 317M main decoder, so their absolute numbers are not comparable to the main tables. The “unified model” motivation (one latent consumed by a joint understanding and generation model) is stated as future work and not demonstrated.
Wiki Connections
- Neural Audio Codec — WavCube is a continuous speech tokenizer (encoder, bottleneck, and acoustic decoder) evaluated on reconstruction against Mel, VAE, and Semantic-VAE latents.
- Self-Supervised Speech — The representation is derived from WavLM-Large, which is fine-tuned in Stage 2 while anchored to a frozen copy.
- Flow Matching — The downstream TTS model is a flow-matching DiT retrained on each candidate latent to compare representations.
- Zero-Shot TTS — Gives controlled zero-shot TTS evidence that the target-latent choice alone shifts WER and speaker similarity.
- GAN Vocoder — The acoustic decoder is a Transformer Vocos-style ISTFT vocoder trained with multi-period and multi-resolution adversarial and feature-matching losses.
- F5-TTS — Provides the unmodified DiT backbone and training recipe, the Mel-spectrogram baseline (reproduced), and the system-level baseline numbers.
- Semantic-VAE — The closest baseline: a semantically aligned VAE latent that WavCube beats on TTS but trails on reconstruction.
- JMAS-VAE — A prior unified speech VAE criticised for relying on fragile dynamic loss weighting, discussed but not compared experimentally.
- DashengTokenizer — An inverted semantic-encoder-plus-acoustic-projection design criticised for keeping full encoder dimensionality, discussed but not compared.
- Ming-UniAudio — A unified-representation speech LLM whose low-dimensional latent needs an extra semantic module for understanding, which motivates WavCube’s single shared latent.
- MiMo-Audio — Source of the Transformer audio decoder and vocoder architecture used as WavCube’s acoustic decoder.
- Emilia — Provides the ~95k h Chinese-English training set for the large-scale TTS comparison.
- E2 TTS — Same-data-scale (95k h) flow-matching baseline in the system-level comparison, using reported numbers.