arXiv · 2026 · Preprint

Zhaoqing Li et al. (The Chinese University of Hong Kong / The Hong Kong Polytechnic University / City University of Hong Kong / The Hong Kong University of Science and Technology / Tsinghua University / Huawei Research Hong Kong) · → Paper · Demo: ✓ · Code: ?

A single 621M to 732M parameter flow-matching MM-DiT, conditioned on layer-wise hidden states of a frozen Qwen2.5-Omni-7B and operating in a frozen general-audio VAE latent, handles bilingual TTS, zero-shot voice cloning, instruction-based gender control and speech-in-scene editing alongside text-to-audio and audio editing, with no phoneme front-end, duration predictor or speaker encoder.

Problem

Unified audio generators that cover both speech and general sound still route tasks through heterogeneous side modules: G2P phoneme front-ends or character encoders for TTS, separate mel encoders for reference or source audio, duration predictors, and distinct conditioning streams for editing versus generation. The authors argue this fragments the latent space and limits cross-task transfer. A second issue they target is shallow text conditioning: most systems feed one final-layer text embedding (T5, CLAP or an LLM) identically to every DiT block, which may underserve compositional prompts that combine speaker attributes, transcript, background events and timestamps. For speech specifically, the question is whether plain-text transcripts read by a frozen multimodal LLM, without phonemes or a duration model, can support intelligible TTS and zero-shot cloning inside a general-audio model.

Method

Every task shares one network and one forward pass. Target audio, source/edit audio and speaker reference audio are all encoded by the same frozen MMAudio continuous VAE (mel-spectrogram based; 40 latent channels at 44.1 kHz or 20 at 16 kHz). The DiT input is a channel concatenation of the noised target latent, a source latent (zeros for pure generation) and a one-channel per-frame task tag: 0 for generation (text-to-audio, TTS, mixed speech-and-sound, timed composition), 1 for editing (source is the pre-edit mix), and 2 for zero-shot TTS (source holds the reference prefix, and the tag separates reference frames from frames to synthesize) (§3.3). The input embedder’s weights for the source and tag channels are zero-initialized so early training behaves like plain denoising (§3.4). There is no duration head; latents are fixed-length and padded, and trailing silence is learned implicitly.

The backbone is a flow-matching MM-DiT of double-stream blocks (d = 1024, 8 heads; 20 blocks in the D20 variant, 24 in D24). Text conditioning comes from a frozen Qwen2.5-Omni-7B Thinker run once on the instruction; hidden states from uniformly spaced LLM layers (28 layers mapped onto 20 or 24 blocks) are linearly projected and fed to the matching DiT block, so shallow blocks see shallow LLM layers (“layer-wise deep LLM fusion”, adapted from text-to-image work). Text tokens are refreshed per block rather than carried through a text MLP; audio and text tokens interact via joint attention (§3.4). TTS transcripts are given as plain text inside instruction templates such as [Speech] A {gender} voice saying "{text}" or [Speech with voice] {text} (Appendix B, Table 10).

Training uses a masked flow-matching loss that zeroes gradients on the reference prefix in zero-shot TTS, 10% condition dropout for classifier-free guidance, and an online GPU-side synthesis pipeline that builds editing and mixing pairs on the fly from raw clips. A two-stage curriculum trains generation tasks only for the first 150K steps, then adds editing (roughly 70% generation, 30% editing), with task-homogeneous mini-batches; in Stage 2 zero-shot TTS and gender-controlled TTS are sampled with probabilities 0.25 and 0.15 (§3.6, §3.7, Appendix D). Data totals about 36M clips (about 57K hours), of which about 50.7K hours are speech from LibriTTS, WenetSpeech and Emilia EN/ZH subsets (Appendix C). Inference uses a 100-step Euler ODE solver with CFG scale 4.5. Trainable sizes are 621M (D20, 44.1 kHz) and 732M (D24, 16 kHz) (§4.1).

Key Results

TTS on Seed-TTS test (Table 2). Intelligibility is measured with Whisper-large-v3 (EN WER) and Paraformer (ZH CER). D24 has the lowest error in all four columns of the table: pure (instruction-only) WER 1.27 vs Audio-Omni 1.35 and UniSonate 1.47; zero-shot WER 1.50 vs ZipVoice 1.70, Audio-Omni 1.77 and F5-TTS 1.83; pure CER 0.92 vs UniSonate 1.25; zero-shot CER 0.89 vs ZipVoice 1.40 and CosyVoice 2 1.45. The D20 (44.1 kHz) variant is weaker on cloning: its zero-shot WER of 1.80 trails ZipVoice (1.70) and Audio-Omni (1.77) and is essentially level with F5-TTS (1.83), and its zero-shot CER of 1.71 trails ZipVoice (1.40), CosyVoice 2 (1.45) and F5-TTS (1.56) while beating E2 TTS (1.97) and MaskGCT (2.27). The specialist baselines are much smaller in some cases (ZipVoice 123M, F5-TTS 336M) and the paper does not say whether baseline numbers were rerun or copied from prior reports.

Warning

Zero-shot cloning is evaluated by WER/CER only: no speaker similarity, no MOS and no listening test are reported for any speech task, so the cloning results show intelligibility, not voice fidelity. The authors themselves note that the general-audio VAE may smooth speaker timbre in zero-shot TTS.

Instruction-based gender control (Table 3). On 300 balanced bilingual prompts, both variants reach 100% gender accuracy (wav2vec2-XLSR classifier) from text instructions alone, with WER 1.21 / CER 1.00 for D24 and 1.47 / 1.02 for D20. D24’s errors are uneven by gender (WER 1.64 male vs 0.74 female).

Speech in mixtures and speech editing (Tables 4, 6). Generating speech plus background from one instruction yields WER 2.04 / CER 3.64 (D24) and 3.44 / 5.80 (D20), measured on the unseparated mixture. In speech-in-scene editing, D24 removes speech with a 99.16% removal rate (D20 95.72%) and keeps WER at 1.35 (insert) and 0.98 (rewrite). No baseline is evaluated on these speech tasks, since the authors report no public benchmark for them.

General audio as context (Tables 1, 5). On AudioCaps text-to-audio, D24 has the lowest FAD in the table (1.558 vs GenAU-L 1.591) and D20 the lowest FD (15.82) and highest IS (12.04, vs MMAudio-L 11.98); GenAU-L has higher CLAP (0.561 vs 0.503), and on KL both UNISON variants (1.459, 1.455) trail GenAU-L (1.290, the lowest), Tango (1.305), Audio-Omni (1.337) and MMAudio-L (1.421). On constructed non-speech editing, D24 has the lowest FD and highest CLAP in every sub-task, though LSD on the Remove sub-task is lower for ZETA (2.09) than for either UNISON variant.

Ablations (Table 8, 80K steps). Deep fusion only (D24-O) gives FD 20.46 / CLAP 0.180 / pure EN WER 4.33, against 22.71 / 0.175 / 4.44 for penultimate-layer-only conditioning, which also has more parameters (975M). Adding both paths (D24-OL) gives the best FD and CLAP (20.18, 0.187) but WER rises to 5.52. A single-stream variant (WER 4.84) and a 3B Qwen encoder (WER 5.61) are both worse. The WER gap between O and L is small (0.11 points) and results are single runs.

Novelty Assessment

The contribution is primarily engineering integration. Layer-wise LLM-to-DiT fusion is taken from text-to-image work (cited by the authors), flow-matching infilling with a masked reference prefix follows the Audiobox / E2 TTS line, and the MM-DiT backbone and VAE come from MMAudio. What is new is putting these together so that bilingual zero-shot TTS, instruction-based voice attributes, speech-in-scene editing and general audio generation share one VAE latent, one forward pass and one checkpoint, with a single mask channel encoding the task and no phoneme or duration modules. The speech evidence is intelligibility-only; the claim that “multi-task training does not degrade TTS quality” is not tested against a single-task model trained on the same data, so it rests on cross-paper comparison with specialists of differing size and data.

Field Significance

Moderate. The paper shows that a plain-text, LLM-conditioned flow-matching model working in a general-audio VAE latent, without G2P or duration modelling, can reach Seed-TTS WER/CER at or below published specialist TTS figures while also doing sound generation and speech editing. Because speaker similarity and human ratings are absent, it is evidence about intelligibility in unified audio models rather than about cloning quality.

Claims

  • supports: Plain-text transcripts encoded by a large frozen LLM can replace phoneme front-ends and explicit duration models in non-autoregressive flow-matching TTS without loss of intelligibility.

    Evidence: With transcripts given only as plain text to a frozen Qwen2.5-Omni-7B and fixed-length padded latents, the D24 model reaches Seed-TTS zero-shot WER 1.50 (EN) and CER 0.89 (ZH), below G2P- or duration-based systems such as MaskGCT (2.62 / 2.27) and F5-TTS (1.83 / 1.56) as reported in the same table. (§3.3, §4.3.2, Table 2)

  • supports: Injecting hidden states from multiple depths of a frozen LLM into matching layers of a diffusion transformer conditions audio and speech generation better than broadcasting a single late-layer embedding.

    Evidence: At 80K steps, per-block deep fusion (D24-O, 732M) gives FD 20.46, CLAP 0.180 and pure EN WER 4.33, against 22.71, 0.175 and 4.44 for penultimate-layer conditioning (975M); replacing the 7B LLM with a 3B one raises WER to 5.61. The WER gain over single-layer conditioning is small and from single runs. (§4.4, Table 8)

  • complicates: Text-conditioning choices that improve semantic alignment for general audio can degrade speech intelligibility in a shared model.

    Evidence: Combining deep fusion with a persistent penultimate-layer text stream (D24-OL) gives the best ablation FD (20.18) and CLAP (0.187) but raises pure EN WER from 4.33 to 5.52. (§4.4, Table 8)

  • complicates: Unified audio models that encode speech through a general-audio latent may trade sample-rate fidelity against cloning intelligibility, and intelligibility-only evaluation leaves their speaker fidelity unverified.

    Evidence: The 44.1 kHz D20 variant has higher zero-shot error than the 16 kHz D24 variant (WER 1.80 vs 1.50, CER 1.71 vs 0.89) and trails ZipVoice and CosyVoice 2 on zero-shot CER; no speaker similarity or listening test is reported, and the authors state the MMAudio VAE, designed for environmental sound, may smooth speaker timbre in zero-shot TTS. (§4.3.2, Table 2, §Limitations)

Limitations and Open Questions

  • No speaker-similarity, MOS or other human evaluation is reported for TTS, zero-shot cloning, gender control or speech editing; all speech results are ASR-based intelligibility, gender-classifier accuracy, VAD removal rate, CLAP and LSD.
  • The frozen MMAudio VAE was designed for environmental sound; the authors acknowledge it caps speech fidelity (formants, prosodic nuance, breathy or whispered voice) and may smooth timbre in cloning.
  • D20 and D24 differ in both depth and sample rate/VAE, so the paper’s attribution of D20’s weaker TTS to capacity and 44.1 kHz difficulty is confounded.
  • There is no single-task TTS control trained on the same data, so the statement that multi-task training does not hurt TTS is not directly tested.
  • Speech-in-scene editing, mixed speech-and-sound generation and gender-controlled TTS are evaluated on constructed sets built by algorithmic mixing, with no baselines; real scenes with reverberation or Lombard effects are not covered.
  • Speech support is limited to English and Chinese, and voice control by instruction is demonstrated only for gender; richer style instructions (emotion, age, speaking rate) are not evaluated.
  • Code release is not mentioned; only a demo page is provided.

Wiki Connections

  • Flow Matching — UNISON trains a flow-matching MM-DiT with a masked loss and 100-step Euler inference as the single backbone for speech, sound and editing.
  • Zero-Shot TTS — evaluated on Seed-TTS prompt-based cloning in English and Chinese via a VAE-encoded reference prefix, reporting intelligibility but no speaker similarity.
  • Multilingual TTS — one model trained on English and Chinese speech (LibriTTS, WenetSpeech, Emilia) and evaluated on both Seed-TTS test sets.
  • Instruction-Conditioned TTS — natural-language instructions set speaker gender with 100% classifier accuracy and drive speech insertion, deletion and rewriting in scenes.
  • Seed-TTS — supplies the EN/ZH test sets used for all TTS and cloning results and the speech for constructed mixing and editing sets.
  • F5-TTS — flow-matching specialist baseline that UNISON D24 beats on zero-shot WER/CER while the D20 variant is roughly level on English and behind on Chinese.
  • ZipVoice — strongest specialist zero-shot baseline in the table; trails D24 but leads D20 on both zero-shot WER and CER.
  • CosyVoice 2 — zero-shot baseline that trails D24 but beats D20 on Chinese zero-shot CER.
  • MaskGCT — phoneme-based zero-shot baseline with higher WER/CER than both UNISON variants.
  • E2 TTS — filler-token flow-matching baseline with higher zero-shot error than both UNISON variants.
  • InstructAudio — unified speech and music instruction model used as a pure-TTS baseline.
  • Qwen2.5-Omni — its frozen 7B Thinker provides the layer-wise text conditioning for every task.