arXiv · 2026 · Preprint

Xingwei Sun et al. (Xiaomi Inc. / Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓

Replaces the frozen text encoder and fixed-length diffusion transformer of a unified speech/music/sound-effect generator with a fully fine-tuned LLM that drives a per-token flow-matching head, cutting Seed-TTS English WER from 12.15% to 2.79% while keeping mixed-scene generation competitive.

Problem

Unified audio-scene generators that produce overlapping speech, music and sound effects from text have poor speech intelligibility. The direct predecessor from the same group, Dasheng AudioGen, conditions a non-autoregressive diffusion transformer on a frozen text encoder. That design fixes the output length, restricts it to one language, and gives 12.15% English WER and over 100% Chinese CER on Seed-TTS (§1, Table 4). Task-unified models such as UniAudio and UniFlow-Audio cover several audio types but generate each type separately, so they cannot render scenes where speech overlaps music and effects (§2.2). The paper argues that the limits come from the generation backbone, not from the structured captions or the latent space, and keeps those two parts.

Method

MiDashengLM-Gen casts scene generation as autoregressive conditional generation over continuous audio latents (§3). The audio representation is the encoder of a custom MiDashengLM-0.6B model, built on the DashengTokenizer architecture. It produces 768-dimensional semantic-acoustic latents at 25 Hz. An MLP adapter groups five consecutive frames into one 5 Hz token (200 ms) projected into the 2048-dimensional space of the LLM (§3.1, §3.3). Decoding goes back through the DashengTokenizer decoder: a 5× convolutional upsampler followed by a Vocos vocoder at 16 kHz. The encoder, adapter, upsampler and decoder are frozen during generation training.

The backbone is Qwen3-1.7B with full fine-tuning. Its input is a structured multi-view caption followed by <|audio_bos|> and the audio token sequence (§3.1, Eq. 2). The caption has six views, each marked by a special token: global description, speech transcript, speaker identity and style, sound effects, music content and recording environment. Views that do not apply are filled with <|unknown|>. Pure TTS data receives the fixed global caption “natural human speech” (§4.1). One LLM therefore encodes both the text and the audio history, and no separate text encoder or cross-attention is used.

Before generation training, an audio-text alignment stage trains the adapter and LLM to caption audio from DashengTokenizer latents, following the MiDashengLM recipe (§3.1, Figure 2). This maps audio latents into the LLM token space.

At each audio position, a 16-layer DiT (hidden size 2048, 8 heads, RoPE self-attention) generates the five 25 Hz frames of that token (§3.2, §3.3). It is conditioned on the LLM hidden state, the timestep and the five frames of the previous token. The head is trained with a conditional flow-matching loss on linear noise-to-data interpolation (§3.2). At inference it runs 10 Euler steps with classifier-free guidance (scale 2.0) against a learned null condition. A stop head on the LLM hidden states decides the output length. It is trained with balanced cross-entropy and hard-negative mining, weighted λ = 0.01 against the flow-matching loss, and fires at probability 0.5 after at least 1 second (§3.2, §3.3). Training data combine a private 77k-hour superset of ACAVCaps (speech, music, sound effects and their mixtures, with speech in many languages) with TTS corpora: Chinese and English Emilia, LibriTTS, LJSpeech, AISHELL-3 and WenetSpeech4TTS (§4.1, Table 2). The TTS portion is only English (11,074 h) and Chinese (11,949 h).

Key Results

Speech intelligibility (Seed-TTS, Table 4). English WER is 2.79% and Chinese CER is 3.87%. Dasheng AudioGen scores 12.15% and >100% on the same sets. Dedicated TTS systems remain clearly ahead: Qwen3-TTS reaches 1.24%/0.77% and Seed-TTS 2.25%/1.12% (§4.3). The paper describes a “2.2×” English gap to Qwen3-TTS (2.79/1.24 ≈ 2.25). The comparison is not like for like. Seed-TTS is a voice-cloning benchmark, and the paper states that MiDashengLM-Gen supports only coarse speaker-style control, with no voice cloning or explicit speaker-identity conditioning (§5). The paper does not say how the prompt audio was handled and reports no speaker similarity.

Multilingual speech (MiniMax-Speech test set, Table 5). WER/CER is 2.42% English, 3.51% Chinese, 3.11% Spanish, 4.38% German, 5.49% Korean, 8.04% Portuguese, 12.53% French, 13.19% Russian and 16.45% Japanese, for a mean of 7.68% (Table 9). MiniMax-Speech is lower in all nine languages. ElevenLabs is lower in eight; MiDashengLM-Gen beats it only on Chinese (3.51% vs. 16.03%). The abstract calls these WERs “highly competitive”, but the table shows it trails both commercial systems in almost every language. Its clear gain is over Dasheng AudioGen, which reaches 99.62% Chinese and 109.21% Japanese error. Speech for languages other than English and Chinese comes only from the ACAVCaps speech subset. The errors there are higher, which matches the data distribution in Table 2 (§4.3).

Emotion (CV3-Eval, Table 6). emotion2vec classification accuracy is compared with F5-TTS, CosyVoice 3-1.5B and CosyVoice3 DiffRO-EMO. In the text-related setting, MiDashengLM-Gen has the highest sad (0.96 vs. 0.68 next) and angry (0.92 vs. 0.84) accuracy, and ties DiffRO-EMO on happy (0.98). In the text-unrelated setting, DiffRO-EMO leads on happy (0.98 vs. 0.78) and angry (0.68 vs. 0.62). MiDashengLM-Gen is marginally highest on sad (0.52 vs. 0.50). The paper does not state which input carries the target emotion (text content, the speech-style caption view, or a reference). The text-related advantage may therefore reflect the LLM reading emotional text, not a controllable emotion mechanism.

Mixed scenes (MECAT, Tables 7-8). Speech-only clips (S00) have the lowest FAD of all five systems (0.87 vs. 1.76 for Dasheng AudioGen) and the highest CLAP (0.35 vs. 0.34). Dasheng AudioGen keeps the better FD (3.93 vs. 4.15) and KL (0.40 vs. 0.44). The same pattern holds in every speech-containing mixed category (S0A, SM0, SMA): FAD is lowest of all systems (1.54, 0.98, 1.88 vs. 1.75, 1.70, 2.17 for Dasheng AudioGen), but FD and KL are worse than Dasheng AudioGen (§4.4).

Non-speech audio (Table 3, Table 7). On AudioCaps the model trails TangoFlux (FAD 5.01 vs. 2.26) and Dasheng AudioGen (3.19). On MusicCaps it has the lowest FD (14.58) and KL (1.24) of the systems compared, and ties Dasheng AudioGen on CLAP (0.33).

Ablations. Removing the audio-text alignment stage raises Seed-TTS English WER from 2.79% to 12.17%, Chinese CER from 3.87% to 14.17%, and mean multilingual WER from 7.68% to 31.73%. Non-speech FAD gets 47-88% worse (§4.5, Table 9). In single-sample overfitting tests at a fixed DiT parameter budget, the DiT converges only when its width is larger than the audio latent dimension. This holds for 768-dim latents with Qwen3 1.7B and 0.6B (Figure 3, Table 10). With 1280-dim latents, however, width 1280 already converges at both 0.6B and 4B, and only width 1248 fails (Table 10). The table therefore supports “at least the latent dimension” in that setting, not the paper’s stated rule that width must strictly exceed it.

Novelty Assessment

The main contribution is a backbone swap inside an existing pipeline. The structured multi-view captions, the DashengTokenizer latent space and decoder, and the training data come from the group’s earlier Dasheng AudioGen, DashengTokenizer and ACAVCaps work. The new parts are a pre-trained LLM, a per-token DiT flow-matching head and a learned stop head. Pairing an autoregressive LM with a per-step continuous diffusion or flow head is already used in speech synthesis (for example DiTAR). The paper neither cites nor compares against such systems, so its “first approach” claim holds only for mixed speech/music/sound-effect scenes. The audio-text alignment pre-stage follows the MiDashengLM captioning recipe. Its ablation is the paper’s most useful evidence: it shows how much intelligibility depends on aligning continuous audio latents with the LLM token space. The width-versus-latent-dimension finding is a single-sample overfitting study that extends a result from image diffusion. Its own Table 10 shows the threshold is “at least equal” in two of three settings. All speech comparisons are objective (WER/CER, emotion2vec), with no listening test and no speaker similarity.

Field Significance

Moderate — The paper shows that an LLM-driven, variable-length continuous-latent generator can bring unified speech/music/sound-effect generation close to dedicated TTS intelligibility in English and Chinese. Its clearest transferable evidence is the ablation showing that audio-text alignment is the main lever for speech intelligibility in such systems. Speaker control, voice cloning and perceptual quality are not addressed.

Claims

  • supports: In LLM-driven generation over continuous audio latents, pre-aligning the audio latents with the LLM’s token space before generation training is critical for intelligible speech.

    Evidence: Skipping the captioning-based alignment stage (random adapter, original Qwen3 weights) raises Seed-TTS English WER from 2.79% to 12.17%, Chinese CER from 3.87% to 14.17%, and mean multilingual WER from 7.68% to 31.73%. Non-speech FAD degrades by 47-88%, a proportionally smaller loss. (§4.5, Table 9)

  • supports: Replacing a frozen text encoder with a jointly fine-tuned LLM backbone substantially improves speech intelligibility in unified speech/music/sound-effect generation.

    Evidence: With the same captions and latent space as Dasheng AudioGen, the LLM plus per-token flow-matching model lowers Seed-TTS English WER from 12.15% to 2.79% and Chinese CER from >100% to 3.87%. Multilingual errors fall from 99.62% to 3.51% (Chinese) and 109.21% to 16.45% (Japanese). (§4.3, Tables 4-5)

  • complicates: Unified audio-scene generators still trail dedicated TTS systems on intelligibility, and the gap widens for languages with little speech training data.

    Evidence: Seed-TTS English WER is 2.79% vs. 1.24% for Qwen3-TTS. On the 9-language set the model trails MiniMax-Speech in every language and ElevenLabs in all but Chinese. Errors reach 13.19% Russian and 16.45% Japanese, languages absent from the English/Chinese-only TTS corpora. (§4.3, Tables 2, 4-5)

  • refines: A flow-matching decoder over high-dimensional continuous audio latents needs a transformer width at least as large as the latent dimension to converge.

    Evidence: In single-sample overfit tests at a fixed DiT parameter budget, 768-dim latents converge only at widths 784 and 1024 (not 752 or 768). 1280-dim latents converge from width 1280 upward at both Qwen3 0.6B and 4B, and fail only at 1248. (§4.5, Figure 3, Table 10)

  • complicates: In mixed-audio generation, gains in Fréchet Audio Distance for speech-containing scenes do not carry over to feature-level distribution metrics.

    Evidence: On MECAT speech-containing categories (S00, S0A, SM0, SMA), the autoregressive model has lower FAD than the non-autoregressive Dasheng AudioGen. Dasheng AudioGen keeps lower FD and KL in each of those categories. (§4.4, Tables 7-8)

Limitations and Open Questions

Warning

The model has no voice cloning or explicit speaker-identity conditioning (only coarse speaker-style control through the caption), and the paper reports no speaker similarity or human listening test. Its Seed-TTS and CV3-Eval comparisons are against voice-cloning systems on cloning benchmarks, evaluated with WER/CER and emotion2vec only.

  • Output length is bounded by the 1-20 s training clips; longer coherent scenes are untested (§5).
  • Speech intelligibility outside English and Chinese is weak (8-16% for Portuguese, French, Russian and Japanese), and the paper attributes this to TTS data limited to those two languages (§4.3, Table 2).
  • Speech-category MECAT results use only the English subset (§4.1). Multilingual mixed-scene quality is not measured objectively.
  • The emotion evaluation does not state which input carries the target emotion, so it is unclear whether the speech-style caption view provides controllable emotion or prosody.
  • The DiT width finding is based on single-sample overfitting runs. Whether the threshold affects full-scale training quality, and not only convergence, is untested.
  • The main training corpus is a private ACAVCaps superset, which limits reproducibility despite the released code and checkpoints.
  • Output is 16 kHz, set by the frozen DashengTokenizer decoder.

Wiki Connections

  • Flow Matching — A per-token DiT head trained with conditional flow matching on linear noise-to-data paths generates each 200 ms latent group, conditioned on the LLM hidden state and sampled with 10 Euler steps and CFG.
  • Multilingual TTS — Trained on speech in many languages (English/Chinese TTS corpora plus multilingual ACAVCaps speech) and evaluated on 9 languages, where intelligibility tracks training-data coverage.
  • Emotion Synthesis — Emotional speech is evaluated by emotion2vec accuracy on CV3-Eval. The model leads on text-related sad and angry speech but trails a dedicated emotion system when emotion is not cued by the text.
  • DashengTokenizer — Same Xiaomi group. The paper uses a frozen encoder built on the DashengTokenizer architecture (from a custom MiDashengLM-0.6B) as its 25 Hz continuous latent space, with the DashengTokenizer decoder for waveform output.
  • Qwen3 — The Qwen3-1.7B LLM is the fully fine-tuned autoregressive backbone; Table 10 also uses Qwen3 0.6B and 4B for the DiT width study.
  • Qwen3-TTS — Dedicated TTS reference on Seed-TTS (1.24%/0.77%) that MiDashengLM-Gen trails by about 2.2× in English, and an MECAT baseline that performs poorly on non-speech categories.
  • Seed-TTS — Provides the English/Chinese intelligibility test sets and is a dedicated-TTS reference row (2.25%/1.12%).
  • MiniMax-Speech — Provides the 9-language test set and is the multilingual baseline that is lower in every language.
  • CosyVoice 3 — Provides the CV3-Eval emotion benchmark, and its 1.5B and DiffRO-EMO variants are the emotion baselines; DiffRO-EMO leads when emotion is not cued by the text.
  • F5-TTS — Flow-matching TTS baseline on CV3-Eval emotion accuracy, trailing MiDashengLM-Gen on text-related sad and angry speech.
  • emotion2vec — Used as the emotion classifier for the CV3-Eval accuracy metric.