arXiv · 2025 · Preprint
Chunyu Qiang et al. (Tianjin University / Kuaishou Technology) · → Paper · Demo: ✓ · Code: ?
Introduces a unified diffusion-transformer framework that conditions both text-to-speech and text-to-music generation on natural language instructions describing timbre, paralinguistic, and musical attributes, removing the reference-audio dependency that both task families otherwise require for attribute control.
Problem
TTS systems typically depend on reference audio for timbre control, expose only limited text-level attribute control, and rarely support multi-speaker dialogue synthesis. Text-to-music (TTM) systems face a parallel limitation: their input conditioning depends on expert-annotated musical descriptors, and control over singer timbre (gender, age) is often absent. Because TTS control conditions (timbre, paralinguistics) and TTM control conditions (genre, instrumentation, rhythm) are heterogeneous, the two tasks have historically been modeled with separate architectures despite sharing the same underlying acoustic generation problem. Prior attempts at a unified audio generation model fall short in one of several ways: Vevo2 unifies speech and singing but still relies on reference audio for attribute control and has no instrumental-music capability; UniAudio requires task-specific input formats and per-task fine-tuning; AudioBox pre-trains across speech, music, and sound effects but only supports speech and sound-effect generation at inference; AudioLDM 2 spans speech, sound, and music but needs different architecture hyperparameters per task. No prior system achieves natural-language, instruction-only control unified across both speech and music generation.
Method
InstructAudio uses a multimodal diffusion transformer (MM-DiT) architecture, adapting the Stable Diffusion 3 design (with modeling inspiration from MM-Audio) to audio generation. The model has two component types: joint diffusion transformer layers, where text and audio modalities interact through joint attention (queries, keys, and values from both modalities are concatenated and processed with scaled dot-product attention, then split back into their respective modalities), and single diffusion transformer layers, which process only audio latents through self-attention to further refine speech and singing quality. The full model comprises 14 joint layers and 6 single layers, with RoPE positional encoding and a flow-matching feedforward dimension of 1024, totaling 1.34B parameters.
The text modality input is built from a standardized instruction-phoneme format: a natural-language instruction description specifying desired acoustic attributes (for speech: gender, age, emotion, style, accent; for music: singer gender/age, genre, instrumentation, melody, emotion), temporally concatenated with a phoneme sequence obtained via grapheme-to-phoneme conversion of the input text or lyrics. For two-speaker dialogue, separate instruction descriptions are provided per speaker, with special tokens [S0]/[S1] prepended to each speaker’s phoneme span to distinguish turns. This identical input structure is used for both the TTS and TTM tasks, which is what allows a single model to be trained jointly across both.
Audio is represented as continuous latents from a pretrained Mel-VAE that extends the authors’ prior SecoustiCodec framework; the mel encoder, mel decoder, and instruct encoder (Qwen2.5-7B) remain frozen during MM-DiT training, while a Zipformer-based phoneme encoder (feedforward dim 512) is trained alongside the diffusion transformer. Training uses conditional flow matching: the model learns a velocity field regressed against the target vector field along a linear interpolation path between Gaussian noise and the VAE latents, optimizing E‖v_θ(t, C_text, x_t) − u(t, x_t)‖². At inference, an ODE solver integrates the learned velocity field to produce the target VAE latents, which are then decoded to audio. Training used 50K hours of speech and 20K hours of music collected from internet sources with an internal annotation pipeline generating instruction descriptions and text/lyrics labels; clips are capped at 5–20 seconds, held to roughly 1:1 Chinese-English and male-female ratios, with over 90% neutral-emotion samples and 0.5% dialogue data, all standardized to 44.1kHz. Training ran on 32 NVIDIA A800 80GB GPUs, batch size 16 per GPU, Adam optimizer, initial learning rate 1e-4.
Key Results
On the Seed-TTS benchmark WER comparison (Table 1), InstructAudio reports the lowest word error rate among all compared TTS systems (1.52% English, 1.35% Mandarin), ahead of ZipVoice (1.70%/1.40%), F5-TTS (1.89%/1.53%), CosyVoice2 (2.57%/1.45%), E2-TTS (2.49%/1.91%), and MaskGCT (2.26%/2.40%), while being the only system among them to support gender/age control, emotion/style/accent control, and dialogue generation simultaneously; CosyVoice1 and CosyVoice2 support only emotion/style/accent control and neither supports dialogue. InstructAudio reaches this while training on the smallest combined dataset (50K speech + 20K music hours) versus 100K–170K speech-only hours for the baselines.
Against CosyVoice2, the strongest instruction-controllable TTS baseline (Table 2), InstructAudio wins on every classification control accuracy category it was tested on (age 86.67% vs. not supported, emotion 83.33% vs. 58.33%, style 86.67% vs. 65%, accent 100% vs. 100%, dialogue 90% vs. not supported), and is the only system tested that supports gender and dialogue control at all. It also achieves higher speaker similarity (0.76 vs. 0.68) and emotion similarity (0.71 vs. 0.53), and lower distortion/error scores across LSD (1.88 vs. 2.57), MCD (5.71 vs. 7.11), MSEP (437.58 vs. 547.87), and voiced/unvoiced mismatch rate (0.33 vs. 0.46). CosyVoice2 achieves a higher subjective MOS (QMOS 3.90 vs. 3.73, NMOS 3.65 vs. 3.46), which the authors attribute to CosyVoice2’s use of reference audio (an additional input modality InstructAudio does not receive).
On the TTM side (Table 3), InstructAudio achieves the best SongEval scores across all five dimensions (Coherence 3.08, Musicality 2.98, Memorability 3.00, Clarity 2.89, Naturalness 2.82) against DiffRhythm+ and ACE-Step, and the best control accuracy for gender (98.89%), age (97.22%), rhythm (94.44%), and atmosphere (95%), while ACE-Step remains stronger on genre (94.44% vs. 92.78%) and instrument (85.56% vs. 83.89%) control and achieves a higher QMOS (3.30 vs. 2.82); InstructAudio has the highest MMOS (2.91). The authors note this TTM comparison constrains all systems to 5–20 second clips to match speech durations for joint training, which disadvantages DiffRhythm+ (optimized for full-length song generation and requiring truncation of its outputs, since it does not natively support sub-90-second synthesis).
Novelty Assessment
The MM-DiT architecture itself is not novel; it is a direct adaptation of Stable Diffusion 3’s diffusion transformer design (with stated inspiration from MM-Audio) to the audio domain, and the flow matching training objective is a standard application of conditional flow matching. The genuine contribution is the standardized instruction-phoneme input format that lets one model jointly serve TTS and TTM without task-specific input schemas, architecture hyperparameters, or fine-tuning stages, combined with removing reference-audio conditioning for timbre and paralinguistic attribute control. This combination has not been demonstrated by prior unified-audio systems: UniAudio still needs inconsistent per-task formats and fine-tuning, AudioBox does not extend to music, and AudioLDM 2 needs per-task architecture changes. The contribution is best characterized as an architectural-integration and first-of-kind capability demonstration rather than a new generative modeling technique. Evaluation is reasonably broad (both objective distortion/similarity metrics and professionally rated subjective scores across TTS and TTM), but the instruction-based TTS and TTM test sets (500 samples each, 100 subjective) are self-constructed and not independently released or verifiable, and the TTM comparison protocol explicitly disadvantages a baseline optimized for a different generation length regime.
Field Significance
Tip
High — the paper demonstrates that a single instruction-conditioned MM-DiT model can serve both TTS and TTM with reference-audio-free attribute control, achieving the strongest reported control accuracy and WER among compared instruction-controllable TTS systems while remaining competitive on music generation quality. This establishes a shared conditioning framework for text-instructed audio generation across speech and music that prior unified systems have not achieved simultaneously.
Claims
-
supports: Natural-language instruction conditioning can control timbre and paralinguistic attributes of synthesized speech at accuracy competitive with or exceeding reference-audio-dependent methods, without requiring a reference audio input.
Evidence: Against CosyVoice2 (which needs reference audio for gender/age control), InstructAudio achieves 100% gender and 86.67% age classification control accuracy using text-only instructions, and outperforms CosyVoice2 on emotion (83.33% vs. 58.33%) and style (86.67% vs. 65%) control accuracy that CosyVoice2 does support. (§3.3, Table 2)
-
complicates: Conditioning speech synthesis on text-only instructions rather than reference audio introduces one-to-many mapping ambiguity that reduces perceived naturalness relative to audio-prompted systems.
Evidence: InstructAudio’s NMOS (3.46) trails CosyVoice2’s NMOS (3.65) despite InstructAudio outperforming CosyVoice2 on nearly every objective control, similarity, and distortion metric; the authors attribute the naturalness gap to the information loss inherent in text-only conditioning versus reference-audio conditioning. (§3.3, §3.4, Table 2)
-
supports: A single instruction-conditioned architecture can jointly serve speech and music generation at competitive quality in both domains, without requiring task-specific architectural branches or per-task fine-tuning.
Evidence: Trained jointly on 50K hours of speech and 20K hours of music using one standardized instruction-phoneme input format and one MM-DiT model, InstructAudio achieves the best SongEval scores across all five evaluated dimensions against dedicated TTM systems (DiffRhythm+, ACE-Step) while simultaneously achieving the lowest Seed-TTS WER among compared TTS systems, despite using less total training data than the largest single-task baselines. (§3.3, Tables 1 and 3)
-
complicates: Constraining audio clip length to enable joint training across generation domains with different natural durations limits the applicability of the resulting model to long-form generation in either domain.
Evidence: To align speech and music training, clips are capped at 5–20 seconds; this disadvantages the TTM comparison against DiffRhythm+, which is optimized for full-length song generation and does not natively support outputs under 90 seconds, requiring the authors to generate longer sequences and truncate them for the evaluation, which they acknowledge “may introduce evaluation bias.” (§3.2, §3.4)
Limitations and Open Questions
Warning
Both training corpora (50K hours of speech, 20K hours of music) are internally collected from internet sources through a proprietary annotation pipeline; neither dataset is named, released, or documented beyond aggregate hour counts and coarse language/gender ratios, so the training data cannot be independently inspected or reproduced.
Text-only instruction conditioning trades naturalness for controllability and audio-input independence: the paper reports lower NMOS than reference-audio-conditioned CosyVoice2 in TTS and lower QMOS than ACE-Step in TTM, both attributed to the one-to-many ambiguity of text-only control. The 5–20 second clip constraint adopted for joint speech-music training limits long-form music generation and is explicitly acknowledged as disadvantaging the DiffRhythm+ comparison. Dialogue generation is validated only through a single classification control accuracy figure (90%), without evaluation of turn-taking naturalness, inter-speaker prosody coherence, or longer multi-turn exchanges. The Latent Audio Codec (Mel-VAE) is used as a frozen, unanalyzed component; the paper explicitly defers analysis of its effect on synthesis quality to a prior technical report rather than examining it here. No code release is indicated; only audio samples are provided via a demo page.
Wiki Connections
- Instruction-Conditioned TTS — introduces a standardized instruction-phoneme input format that conditions TTS output attributes (timbre, emotion, style, accent) directly on natural language descriptions, without reference audio.
- Flow Matching — trains an MM-DiT architecture with a conditional flow matching objective, adapting the Stable Diffusion 3 rectified-flow recipe to joint speech and music latent generation.
- Multilingual TTS — trains and evaluates the same instruction-conditioned model jointly across English and Chinese, reporting per-language WER on the Seed-TTS benchmark.
- Emotion Synthesis — controls emotion as one of several instruction-conditioned attributes, measuring both emotion classification accuracy and emotion similarity against ground truth.
- Subjective Evaluation — reports professionally-rated QMOS, NMOS, and MMOS scores alongside human classification-accuracy listening tests for both the TTS and TTM outputs.
- F5-TTS — used as a non-instruction-controllable TTS baseline in the Seed-TTS WER comparison.
- CosyVoice 2 — the primary instruction-controllable TTS baseline, compared head-to-head on control accuracy, similarity, distortion, and MOS metrics.
- CosyVoice — used as a TTS baseline supporting only emotion/style/accent control without gender, age, or dialogue control.
- ZipVoice — used as a non-instruction-controllable TTS baseline in the Seed-TTS WER comparison.
- MaskGCT — used as a non-instruction-controllable TTS baseline in the Seed-TTS WER comparison.
- Vevo2 — discussed as the closest prior unified speech/singing framework, contrasted for its dependence on reference audio and lack of instrumental music generation.
- SecoustiCodec — the prior codec framework that this paper’s Mel-VAE latent audio codec extends.
- UniAudio — discussed as a prior unified audio generation framework, contrasted for requiring inconsistent per-task input formats and task-specific fine-tuning.