ACL · 2026 · Conference
Chunyu Qiang et al. (Tianjin University / Kuaishou Technology / Institute of Automation, Chinese Academy of Sciences) · → Paper · → Preprint · Demo: ✓ · Code: ✗
Extends a speech-and-music instruction-driven flow-matching MM-DiT to sound effects by feeding it duration-scaled runs of a learnable [SFX] token in place of phonemes, trained with a speech-then-music-then-effects curriculum, so that one 1.34B model generates speech, music, and sound effects from reference-free natural-language instructions.
Problem
Text-to-speech, text-to-music, and text-to-audio models are trained separately and controlled through different interfaces: TTS systems usually need reference audio for timbre and phoneme-aligned text, music models rely on lyrics or tags, and sound-effect models take open-ended captions. Earlier unified models either still depend on reference audio (Vevo2, CosyVoice for speech and singing) or use inconsistent per-task input formats or task-specific fine-tuning (UniAudio, Audiobox). The authors’ own predecessor, InstructAudio, unified speech and music under one instruction-plus-phoneme interface but left out sound effects. The difficulty the paper names is structural: speech and music are tied to a discrete unit sequence (phonemes or lyrics) that fixes their timing, while sound effects have no transcript and no fixed temporal units. The authors argue that naively mixing all three data types causes negative transfer, with the variability of environmental sounds harming speech articulation.
Method
UniSonate is a dual-stream Multimodal Diffusion Transformer (MM-DiT) trained with a conditional flow-matching objective: the network regresses the velocity between Gaussian noise and the clean latent along a linear interpolation path, and at inference latents are produced by integrating the predicted field with an Euler ODE solver (§3.1, Eq. 1). Audio at 44.1kHz is compressed by a pre-trained Mel-VAE into continuous latents at 43 Hz (1024x downsampling), and these latents form the audio stream. The model has about 1.34B parameters: 14 Joint Diffusion Transformer layers, in which text and audio tokens each pass through their own self-attention and then share a joint attention over concatenated queries, keys, and values, followed by 6 Single Diffusion Transformer layers that refine the audio stream alone. RoPE provides positional information (§3.1, §4.2).
The text stream follows what the paper calls the Instruction-Content Alignment paradigm, which extends InstructAudio’s instruction-plus-phoneme format (§3.2). The conditioning sequence is the concatenation of an instruction embedding and a content embedding. A frozen Qwen2.5-7B encodes the natural-language instruction: speaker gender, age, emotion, style, and accent for speech, with a separate description per speaker and [S0]/[S1] speaker-ID tokens in the content for two-speaker dialogue; genre, instrumentation, tempo, mood, and vocal characteristics for music; an event or scene description for sound effects. For speech and music, the content is the phoneme sequence of the transcript or lyrics (from a G2P model), encoded by a 512-dimensional Zipformer-based content encoder. No reference audio is used for any modality.
The new component is dynamic token injection for sound effects, which have no transcript. The content sequence becomes repeated copies of a learnable [SFX] token, with the count set to the floor of the target duration multiplied by a global factor: the average number of phonemes per second in the speech corpus (§3.2, Eq. 2-3). The authors chose repeated tokens over a single duration embedding so that the token sequence has a physical length matching the audio, giving joint attention a temporal anchor that mimics the monotonic phoneme-to-audio alignment of speech. This lets one architecture with one attention mechanism handle all three modalities and sets sound-effect duration through the token count.
![Figure 2: The overall architecture of UniSonate. The framework employs a dual-stream MM-DiT based on conditional flow matching. The input follows the Instruction-Content Alignment paradigm, unifying natural language instructions with content sequences, utilizing phonemes for speech/music and special token Injection (via learnable [SFX] tokens) for sound effects. These semantic conditions interact with acoustic latents (compressed by a Mel-VAE) through Joint Diffusion Transformer layers to enable unified audio generation.](../papers/assets/2604.22209/figure-2.png)
Training uses a three-stage curriculum (§3.3, Algorithm 1): one epoch on speech only, two more epochs on speech plus music, then speech, music, and sound effects together until convergence. The data are 50K hours of speech (balanced 1:1 Chinese/English and by gender, including 0.5% dialogue data) and 20K hours of music, both internet-sourced and the same as InstructAudio’s, plus 1.5M newly added sound-effect clips. All clips are 2 to 20 seconds at 44.1kHz, with instructions produced by an internal processing pipeline (§4.1). Training ran on 32 A800 GPUs with Adam at a learning rate of 1e-4 (§4.2).
Key Results
Speech intelligibility (Seed-TTS test set, Table 3). UniSonate has the lowest WER among the systems compared, at 1.47% EN and 1.25% ZH. The other systems score: InstructAudio 1.52/1.35, ZipVoice 1.70/1.40, F5-TTS 1.89/1.53, CosyVoice2 2.57/1.45, MaskGCT 2.26/2.40, E2-TTS 2.49/1.91, CosyVoice1 4.29/3.63. The ZH figure equals the ground-truth WER (1.25), and the EN figure is below it (2.14). The protocol differs by system: UniSonate is driven by neutral text descriptions with randomized speakers, while the reference-based baselines clone a prompt speaker (Appendix A.1). The WER numbers therefore compare intelligibility across different generation settings.
Instruction-based speech control (Table 2; control accuracy from human annotators on 500 internal instruction-phoneme pairs). Against InstructAudio, UniSonate ties on gender (100%), age (86.67%), and accent (100%), is higher on dialogue (93.33% vs 90.00%), and is lower on emotion (80.00% vs 83.33%) and style (80.00% vs 86.67%). CosyVoice2 scores 58.33% on emotion and 65.00% on style; it has no gender, age, or dialogue control and gets reference audio plus mapped control tags. UniSonate has the best speaker similarity (0.77 vs 0.76 InstructAudio, 0.68 CosyVoice2), but its emotion similarity is below InstructAudio’s (0.67 vs 0.71; CosyVoice2 0.53). It is best on all four distortion metrics: LSD 1.79, MCD 5.46, MSEP 422.36, and MR 0.31, against InstructAudio’s 1.88/5.71/437.58/0.33 and CosyVoice2’s 2.57/7.11/547.87/0.46. On listener scores, CosyVoice2 is higher on both QMOS (3.90 vs 3.83) and NMOS (3.65 vs 3.50). UniSonate is above InstructAudio on both (3.73 QMOS, 3.46 NMOS). The authors call the NMOS “comparable” and attribute CosyVoice2’s lead to its reference-audio guidance.
Music (internal 500-sample test set, Table 4). UniSonate scores highest on all five SongEval dimensions: coherence 3.18, musicality 3.07, memorability 3.10, clarity 2.99, and naturalness 2.90. InstructAudio scores 3.08/2.98/3.00/2.89/2.82, ACE-Step 2.89/2.87/2.83/2.77/2.71, and DiffRhythm+ lower still. UniSonate also has the highest musicality MOS (3.01 vs 2.91 InstructAudio, 2.88 ACE-Step, 2.79 DiffRhythm+). Control accuracy is mixed:
- Behind ACE-Step: genre (93.89% vs 94.44%) and instrument (85.00% vs 85.56%).
- Tied with InstructAudio: vocal gender (98.89%).
- Highest: vocal age (97.78%).
- Behind InstructAudio: rhythm (93.33% vs 94.44%, tied with DiffRhythm+) and atmosphere (94.44% vs 95.00%).
On music quality MOS, UniSonate (2.88) is behind ACE-Step (3.30) and DiffRhythm+ (3.04) and above InstructAudio (2.82).
Sound effects (AudioCaps test, Table 5). UniSonate does not lead on any metric. Its FAD is 4.21, behind GenAU-L (2.07), Tango-FT (2.68), EzAudio-XL (3.64), and Stable Audio (4.19) and ahead only of AudioLDM-L (4.32). Its CLAP score (0.156) and KL (2.44) are the worst of the six systems. Its FD (30.21) is better only than Stable Audio’s (39.14), and its IS (8.22) is higher only than AudioLDM-L’s (8.17). The authors describe this as competitive fidelity for a unified model.
Joint vs single-domain training (Tables 6-7). The same architecture trained on speech only is worse on every TTS metric than the jointly trained model: WER-EN 2.24% vs 1.47%, WER-ZH 1.40% vs 1.25%, speaker similarity 0.63 vs 0.77, emotion similarity 0.51 vs 0.67, and MCD 8.70 vs 5.46. A music-only variant scores below the joint model on every SongEval dimension, by 0.06 to 0.07 (for example, coherence 3.11 vs 3.18). The TTS-only variant’s EN WER (2.24%) is also well above InstructAudio’s (1.52%), which was trained on speech and music without sound effects. The ablation therefore contrasts speech-only training with three-domain training and does not isolate the contribution of the sound-effect data.
Novelty Assessment
The main new mechanism is dynamic token injection: the content sequence for a transcript-free modality is a run of learnable [SFX] tokens whose length is set by the target duration and the speech corpus’s phoneme rate, so sound effects enter a phoneme-driven MM-DiT without architectural changes and with explicit duration control. The rest of the system is inherited from InstructAudio: the MM-DiT backbone, the frozen LLM instruction encoder, the instruction-plus-phoneme format, the speech and music data, and the 1.3B scale. The three-stage curriculum is a training-recipe addition. The contribution is therefore an incremental extension of a prior unified speech-and-music model to a third modality. The paper also reports cross-domain positive transfer, but its evidence is a speech-only vs. three-domain comparison, with no ablation of the token-injection design itself or of the curriculum ordering.
Field Significance
Moderate — UniSonate demonstrates that one reference-free, instruction-driven flow-matching MM-DiT can cover speech (Chinese and English, including two-speaker dialogue), music, and sound effects. Relative to its speech-and-music predecessor, adding sound effects coincided with better WER, distortion, and SongEval scores in the authors’ evaluation, though emotion and style control accuracy dropped. Its sound-effect quality trails dedicated text-to-audio models on every reported metric. The paper’s most reusable data point for speech generation is that training on music and sound effects alongside speech coincided with better intelligibility and spectral fidelity than speech-only training at identical architecture. This is a single controlled comparison rather than a mechanism-level account.
Claims
- supports: Training one generative model on speech jointly with other audio domains can improve speech intelligibility and fidelity over training the same architecture on speech alone.
Evidence: With architecture and hyperparameters held fixed, the jointly trained model improves on its speech-only counterpart in WER-EN (2.24% to 1.47%), WER-ZH (1.40% to 1.25%), speaker similarity (0.63 to 0.77), and MCD (8.70 to 5.46). A music-only variant likewise trails the joint model on all five SongEval dimensions. (§4.3.4, Tables 6-7)
- supports: Audio without a transcript can be integrated into a phoneme-conditioned non-autoregressive generator by replacing the phoneme sequence with a run of placeholder tokens whose count encodes the target duration.
Evidence: Learnable [SFX] tokens, repeated in proportion to target duration times the corpus-average phoneme rate, let sound effects share the MM-DiT’s joint attention without architectural change. The model then generates AudioCaps sound effects (FAD 4.21, CLAP 0.156), though below dedicated text-to-audio models. (§3.2, §4.3.3, Table 5)
- complicates: Unifying speech, music, and sound effects in one model does not yet match specialist quality on the unstructured modality.
Evidence: On AudioCaps, the unified model’s FAD (4.21) is worse than GenAU-L (2.07), Tango-FT (2.68), EzAudio-XL (3.64), and Stable Audio (4.19), and its CLAP score (0.156) and KL (2.44) are the worst of the six systems compared. (§4.3.3, §Limitations, Table 5)
- complicates: Reference-free, instruction-only speech generation can trail reference-guided systems on listener-rated quality and naturalness even when it leads on objective intelligibility and distortion metrics.
Evidence: The instruction-only model has lower WER and distortion than reference-prompted CosyVoice2 but lower QMOS (3.83 vs 3.90) and NMOS (3.50 vs 3.65). The authors attribute this to one-to-many ambiguity in text descriptions. (§4.3.1, §Limitations, Table 2)
- complicates: Adding a new generation domain to a unified instruction-controlled model can trade off some fine-grained speech attribute control while improving other speech metrics.
Evidence: Compared with its speech-and-music predecessor, the three-domain model improves dialogue control (93.33% vs 90.00%) and all distortion metrics but loses emotion accuracy (80.00% vs 83.33%), style accuracy (80.00% vs 86.67%), and emotion similarity (0.67 vs 0.71). (§4.3.1, Table 2)
Limitations and Open Questions
The two named contributions are not ablated
No experiment compares dynamic token injection with an alternative such as a single duration embedding, removes the curriculum, or varies its stage order. The positive-transfer evidence contrasts speech-only training with speech, music, and sound-effect training. It cannot separate the effect of the added sound-effect data from that of the music data that InstructAudio already used. The speech-only variant’s EN WER (2.24%) is also well above InstructAudio’s (1.52%).
- Sound-effect quality: sound effects trail specialized text-to-audio models (FAD 4.21 vs 2.07 for GenAU-L), and the authors suggest the unified representation may not capture the diversity of unstructured acoustic scenes (§Limitations).
- Clip length: training and evaluation cover only 2 to 20 second clips. Long-form output (full songs, audiobooks) is out of reach, and there is no hierarchical long-term planning (§Limitations).
- Instruction-only ambiguity: relying on text instructions alone leaves a one-to-many ambiguity, which the authors link to a naturalness gap against reference-conditioned systems (§Limitations).
- Inference cost: as a 1.3B model with multi-step ODE sampling, UniSonate is computationally heavy at inference and not suited to real-time, low-latency synthesis (§Limitations).
- WER protocol: Seed-TTS WER for UniSonate uses neutral descriptions with randomized speakers, while the baselines clone reference speakers. The WER comparison therefore spans different task settings (Appendix A.1).
- Listener pool and test sets: speech MOS raters were 20 native Chinese speakers rating outputs that include English (Appendix A.3). The instruction-control and music test sets are internal and not released (Appendix A.4).
- Sound-effect duration control: the paper presents duration control as a benefit of token injection but reports no measurement of duration accuracy for generated sound effects.
- Release status: training data are internal, and model weights are described only as planned for future release (§Ethical Considerations).
Wiki Connections
- Flow Matching — UniSonate is an MM-DiT trained with a conditional flow-matching objective and sampled with an Euler ODE solver, here extended to speech, music, and sound effects in one model.
- Instruction-Conditioned TTS — Speech is controlled only through natural-language descriptions of gender, age, emotion, style, accent, and per-speaker dialogue roles, with human-judged control accuracy reported for each attribute.
- Emotion Synthesis — Emotion is an instructed attribute evaluated by listener control accuracy (80.00%) and emotion2vec similarity, where UniSonate trails its predecessor.
- Multilingual TTS — Trained on balanced Chinese/English speech and evaluated separately on the Seed-TTS English and Chinese test sets.
- Subjective Evaluation — Reports QMOS, NMOS, and MMOS from 20 listeners alongside human-annotated control-accuracy tests for speech and music.
- InstructAudio — The direct predecessor, which unified speech and music under an instruction-plus-phoneme MM-DiT; UniSonate extends it to sound effects via dynamic token injection and evaluates it as the main unified baseline.
- CosyVoice 2 — Instruction-control baseline given reference audio and mapped control tags; it leads UniSonate on QMOS and NMOS but trails on control accuracy, WER, and distortion.
- F5-TTS — Flow-matching zero-shot TTS baseline in the Seed-TTS WER comparison (1.89/1.53 vs UniSonate’s 1.47/1.25).
- ZipVoice — Flow-matching TTS baseline (WER 1.70/1.40), and the work cited for the Zipformer-based design of UniSonate’s phoneme content encoder.
- E2 TTS — Non-autoregressive flow-matching TTS baseline in the Seed-TTS WER comparison.
- Seed-TTS — Provides the English and Chinese test sets used for UniSonate’s intelligibility evaluation.
- Audiobox — Earlier flow-matching model for speech, music, and sound that the authors contrast as lacking a single consistent instruction-only interface.
- AudioLDM — Latent-diffusion text-to-audio baseline in the sound-effect comparison, the only system UniSonate beats on FAD (4.21 vs 4.32).