arXiv · 2026 · Preprint
Junjie Zheng et al. (Giant Network / Shanghai Conservatory of Music) · → Paper · Demo: ✓ · Code: ✗
Trains one 0.3B conditional flow-matching DiT for both zero-shot speech and singing synthesis. The condition is split into content, melody and timbre streams, and speech replaces the MIDI melody stream with a learned null token, so melody constrains singing without forcing a pitch contour onto speech.
Problem
TTS and singing voice synthesis (SVS) are usually built as separate systems. Speech prosody is inferred from linguistic context, while singing needs explicit pitch and rhythm control (MIDI notes or F0). Training one model on mixed speech and singing data with a single undifferentiated condition forces melody-related inputs to mean different things in the two modes: a hard constraint for singing, an unnecessary restriction for speech. The authors argue this produces gradient interference on melody-related parameters. The prior unified system they compare against (Vevo2) uses autoregressive modeling with explicit F0 conditioning, which they describe as trading natural speech prosody against singing controllability (§2).
Method
UniVoice is a conditional flow-matching (CFM) generator under the optimal-transport path. It operates on continuous 48-dimensional SongBloom VAE latents at 25 Hz, and the same VAE decodes them back to 48 kHz audio (§3.1, §3.2). The backbone is a Diffusion Transformer adapted from the Wan-DiT video architecture (24 layers, hidden size 1024, 16 heads, about 0.3B / 329M parameters, RoPE and FlashAttention-2). The authors picked this backbone so the audio model can plug into Wan-based audio-visual pipelines (§3.2, Table 3).
The condition is factorized into four parts (§3.3):
- Content: IPA phoneme sequences, shared across speech and singing, encoded by a 4-layer ConvNeXt as in F5-TTS.
- Melody: for singing, a 6-layer Conformer turns MIDI note sequences (pitch plus duration) into frame-level embeddings. For speech, a single learned null embedding is broadcast across time.
- Timbre: a 3-10 s reference clip is encoded by the same VAE and placed before the target along the time axis. Speaker identity is picked up through self-attention (in-context prompting, following E2 TTS), with no speaker embedding.
- Task token: a learned speech/singing embedding added to the timestep embedding. It drives AdaLN scale and shift at every layer, so there are no task-specific heads.
The noisy latent and the condition streams are concatenated along the channel axis and linearly projected into the DiT; there is no cross-attention (§3.2). The paper gives two theoretical arguments. First, routing melody gradients only from singing samples removes a negative cross-task gradient inner product on the melody subspace (Proposition 1). Second, at convergence the learned null token approximates the melody-marginalized, Bayes-optimal speech velocity field (Proposition 2). The authors also read the scheme as axis-selective classifier-free guidance (§3.4, Appendix A).
Training uses about 65k hours: 30k hours of speech, a balanced Emilia subset of 15k Chinese plus 15k English, and 35k hours of singing. The singing comes from in-the-wild songs (vocals isolated by source separation) and studio dry recordings, with lyrics aligned by ASR and MIDI extracted by a pre-trained model. Batches mix speech and singing at their natural ratio of about 46/54. Text, melody and prompt are each dropped with p = 0.1 for CFG. Training runs 100 epochs on 8 A800 GPUs (§3.5). Inference uses a 32-step Euler solver with text guidance 5 and audio/melody guidance 1 (§3.5).
The paper also introduces UniSinging-Eval: 60 songs in 12 styles (Pop, Hip-Hop/Rap, Rock, Jazz, Bossa Nova, Afrobeats and others), giving 900 phrase-level clips (about 2 h of song audio). The songs were re-sung by humans, and professional composers manually corrected the MIDI. Lyrics come in three rewrite levels: original, extensively rewritten, and elastic (±5 words) (§4, Appendix C, Table 4).
Key Results
All numbers are from Table 1. Speech is evaluated on the Emilia test set with a cross-sentence protocol. Singing uses MIDI-conditioned cross-sentence generation; the singing split is presumably UniSinging-Eval, though Table 1 does not name it. Human ratings use 20 listeners per sample on a 1-5 scale (§5.1).
| System | Speech PER ↓ | Speech SIM ↑ | Speech S-MOS ↑ | Speech N-MOS ↑ | Singing PER ↓ | Singing SIM ↑ | Singing S-MOS ↑ | Singing N-MOS ↑ |
|---|---|---|---|---|---|---|---|---|
| F5-TTS (0.3B, 100k h speech) | 5.21 | 72.73 | 3.85 | 3.41 | - | - | - | - |
| CosyVoice3 (0.5B, 1M h speech) | 5.30 | 74.94 | 3.75 | 3.19 | - | - | - | - |
| Vevo1.5 (1B, 101k + 7k h) | 14.10 | 59.07 | 2.85 | 2.77 | 45.07 | 36.33 | 2.62 | 2.69 |
| Soul-X-Singer (0.7B, 42k h singing) | - | - | - | - | 26.22 | 41.71 | 3.19 | 3.24 |
| UniVoice (0.3B, 30k + 35k h) | 5.26 | 67.42 | 3.76 | 3.07 | 16.22 | 35.70 | 3.19 | 3.25 |
- Speech: UniVoice’s PER (5.26%) sits between F5-TTS (5.21%) and CosyVoice 3 (5.30%), and is much lower than Vevo1.5 (14.10%). On every other speech metric it trails the dedicated TTS systems: SIM is 67.42 against 72.73 and 74.94, and N-MOS is 3.07 against 3.41 and 3.19. Its speech S-MOS (3.76) is second to F5-TTS (3.85). The text calls 3.76 the “highest speech naturalness score”, which Table 1 does not support, both because F5-TTS is higher and because S-MOS is the similarity MOS.
- Singing: UniVoice has the lowest singing PER (16.22% vs. 26.22% for SoulX-Singer and 45.07% for Vevo1.5). Singing N-MOS is 3.25 vs. 3.24, effectively level with SoulX-Singer. Singing S-MOS ties SoulX-Singer at 3.19. On singing SIM it ranks third (35.70, behind SoulX-Singer’s 41.71 and Vevo1.5’s 36.33). The singing advantage is therefore concentrated in intelligibility.
- Ablations (Table 2): Removing factorized conditioning causes the largest drop: speech PER 5.26 to 12.31, singing PER 16.22 to 23.45, speech N-MOS 3.07 to 2.31. Removing the task token gives speech PER 8.34 and singing PER 19.92. Replacing the learned null melody token with a fixed zero vector gives speech PER 7.86 and singing PER 18.64. Removing the melody encoder barely changes speech (PER 6.27) but hurts singing badly (PER 23.21, S-MOS 2.03).
Caveats on fairness and reporting. Table 1 labels the unified baseline “Vevo1.5” but cites the Vevo2 paper. The abstract and introduction give its singing PER as 24.72%, while Table 1 reports 45.07%. The full-model row in Table 2 differs slightly from Table 1 (speech S-MOS 3.74 vs. 3.76, singing N-MOS 3.23 vs. 3.25). Baselines differ widely in training data and size. No results are broken down by UniSinging-Eval difficulty level, by style, or by language, and no variance or significance tests are reported.
Novelty Assessment
The generator itself is established: CFM on VAE latents with a DiT, in-context audio prompting in the style of E2 TTS, and a ConvNeXt text encoder in the style of F5-TTS. The new part is the conditioning design for joint speech/singing training. It has separate condition streams, a learned null melody token for speech in place of an explicit F0/MIDI constraint, and an AdaLN task token. Learned null embeddings are familiar from classifier-free guidance; applying one per condition axis to separate modalities is a sensible, modest extension. The ablations support each component. The propositions are informal arguments resting on stated assumptions, not empirical evidence. UniSinging-Eval, with its multi-style coverage, composer-corrected MIDI and lyric-rewrite levels, is a useful evaluation contribution, but the paper reports no per-level or per-style results on it. Overall the contribution is engineering integration plus a benchmark, not a new architecture.
Field Significance
moderate: The paper shows that a single flow-matching model can keep speech intelligibility close to dedicated zero-shot TTS systems while beating a unified autoregressive baseline and a dedicated singing model on singing intelligibility. The key ingredient is modality-specific handling of the melody condition. It is a concrete data point that speech and singing can share one generator when the condition space is separated, but the remaining speaker-similarity gap and some reporting inconsistencies limit how far the result can be pushed. UniSinging-Eval provides a multi-style test set for unified speech-singing systems.
Claims
- supports: Separating content, melody and timbre conditions reduces cross-task interference when one generator is trained jointly on speech and singing.
Evidence: Removing factorized conditioning raises speech PER from 5.26% to 12.31% and singing PER from 16.22% to 23.45%, with speech N-MOS falling from 3.07 to 2.31, the largest degradation among the ablations. (§5.3, Table 2)
- supports: For speech in a unified speech-singing generator, a learned null melody embedding works better than a fixed zero vector for leaving prosody to be inferred from context.
Evidence: Replacing the learned null melody token with a fixed zero vector raises speech PER from 5.26% to 7.86% and singing PER from 16.22% to 18.64%. (§5.3, Table 2)
- supports: Explicit symbolic melody conditioning matters for singing intelligibility and quality but contributes little to speech generation in a joint model.
Evidence: Removing the MIDI melody encoder moves speech PER only from 5.26% to 6.27% but raises singing PER to 23.21% and lowers singing S-MOS from 3.19 to 2.03. (§5.3, Table 2)
- complicates: Unified speech-singing models can match dedicated TTS on intelligibility while still trailing on speaker similarity and naturalness.
Evidence: UniVoice’s speech PER (5.26%) is within 0.1 points of F5-TTS and CosyVoice3, but its speech SIM (67.42) trails 72.73 and 74.94 and its speech N-MOS (3.07) trails 3.41 and 3.19. The authors attribute the gap to unified training and less speech data. (§5.2, §6, Table 1)
- complicates: Gains in singing intelligibility from unified training do not necessarily come with gains in singer similarity.
Evidence: UniVoice has the lowest singing PER (16.22% vs. 26.22% and 45.07%) but the lowest singing SIM of the three singing-capable systems (35.70 vs. 41.71 for Soul-X-Singer and 36.33 for Vevo1.5). (§5.2, Table 1)
Limitations and Open Questions
Warning
The headline comparisons have reporting inconsistencies. The Vevo1.5/Vevo2 singing PER is 24.72% in the abstract and introduction but 45.07% in Table 1. The baseline is labeled Vevo1.5 while citing the Vevo2 paper. The text calls UniVoice’s speech S-MOS (3.76) the highest, but Table 1 shows F5-TTS at 3.85. The full-model scores differ slightly between Table 1 and Table 2. Code, checkpoints and UniSinging-Eval are promised but not yet released, so the results cannot currently be reproduced.
- Speaker similarity is the clearest weakness: speech SIM trails both dedicated TTS baselines, and singing SIM trails both singing-capable baselines (§6, Table 1).
- Evaluation covers only English and Chinese, with no per-language breakdown. The authors note that rap and spoken singing are not explicitly modeled. 32-step ODE sampling is slower than few-step generators (§6).
- The 35k-hour singing corpus is in-house: separated in-the-wild songs and dry recordings with automatically extracted MIDI. Its composition and licensing are not described in detail.
- The paper does not state whether the Emilia test speakers or the UniSinging-Eval singers overlap with training data. UniSinging-Eval difficulty levels are defined but not reported separately.
- The theoretical propositions rest on an assumed conditional independence of speech audio and melody, and the paper does not test the claimed gradient conflict directly (for example by measuring gradient cosine similarity).
- Cross-modal timbre consistency is listed as a benchmark dimension in §4, but no cross-modal consistency result is reported.
Wiki Connections
- Flow Matching — The generator is an OT-path conditional flow-matching DiT on VAE latents, extended to serve speech and singing as two conditional modes of one flow.
- Singing Voice Synthesis and Conversion — Proposes MIDI-conditioned singing generation inside a shared speech/singing model and introduces the 12-style UniSinging-Eval benchmark.
- Zero-Shot TTS — Speaker or singer timbre comes from a 3-10 s in-context audio prompt without fine-tuning or speaker embeddings, evaluated with a cross-sentence protocol.
- Multilingual TTS — Trained on balanced Chinese and English speech and evaluated on both languages, though no per-language results are reported.
- Subjective Evaluation — Reports human S-MOS and N-MOS ratings (20 listeners per sample) for speech and singing alongside PER and WavLM SIM.
- Evaluation Metrics — Contributes UniSinging-Eval, a multi-style singing test set with composer-corrected MIDI and three lyric-rewrite difficulty levels.
- Vevo2 — Main unified speech-singing baseline (labeled Vevo1.5 in Table 1); UniVoice reports much lower speech and singing PER.
- F5-TTS — Dedicated flow-matching TTS baseline whose ConvNeXt text encoder UniVoice reuses; UniVoice matches its speech PER but trails on SIM and MOS.
- CosyVoice 3 — Large-scale dedicated TTS baseline; UniVoice’s speech PER is comparable and its SIM is lower.
- SoulX-Singer — Dedicated zero-shot singing baseline; UniVoice has lower singing PER, ties on S-MOS, roughly matches on N-MOS, and trails on SIM.
- E2 TTS — Source of the in-context timbre prompting scheme that UniVoice adopts.
- Emilia — Source of the 30k-hour speech training subset and the speech test set.
- Flow Matching for Generative Modeling — Provides the conditional flow-matching objective UniVoice is trained with.
- DiTSinger — Cited DiT-based singing synthesis work in the same architectural family.