arXiv · 2025 · Preprint
Zongcai Du et al. (Migu Music, China Mobile Communications Corporation) · → Paper · Demo: ✓ · Code: ?
Scales singing voice synthesis with a Diffusion Transformer trained on an LLM-bootstrapped 530-hour Chinese corpus, and replaces explicit phoneme-duration labels with an implicit character-span attention mask.
Problem
Singing voice synthesis (SVS) is constrained by two compounding limitations: high-quality paired singing corpora are scarce and costly to collect, and it is unclear how SVS quality scales with model and data size. Prior diffusion- and flow-based SVS systems typically rely on monotonic attention or explicit phoneme-level duration prediction for phoneme-to-acoustic alignment, both of which require accurate duration annotation or post-processing and degrade under timing uncertainty.
Method
The paper addresses data scarcity with a two-stage construction pipeline. In the Recording-fitting Phase, a small, fixed set of melodies is paired with diverse LLM-generated lyric variants and recorded by professional singers, producing a compact corpus used to train melody-specific models called PseudoSinger. In the Data Expansion Phase, each PseudoSinger synthesizes further singing audio for newly generated lyrics while keeping its melody fixed, scaling a base model (pretrained on M4Singer) into 20 PseudoSinger models covering 500 disjoint melody groups, for a total of roughly 530 hours of Chinese singing data.
DiTSinger itself is a transformer-based latent diffusion model that predicts noise in the mel-spectrogram domain. Waveforms are converted to 80-bin mel-spectrograms and tokenized into latents via a convolutional downsampler. Fine-grained conditioning (pitch, phonemes, word durations, slur indicators) is embedded, summed, and passed through a Transformer condition encoder; coarse-grained conditioning (speaker identity, diffusion timestep) is injected via AdaLN, with timbre represented by a learnable embedding table rather than a reference encoder (the corpus has only 40 singers). The denoising network stacks DiT blocks, each combining multi-head self-attention with RoPE and QK-norm, masked multi-head cross-attention onto the fine-grained phoneme conditions, and a feedforward branch, all modulated by AdaLN and learnable residual scaling.
To avoid phoneme-level duration labels, the cross-attention in each DiT block is constrained by an implicit alignment mechanism: each phoneme inherits its character’s known start time and duration, extended backward by a tunable offset δ, and this interval defines an additive attention bias mask that zeroes attention within the valid span and sets it to −∞ elsewhere. The mask is fixed and applied identically during both training (supervised solely by the diffusion reconstruction loss) and inference, encouraging soft, localized phoneme-to-acoustic attention without duration supervision. Training uses classifier-free guidance with 0.1 probability of dropping fine-grained conditions, AdamW (lr = 1e-3) on 4 A100 GPUs for 100,000 iterations (batch size 8, 6-step gradient accumulation), and inference uses DPM-Solver with guidance scale 4.0.
Key Results
Table 1 sweeps the number of PseudoSinger groups from 1 to 50: quality improves as groups increase from 1 (base model only, unstable articulation) up to 20 groups (MOS 4.05 ± 0.06, MCD 3.12, FFE 0.19, F0RMSE 11.48), then degrades at 50 groups (MOS 3.81, MCD 3.65) as each PseudoSinger sees fewer melodies and generalizes worse. Table 2 compares DiTSinger variants against DiffSinger (retrained on the paper’s own data), StyleSinger, and TCSinger (both conditioned on a reference clip from the same singer): DiTSinger L2 (Large depth/width, half-resolution) achieves the best synthesized-system scores (MOS 4.02 ± 0.06, MCD 3.03, FFE 0.15, F0RMSE 11.18), improving MOS by 0.22 and roughly halving F0RMSE relative to the retrained DiffSinger baseline (MOS 3.80, F0RMSE 14.15). Human recordings (MOS 4.35) and HiFi-GAN vocoder reconstruction of ground-truth mel-spectrograms (MOS 4.12) remain upper bounds. Figure 3’s scaling study shows MCD improving with both architectural scale (depth/width/resolution) and data scale (30h to 530h); notably, a half-resolution Small model (S2) outperforms a higher-FLOP Base model (B4), indicating that spectrogram resolution matters more than raw compute in this regime.
Novelty Assessment
The implicit alignment mechanism is the paper’s most genuinely novel contribution: constraining cross-attention with a fixed, character-span attention bias is a lightweight way to remove duration-label supervision that is distinct from prior monotonic-attention or duration-predictor approaches in SVS. The DiT backbone itself (RoPE, QK-norm, AdaLN) applies established diffusion transformer components from other domains to SVS rather than introducing new architectural mechanisms, making that portion of the contribution primarily engineering integration. The systematic depth/width/resolution and data-scale study is a useful empirical contribution to a domain where such scaling analyses are uncommon, though it is conducted on a single in-house Chinese corpus. The two-stage PseudoSinger data pipeline is a practical, if domain-specific, solution to SVS data scarcity rather than a fundamentally new data-augmentation paradigm, and the resulting 530-hour corpus is not released.
Field Significance
Moderate — this paper contributes a specific, testable mechanism (character-span implicit alignment) for removing duration supervision in singing voice synthesis, and demonstrates that diffusion transformer scaling behavior extends to the SVS domain across both model size and dataset scale. Its evaluation is confined to a single in-house Chinese corpus with baselines under mixed retraining conditions, which limits how far its comparative numbers generalize.
Claims
- supports: Constraining cross-attention to character-level temporal spans via a fixed additive attention bias can substitute for explicit phoneme-level duration labels in phoneme-to-acoustic alignment for singing synthesis.
Evidence: The implicit alignment mask (Eq. 6–8) is derived only from character-level start times and durations, extended by a tunable offset δ, and is applied identically at training and inference under diffusion reconstruction loss alone; the resulting model achieves the best MCD (3.03) and F0RMSE (11.18) among all compared systems. (§2.3, Table 2)
- supports: Diffusion transformer scaling trends observed in other modalities (quality improving with depth, width, and resolution) extend to singing voice synthesis.
Evidence: Sweeping DiTSinger from Small (depth 4, width 384) to Large (depth 16, width 768) and data from 30h to 530h produces consistent MCD improvements, with a half-resolution Small variant (S2) outperforming a higher-FLOP Base variant (B4), showing resolution has an outsized effect relative to raw compute. (§3.2, Fig. 3)
- complicates: Bootstrapping large singing corpora from a small number of melody-specialized synthesis models has a data-regime sweet spot rather than improving monotonically with the number of source models.
Evidence: MOS peaks at 20 PseudoSinger groups (4.05 ± 0.06) and then falls to 3.81 at 50 groups, with MCD rising from 3.12 to 3.65, as each group’s melody coverage shrinks and generalization to unseen lyrics worsens. (§3.3, Table 1)
- complicates: Cross-system comparisons in singing voice synthesis are hard to interpret when baselines differ in whether they are retrained on the evaluation corpus or conditioned via a reference clip.
Evidence: DiffSinger is retrained on the paper’s own 530h corpus, while StyleSinger and TCSinger are instead conditioned on a reference clip from the same singer rather than retrained, so the MOS/MCD gaps in Table 2 partly reflect this mismatch in evaluation protocol rather than architecture alone. (§3.1, §3.4, Table 2)
Limitations and Open Questions
Warning
All training and evaluation data is an unreleased, in-house Chinese corpus (M4Singer plus a custom 530h PseudoSinger-generated set), and no code release is mentioned; results cannot be independently reproduced or verified on public data.
The authors themselves note that experiments are limited to Chinese-language singing and that the model does not account for singing techniques such as vibrato or falsetto as an explicit conditioning factor. Speaker coverage is limited to 40 professional vocalists represented by a fixed lookup embedding table rather than a reference encoder, so the model has not been evaluated for generalization to unseen singers. Stated future work targets multilingual and multi-scenario adaptability and adds reference-timbre and singing-style conditioning.
Wiki Connections
- Singing Voice Synthesis — proposes a scalable data-bootstrapping pipeline and a diffusion transformer architecture specifically for Chinese singing voice synthesis, with a systematic model- and data-scaling study.
- Diffusion TTS — applies a Diffusion Transformer (DiT) with RoPE and QK-norm to the mel-spectrogram latent domain, extending diffusion-based generation techniques to singing voice synthesis with an implicit, duration-label-free alignment mechanism.
- Subjective Evaluation — reports MOS listening tests with 95% confidence intervals as the primary quality measure alongside objective MCD, FFE, and F0RMSE metrics.