arXiv · 2025 · Preprint
Ryu et al. (Korea University) · → Paper · Demo: ✓ · Code: ✗
Introduces phoneme-level energy sequence conditioning for explicit dynamics control in singing voice synthesis, achieving over 50% reduction in energy mean absolute error compared to a baseline lacking energy conditioning and outperforming an implicit energy predictor approach.
Problem
Most singing voice synthesis (SVS) systems achieve high perceptual quality but generate expressive output through probabilistic modeling conditioned only on the musical score, giving users no mechanism to precisely control dynamics (temporal loudness variation). Implicit energy predictors borrowed from text-to-speech, such as those in FastSpeech 2, model energy as an auxiliary target during training but cannot be driven directly at inference time, offering only marginal controllability gains. No prior SVS work had explicitly modelled dynamics as a user-controllable input, leaving a gap between system capability and user intent.
Method
The system follows a DiffSinger-inspired two-stage design: an encoder stage maps phoneme-level input sequences into a hidden representation, which a diffusion-based decoder converts to a mel-spectrogram, then a pre-trained HiFi-GAN vocoder produces the waveform. Inputs are lyrics, musical notes, duration, and a phoneme-level energy sequence, all of length L (number of phonemes).
A length regulator expands phoneme-level sequences to frame-level length T by repeating each token according to its duration annotation in seconds converted to frame count. A Feed-Forward Transformer (FFT) block sums the embedding vectors of the four input streams and applies a 2-layer 1D convolutional network to capture adjacent-frame context, producing the conditioning representation H_c. The mel-spectrogram decoder is a U-Net-based Denoising Diffusion Probabilistic Model (DDPM) with 100 forward steps and linear noise scheduling (max beta 0.06). HiFi-GAN is fine-tuned and used only at inference; it is excluded from evaluation metrics.
The energy sequence is derived without manual annotation. Frame-level energy is the square root of the mean squared mel-spectrogram amplitude across channels. Phoneme-level energy averages these frame values within each phoneme boundary, reducing the input from roughly 1,000 frame values per utterance to L phoneme values, a length matching the lyric and note sequences. This aggregation trades some precision for usability: a user specifies one loudness value per phoneme rather than per frame.
Training uses the standard DDPM L1 noise-prediction objective on the GTSinger Chinese subset (two speakers, ~16 hours, 7,082 training samples). Embedding dimension is 256; training ran ~113,600 steps on a single RTX A5000 GPU.
Key Results
Results are reported on the 57-sample GTSinger Chinese test set (Table I, Table II). The baseline DDPM model conditioned only on lyrics and notes yields energy MAE of 0.33 and MOS of 3.43 ± 0.17. Adding the phoneme-level energy sequence reduces energy MAE to 0.14 (57.6% reduction) and raises MOS to 3.78 ± 0.19; the frame-level model achieves the lowest energy MAE at 0.03 but lower MOS at 3.57 ± 0.18. The phoneme-level model also improves F0 MAE from 10.67 to 9.73 and MCD from 12.89 to 12.07, though the authors consider these secondary to controllability.
An ablation (Table III) shows that attaching an energy predictor to the baseline reduces energy MAE only marginally (0.33 to 0.30), confirming that implicit modeling provides far weaker control than explicit conditioning. Ground-truth audio reconstructed through HiFi-GAN scores 4.02 ± 0.2 MOS, setting the ceiling. MOS evaluation used 10 human listeners.
Novelty Assessment
The core contribution is replacing an implicit energy predictor with an explicit phoneme-level energy conditioning channel in an otherwise standard DiffSinger-style pipeline. The architecture itself (DDPM decoder, length regulator, FFT block) is not new; the innovation is in showing that summing an energy embedding directly with lyric and note embeddings provides effective and user-accessible dynamics control. The phoneme-level aggregation is a practical engineering decision to reduce control dimensionality from frame to phoneme granularity.
The claim of first user-driven dynamics control in SVS is plausible given the cited prior art, though the comparison set is limited (no SOTA SVS baselines, single language, 10 MOS listeners). The evaluation design is honest about this scope: the stated goal is demonstrating controllability over the baseline rather than matching state-of-the-art quality.
Field Significance
Moderate - This paper addresses a genuine gap: dynamics controllability has been largely unexplored as an explicit conditioning target in SVS despite its musical importance. The mechanism is simple (embedding summation, annotation-free energy extraction) and can be integrated into any score-conditioned SVS system, which gives the contribution practical transferability. However, the evaluation scale is small and the baseline architecture is not competitive with current SVS systems, so the quantitative results should be interpreted as proof-of-concept rather than field-setting benchmarks.
Claims
-
supports: Explicit conditioning on a signal-derived attribute provides substantially more controllability than implicit predictors trained on the same attribute as an auxiliary loss.
Evidence: Adding an energy predictor to the baseline reduces energy MAE from 0.33 to 0.30, while replacing the predictor with explicit phoneme-level energy input achieves 0.14 - a 57.6% reduction over baseline vs. 9% for the predictor. (§V.B, Table III)
-
supports: Phoneme-level aggregation of frame-level features enables user-friendly control in singing synthesis at acceptable precision cost.
Evidence: Phoneme-level energy (L values, matching lyric and note sequence length) achieves energy MAE of 0.14 vs. 0.03 for frame-level (T values, roughly 1,000 per utterance), with higher MOS (3.78 vs. 3.57), demonstrating that coarser granularity trades minor precision loss for substantially reduced control burden. (§III.B, §V.A, Table I, Table II)
-
supports: Ground-truth signal features can substitute for manually annotated expressive attributes as conditioning signals in singing voice synthesis.
Evidence: Frame-level energy is extracted directly from ground-truth mel-spectrograms without human annotation, yet conditioning on it enables effective dynamics control, reducing annotation costs relative to technique-labelled datasets used in prior controllable SVS work. (§I, §III.B)
-
complicates: Perceptual quality and controllability precision do not align monotonically across conditioning granularities in SVS.
Evidence: The frame-level model achieves the best energy MAE (0.03) but lower MOS (3.57 ± 0.18) than the phoneme-level model (MOS 3.78 ± 0.19, energy MAE 0.14), suggesting that very tight energy matching may over-constrain generation in ways that reduce perceived naturalness. (§V.A, Table I, Table II)
Limitations and Open Questions
Warning
MOS evaluation involved only 10 listeners, which the authors acknowledge constrains statistical reliability and generalizability. Results should be treated as directional rather than definitive.
The study uses only the Chinese subset of GTSinger and two speakers, leaving multilingual and speaker-diversity generalisability untested. The baseline DDPM architecture is explicitly noted to fall below state-of-the-art SVS performance, so the absolute quality figures do not reflect what the conditioning mechanism would achieve on a competitive backbone. Expressive attributes beyond dynamics (timbre, vibrato, advanced singing techniques) are not addressed. Phoneme-level energy may not capture fine-grained intra-phoneme temporal variation present in skilled vocal performance.
Wiki Connections
- Singing Voice Synthesis - the paper’s core contribution, explicit phoneme-level energy conditioning, targets dynamics control in a DiffSinger-style singing voice synthesis pipeline.
- Diffusion TTS - the mel-spectrogram decoder uses a U-Net DDPM, following the DiffSinger-style pipeline that this paper extends with energy conditioning.
- Prosody Control - the paper introduces explicit phoneme-level energy conditioning as a mechanism for user-driven dynamics (loudness) control in singing, addressing a prosody attribute not previously made controllable in SVS.
- GAN Vocoder - HiFi-GAN is fine-tuned and used for waveform generation from the predicted mel-spectrogram; it is excluded from training and objective evaluation.
- Subjective Evaluation - MOS with 10 human raters is reported for all system variants; the small panel size is flagged as a limitation by the authors.