IJCNLP-AACL · 2025 · Conference
Pan, Yao, Zhang, Guo, Lu, Zhu & Zhao (Zhejiang University) · → Paper · Demo: ✗ · Code: ✓ (companion repo of curated resources)
The first comprehensive survey of deep-learning-based singing voice synthesis (SVS), organizing the field by task type, architectural paradigm (cascaded vs. end-to-end), core modeling and control techniques, and available datasets, tools, and evaluation benchmarks.
Problem
Singing voice synthesis has advanced rapidly under the influence of large language models and modern generative paradigms, but unlike text-to-speech, the field lacked a systematic survey tying together its task taxonomy, competing architectural paradigms, representation and control techniques, and supporting resources (datasets, annotation tools, evaluation metrics). The scattered nature of this literature makes it hard for newcomers and practitioners to orient themselves or to see which design choices remain unsettled.
Method
The survey organizes SVS research along four axes. First, it categorizes prevailing SVS tasks into four types: high-fidelity synthesis, controllable synthesis, singing style transfer, and text-to-song generation. Second, it classifies SVS system architectures by whether they use a vocoder, distinguishing cascaded designs (an acoustic model predicts intermediate features such as mel-spectrograms, F0, and duration, followed by a separate vocoder) from end-to-end designs (waveform generated directly, without hand-crafted intermediate interfaces), and separately discusses the autoregressive-vs-non-autoregressive axis within each. Third, it reviews core technologies for singing modeling (content, acoustic, and semantic representations) and control (audio-based style transfer and text-based control via concatenation, cross-attention, adaptive normalization, classifier-free guidance, and mixture-of-experts routing). Fourth, it catalogs open-source datasets (comparing language, singer count, duration, and annotation richness across 16 corpora), annotation tools for alignment and pitch/duration transcription, and evaluation metrics grouped by the attribute they assess (accuracy, expressiveness/naturalness, sound quality, similarity, controllability). Two appendices extend the review: one on training strategies (data augmentation, pre-train-and-fine-tune, multi-stage training, and inference acceleration) and one discussing open problems (novel task directions, architecture trade-offs, representation choices, and data scarcity), plus a dedicated section on the emerging role of multimodal large language models in SVS data annotation, content generation, expressiveness guidance, cross-modal alignment, and automated evaluation. The survey conducts no original experiments; all conclusions are drawn from synthesizing the reviewed literature.
Key Results
Not applicable — this is a literature review with no original experiments or reported quantitative results. Its output is a structured taxonomy and synthesis of the state of SVS research as of its 181 reviewed references, accompanied by a maintained companion repository of the underlying paper list.
Novelty Assessment
The paper introduces no new model or architecture; its contribution is organizational and synthetic. It is, by the authors’ own framing, the first survey to systematically unify SVS task types, the cascaded/end-to-end architectural split, core representation and control techniques, and supporting resources (datasets, annotation tools, evaluation metrics) into a single reference, and it extends this synthesis with a forward-looking discussion of open architectural and representational trade-offs and a dedicated treatment of how multimodal LLMs are beginning to reshape SVS annotation, generation, and evaluation.
Field Significance
high — As the first comprehensive survey of deep-learning-based SVS, accepted as an oral presentation and paired with a maintained companion repository, this paper functions as a reference map for a fast-moving but previously unconsolidated subfield.
Its value lies less in any single finding and more in giving researchers and engineers a structured entry point: a shared task taxonomy, an explicit accounting of the cascaded-vs-end-to-end and autoregressive-vs-non-autoregressive design trade-offs, and a catalog of the datasets, tools, and metrics needed to train and evaluate new SVS systems.
Claims
- supports: Cascaded speech/singing synthesis pipelines can retain a practical performance advantage over end-to-end designs in low-resource settings, because their hand-crafted intermediate targets (e.g., mel-spectrograms, F0) provide auxiliary supervision that raises the performance floor, even though end-to-end designs offer a higher ceiling once enough data and compute are available.
Evidence: The survey’s architectural analysis attributes the continued strength of cascaded SVS systems to auxiliary supervision from hand-crafted acoustic targets, contrasted with end-to-end systems’ greater data requirements, training instability, and alignment sensitivity. (§B.2)
- complicates: There is no settled consensus on whether continuous or discrete acoustic representations are preferable for singing/speech generation, since each trades off different capabilities rather than one dominating the other.
Evidence: The survey characterizes discrete representations as enabling autoregressive/LLM-style next-token modeling, streaming inference, and precise control at the cost of smoothing fine vocal detail, while continuous representations preserve finer acoustic texture and phase structure but depend on more powerful decoders and are more alignment-sensitive. (§B.3)
- complicates: Increasing a generative speech/singing model’s controllability does not straightforwardly translate into better audio fidelity, and increasing guidance strength at inference time alone does not produce genuine controllability.
Evidence: The survey’s discussion of the quality-controllability trade-off notes that overemphasis on control can degrade fidelity, that disentangling content, acoustic, and semantic factors for cross-lingual/cross-style transfer remains difficult, and that relying on a single encoder or higher inference-time guidance strength is insufficient for genuine controllability. (§B.3)
- supports: Singing voice synthesis remains substantially more data-constrained than speech synthesis, both in scale and in annotation quality, motivating data augmentation and self-/weakly-supervised training strategies over reliance on large clean corpora.
Evidence: The survey contrasts open-source singing datasets (largest corpora on the order of tens to ~80 hours) against speech datasets exceeding 100,000 hours, and cites long-tail style/language coverage and inaccurate annotations (e.g., lyric timestamps) as the primary bottleneck motivating augmentation and weak-supervision research directions. (§B.4, Table 1)
Limitations and Open Questions
The survey explicitly scopes itself to deep-learning-based SVS approaches, excluding detailed treatment of traditional methods such as waveform concatenation and statistical parametric synthesis. As a review rather than an empirical study, its conclusions synthesize claims and comparisons reported across the underlying literature (which the survey itself does not independently re-verify) rather than presenting new controlled experiments, and its appendix discussion of MLLM potential in SVS is explicitly speculative and forward-looking rather than based on the survey’s own validation.
Wiki Connections
- Singing Voice Synthesis — provides a comprehensive taxonomy of SVS tasks, architectures (cascaded vs. end-to-end), modeling/control techniques, and supporting resources, intended as a field-level reference.
- Seed-TTS — cited as an example of large-scale versatile speech generation models informing the survey’s discussion of MLLM-driven evaluation.
- CosyVoice — cited as an example of end-to-end design adoption trends contrasted against the SVS community’s continued use of cascaded pipelines.
- DiTAR — discussed as a hybrid autoregressive/non-autoregressive architecture offering a practical avenue for SVS modeling.
- Qwen3-Omni — cited as an example multimodal large language model relevant to the survey’s discussion of MLLM contributions to SVS.