arXiv · 2025 · Preprint
Junchuan Zhao et al. (National University of Singapore) · → Paper · Demo: ✓ · Code: ✗
Adapts the non-autoregressive MaskGCT codec-language architecture to zero-shot singing voice synthesis, adding a coarse-to-fine contrastive objective and an integrated transcription module to stop pitch information from leaking through the timbre prompt.
Problem
Discrete-token, prompt-based in-context synthesis (as popularized by VALL-E-style TTS and extended to non-autoregressive masked generative models such as MaskGCT) enables zero-shot voice cloning by conditioning on a short acoustic prompt. Directly transplanting this recipe to singing voice synthesis (SVS) is harder than it looks: singing requires the model to follow an explicit pitch/melody sequence from the musical score, but the acoustic prompt used for timbre conditioning also carries prosodic cues (pitch contour, timing). Because singing datasets are far smaller and less diverse than the large multi-speaker speech corpora that make TTS prompting work, this timbre-melody entanglement is more severe in SVS: the prompt’s own prosody leaks into the output and competes with the explicit pitch condition, degrading melody controllability. Prior discrete-token SVS systems either lack explicit melody conditioning (Make-A-Voice) or rely on external melody-aware tokenizers built from auxiliary melody audio (Vevo 1.5/2.0), which introduces its own melody-lyrics alignment ambiguity.
Method
CoMelSinger keeps MaskGCT’s two-stage, non-autoregressive masked generative pipeline: a Text-to-Semantic (T2S) module converts lyric (phoneme) tokens plus a semantic prompt into semantic tokens, and a Semantic-to-Acoustic (S2A) module predicts acoustic (codec) tokens conditioned on the semantic tokens, an acoustic prompt, and a frame-aligned pitch sequence. The key adaptation is a new pitch embedding that is added element-wise to the semantic token embedding before being fed to the S2A model’s LLaMA-style backbone, giving the model an explicit, frame-synchronized melody signal alongside the lyric content.
To suppress prosody leakage from the acoustic prompt, the paper introduces a coarse-to-fine contrastive learning strategy applied during S2A fine-tuning. At the sequence level, a symmetric contrastive loss pulls together mean-pooled acoustic embeddings generated from the same semantic and pitch condition but different same-singer acoustic prompts, while treating embeddings from different pitch sequences as negatives — encouraging prompt-invariant global melody consistency. At the frame level, a token-wise contrastive loss uses pitch-perturbed pairs (50% of pitch tokens randomly shifted by up to a semitone-scale offset) and a soft label matrix that jointly encodes pitch and semantic alignment, enforcing local pitch fidelity independent of the prompt.
A third component is a lightweight, encoder-only Singing Voice Transcription (SVT) module (4 Transformer layers, 512-dim hidden size, 8 heads) that predicts frame-level pitch tokens directly from acoustic codec tokens. Unlike prior SVT use (e.g., ROSVOT), which treats transcription as an offline data-cleaning step, CoMelSinger keeps the pretrained SVT frozen and uses it in the training loop to supply pseudo pitch labels as auxiliary frame-level supervision (cross-entropy plus a segment-transition loss and a soft duration loss) for the S2A model. Training combines this SVT loss with the mask-prediction loss and the coarse-to-fine contrastive loss, and only the S2A model’s DiffLlama-style diffusion estimator is fine-tuned, using LoRA (r=16) for parameter efficiency; the T2S model, codecs, and SVT module stay frozen. At inference, ground-truth score durations (rather than a learned duration predictor) directly determine output length.
Key Results
On seen-singer evaluation (M4Singer/Opencpop, Table II), CoMelSinger outperforms DiffSinger, VISinger2, StyleSinger, SPSinger, and Vevo 1.5 on SMOS (4.22), SECS (0.912), and F0-RMSE (0.042), while remaining competitive on MCD (4.17) and SingMOS (4.32). Against the ground-truth-with-codec upper bound, the gap is small, suggesting most of the remaining loss comes from architecture rather than the codec itself.
In the zero-shot (unseen-singer) setting (OpenSinger, Table III), CoMelSinger shows only mild degradation relative to the seen-singer condition, while baselines drop more sharply on SMOS, SECS, and F0-RMSE. CoMelSinger’s F0-RMSE (0.048) is substantially lower than the next-best baseline (SPSinger, 0.065) and Vevo 1.5 (0.094).
The ablation (Table V) shows the coarse-to-fine contrastive loss and the SVT supervision are both necessary and complementary: removing the full contrastive objective raises F0-RMSE from 0.042 to 0.08 and drops SingMOS from 4.32 to 4.12; removing SVT alone is worse still (F0-RMSE 0.194, SingMOS 3.95); removing both (reducing to a MaskGCT-based SVS baseline) gives the largest degradation (F0-RMSE 0.21, SingMOS 3.83). A separate fine-tuning-strategy comparison (Table VII) finds LoRA (6.51% trainable parameters) outperforms full fine-tuning (100% trainable) on F0-RMSE, SingMOS, and SECS, while other parameter-efficient variants (prefix tuning, pitch-only tuning, progressive unfreezing) trail LoRA on at least one axis.
A separate diagnostic experiment (Table I) measures prosody similarity between MaskGCT-generated TTS output and its acoustic prompt on LibriTTS (English) and AISHELL-3 (Mandarin): paired prompts consistently produce lower pitch/energy/jitter differences than unpaired prompts from the same speaker, which the paper uses as direct evidence that prosody leakage occurs in prompt-based masked generative TTS, not just in the SVS extension.
Novelty Assessment
The backbone (MaskGCT, LLaMA-style masked generative codec transformer) and the general strategy of replacing text conditioning with lyric-plus-pitch tokens are not new; Make-A-Voice already explored discrete-token SVS conditioning, and MaskGCT itself is an existing TTS system. The genuinely new contributions are narrower but real: (1) the coarse-to-fine contrastive objective that explicitly regularizes pitch-prompt redundancy at both sequence and frame granularity, and (2) folding SVT-derived pseudo pitch labels into the training loop as a differentiable auxiliary loss rather than an offline annotation tool. Both are demonstrated with clean, targeted ablations that isolate their individual contributions. The fine-tuning-strategy comparison (LoRA vs. full fine-tuning vs. prefix tuning, etc.) is a useful secondary empirical contribution but is standard parameter-efficient fine-tuning applied to a new domain rather than a new method.
Field Significance
Moderate — the paper provides direct evidence, via a controlled diagnostic experiment on standard TTS corpora (LibriTTS, AISHELL-3), that prosody leakage from the acoustic prompt is a real, measurable phenomenon in prompt-based masked generative codec models, not merely a hypothesized SVS-specific issue. It also demonstrates that explicit contrastive disentanglement plus integrated transcription supervision can substantially close this gap within a zero-shot SVS system, using targeted ablations to attribute the improvement to specific components rather than to overall system tuning.
Claims
- supports: Prompt-based conditioning in masked generative or codec-language speech synthesis models causes measurable leakage of prosodic attributes from the acoustic prompt into the synthesized output, independent of the target language.
Evidence: Paired-prompt outputs show consistently lower pitch/energy/jitter differences than unpaired-prompt outputs from the same speaker on both LibriTTS (English) and AISHELL-3 (Mandarin) when synthesizing with MaskGCT. (§V.A, Table I)
- supports: Explicit contrastive regularization between the acoustic prompt and an external control signal (e.g., melody/pitch) reduces attribute leakage and improves controllability in prompt-based zero-shot synthesis.
Evidence: Removing the coarse-to-fine (sequence + frame level) contrastive loss increases F0-RMSE from 0.042 to 0.08 and lowers SingMOS from 4.32 to 4.12 on the seen-singer test set, with sequence-level and frame-level components independently ablated to show complementary effects on speaker-identity and pitch-detail metrics respectively. (§V.D, Table V)
- supports: Integrating auxiliary transcription-derived frame-level supervision directly into a synthesis model’s training loop, rather than using it only as an offline data-cleaning step, improves fine-grained attribute alignment.
Evidence: Removing the in-loop SVT auxiliary loss produces the largest single-component degradation in the ablation, raising F0-RMSE from 0.042 to 0.194 and lowering SingMOS from 4.32 to 3.95. (§V.D, Table V)
- refines: Parameter-efficient fine-tuning can match or exceed full fine-tuning when adapting a large pretrained codec-based speech model to a lower-resource downstream domain, provided the low-rank capacity is placed appropriately.
Evidence: LoRA fine-tuning of the S2A diffusion estimator (6.51% trainable parameters) achieves lower F0-RMSE (0.053) and higher SECS (0.92) than fully fine-tuning the same backbone (100% trainable, F0-RMSE 0.099, SECS 0.859) when adapting MaskGCT to singing voice synthesis. (§V.D, Table VII)
Limitations and Open Questions
Warning
The zero-shot evaluation set (OpenSinger) has no native music-score annotations, so the authors pair OpenSinger audio with M4Singer’s pitch/duration sequences to construct test inputs. This means the “unseen singer” evaluation is not evaluating on musically native score-audio pairs, which could understate or overstate melody-control difficulty relative to a genuinely paired unseen-singer benchmark.
Beyond that, evaluation is confined to Mandarin singing corpora (M4Singer, Opencpop, OpenSinger); no cross-lingual or multilingual SVS results are reported, despite the underlying MaskGCT backbone being multilingual-capable. Overall S2A/T2S model size is not reported (only the small SVT module’s dimensions are given), limiting reproducibility of compute requirements. The subjective evaluation panel is modest (20 musically trained participants), typical for SVS papers but smaller than typical large-scale TTS MOS studies. Finally, the sequence-level contrastive objective depends on having multiple same-singer utterances available per training batch, which the curated corpora used here provide but which may not hold for less structured or lower-resource singing data.
Wiki Connections
- Singing Voice Synthesis — adapts a codec-language masked generative TTS architecture (MaskGCT) to zero-shot singing voice synthesis with explicit melody/pitch conditioning.
- Zero-Shot TTS — extends prompt-based zero-shot in-context conditioning, established for TTS via codec language and masked generative models, to the singing voice synthesis domain where prosodic constraints are stricter.
- Prosody Control — introduces frame-aligned pitch/duration tokens as an explicit melody-control signal, decoupled from timbre via a dedicated contrastive training objective.
- Disentanglement — the coarse-to-fine contrastive loss is a training-time mechanism specifically designed to separate pitch/melody information from timbre carried in the acoustic prompt.
- Neural Audio Codec — builds directly on MaskGCT’s pretrained semantic and RVQ acoustic codecs, reusing them frozen while fine-tuning only the acoustic token predictor.
- Subjective Evaluation — reports MOS-Q, MOS-N, and SMOS listening tests with musically trained raters alongside automatic SingMOS, F0-RMSE, MCD, and SECS metrics.
- Vevo2 — used as the strongest baseline system in both seen- and zero-shot evaluation tables; CoMelSinger reports lower F0-RMSE and comparable-or-better SMOS/SECS against it.
- CosyVoice — cited as one of the LLM-style discrete-token TTS systems whose prompt-based in-context learning paradigm motivates this paper’s approach to SVS.
- CosyVoice 2 — cited alongside CosyVoice as part of the family of large-scale codec-language TTS systems this paper adapts prompt-based conditioning from.