arXiv · 2026 · Preprint
Yukun Chen et al. (Xi’an Jiaotong University / Nanyang Technological University / Ant Group / Zhejiang University) · → Paper · Demo: ✗ · Code: ✓
Adapts an autoregressive diffusion speech model to Mandarin singing voice synthesis that runs on symbolic scores (lyrics, MIDI pitches, note values and BPM) without durations. The model sets the timing and decides when to stop, so it needs no duration predictor and no time-aligned acoustic cue.
Problem
Singing voice synthesis (SVS) systems usually need timing before they generate audio. Duration-based models (DiffSinger, VISinger, TechSinger) take phoneme-level durations as input or predict them. Diffusion Transformer (DiT) systems need a fixed-length canvas, so they rely on a duration predictor (DiTSinger, TCSinger 2) or on a time-aligned melody cue such as reference audio or an F0 curve (YingMusic-Singer, SoulX-Singer). Composers write intent-level scores with note values and tempo, not durations in seconds, so these input formats do not match real composition work. Most systems also feed lyrics and melody as two separate streams. That makes the lyric-to-note alignment ambiguous under melisma, where one syllable spans several notes. A second problem is data: open SVS corpora with score annotations stay under 100 hours each, far below what modern scaled architectures need.
Method
VocalRender has three parts:
- Score-native input. The score is serialized into one interleaved token sequence: the global BPM, then each lyric syllable followed directly by its (MIDI pitch, note value) pairs. A melismatic syllable is simply followed by several pitch-note-value pairs, so the syllable-to-note binding is explicit and needs no phoneme-level alignment. Note values are symbolic categories (quarter, eighth, dotted), not durations.
- Continuous latent space. A pretrained audio VAE, kept frozen during generation training, encodes the waveform into 25 Hz continuous latents (16 kHz input, 48 kHz reconstruction). No discrete codec is used.
- Autoregressive diffusion model (ARDM). The method follows DiTAR (DiTAR) and uses VoxCPM2 as the backbone, initialized from its speech-pretrained weights. Latents are grouped into patches of 4 frames (6.25 Hz LM rate). An aggregation encoder embeds each patch. A decoder-only AR transformer (~1.7B parameters) reads the score prompt and the patch history. A local DiT (~0.6B) generates the next patch from the AR hidden state and the previous patch. A binary stop predictor on the AR hidden state decides the output length.
The model is trained end-to-end with a flow-matching loss on the latent patches plus the stop loss (λ_stop = 1). The authors call the generator a “diffusion” model, but the objective is flow matching. Timbre comes from in-context prompting: latent patches from a short reference clip of the same song (2 to 8 s during training) are placed before the score tokens. Inference uses 10 sampling steps with classifier-free guidance scale 2.0.
Two variants are trained. VocalRender is trained on CrawlSinger-OS, a corpus of more than 2,300 hours built from Muse (Chinese subset), MuChin, SongFormDB and OpenSinger with the SingCrawl pipeline: vocal separation, slicing, ASR or annotation-guided lyric transcription, retrained SOFA forced alignment, and ROSVOT pitch transcription quantized to note values. It is pretrained on the synthetic subset for 40k steps and then fine-tuned on the real subset for 20k steps. VocalRender-Pro is trained on CrawlSinger (more than 5,600 hours, in-house) for 160k steps.
Key Results
All results are for Mandarin, on Opencpop (filtered for lyric overlap with crawled training data) and on CrawlSinger-Eval, a set of songs published after 2026-06-01. The baselines are TCSinger, TechSinger, Vevo2 and SoulX-Singer. Baselines that need durations receive nominal durations computed from note values and BPM, not ground-truth-aligned durations, so every system gets the same score-level information (§Experimental Setup).
Objective results (Table 1).
- WER. VocalRender reaches 4.44 on Opencpop and 4.52 on CrawlSinger-Eval, lower than every baseline (best baseline: SoulX-Singer at 5.02 / 5.03). VocalRender-Pro is lower still (3.88 / 4.45). On CrawlSinger-Eval both variants are below the ground-truth WER of 5.17. On Opencpop the ground truth is 3.81.
- Speaker similarity. SIM is tightly clustered among the strong systems. On Opencpop: VocalRender-Pro 0.929, SoulX-Singer 0.928, VocalRender 0.922. On CrawlSinger-Eval: VocalRender-Pro 0.926, VocalRender 0.919, SoulX-Singer 0.918.
- Rhythm and melody. SoulX-Singer leads on both IOU and RPA on both sets (Opencpop 0.63 / 0.77; CrawlSinger-Eval 0.54 / 0.70). VocalRender scores 0.62 / 0.72 and 0.43 / 0.63. VocalRender-Pro scores 0.62 / 0.75 and 0.48 / 0.67.
- Automatic quality. On Opencpop, VocalRender’s SingMOS of 4.59 equals the ground truth, and SoulX-Singer has the highest CE (6.01). On CrawlSinger-Eval, VocalRender has the highest SingMOS (4.53) and CE (5.85). The authors note that SingMOS and CE vary little across frontier systems.
Subjective results (Table 2; values recovered from the PDF). VocalRender is the CMOS reference (0). All baselines score negative on naturalness (N-CMOS): SoulX-Singer −0.42±0.28, Vevo2 −0.93, TechSinger −0.99, TCSinger −1.20. They also score negative on prosody similarity (PS-CMOS): SoulX-Singer −0.32±0.24, with the other baselines between −1.16 and −1.65. VocalRender-Pro scores slightly above VocalRender on both (+0.13±0.27 and +0.06±0.22), so its intervals include zero. For score following (MS-MOS, 1 to 4), VocalRender scores highest at 2.96±0.22, ahead of SoulX-Singer (2.84±0.22) and VocalRender-Pro (2.71±0.27). The VocalRender and SoulX-Singer intervals overlap. Each test uses 20 randomly drawn samples or pairs and more than 20 listeners. Musically trained listeners rate PS-CMOS and MS-MOS; N-CMOS adds amateur listeners (§Appendix A).
So the objective and subjective results disagree on rhythm and melody. SoulX-Singer, which receives a strict predefined time alignment, wins on IOU and RPA. Listeners prefer VocalRender’s prosody and judge that it follows the score better.
Ablations (Table 3, deltas vs. VocalRender). Training only on GTSinger + M4Singer instead of CrawlSinger-OS barely changes WER (+0.12 / +0.03). SIM drops by about 0.03 and RPA collapses (−0.45 / −0.40). Replacing the interleaved prompt with a cascaded one (all lyrics, then all notes) lowers IOU (−0.05 / −0.04) and raises WER (+0.11 / +0.16), while RPA stays nearly unchanged (+0.02 / +0.01). This suggests interleaving mainly fixes rhythm alignment, not pitch.
Novelty Assessment
The contribution is mainly engineering integration. The ARDM formulation (AR transformer plus local DiT over continuous latent patches, with a stop predictor) comes from DiTAR, and the model is initialized from the existing VoxCPM2 TTS backbone. The paper is a transfer of this architecture to SVS, which the authors call the first ARDM-based SVS model. The genuinely new design choice is the interleaved syllable-then-notes score serialization. It is simple, and the ablation shows it helps rhythm alignment. The data pipeline (SingCrawl applied to open sources) is a substantial practical contribution, but the paper does not say whether CrawlSinger-OS will be released.
The evaluation is reasonably careful. It uses an out-of-domain test set dated after the training data, filters Opencpop for leakage, and gives all systems the same score-level information. Two caveats apply. First, giving duration-based baselines nominal BPM-derived durations matches the score-native use case but works against systems designed for aligned durations. Second, the subjective tests use only 20 samples each, and the score-following lead over SoulX-Singer falls within the confidence intervals.
Field Significance
Moderate — The paper shows that an autoregressive continuous-latent generator with a learned stop decision can synthesize singing from symbolic scores alone. In listening tests it matches or beats duration-aligned DiT systems on naturalness and score following, while those systems still lead on objective rhythm and pitch alignment. It is a useful data point that explicit duration modeling may not be necessary for SVS when a speech-pretrained AR backbone and a few thousand hours of annotated singing are available. The evidence is limited to Mandarin.
Claims
- supports: Autoregressive generation of continuous acoustic latents with a learned stopping decision can replace explicit duration prediction in singing voice synthesis from symbolic scores.
Evidence: VocalRender generates VAE latent patches with an AR transformer plus local DiT and a stop predictor. With no duration input, it has the lowest WER of all baselines on both test sets (4.44 / 4.52 vs. 5.02 / 5.03 for SoulX-Singer), and all baselines score below it on naturalness CMOS (best baseline −0.42). (§Architecture; §Main Results, Table 1, Table 2)
- complicates: Objective rhythm and pitch alignment metrics can disagree with listener judgments of score adherence in singing voice synthesis.
Evidence: SoulX-Singer, which receives strict predefined temporal alignment, has the highest IOU and RPA on both test sets. Listeners still rate VocalRender higher on prosody similarity (SoulX-Singer at −0.32 PS-CMOS) and on score following (MS-MOS 2.96 vs. 2.84). (§Main Results, Table 1, Table 2)
- refines: Interleaving each lyric unit with its own notes in a single input sequence mainly improves rhythm alignment, not pitch accuracy, compared with feeding lyrics and melody as separate streams.
Evidence: Replacing the interleaved prompt with a cascaded lyrics-then-notes prompt lowers IOU by 0.05 / 0.04 and raises WER by 0.11 / 0.16, while RPA changes by only +0.02 / +0.01. (§Ablation Study, Table 3)
- complicates: A speech-pretrained backbone can keep singing intelligibility high on small singing corpora, while melody accuracy and timbre similarity still depend on large-scale singing data.
Evidence: Training without CrawlSinger-OS (GTSinger + M4Singer only) raises WER by only 0.12 / 0.03, but SIM drops by more than 0.03 and RPA by 0.45 / 0.40. The authors attribute the stable WER to the speech-pretrained VoxCPM2 initialization. (§Ablation Study, Table 3)
- complicates: Automatic singing-quality predictors can lack the resolution to separate frontier singing synthesis systems that listeners still distinguish.
Evidence: SingMOS and Audiobox Aesthetics CE values cluster closely across SoulX-Singer, VocalRender and VocalRender-Pro, and VocalRender’s SingMOS equals the ground truth on Opencpop (4.59). Naturalness CMOS still separates SoulX-Singer from VocalRender by −0.42. (§Main Results, Table 1, Table 2)
Limitations and Open Questions
- Mandarin only. The paper does not test whether the score-native approach carries over to other languages or to non-pop singing styles.
- Training data is mostly synthetic. About 2,013 of CrawlSinger-OS’s roughly 2,317 hours come from Muse, which is generated by SunoV5 (Table 4). The authors’ embedding analysis shows that this synthetic audio differs from real singing in musical and semantic space and has a quantized pitch distribution (§Appendix B, Figures 8 to 10). VocalRender uses it only for pretraining before fine-tuning on the real subset.
- The strongest variant cannot be reproduced. VocalRender-Pro is trained on the 5,600-hour in-house CrawlSinger corpus. The paper does not state whether CrawlSinger-OS will be released, although code is linked.
- Train-inference mismatch in scores. Training scores are automatic audio-centric transcriptions, which record every realized ornament and melisma. Composers write intent-centric scores. Transcription also ignores key, harmony and meter, which can produce spurious chromatic notes and fragmented note sequences (§Discussion). The paper reports no evaluation on composer-written scores, so the size of this mismatch is not measured.
- Small subjective tests. Each test uses 20 samples or pairs. The MS-MOS lead over SoulX-Singer and the VocalRender-Pro CMOS gains fall within the reported confidence intervals.
- Score following trails on objective alignment. Without forced timing, VocalRender stays behind SoulX-Singer on IOU and RPA, especially out of domain (IOU 0.43 vs. 0.54).
- Zero-shot timbre cloning is not verified. The paper does not show that test singers are absent from training, so its speaker-similarity results do not establish unseen-singer cloning.
Wiki Connections
- Singing Voice Synthesis — Proposes a score-native, duration-free SVS system that takes lyrics, MIDI pitches, note values and tempo directly.
- Flow Matching — The local DiT that generates each latent patch is trained with a flow-matching objective, although the paper describes the model as diffusion.
- Subjective Evaluation — Reports naturalness CMOS, prosody-similarity CMOS and a score-following MOS from more than 20 listeners, and these results diverge from the objective alignment metrics.
- DiTAR — Source of the autoregressive diffusion formulation (AR transformer plus local DiT over continuous patches) that VocalRender transfers to singing.
- VoxCPM2 — Backbone architecture and speech-pretrained initialization for VocalRender’s ARDM.
- SoulX-Singer — Strongest baseline. It leads on objective rhythm and pitch alignment but trails VocalRender on all three listening tests.
- Vevo2 — Baseline with a cascaded AR plus DiT design. Its WER degrades sharply on the out-of-domain set.
- YingMusic-Singer — Cited as a DiT SVS system that depends on a time-aligned acoustic melody cue, the input requirement VocalRender removes.
- DiTSinger — Cited as a DiT SVS system that still needs a duration predictor to fix the output length.
- TCSinger 2 — Cited as a DiT-era singing model with a duration-predictor frontend.
- Meta Audiobox Aesthetics — Provides the CE automatic quality metric used in the objective evaluation.