ACL · 2026 · Conference
Yufei Shi et al. (University of Science and Technology of China / Independent Researcher) · → Paper · → Preprint · Demo: ✓ · Code: ✓
Defines speech-singing code-switching synthesis, where a single model decides from the text alone when to speak and when to sing within one utterance, and adapts CosyVoice 2 to do it with synthetic code-switching data, a two-stage curriculum, and an interleaved pitch-token “plan-then-generate” decoding scheme.
Problem
Speech and singing synthesis are usually separate systems: TTS models produce only spoken prosody, while singing voice synthesis and song generation follow scores or lyrics in singing mode only. Unified audio generators such as UniAudio handle both modes but produce one mode per input, chosen by an instruction prompt, and Vevo2 needs reference audio to decide the output mode. The one system that mixes the two inside an utterance (Bark) relies on explicit inline tags. No prior system switches between speaking and singing automatically from textual semantics (for example, a narrator who hums or breaks into a remembered song mid-story), and no training data or benchmark exists for this task. The authors call it Speech-Singing Code-Switching (SCS) synthesis; “code-switching” here means switching vocal mode, not language. All data and evaluation are in English.
The authors also argue that the supervised semantic tokens of CosyVoice 2-style systems discard fine-grained pitch, which flattens speech prosody and singing melody, and that chromagram-based prosody tokens (as in Vevo2) are too coarse at 12-semitone resolution for speech.
Method
UniVocal keeps the CosyVoice 2 pipeline: a text-to-vocal language model (a 24-layer causal Transformer, about 0.5B parameters) generates 25 Hz semantic speech tokens, a flow-matching module turns them, together with an audio prompt, into mel-spectrograms, and a pretrained HiFi-GAN vocoder produces the waveform (§3.2).
Task instructions. CosyVoice 2’s natural-language instruction prefix (terminated by <|endofprompt|>) is reused as a global task signal. Regular TTS uses no instruction; singing uses one of 100 Gemini-written style templates filled with 1 to 5 style tags from the song metadata; SCS uses one of three fixed scenario instructions (“Generate a monologue / a podcast / an audiobook”). No segment-level tags such as <sing> appear in the text, so where to switch must be inferred from the text content (§3.2, Appendix A.3).
Refined cent tokens with chain-of-thought (CoT) decoding. F0 is converted to cents relative to A4 (440 Hz), folded into one octave with a modulo-1200 operation, and rounded up to an integer, giving 1,200 pitch tokens plus one unvoiced token (1,201 new vocabulary entries with randomly initialized embeddings) (§3.1). At every 25 Hz frame the LM first predicts a cent token and then the semantic token conditioned on it, so the joint probability factorizes as cent-then-semantic per step; logit masks enforce the alternation at inference (§3.2, Eq. 3). The authors describe this as drafting a pitch contour (“planning” prosody or melody) before content. The flow-matching module gets a new randomly initialized embedding layer so the cent tokens become an extra condition, and it is fine-tuned on stage-2 data for 2 days (§3.2, Appendix B.2). The cent tokens are optional: a standard configuration omits them (used for SCS and regular TTS) and an expressive configuration uses CoT decoding (used for empathetic TTS and singing). Reported results use whichever configuration the authors consider best for each domain (§4.3.3).

Synthetic code-switching data. Gemini 2.5 Pro (Gemini 2.5) writes first-person monologue, podcast and audiobook scripts that are mostly spoken but include sung or hummed phrases. Speech is kept prose-like while lyrics are lyrical and repetitive (implicit cues); about 50% of samples also get a transitional trigger phrase before the singing (explicit cues, such as “And that reminds me of a lyric…”) (§3.3, Appendix A.1.2). The stage-1 model then synthesizes each segment separately, with speech conditioned on emotion-specific reference audio from Expresso and EmoVoice-DB (9 speakers, 9 emotions) and singing conditioned only on the same speaker’s embedding; segments are concatenated with 0.25 s silences. Samples with Whisper-v3 WER at or above 20% are discarded (about 15% of generated samples). The result is 11,769 samples (261.9 h) from 9 speaker timbres, of which roughly 1,200 are held out as SCSBench (Appendix A.1, Table 9).
Two-stage curriculum. Stage 1 continues pre-training CosyVoice 2’s LM on all of LibriTTS (960 h) and a curated 3,700 h English singing set derived from Suno-generated songs (source separation, DNSMOS/SRMR filtering, dereverberation, FastWhisper lyrics, phonemes-per-second filtering), with a 4:1 singing-to-speech ratio, to align the two modes in one latent space. Stage 2 is supervised fine-tuning on a 1:1:1 mix of synthetic SCS data, a 200 h LibriTTS subset, and 200 h of Suno singing plus 10 h of GTSinger (§3.4, §4.1, Appendix A.2). Training takes about 6 days on 4 A800 GPUs (5 days stage 1, 1 day stage 2) (§4.2, Appendix B.2).
Key Results
Mode switching on SCSBench (Table 1). Baselines are cascades in which Gemini 2.5 Pro first splits the text into speech and singing segments: Gemini + Bark, and Gemini + CosyVoice 2 (speech) + LeVo (singing). F1(O) is judged by Gemini 2.5 Pro with in-context calibration, F1(S) by human annotators.
| Subset | Metric | Gemini + Bark | Gemini + Cosy2 + LeVo | UniVocal |
|---|---|---|---|---|
| Implicit | F1(O) / F1(S) | 0.414 / 0.142 | 0.752 / 0.685 | 0.626 / 0.595 |
| Explicit | F1(O) / F1(S) | 0.533 / 0.250 | 0.572 / 0.489 | 0.714 / 0.635 |
| Mixed | F1(O) / F1(S) | 0.465 / 0.199 | 0.607 / 0.566 | 0.871 / 0.810 |
UniVocal has the highest F1 on the Explicit and Mixed subsets under both judges, but on the Implicit subset (no trigger phrases) the Gemini + CosyVoice 2 + LeVo cascade is higher on both F1(O) and F1(S).
Speech quality on SCSBench (Table 2). UniVocal has the highest UTMOS on all three subsets (4.36 / 4.41 / 4.36 vs. at most 3.62 for the baselines) and the lowest WER on Implicit (5.83 vs. 17.97) and Mixed (10.90 vs. 12.43), but not on Explicit, where the CosyVoice 2 + LeVo cascade has 8.18 vs. UniVocal’s 8.80. Speaker similarity is lower than that cascade on every subset (0.650 / 0.643 / 0.652 vs. 0.758 / 0.763 / 0.773; not computed for Bark, which uses fixed voices). The authors attribute the SIM gap to poor-quality singing training data. On an intra-sample consistency analysis (pairwise speaker similarity between five temporal segments of each sample), UniVocal shows more uniform similarity across segments than the cascade, which switches models between speech and singing; this is shown only as heatmaps (§5.1, Figure 3, Appendix D.1, Figure 4).
Regular and empathetic TTS (Table 3). On Seed-TTS test-en, UniVocal (WER 2.69, SIM 0.703, UTMOS 4.21) has the highest UTMOS (CosyVoice 2: 4.18; F5-TTS and Vevo 1.5: 3.68), the second-lowest WER behind F5-TTS (2.15; CosyVoice 2 2.96, Vevo 1.5 12.58), and the lowest SIM of the four systems (F5-TTS 0.755, CosyVoice 2 0.744, Vevo 1.5 0.718). On a 50-sentence textual empathy set, with all systems conditioned only on speaker timbre so emotion must come from the text, human 3-point E-MOS / P-MOS rise from 1.78 / 1.74 (CosyVoice 2) to 2.26 / 2.22, a gain of 0.48 on both. ElevenLabs multilingual-v2 remains ahead on E-MOS (2.30), P-MOS (2.47) and WER (0.24 vs. UniVocal’s 0.32). Gemini-judged scores give the same system ranking (ElevenLabs > UniVocal > CosyVoice 2) (Appendix D.2, Table 13).
Singing (Table 4). Against Vevo 1.5, YuE and LeVo (the latter two scored on vocal tracks): on GTSinger short phrases UniVocal has the highest AES (5.44) and QUA (10.70) and the lowest WER (18.07 vs. 22.79 for Vevo 1.5, the next best), while its SIM (0.703) is slightly below Vevo 1.5 (0.709). On the 27-track Fullsong set it leads all objective metrics (AES 5.58, WER 35.88, SIM 0.72, QUA 7.75), but in human ratings LeVo is ahead on both N-MOS (2.41 vs. 2.23) and M-MOS (2.34 vs. 2.18), and YuE is also ahead on M-MOS (2.24); UniVocal is above Vevo 1.5 (2.17 / 2.08) and essentially level with YuE on N-MOS (2.23 vs. 2.22). Absolute Fullsong WERs are high for every system (35.88 to 77.60).
Ablations (Table 5). Removing CoT (the standard configuration) raises SCSBench-Mixed F1 from 0.716 to 0.810 but raises its WER from 5.99 to 10.90, and lowers E-MOS (2.26 to 2.03), P-MOS (2.22 to 1.84), Fullsong N-MOS (2.23 to 2.20) and M-MOS (2.18 to 1.86). Training on the full mixture in one stage (w/o CL) drops SCSBench-Mixed F1 to 0.496 and raises its WER to 14.46, while empathy and singing MOS stay close to the full model. Generated cent tokens correlate with cent tokens extracted from the final audio (SRCC 0.633 / 0.679, LCC 0.604 / 0.628 on the empathy and Fullsong sets; Table 6). A stage-1-only model without CoT gives E-MOS similar to w/o CoT, which the authors read as singing data in stage 1 driving much of the empathy gain (§5.2). In a reduced-training resolution ablation, 1,200 bins beat 12 bins on Gemini-judged E-MOS and P-MOS (1.85 / 2.06 vs. 1.57 / 1.63), but on the Fullsong set 12 bins has the highest AES (3.61) and 480 bins the lowest WER (49.72%) (Appendix D.3, Table 14).
Real-world generalization (Table 8). On about 30 minutes of real human speech-singing recordings collected from the internet, UniVocal’s mode-switching F1 is 0.201 vs. 0.452 for the Gemini + CosyVoice 2 + LeVo cascade. After one explicit cue phrase is inserted by hand into each sample’s text (Enhanced SCS), UniVocal reaches 0.730 vs. 0.691.
Fairness of comparisons. SCSBench is a held-out split of the same synthetic pipeline that produced UniVocal’s stage-2 training data, using the same 9 speaker timbres, scenarios and instruction formats, so UniVocal is evaluated in-distribution while the cascades are not. The cascades also depend on Gemini’s text segmentation, and the Gemini-based F1(O) uses the same model family that wrote the scripts. Results for SCS and regular TTS use the standard configuration and those for empathy and singing use the expressive one, so no single configuration achieves all reported numbers.
Novelty Assessment
The task framing is new: the paper defines speech-singing code-switching driven only by text semantics and builds a training set and benchmark for it. The interleaved cent-token decoding, where a high-resolution, octave-folded pitch token is predicted before each semantic token, is a new generation procedure for a semantic-token LM TTS and is the main architectural contribution; the “planning” interpretation is supported by the correlation analysis and ablation, though the gains are concentrated in subjective expressiveness and come with a switching-accuracy cost. The rest is engineering on an existing backbone: CosyVoice 2’s LM, flow-matching decoder and instruction mechanism are reused, the curriculum is continued pre-training plus SFT, and the data pipeline combines an LLM script writer with the authors’ own stage-1 model, an approach whose ceiling is set by that model and by the Suno-derived singing data.
Field Significance
Moderate — The paper shows that a semantic-token LM TTS system can be extended, with a few hundred hours of synthetic data, to choose between speaking and singing from text alone, and that explicit trigger phrases are what make this reliable: implicit-only switching trails a cascade baseline and falls to an F1 of 0.201 on real recordings without an added cue. The octave-folded cent token is a simple, codec-free way to inject pitch into a semantic-token LM, with evidence that it helps expressive speech and singing at some cost in mode-switching accuracy. Because SCSBench is synthetic and in-distribution for the proposed model, the benchmark numbers are better read as a proof of concept than as a measure of real-world performance.
Claims
- supports: Predicting an explicit high-resolution pitch token before each semantic token in a speech-token language model improves the perceived expressiveness of both emotional speech and singing.
Evidence: Removing the interleaved cent-token CoT lowers human E-MOS from 2.26 to 2.03, P-MOS from 2.22 to 1.84 and Fullsong M-MOS from 2.18 to 1.86; generated cent tokens correlate with those extracted from the output audio (SRCC 0.633 to 0.679). (§5.2, Tables 5 and 6)
- complicates: Adding explicit pitch planning to a speech-token language model can trade off against alignment-critical behaviour, so a single decoding configuration may not be best for both expressiveness and structural accuracy.
Evidence: With CoT, SCSBench-Mixed mode-switching F1 drops from 0.810 to 0.716 relative to the configuration without cent tokens, so the authors use different configurations for switching/TTS and for empathy/singing. (§4.3.3, §5.2, Table 5)
- supports: Sequentially aligning speech and singing in a shared model before teaching mixed-mode generation is more effective than training on all data jointly.
Evidence: Skipping the stage-1 alignment and training on the full mixture in one stage lowers SCSBench-Mixed F1 from 0.716 to 0.496 and raises WER from 5.99 to 14.46, while empathy and singing MOS stay close to the full model. (§5.2, Table 5)
- complicates: A text-driven vocal-mode switching model trained on LLM-scripted synthetic data relies on explicit lexical cues and generalizes poorly to purely implicit switches in real speech.
Evidence: On the implicit-only SCSBench subset UniVocal trails a Gemini-segmented CosyVoice 2 + LeVo cascade (F1(S) 0.595 vs. 0.685), and on real recordings its F1 is 0.201 vs. 0.452, rising to 0.730 only after one explicit cue is inserted per sample. (§5.1, §5.4, Tables 1 and 8)
- refines: A single model that generates both speaking and singing segments keeps speaker identity more consistent across mode switches than a cascade of separate speech and singing models, even when the cascade scores higher on global speaker similarity.
Evidence: The CosyVoice 2 + LeVo cascade has higher global SIM on SCSBench (0.758 to 0.773 vs. 0.643 to 0.652), but pairwise similarity between five temporal segments of each sample is more uniform for UniVocal (heatmaps only, no summary number). (§5.1, Table 2, Figure 3)
Limitations and Open Questions
Synthetic, in-distribution benchmark
SCSBench is a held-out split of the same synthetic data UniVocal is fine-tuned on (same pipeline, 9 speaker timbres, scenarios and instructions), produced by the authors’ own stage-1 model. On real speech-singing recordings, mode-switching F1 falls to 0.201, below the cascade baseline (0.452), and recovers only after hand-inserted explicit cues.
- Singing data quality. Singing training data come from Suno-generated songs passed through source separation and ASR; the authors report persistent electronic-tone artifacts, lyric misalignment and weak agreement between style tags and the performance, which they say cap singing fidelity and probably explain the lower speaker similarity (§Limitations, Appendix A.2).
- Evaluator reliability. The objective F1(O) relies on Gemini 2.5 Pro with an in-context calibration prompt. Its sample-level correlation with human labels is modest (Pearson r = 0.343, Spearman ρ = 0.346 on 243 samples); the authors report agreement in system ranking but acknowledge limited sample-level resolution (§Limitations, Appendix C.2.4). The same model family writes the training scripts and segments text for the baselines.
- Coarse subjective scale. All MOS-style ratings use a 3-point scale (1.0 to 3.0), and many differences are a few hundredths of a point (for example N-MOS 2.23 vs. 2.20 vs. 2.17); no confidence intervals or significance tests are reported.
- Configuration switching. Reported results mix the standard and expressive configurations by domain, and the CoT gains in expressiveness come with lower mode-switching F1.
- Scope. English only; fixed scenario instructions (monologue, podcast, audiobook); singing in the SCS data is conditioned only on a speaker embedding, without melody or style control. The resolution ablation was run with reduced training (3 LM epochs, 1 flow-matching epoch) and does not show 1,200 bins winning on the singing metrics.
- Open questions. Whether implicit switching can be learned from real rather than synthetic code-switching data, and whether the cent-token CoT transfers to other semantic-token TTS backbones (the authors call the framework model-agnostic but test only CosyVoice 2).
Wiki Connections
- Singing Voice Synthesis and Conversion — Generates singing from lyrics plus style instructions and also mixes singing and speech within one utterance, evaluated with musically trained raters.
- Autoregressive Codec TTS — Extends a semantic-token AR LM with interleaved pitch tokens predicted before each speech token.
- Flow Matching — Fine-tunes CosyVoice 2’s flow-matching decoder with a new embedding layer that takes the cent tokens as an extra condition.
- Emotion Synthesis — Reports that emotion inferred from text alone improves over the CosyVoice 2 backbone (human E-MOS 1.78 to 2.26), which the authors attribute to singing-data pre-training amplified by pitch-token planning.
- Evaluation Metrics — Introduces SCSBench and an in-context-calibrated LLM judge for speech-vs-singing mode-switching F1, validated against human labels.
- Subjective Evaluation — Uses human F1 labels for mode switching and 3-point E-MOS, P-MOS, N-MOS and M-MOS ratings with at least three annotators per sample.
- CosyVoice 2 — Backbone that UniVocal continues training from, and a baseline for regular TTS, empathetic TTS, and (as the speech half of a cascade) code-switching.
- Vevo2 — Unified speech and singing model that needs reference audio to pick the mode; UniVocal contrasts its chromagram tokens with the finer cent token.
- F5-TTS — Baseline on Seed-TTS test-en, where it has lower WER and higher SIM than UniVocal.
- Seed-TTS — Source of the Seed-TTS test-en benchmark used to check that regular TTS quality is preserved.
- Gemini 2.5 — Writes the code-switching scripts, segments text for the cascade baselines, and serves as the calibrated F1(O) judge.
- HiFi-GAN — Pretrained vocoder used unchanged to render the mel-spectrograms.
- LibriTTS — Speech training data for both curriculum stages.
- Meta Audiobox Aesthetics — Provides the AES aesthetic-quality metric used for singing evaluation.