arXiv · 2026 · Preprint

Yizhong Geng et al. (Beijing University of Posts and Telecommunications / Li Auto) · → Paper · Demo: ✓ · Code: ✗

MeloCodec adds a separately pre-trained, quantized chromagram branch to an RVQ singing-voice codec, and fuses its melody tokens with the acoustic stream only after that branch is frozen. This “Tokenize-then-Fuse” design improves pitch accuracy at low bitrate and allows pitch shifting with little timbre loss.

Problem

Neural audio codecs act as tokenizers for LLM-based audio generation. Recent codecs add semantic priors (SSL features or ASR supervision) to make tokens more linguistically aligned. Singing depends on precise pitch, and these semantic priors do not protect it. Implicit disentanglement codecs such as FACodec rely on multi-objective constraints that the authors describe as unstable and prone to information leakage. The authors also argue that naively concatenating an explicit acoustic prior such as a chromagram with the waveform encoder output fails. The acoustic stream already carries a superset of the melody information, so the decoder learns to ignore the prior (shortcut learning). In addition, the joint gradients conflict and destabilize training. The arXiv comment states that the paper was accepted at IEEE ICME 2026; this page summarizes the arXiv version.

Method

MeloCodec has two streams (§II.A). The Tokenize Branch (§II.B) extracts a 12-bin chromagram from the waveform with an STFT followed by pitch-class projection. A convolutional melody encoder maps the chromagram to latent features. An RVQ quantizes these features into melody tokens M_q, and a symmetric melody decoder reconstructs the chromagram. This branch is pre-trained on its own with a VQ-VAE objective (chromagram reconstruction plus a commitment term). The quantization acts as an information bottleneck that strips timbral detail from the melodic prior.

The Fuse Branch (§II.C) encodes the raw waveform with a convolutional acoustic encoder. It linearly projects M_q into the same space, concatenates the two, and quantizes the result with a second RVQ into unified tokens U_q. The acoustic decoder reconstructs the waveform from U_q alone. An auxiliary head predicts melody tokens from U_q, and these are decoded through the melody decoder into a chromagram. The resulting auxiliary loss forces the unified tokens to keep melodic structure. Only U_q is transmitted, so the reported bitrates count unified tokens only, and the melody tokens are not a side channel (§III.A).

The architecture of MeloCodec implementing the 'Tokenize-then-Fuse' paradigm. The framework consists of a Tokenize Branch (Green) for discrete structural modeling and a Fuse Branch (Blue) for acoustic reconstruction. The switch K governs the data flow across the robust two-stage training strategy, where dashed boxes explicitly delimit the scope of active gradient updates: (1) In Stage 1, K routes the quantized melody M q directly to the decoder to pre-train the bottleneck; (2) In Stage 2, K switches to the predicted auxiliary path, allowing gradients to update the Fuse Branch and fine-tune the Melody Decoder, while the pre-trained Melody Encoder and RVQ remains frozen.

Training has two stages (§II.D). Stage 1 trains only the Tokenize Branch. Stage 2 freezes the melody encoder and its RVQ. It fine-tunes the melody decoder, because the authors observed that a frozen decoder caused optimization conflicts, and trains the Fuse Branch end-to-end. The Stage 2 loss combines an L1 waveform loss and a multi-scale STFT loss, a hinge adversarial loss from a multi-scale discriminator (MSD) with feature matching, and the auxiliary chromagram loss. Codebooks have 1,024 entries with EMA updates, and the token rate is 75 Hz. Two operating points are evaluated: Nq = 8 (about 6.0 kbps) and Nq = 2 (about 1.5 kbps). Inference uses a sliding window with 50% overlap. Model size is not reported.

Key Results

All models train on an internal 5,500-hour Mandarin singing corpus at 44.1 kHz and are tested on the held-out Opencpop set (§III.A). EnCodec, DAC and X-Codec are fine-tuned on the same internal data, which makes the comparison with them reasonably controlled. MuCodec uses official weights as a zero-shot reference, and it runs at a different operating point (1.33 kbps, 25 Hz, 10,000-entry codebooks, Nq = 4).

  • High bitrate, about 6.0 kbps (Table I). MeloCodec gives the best value of every system in all five columns. ViSQOL is 4.08, against 3.97 (DAC), 3.92 (EnCodec) and 3.34 (X-Codec). STOI is 0.85, against 0.83 / 0.82 / 0.81. F0-RMSE is 0.96, against 1.23 / 1.92 / 1.35. Chroma-Sim is 0.97, against 0.96 / 0.93 / 0.95. SPK-SIM is 0.99, against 0.98 / 0.97 / 0.96. Apart from F0-RMSE, these margins over DAC are small.
  • Low bitrate, about 1.5 kbps (Table I). MeloCodec again gives the best value in all five columns. ViSQOL is 3.38, against 3.05 (DAC), 2.81 (EnCodec), 2.74 (X-Codec) and 2.62 (MuCodec). STOI is 0.78, against 0.71 / 0.68 / 0.66 / 0.57. F0-RMSE is 1.64, against 4.12 / 4.87 / 4.68 / 2.68. Chroma-Sim is 0.95, against 0.84 / 0.81 / 0.83 / 0.92. SPK-SIM is 0.94, against 0.91 / 0.88 / 0.87 / 0.80. The pitch gain is largest here: F0-RMSE falls by 2.48 relative to DAC. MuCodec keeps pitch better than the RVQ baselines but has the lowest ViSQOL, STOI and SPK-SIM of the five systems.
  • MUSHRA (Figure 2). Fifteen expert listeners each rated 30 samples. The authors report the highest mean score for MeloCodec at both bitrates, close to the reference at high bitrate. The scores are only plotted with 95% confidence intervals, so no numbers are given in the text.
  • Ablation (Table II). At 1.5 kbps, adding only an auxiliary melody loss to a single-stream codec lowers F0-RMSE from 4.12 to 2.10, and ViSQOL changes from 3.05 to 3.25. Direct end-to-end dual-stream fusion with MLP projection and normalization is worse than the acoustic-only codec on every metric: F0-RMSE 5.95 and ViSQOL 2.34. The authors report codebook utilization below 5% early in training and a diverging chroma validation loss for this variant. The full Tokenize-then-Fuse model is best in every column at both bitrates. The “Acoustic-only” rows report the same values as the DAC rows in Table I.
  • Pitch shifting (Table III). The chromagram prior is shifted by ±2 or ±4 semitones while the acoustic features stay fixed. Target F0 correlation is 0.72 / 0.70 for MeloCodec, against 0.40 / 0.27 for Direct Fusion. Target F0-RMSE is 3.85 / 4.50, against 4.10 / 6.80. SPK-SIM degradation is 1.6% / 3.9%, against 7.6% / 14.0%. The authors acknowledge that absolute pitch precision stays limited, because the unchanged acoustic stream still carries the original pitch.

Novelty Assessment

The architectural contribution is a training procedure. It pre-trains and quantizes an explicit DSP prior (the chromagram), freezes it, and fuses it with the acoustic latent before the final RVQ, with an auxiliary loss that predicts the prior back from the fused tokens. Each part is familiar: VQ-VAE pre-training, RVQ, concatenation fusion, an auxiliary reconstruction loss, and DAC-style GAN losses. The new element is the ordering (quantize the prior first, fuse second) and the demonstration that this ordering avoids the collapse seen with direct fusion. The ablation in Table II supports this design better than the main comparison does. It isolates the auxiliary loss from the two-stage fused design and shows that direct fusion collapses.

The evidence has clear limits. There is one language and one test set (Mandarin, Opencpop), the training data is internal, and model size and compute are not reported. The pitch-shift experiment compares only against the Direct Fusion ablation, not against other controllable codecs. Claims that the design is a “scalable blueprint” for physical priors in general go beyond the experiments, which use only one prior (the chromagram) and only singing.

Field Significance

Moderate — MeloCodec shows that an explicit pitch-class prior can be added to an RVQ codec when the prior is quantized and frozen before fusion, while naive joint fusion collapses. The pitch-accuracy gain is largest at low bitrate. The evidence covers a single Mandarin singing test set, and the work reads as a well-ablated sub-area step for singing-voice tokenizers rather than a general codec advance.

Claims

  • supports: Explicit melodic priors improve pitch preservation in neural codecs for singing, with the largest benefit at low bitrates where standard RVQ codecs lose pitch structure.

    Evidence: At about 1.5 kbps on Opencpop, MeloCodec reaches F0-RMSE 1.64 against 4.12 for a DAC baseline fine-tuned on the same data, and 4.87 for EnCodec. At about 6.0 kbps the gap shrinks to 0.96 against 1.23. (§III.B, Table I)

  • complicates: Directly fusing a deterministic acoustic prior with learned waveform latents in end-to-end codec training can collapse, because the decoder learns to bypass the prior.

    Evidence: The one-stage Direct Fusion variant is worse than the acoustic-only codec on every metric (F0-RMSE 5.95 against 4.12 at 1.5 kbps). Codebook utilization drops below 5% early in training, and the auxiliary chroma validation loss diverges while the training loss falls. (§III.D, Table II)

  • refines: An auxiliary reconstruction loss alone recovers much of the pitch accuracy of an explicit prior, but quantizing the prior and pre-training it before fusion gives further gains in both pitch and overall quality.

    Evidence: At 1.5 kbps, adding an auxiliary melody loss to a single-stream codec lowers F0-RMSE from 4.12 to 2.10. The full two-stage Tokenize-then-Fuse model reaches 1.64 and raises ViSQOL from 3.25 to 3.38. (§III.D, Table II)

  • supports: A separately quantized melodic stream inside a codec can support pitch manipulation at the token level with limited speaker-similarity loss.

    Evidence: Shifting the chromagram prior by ±2 / ±4 semitones gives target F0 correlation 0.72 / 0.70 and SPK-SIM degradation 1.6% / 3.9%. Direct Fusion gives 0.40 / 0.27 and 7.6% / 14.0%. (§III.C, Table III)

  • complicates: Pitch control through an explicit prior remains imprecise when the acoustic stream still encodes the original pitch.

    Evidence: Under pitch shifting, target F0-RMSE stays elevated at 3.85 to 4.50 for MeloCodec, which the authors attribute to constraints from the fixed acoustic branch. (§III.C, Table III)

Limitations and Open Questions

Warning

All training uses an internal 5,500-hour Mandarin singing corpus, and the only test set is Opencpop (Mandarin pop singing). No code is released and model size is not reported, so neither the generality across languages or singing styles nor the reproducibility of the results can be checked.

  • No speech, multilingual, or polyphonic or accompanied evaluation is reported. Generality beyond the chromagram prior is asserted, not tested.
  • MUSHRA results appear only as a plot, with no numbers in the text.
  • MuCodec runs at a different bitrate, token rate and codebook configuration, and it is not fine-tuned on the internal data, so its comparison is less controlled than the comparisons with EnCodec, DAC and X-Codec.
  • The pitch-shift test has no baseline outside the paper’s own ablation. It is unclear how the method compares with codecs or decoders that condition directly on F0.
  • No downstream generation experiment (for example, an LM trained on MeloCodec tokens for singing voice synthesis) tests the stated motivation that the codec serves as a tokenizer for LLM-based audio generation.

Wiki Connections

  • Neural Audio Codec — Adds an explicit, pre-quantized chromagram prior to an RVQ codec. The paper’s main evidence concerns pitch preservation at low bitrate.
  • Singing — Targets singing-voice representation and evaluates reconstruction and pitch manipulation on Opencpop.
  • Disentanglement — Separates melody into its own bottlenecked token stream, and an ablation on pitch-shift timbre stability tests the separation against direct fusion.
  • Prosody Control — Shifts the melodic prior by semitones to change pitch, evaluated by target F0 correlation and speaker-similarity degradation.
  • Subjective Evaluation — Reports a MUSHRA test with 15 expert listeners on 30 samples at two bitrates.
  • EnCodec — Used as a baseline fine-tuned on the same singing data. Its pitch accuracy degrades sharply at 1.5 kbps.
  • SpeechTokenizer — Cited as a semantic-prior codec that fuses SSL features. MeloCodec argues that such priors neglect pitch-sensitive acoustics.
  • NaturalSpeech 3 — Its FACodec is cited as the implicit-disentanglement alternative that MeloCodec positions itself against.
  • Spark-TTS — Its BiCodec is cited as another semantic-prior tokenizer in the motivation.
  • CosyVoice 2 — Cited for ASR-supervised tokenization as a semantic prior.
  • Vevo2 — Cited as related work that uses explicit chromagram-style priors for speech and singing generation.
  • CoMelSinger — Discrete-token singing synthesis with melody control, cited for the shortcut-learning and gradient-conflict problems that motivate the Tokenize-then-Fuse ordering.
  • HQ-SVC — Prior singing voice conversion work from overlapping authors, cited for the optimization difficulty of fusing heterogeneous features.