arXiv · 2025 · Preprint
Xueyan Li et al. (Shanghai Artificial Intelligence Laboratory) · → Paper · Demo: ? · Code: ✓
Proposes a generative feedback framework that fine-tunes an audio-language model to produce multi-dimensional text-and-audio critiques of singing performances, replacing single-scalar reward scores used to evaluate singing voice synthesis.
Problem
Reward systems used to evaluate and optimize singing voice synthesis (SVS) typically compress quality assessment into a single numerical score. This oversimplifies singing quality, obscures which dimensions (phrasing, expressiveness, melody, auditory quality) drive a score, and complicates statistical analysis of failure modes. Rule-based rewards built on explicit music-theoretic criteria are interpretable but generalize poorly to unseen singers or novel styles. Learned, data-driven reward models trained on human preferences avoid hand-crafted rules and can be paired with RL fine-tuning algorithms such as PPO, but they still typically require large volumes of costly, expertly-annotated data and still output a single scalar. The paper targets all three limitations at once: interpretability, generalization, and annotation cost.
Method
The framework treats SVS evaluation as a generation problem rather than a regression problem: given a singing audio clip and a contextual text prompt (song background plus a stylistic critic persona), an audio-language model generates both a text critique and a spoken audio critique covering dimensions such as melody, content, and auditory quality.
The backbone is an existing pretrained audio-language model (Qwen2.5-Omni-7B or Kimi-Audio), fine-tuned with LoRA rather than trained from scratch. Model parameters are split into a shared LLM backbone (θ_shared) plus separate text and audio generation heads (θ_text, θ_audio). Input music and persona text are tokenized (music into discrete audio tokens via the backbone’s audio tokenizer) and passed through the shared backbone to produce a joint hidden representation, from which the two heads autoregressively generate the target text sequence and target audio token sequence respectively. Training minimizes a weighted sum of the text and audio cross-entropy losses, with gradients from both heads flowing back to jointly update the shared backbone so that the shared representation is informative for both modalities.
Training data combines two sources. Human Reaction Data is mined from Bilibili reaction videos: an ASR module with timestamps extracts spoken commentary, the interval between consecutive comments is treated as the music segment under review (capped at 30 seconds), and AudioSep (a prompt-based source-separation tool) removes reviewer speech that overlaps with the music, yielding (music, reaction text, reaction speech) triplets. MLLM-generated Data covers ten genres with representative songs; a multimodal LLM (Gemini) generates persona-conditioned critiques from structured prompts and per-song metadata, giving standardized, systematic coverage. A FAISS-based similarity filter discards near-duplicate samples (cosine similarity > 0.95) before a 10% evaluation split is held out. Text-only fine-tuning uses LoRA rank 8 (lr 1e-4) on Qwen2.5-Omni-7B; joint audio-text fine-tuning uses LoRA rank 16 (lr 1e-5) on Kimi-Audio, with the text/audio loss balance weight λ = 2/3. All runs use a single NVIDIA A800 GPU for 3 epochs (about 3.5 hours for a single data source, about 7 hours for the hybrid dataset).
Key Results
The paper evaluates the fine-tuned critique model on its own LLM-based benchmark: single-choice questions (SCQ) testing core musicianship (vocal technique, emotion/expression, musical knowledge, instrumentation) and open-ended questions (OEQ) scored by an LLM judge on completeness, accuracy, and novelty (weighted 0.2/0.6/0.2). Fine-tuning on Qwen2.5-Omni-7B with the hybrid dataset raises SCQ accuracy to 0.65, exceeding proprietary baselines GPT-4o-Audio (0.583), Gemini-2.5-Pro (0.450), and the un-tuned base model (0.200). Training on human-only data alone pushes SCQ accuracy even higher (0.60, still below the hybrid result of 0.65) but OEQ average collapses to 0.375, well below the untrained base model’s OEQ average of 0.683; training on MLLM-only data keeps OEQ balanced (0.739 average) but leaves SCQ accuracy at only 0.375. The hybrid configuration is reported as the best trade-off between the two failure modes (SCQ 0.65, OEQ weighted average 0.577), though its OEQ average is not the highest among the tested configurations. A separate ablation isolates a data-quality confound in the Human Reaction Data: because reviewer speech sometimes overlaps the music being reviewed, the raw audio input can closely resemble the target critique, and training on this unseparated audio yields substantially higher validation loss than training on AudioSep-separated audio.
Novelty Assessment
The architectural contribution is limited: the framework fine-tunes existing pretrained audio-language models (Qwen2.5-Omni, Kimi-Audio) with LoRA rather than introducing a new model design. The genuine contributions are (1) reframing SVS/singing evaluation as multi-dimensional generative text-and-audio feedback instead of scalar reward regression, (2) a hybrid data-construction pipeline that combines mined human reaction commentary with MLLM-generated critiques, including a source-separation preprocessing step that resolves a target-leakage confound in the human data, and (3) an accompanying LLM-based SCQ/OEQ benchmark for measuring critique quality. A notable methodological gap given the paper’s own motivation (moving past unreliable, hard-to-interpret evaluation) is that the benchmark’s OEQ component is itself scored by an LLM judge rather than validated against human ratings, and the paper reports no downstream experiment in which the resulting feedback signal is actually used to fine-tune an SVS model via RL, despite motivating the work with that use case.
Field Significance
Moderate — the paper contributes a data pipeline and matching evaluation benchmark for framing singing-performance evaluation as generative multi-modal feedback rather than scalar reward regression, and demonstrates that combining authentic human commentary with standardized synthetic critiques addresses complementary failure modes. Its significance is constrained by self-referential validation (an LLM-judge benchmark built from the same data-generation process it evaluates) and by not yet closing the loop with an actual RL fine-tuning experiment on an SVS model, which is the framework’s stated end use case.
Claims
- supports: Framing generative-model evaluation as multi-dimensional language-and-audio feedback, rather than a single scalar reward, can preserve interpretability across separate quality dimensions while remaining usable as a training signal.
Evidence: The proposed framework generates parallel text and audio critiques spanning melody, content, and auditory-quality dimensions instead of a single numeric reward, directly targeting the interpretability limits of prior scalar SVS reward models. (§1, §2.2)
- supports: Combining authentic human-generated feedback data with standardized synthetic (MLLM-generated) feedback data captures complementary properties that neither source provides alone.
Evidence: Human-only training raises SCQ accuracy to 0.60 (above GPT-4o-Audio’s 0.583) but collapses OEQ average to 0.375, while MLLM-only training preserves balanced OEQ (0.739) with weaker SCQ (0.375); hybrid training achieves the best combined trade-off (SCQ 0.65, OEQ weighted average 0.577). (§3, Table 1)
- complicates: Reaction or commentary data mined from in-the-wild recordings can leak target-like signal into the model’s input, letting a feedback-generation model shortcut-learn instead of genuinely inferring critique from the underlying performance.
Evidence: In the raw Human Reaction Data, the reviewer’s overlapping speech closely resembled the target critique text; removing it with source separation (AudioSep) produced significantly lower validation loss than training on the unseparated audio. (§Appendix F, Figure 5)
- supports: Joint autoregressive supervision of text and audio outputs from a shared LLM backbone can induce expressive, affect-laden audio-generation behavior beyond what the base model exhibits.
Evidence: Case studies comparing the base Kimi-Audio model to fine-tuning checkpoints show the final model producing emotionally intonated spoken reactions and, in one case, spontaneous humming in response to musical input, capabilities absent from the base model. (§3 “Multi-modal Supervision”, §Appendix E)
Limitations and Open Questions
Warning
The framework’s own evaluation relies entirely on an LLM-as-judge benchmark (SCQ accuracy plus LLM-judged OEQ completeness/accuracy/novelty) rather than human listening tests, and no experiment fine-tunes an SVS model via RL using the generated feedback signal, so the paper’s central motivating claim (that this feedback can guide generative model improvement) is not directly demonstrated.
Additional limitations noted or evident in the paper: training and evaluation data are sourced mainly from a single Chinese video platform (Bilibili) plus Gemini-generated critiques in Chinese and English, leaving generalization to other languages and critique cultures untested; the authors themselves report that generated text can be overly concise and synthesized-speech clarity/articulation varies, attributing this to short, unstructured commentary in the Human Reaction Data (§Appendix E); and the full SCQ benchmark is not released, with only illustrative example questions provided (Table 4).
Wiki Connections
- Spoken Language Model — fine-tunes existing audio-language model backbones (Qwen2.5-Omni, Kimi-Audio) to consume an external singing/music recording and jointly generate text and spoken audio output, applying the spoken-language-model paradigm to a critique/feedback task rather than conversational response.
- Evaluation Metrics — proposes replacing single-scalar reward scores for singing voice synthesis with multi-dimensional language-and-audio feedback, and introduces a matching SCQ/OEQ LLM-based benchmark to measure feedback quality.
- RLHF Speech — positions the generative feedback model as a richer alternative to scalar reward models that are used to fine-tune SVS systems via RL algorithms such as PPO, though this paper does not itself run an RL fine-tuning experiment.
- Singing Voice Synthesis and Conversion — builds an evaluation and feedback framework specifically targeting singing voice synthesis output quality, not general audio or speech quality.