arXiv · 2026 · Preprint
Shen et al. (Meta Superintelligence Labs) · → Paper · Demo: ? · Code: ?
Introduces GSRM, a reasoning-centric reward model that grounds speech-naturalness judgments in explicitly extracted acoustic evidence before producing chain-of-thought reasoning, then uses it as a verifier to run online RLHF that measurably improves the naturalness of a full-duplex speech LLM’s generations.
Problem
Speech language models such as GPT-4o Voice Mode and Gemini Live still produce audio that sounds less natural than human speech, with gaps in prosody, expressiveness, and emotional nuance. Closing that gap requires a reliable evaluator of speech naturalness, but existing evaluators (MOS predictors such as MOSNet, NISQA, UTMOSv2) regress raw audio to a single scalar score. A single regressor forces naturalness, an inherently multi-dimensional perceptual quality spanning expressiveness, intonation, and pacing, into one number, which the paper argues limits both interpretability and generalization to out-of-domain speech. Generative reward models (GRMs) that produce reasoning traces alongside judgments have recently improved reward modeling in text domains, but the paper argues that speech GRMs remain under-explored: concurrent work such as WavReward and SpeechJudge synthesizes chain-of-thought (CoT) reasoning using a teacher speech LLM conditioned directly on raw audio, which the authors argue mediates reasoning through implicit, unverifiable acoustic representations. No prior work, per the paper, had used a generative speech reward model to run online reinforcement learning that improves the naturalness of a full-duplex, interactive speech LLM.
Method
GSRM (Generative Speech Reward Model) decomposes speech naturalness judgment into two explicit stages: (1) extracting interpretable paralinguistic acoustic features from raw audio, and (2) generating feature-grounded chain-of-thought (CoT) reasoning that maps this evidence to quantitative ratings. The acoustic feature extraction stage performs phoneme-level forced alignment between an utterance’s transcript and waveform, retains only vowel segments (treated as the acoustically stable carriers of prosodic information), and computes six low-level features per vowel: pitch level, pitch variation, pitch slope, intensity level, intensity variation, and duration. Continuous values are speaker- and vowel-type normalized, then discretized into ordinal categories (e.g., very low to very high) to form an utterance-level acoustic feature log.
To synthesize training data, a teacher text LLM (GPT-4o) is prompted with each utterance’s transcript and its vowel-level acoustic feature log to generate a structured, four-dimension evidence analysis (inferred context, non-speech vocalizations/fillers, positive attributes, potential issues). The utterance-level evidence logs for a full recording are concatenated with a synthesized global judgment CoT that maps the entire evidence log to the oracle human ratings for seven sub-metrics (expressive intensity, expressive correctness, intonation, non-speech vocalizations, mispronunciation, pacing) plus an overall human-likeness score. This produces a complete CoT trajectory used to supervise-fine-tune (SFT) Qwen2.5-Omni-7B (full-parameter, 10 epochs) to map raw audio directly to a CoT trace and final ratings.
At inference time, GSRM is deployed as a speech-in, text-out model: given raw audio it produces a CoT reasoning trace followed by numerical sub-metric and overall ratings. Test-time scaling (averaging predictions across multiple random samples per input) is used to reduce prediction variance. To apply GSRM as a reward signal for online RLHF, the authors extend it beyond acoustic naturalness to also judge semantic quality (language complexity, contextual awareness, spontaneity) of an in-house full-duplex speech LLM’s spoken responses, using automatically generated labels from a teacher model (GPT-OSS-120B) rather than direct human annotation for this semantic sub-model. In the RLHF loop, a speech LLM generator produces candidate spoken responses to a user query; each response is transcribed via Whisper ASR to isolate semantic content, and both the audio and its transcript are scored by GSRM across acoustic and semantic sub-dimensions. A reward aggregator uniformly averages these sub-metric scores (with GSRM run multiple times per response to reduce reward variance) into a scalar reward used to update the generator online via GRPO.
The training data comes from two datasets curated for this work: ConvTTS, 6,579 two-channel dialogue samples generated by an in-house controllable TTS system (4,579 train / 1,000 dev / 1,000 test), rated by an average of 6.8 human annotators per sample for audio naturalness; and FDX-Conv, 490 dialogue samples generated by a separate in-house full-duplex speech LLM, used exclusively as an out-of-domain (OOD) generalization benchmark since it comes from a different generating system than ConvTTS.
Key Results
On the main naturalness-prediction benchmark (Table 5), human inter-rater consistency (an oracle upper bound on any predictor) is Pearson 0.533 / Spearman 0.517 on the ConvTTS validation set and 0.532 / 0.489 on the FDX-Conv OOD set. GSRM reaches Pearson 0.401 / Spearman 0.375 on validation and Pearson 0.465 / Spearman 0.427 on the OOD set, approaching the human ceiling on OOD data and substantially exceeding all baselines, including NISQA and UTMOSv2 (both near-zero or negative correlation), a WavLLM-based regressor (0.331 val / 0.222 OOD), and an AES-based regressor built on Meta Audiobox Aesthetics (0.384 val / 0.236 OOD). A pilot study (Table 2) further shows that prompting a frontier text-only LLM (Gemini-2.5-Pro) with explicit extracted acoustic features outperforms prompting a frontier speech LLM directly with raw audio in voice mode, which the authors interpret as evidence that the bottleneck for frontier models is extracting paralinguistic cues, not reasoning over them.
Ablations (§6.3) show that removing the utterance-level evidence log and training on the global judgment CoT alone drops OOD Pearson by about 0.08, and that among the six acoustic feature groups, removing pitch-related features causes the largest degradation (over 0.12 Pearson), with duration having a moderate effect and intensity the least. Among sub-metrics, intonation and pacing contribute the most to naturalness prediction. Scaling experiments (Figure 4) show OOD Pearson improves steadily with more SFT training data (500 to 4,579 samples) and with more test-time inference samples (K = 1 to 16), with the largest gains occurring between K = 1 and K = 4.
Applied as an online RLHF verifier for an in-house full-duplex speech LLM (GRPO, 9.2K training prompts, up to 2,000 steps), the RLHF-trained model beats the base SFT model in human A/B pairwise evaluation (5 raters per pair, held-out prompts) on overall naturalness in 82% of comparisons (12% base model wins, 6% ties), and also wins on tone (74%), pacing (60%), and intonation (66%) (Table 6).
Novelty Assessment
The core novelty is architectural: GSRM’s two-stage decomposition, explicit sufficient-statistic acoustic feature extraction followed by feature-grounded CoT reasoning, is a genuinely new design for speech reward modeling that differs concretely from the closest concurrent systems (WavReward, SpeechJudge), which synthesize CoT via a teacher speech LLM conditioned on raw audio and therefore inherit whatever acoustic understanding that teacher happens to encode. GSRM’s explicit feature layer instead makes the evidence auditable and enables systematic ablation of which acoustic cues drive judgments, which the paper backs with feature- and sub-metric-level ablations. The paper is also a genuine dataset contribution: 31k expert ratings across a purpose-built 6,579-sample dialogue corpus (ConvTTS) plus a distinct out-of-domain benchmark (FDX-Conv) from a different generating system, with measured inter-rater consistency used as an interpretive ceiling. The RLHF application is best read as the paper’s evaluation contribution rather than a new training algorithm: it uses a standard GRPO loop, but is, per the authors’ own related-work framing, the first to close the loop by using a generative speech reward model to run online RL on a full-duplex, interactive speech LLM and validate the naturalness gain against human raters. The appendix RLVR ablation for GSRM itself (Appendix E.2) is an honest negative result within the paper: naive RL fine-tuning of the reward model collapses to near-constant predictions under an L2 distance reward and underperforms plain SFT even under an RBF-kernel reward.
Field Significance
High significance
GSRM demonstrates that decomposing speech naturalness judgment into explicit, auditable acoustic evidence plus grounded chain-of-thought reasoning meaningfully narrows the gap toward human-level correlation, and it closes the loop by using that reward model to run online RLHF on a production-scale full-duplex speech LLM, measuring the resulting naturalness gain with human A/B testing rather than only automatic metrics. It provides a concrete architectural template, a large human-annotated naturalness dataset design, and a documented negative result on RL fine-tuning the reward model itself, all of which are directly reusable for future speech RLHF work.
Claims
- supports: Decomposing speech-quality judgment into explicit acoustic evidence extraction followed by grounded reasoning improves agreement with human naturalness ratings over black-box scalar regressors, particularly for out-of-domain generalization.
Evidence: GSRM reaches OOD Pearson 0.465 versus 0.236 for the best scalar regressor baseline (AES-based) and 0.222–0.245 for WavLLM- and UTMOSv2-based baselines (§6.2, Table 5).
- supports: Conditioning a reasoning model on explicitly extracted, structured acoustic features is more effective for eliciting reliable paralinguistic judgments than prompting a frontier model directly with raw audio.
Evidence: Gemini-2.5-Pro prompted with textual acoustic-feature descriptions (no audio access) outperforms Gemini-2.5-Pro prompted directly with speech in both in-domain and OOD naturalness prediction (§3, Table 2).
- supports: A generative reward model trained on human naturalness ratings can serve as an effective online RLHF verifier, producing generation-quality gains a human panel can detect.
Evidence: After GRPO training of an in-house full-duplex speech LLM against GSRM as the reward source, the RLHF-trained model wins 82% of human A/B pairwise comparisons for overall naturalness against the base SFT model (§6.4, Table 6).
- refines: For generative speech reward models, prediction reliability can be improved at inference time by increasing the number of sampled judgments, without any additional training.
Evidence: OOD Pearson correlation increases monotonically as the number of test-time inference samples K grows from 1 to 16, with the largest gains between K = 1 and K = 4 (§6.2, Figure 4b).
- complicates: Even the best current generative speech reward models remain well short of human inter-rater agreement, and naively applying reinforcement learning to train the reward model itself can be unstable.
Evidence: GSRM’s validation Pearson (0.401) trails the human inter-rater oracle (0.533), and an RL-trained variant of GSRM using an L2-distance reward collapses to near-constant score predictions (reward hacking), underperforming plain supervised fine-tuning (§6.2, Table 5; Appendix E.2, Table 11).
Limitations and Open Questions
Reproducibility depends on unreleased, proprietary systems
GSRM’s training and evaluation datasets (ConvTTS, FDX-Conv, the 9.2K-prompt RLHF set) are generated by in-house TTS and full-duplex speech LLM systems that are not described in architectural detail or released, and the paper reports no code, model weight, or dataset release. The reported gains are therefore difficult for outside researchers to independently reproduce or extend.
Beyond reproducibility, several limitations are visible directly in the paper’s own results. GSRM’s absolute correlation with human raters remains meaningfully below the human inter-rater ceiling even in-domain (Pearson 0.401 vs. 0.533 oracle), so a gap in reward reliability persists despite outperforming all tested baselines. The CoT reasoning traces used to train GSRM are synthesized by a teacher text LLM (GPT-4o) conditioned on the extracted acoustic features and oracle ratings, not authored or verified by the human raters themselves, so systematic biases in GPT-4o’s acoustic reasoning could propagate into GSRM’s judgments. The semantic-judgment extension of GSRM (used in the RLHF loop) is trained on labels from an automated teacher model (GPT-OSS-120B) rather than direct human annotation, and the paper reports that its accuracy at identifying lack of contextual awareness in lower-quality samples remains an open area for improvement (§E.3). Finally, the paper’s own reinforcement-learning ablation for training GSRM (rather than using GSRM to train a generator) shows that naive RL objectives for the reward model are prone to instability, which the authors attribute to Qwen2.5-Omni-7B lacking sufficient prior speech-judgment knowledge without an SFT warm-up stage.
Wiki Connections
- RLHF Speech — GSRM is the reward and verifier model used to run online RLHF (GRPO) that measurably improves the naturalness of a full-duplex speech LLM’s generations, extending speech RLHF beyond single-utterance TTS to interactive dialogue.
- Spoken Language Model — GSRM is built by fine-tuning Qwen2.5-Omni-7B, an adapted LLM that consumes external raw audio (speech it did not itself generate) and produces text-based reasoning and ratings.
- Evaluation Metrics — introduces a generative-reward evaluation methodology, feature-grounded CoT judgments scored against human ratings via Pearson, Spearman, and MSE, that substantially outperforms existing scalar naturalness predictors.
- Subjective Evaluation — both trains and validates the reward model against large-scale human ratings (31k expert annotations across ConvTTS and FDX-Conv) and evaluates the RLHF-trained system via human A/B listening tests.
- WavReward — the closest concurrent generative reward model for spoken dialogue systems; GSRM differs by grounding CoT synthesis in explicit extracted acoustic features rather than a teacher speech LLM’s implicit audio understanding.
- Meta Audiobox Aesthetics — its AES Transformer backbone is fine-tuned as the strongest scalar-regressor baseline in this paper’s experiments, which GSRM outperforms especially on out-of-domain generalization.
- UTMOS — UTMOSv2, a direct descendant of this MOS-prediction line, is used as a baseline naturalness predictor that GSRM substantially outperforms on both in-domain and OOD correlation.
- AudioJudge — its finding that frontier speech LLMs struggle to reliably judge paralinguistic attributes under prompting motivates this paper’s pilot study, which reproduces a similar gap for Gemini-2.5-Pro speech prompting.
- Qwen2.5-Omni Technical Report — provides the 7B multimodal backbone that GSRM fine-tunes for its speech-in, text-out reward modeling task.