arXiv · 2025 · Preprint
Zhiyu Lin et al. (The Chinese University of Hong Kong, Shenzhen / Li Auto Inc.) · → Paper · Demo: ✓ · Code: ✓
Introduces DeEAR, a framework that decomposes speech expressiveness into Emotion, Prosody, and Spontaneity sub-scores and fuses them with a human-preference-aligned model, then uses the resulting metric to curate a bilingual dataset that measurably improves the expressiveness of a fine-tuned speech-to-speech model.
Problem
Speech-to-speech and conversational TTS systems increasingly produce intelligible output, but the field lacks a reliable, scalable way to measure whether that output actually sounds expressive rather than robotic. Subjective MOS ratings are accurate but costly and unscalable; low-level acoustic features (pitch, energy) miss perceptual subtleties; and emotion recognition captures only one facet of what listeners perceive as expressive speech. Without a trustworthy automatic metric, it is difficult to benchmark competing dialogue systems fairly or to select training data that will actually make a model sound more expressive.
Method
DeEAR treats expressiveness as a composite human-preference construct rather than a single learnable target, on the premise that scoring an abstract perceptual quality directly is unreliable given limited labeled data. The framework follows a four-stage pipeline (§2, Figure 1): decompose expressiveness into concrete sub-tasks, build a specialized proxy model for each, learn a fusion function that maps the proxy scores onto holistic human judgments, then distill and decouple the system for efficient deployment.
Three dimensions are proxied with different strategies matched to their data availability (§2.2). Emotion intensity (arousal) is data-rich, so DeEAR fine-tunes a wav2vec2-large-robust model already pre-trained for emotion recognition, adding 12,000 Chinese samples from CNSCED and 2,000 English samples from IEMOCAP for bilingual coverage. Prosodic richness lacks large labeled data, so DeEAR uses Gemini 2.5 Pro as an LLM proxy annotator, validated against human ratings (SRCC = 0.73) before being used to generate training labels at scale. Spontaneity is scored with a two-stage knowledge-guided approach: a heuristic combines a dataset-level base spontaneity category with a DNSMOS-derived acoustic-quality term, penalizing “hyper-clean” audio that is perceptually incongruent with its expected style (the “speech uncanny valley” effect), and the resulting pseudo-labels are then distilled into a fine-tuned wav2vec2-large-robust scorer.
The three proxy scores are combined by a separate, lightweight XGBoost model trained on 480 human-annotated clips (three raters per clip) to predict a single holistic expressiveness score, keeping the preference-fusion logic interpretable and independently updatable (§2.3). For efficient deployment, the three proxy models are distilled into a single student, DeEAR-Base, built on a wav2vec2-large-xlsr-53 backbone with three regression heads trained jointly on 20,000 pseudo-labeled utterances; the XGBoost fusion layer is kept as a separate, decoupled module on top of DeEAR-Base’s outputs (§2.4). All scores are reported on a 0-100 scale.
DeEAR is then used for two downstream applications: automated benchmarking of speech generation systems, and evaluation-driven data curation, in which utterances are filtered by their DeEAR expressiveness score to build a training set of curated high-expressiveness speech (ExpressiveSpeech, §3), which is then used to fine-tune an existing speech-to-speech model.
Key Results
DeEAR’s overall expressiveness score correlates strongly with human perception: PCC = 0.91 and SRCC = 0.86 against averaged ratings from three graduate raters on four 100-utterance test sets spanning real-world conversation, professional recordings, and TTS-generated speech (§4.1, Table 2). The Spontaneity sub-scorer shows the strongest individual correlation (PCC = 0.84, SRCC = 0.84), while Emotion and Prosody are weaker (PCC 0.72 and 0.70 respectively).
For system-level benchmarking, DeEAR ranks seven SOTA speech-to-speech dialogue systems (Doubao, Grok-4 Voice, GPT-4o Audio, Sesame, Step Audio 2, Qwen2.5-Omni, Gemini-2.5 Pro) with SRCC = 0.96 against a human ranking from four native-speaker raters on a 3-point scale, and reproduces a nearly 60-point overall-score gap between the top (Doubao) and bottom (Qwen2.5-Omni) systems (§4.2, Table 3).
For data curation, ExpressiveSpeech (14K utterances, 51 hours, curated from Expresso, NCSSD, M3ED, MultiDialog, and IEMOCAP at a DeEAR threshold of 63.5) reaches an average DeEAR score of 80.2, well above any individual source dataset (39.4-62.9, Table 1). Fine-tuning a baseline speech-to-speech model (architecture analogous to MinMo and Qwen2.5-Omni, a 7B LLM plus a 1.5B audio language model with an S3 tokenizer) on this dataset for a single epoch raises the overall expressiveness score from 2.0 to 23.4, with statistically significant gains (p < 0.001) on both in-domain and out-of-domain test sets (§4.3.2, Table 4). Human A/B preference tests corroborate this: 10 native speakers preferred the fine-tuned model’s output 78.5% of the time versus 10% for the baseline.
Novelty Assessment
The contribution is primarily in evaluation methodology and dataset curation rather than in generative or evaluation-model architecture. Each individual component, fine-tuned wav2vec2 regressors, an LLM used as a scoring proxy, XGBoost as a fusion layer, and knowledge distillation for deployment efficiency, is an established technique. What is genuinely new is the combination: decomposing a hard-to-model holistic construct into psychologically motivated sub-dimensions matched to task-specific data strategies, then learning an interpretable, decoupled fusion function from a small amount of human preference data (480 clips) rather than trying to regress the holistic score end-to-end. The data-efficiency claim (SRCC = 0.86 from under 500 annotated samples) is a real and useful practical result, though it has not been validated against an independent, larger-scale challenge in the way that established metrics like DNSMOS or UTMOS have been.
Field Significance
Moderate — the paper targets a genuinely unaddressed gap (a reliable, scalable, human-aligned expressiveness metric for conversational and TTS speech) and demonstrates that such a metric can be built with limited annotated data and used to drive a measurable improvement in a downstream speech-to-speech model. Its evidence base, however, is modest in scale: the main validity check uses three raters and 400 test utterances, the benchmarking human reference uses four raters, and the data-curation A/B test uses ten raters, all self-administered by the authors rather than run as an independent, community-scale shared task. The contribution is best read as a promising evaluation-driven data curation methodology rather than an established or widely validated metric.
Claims
- supports: Decomposing a holistic, subjective speech quality construct into interpretable sub-dimensions and fusing them with a lightweight, separately-trained model can approximate human preference judgments with high data efficiency.
Evidence: The XGBoost preference-fusion function, trained on only 480 human-annotated clips using three proxy sub-scores as input, reaches SRCC = 0.86 and PCC = 0.91 against averaged human expressiveness ratings. (§2.3, §4.1, Table 2)
- supports: An automatic expressiveness metric aligned with human judgment can reliably rank competing speech generation systems, offering a scalable substitute for comparative listening tests at the system level.
Evidence: DeEAR ranks seven SOTA speech-to-speech dialogue systems with SRCC = 0.96 against human rankings and reproduces the near-60-point gap between the highest- and lowest-scoring systems. (§4.2, Table 3)
- supports: Filtering training data with an automatic expressiveness score can substantially increase the expressiveness of a fine-tuned speech generation model without any architectural change to that model.
Evidence: Fine-tuning an existing speech-to-speech model on 51 hours of DeEAR-selected utterances (threshold 63.5) for a single epoch raised the overall expressiveness score from 2.0 to 23.4 (p < 0.001), corroborated by a 78.5%-vs-10% human A/B preference for the fine-tuned model. (§4.3.2, Table 4)
- complicates: Using a general-purpose LLM as a scalable proxy annotator for a subjective perceptual dimension yields only moderate alignment with human judgment, so its labels should be treated as a noisy training signal rather than as ground truth.
Evidence: Gemini 2.5 Pro, used to generate prosodic-richness training labels, itself reaches only SRCC = 0.73 against human ratings, and the resulting Prosody sub-scorer is the weakest of the three dimensions against holistic human perception (PCC = 0.70, SRCC = 0.68). (§2.2.2, §4.1, Table 2)
- complicates: Heuristic pseudo-labeling of a perceptual attribute at the dataset level, rather than the individual-utterance level, caps the precision of any model subsequently distilled from those labels.
Evidence: The spontaneity pseudo-label heuristic assigns a single manually-set categorical base level per source dataset, combined with a DNSMOS-derived acoustic-quality adjustment, so utterance-level variation in true spontaneity within a dataset is not captured by the label itself. (§2.2.3)
Limitations and Open Questions
Warning
The same DeEAR metric is used both to select the ExpressiveSpeech training data and to score the resulting fine-tuned model’s expressiveness gains (§4.3), so the headline 2.0-to-23.4 improvement is partially self-referential. The independent human A/B preference test (§4.3.2) mitigates but does not fully resolve this circularity, since the curation criterion and the primary evaluation criterion are the same underlying construct.
Beyond this, the human validation studies throughout the paper are small in scale and drawn from a narrow rater pool: three graduate students in speech processing for the main validity check (§4.1), four native speakers for the system-benchmarking reference ranking (§4.2), and ten native speakers for the data-curation A/B test (§4.3.2). None of these studies is run as an independent, larger-scale challenge in the way established metrics like DNSMOS or UTMOS have been validated. The Prosodic Richness dimension depends on a closed, versioned commercial LLM (Gemini 2.5 Pro) as its label source, which introduces a dependency on a model whose behavior is outside the authors’ control and could change over time. The ExpressiveSpeech dataset is released under a CC BY-NC-SA 4.0 license inherited from its source corpora, restricting it to non-commercial use.
Wiki Connections
- Evaluation Metrics — introduces a new composite metric (DeEAR) that targets expressiveness specifically, a dimension not well covered by existing WER-, DNSMOS-, or MOS-style metrics.
- Subjective Evaluation — validates its automatic metric against three separate human-rated studies (a validity check, a system-ranking reference, and an A/B preference test) rather than relying on automated scoring alone.
- Speech-to-Speech — benchmarks seven SOTA speech-to-speech dialogue systems and uses evaluation-driven data curation to improve the expressiveness of a fine-tuned speech-to-speech model.
- Self-Supervised Speech — builds all three of its perceptual proxy scorers, and its distilled DeEAR-Base student, on fine-tuned wav2vec2 self-supervised backbones.
- Emotion Synthesis — treats emotion intensity as one of three core expressiveness dimensions and curates training data specifically to increase emotional expressiveness in generated speech.
- Qwen2.5-Omni Technical Report — used both as one of the seven benchmarked S2S systems (scoring lowest on DeEAR’s automated ranking) and as an architectural reference point for the Expressive-FT baseline model.
- MinMo — cited as an architectural analog for the Expressive-FT baseline S2S model that DeEAR-curated data is used to fine-tune.
- EXPRESSO — one of the five source datasets curated (via DeEAR filtering) into the ExpressiveSpeech training set, and the highest-scoring individual source dataset before curation.