arXiv · 2026 · Preprint
Xu et al. (Meta) · → Paper · Demo: ? · Code: ?
Introduces a dual-channel automatic naturalness predictor for two-speaker, multi-turn conversational speech, and shows that single-utterance MOS predictors (NISQA, UTMOSv2) correlate poorly, sometimes negatively, with human conversational naturalness ratings.
Problem
Automatic naturalness predictors such as NISQA and UTMOSv2 were designed and trained to score single-speaker, single-utterance audio for sentence-level fluency and quality. As spoken conversational agent systems move toward multi-turn, two-speaker, and full-duplex interaction, naturalness depends on conversational phenomena, smooth turn-taking, appropriate filler words, and expressions suited to conversation dynamics, that single-utterance predictors were never trained to capture. The paper first quantifies this gap directly: it segments conversational recordings into utterances, applies existing predictors, aggregates the scores, and correlates them against human conversational-naturalness ratings. It then asks whether a predictor explicitly designed for two-channel, multi-turn conversational input can do better.
Method
The authors build two internal, human-rated datasets of dual-channel (user + system) conversation recordings: ConvTTS, 6,579 conversations synthesized by an internal offline TTS model in a user-system dialogue format, and FDX-Conv, 490 real recordings between an internal full-duplex speech generation model and human workers, used only for out-of-domain (OOD) evaluation. Each recording carries naturalness ratings from at least 5 human raters on a 1–5 scale, collected along two axes: conversation naturalness (human-likeness of the exchange between both speakers) and system naturalness (human-likeness of the agent’s channel, rated while listening to both channels).
The proposed predictor extracts frame-level features from a frozen pretrained transformer encoder, computes a learned weighted sum over all hidden layers, and mean-pools over 30-second segments. In the dual-channel configuration, embeddings from the user and system audio channels are extracted independently and concatenated along the feature dimension before being passed to a three-layer MLP (hidden size 768, dropout 0.1, GeLU) that regresses a MOS-style naturalness score; segment-level predictions are averaged for the final conversation score. A single-channel variant uses only the system-channel embedding. Three pretrained encoders are compared as the feature backbone: WavLM-large (self-supervised, 94k hours), Audiobox-Aesthetics (WavLM-base backbone, trained on 562 hours of annotated quality-aesthetics labels), and Whisper large-v3 encoder (680k hours of weakly-supervised multilingual ASR pretraining). Models are trained with Adam and MSE loss against the human MOS labels, using batch sizes and learning rates tuned per encoder, for 30 epochs with best-validation-loss checkpoint selection. To improve robustness under distribution shift, the internal TTS model (conditioned on reference audio for both channels, in a manner the authors liken to zero-shot TTS) is used to generate 5,000 hours of additional synthetic conversational audio from Llama-3.1-405B-generated transcripts, inheriting the naturalness label of the reference conversation it was conditioned on; the predictor is pretrained on this augmented set before being fine-tuned on the real ConvTTS data.
Key Results
Existing single-utterance predictors fail on conversational naturalness: NISQA and UTMOSv2 achieve negative Pearson correlation coefficients (PCC) with human conversation-naturalness ratings across all aggregation statistics (mean, min, max, median) on ConvTTS, and mostly negative PCC on system naturalness as well (Table 2, Table 3). The authors attribute this to existing predictors penalizing informal, “chit-chatty” speech that human raters judge as more conversationally natural.
The proposed predictor closes this gap substantially. For conversation naturalness on ConvTTS, the best configuration (Whisper encoder, dual-channel) reaches PCC 0.482 / SRC 0.451 (Table 4), versus negative correlations for the existing baselines. For system naturalness in-domain, the same configuration reaches PCC 0.570 / SRC 0.560 (Table 5). Dual-channel input consistently outperforms single-channel input in-domain for all three encoders and both naturalness targets (Table 4, Table 5). Whisper consistently outperforms both WavLM and Audiobox-Aesthetics as the backbone across every setting tested (§6.3).
Under OOD evaluation on FDX-Conv (real full-duplex recordings), the pattern partially reverses: single-channel modeling outperforms dual-channel modeling for system naturalness (best Whisper single-channel PCC 0.362 vs. dual-channel 0.290; Table 5), which the authors attribute to a domain mismatch between ConvTTS’s synthetic user audio and FDX-Conv’s real user audio. The best OOD result (0.362 PCC) still exceeds the best existing baseline, UTMOSv2, on the same OOD set (0.31 PCC, Table 3). An ablation that removes explicit channel separation (merging user and system audio into one channel before encoding) underperforms the dual-channel design on both ConvTTS and FDX-Conv, with the largest gap OOD (PCC drops 62.07% relative on FDX-Conv system naturalness; Table 7). Pretraining on the 5,000-hour synthetic augmentation set before fine-tuning on real ConvTTS data yields a 23.45% relative PCC improvement on OOD system naturalness, with negligible or slightly negative change on the in-domain ConvTTS metrics (Table 6).
Novelty Assessment
The contribution is primarily an evaluation methodology and a systematic empirical study, not a new architecture: the model is a standard frozen-encoder-plus-MLP MOS-predictor design, applied with a dual-channel input arrangement rather than a novel structural mechanism. The genuine novelty is the framing itself: separating naturalness into conversation-level and system-level targets, collecting matched human ratings for both, demonstrating quantitatively that existing single-utterance predictors transfer poorly (including negative correlations) to this setting, and showing via controlled ablation that explicit two-speaker channel separation, not just more conversational training data, is what drives the improvement. The synthetic data augmentation pipeline is a straightforward application of the internal TTS system rather than a new augmentation method.
Field Significance
moderate — this paper documents a concrete blind spot in existing automatic naturalness/MOS predictors (negative correlation with human judgment on multi-turn conversational speech) and proposes a simple, replicable dual-channel remedy with a systematic encoder comparison and an ablation isolating the channel-separation effect. Its datasets and speech generation systems are internal and not released, which limits reproducibility and external verification of the reported correlations.
Claims
- supports: Automatic naturalness predictors trained for single-speaker, single-utterance speech transfer poorly, and can be negatively correlated with human judgment, when applied to multi-turn, two-speaker conversational speech.
Evidence: NISQA and UTMOSv2 show negative PCC with human conversation-naturalness ratings across all aggregation statistics on ConvTTS, and mostly negative PCC for system naturalness as well (§4, Table 2, Table 3).
- supports: Modeling both interlocutors’ audio channels jointly, rather than only the target speaker’s channel, improves automatic prediction of naturalness in conversational speech.
Evidence: Dual-channel input modeling outperforms single-channel modeling across all three encoders (WavLM, Audiobox-Aesthetics, Whisper) for both conversation naturalness and system naturalness on the in-domain ConvTTS test set (§6.4, Table 4, Table 5).
- supports: Preserving explicit per-speaker channel separation in the input representation, rather than merging channels before encoding, is more important than simply exposing both speakers’ audio to the model.
Evidence: An ablation that combines user and system audio into a single merged channel underperforms the separated dual-channel design on both ConvTTS and FDX-Conv, with a 62.07% relative PCC drop on OOD system naturalness (§6.6, Table 7).
- complicates: The benefit of dual-channel conversational modeling is contingent on matched acoustic domains between the two channels, and can reverse when one channel is out-of-domain relative to training.
Evidence: On the OOD FDX-Conv set, where user audio is real human speech rather than synthetic TTS output as in the ConvTTS training domain, single-channel (system-only) modeling outperforms dual-channel modeling, attributed to the mismatch introduced by the real, out-of-domain user channel (§6.4, Table 5).
- supports: Large-scale synthetic pretraining data can improve a naturalness predictor’s robustness to distribution shift without necessarily improving its in-domain accuracy.
Evidence: Pretraining on 5,000 hours of TTS-generated synthetic conversational audio before fine-tuning on real ConvTTS data yields a 23.45% relative PCC gain on the OOD FDX-Conv system-naturalness test, while producing negligible or slightly negative change on in-domain ConvTTS metrics (§6.5, Table 6).
Limitations and Open Questions
The two datasets underlying all results, ConvTTS and FDX-Conv, are internal and generated with an internal offline TTS model and an internal full-duplex speech generation model respectively; neither the datasets, the generation systems, nor a public naturalness-predictor checkpoint are described as released, which limits independent verification and reuse. The OOD evaluation is conducted on only 490 recordings from a single full-duplex system, so the domain-mismatch findings (reversal of the dual-channel benefit) may not generalize to other full-duplex or dialogue-system architectures. Reported correlations, even for the best configuration, remain moderate in absolute terms (PCC around 0.48–0.57 in-domain, 0.29–0.36 OOD), indicating substantial remaining gap between the automatic predictor and human agreement on conversational naturalness.
Wiki Connections
- Evaluation Metrics — proposes a new automatic predictor and demonstrates that existing single-utterance MOS predictors do not transfer to multi-turn conversational naturalness assessment.
- Subjective Evaluation — collects dedicated human naturalness ratings (at least 5 raters per recording, 1–5 scale) along two axes, conversation- and system-level naturalness, as training and validation targets for the automatic predictor.
- Self-Supervised Speech — evaluates a WavLM-large self-supervised encoder as one of three pretrained backbones for the naturalness predictor, alongside the weakly-supervised Whisper encoder and the Audiobox-Aesthetics model.
- Moshi — cited as an example of the full-duplex spoken dialogue models that motivate the need for conversation-level naturalness evaluation.
- OmniFlatten — cited alongside Moshi as a full-duplex, listen-while-speaking dialogue system whose outputs existing single-utterance predictors are not designed to evaluate.
- Meta Audiobox Aesthetics — one of the three pretrained encoder backbones compared for the naturalness predictor, using its WavLM-base quality-assessment pretraining as feature extractor.
- Full-Duplex-Bench v1.5 — cited as a related conversational speech generation benchmark, contrasted for targeting specific phenomena like overlap handling rather than the holistic naturalness this paper addresses.