arXiv · 2026 · Preprint
Hallur et al. (Johns Hopkins University / Amazon) · → Paper · Demo: ✗ · Code: ✗
Builds stratified human reference distributions for conversational F0, speaking rate, and pausing from over 4,000 hours of dyadic English conversation, and proposes scoring a spoken dialogue system’s output waveform as percentile deviations against the matched human stratum rather than against pooled statistics.
Problem
Evaluation of speech-to-speech dialogue agents relies mostly on text-based measures, task success, or subjective ratings, none of which directly quantify speech-native interactional behaviour such as pitch expressivity, pace, and pausing. Where prosody is checked at all, pooled human statistics are a natural reference, but F0 and timing shift systematically with speaker traits and interaction state. An output can then sit inside a pooled human range while still being implausible for the relevant speaker profile or conversational state (for example, narrow pitch excursions in a high-arousal exchange). Prior work on conversational prosody and timing has also tended to study single dimensions or small corpora, leaving no large-scale, multidimensional reference regime usable for system evaluation.
Method
The reference data is the Seamless Interaction corpus (4,065.04 hours, 64,739 interactions from 5,098 sessions and 4,284 participants, covering both naturalistic interactions with untrained participants and improvised interactions with trained actors). All metrics are computed per speaker channel.
Prosodic metrics come from Praat’s autocorrelation F0 tracker (via parselmouth) with 75-500 Hz bounds. Channels with voiced ratio below 0.05 are excluded, and voiced F0 values are trimmed to the 10th-90th percentile within each channel before computing the trimmed mean, standard deviation, and range (SD and range are jointly called F0 expressivity). Temporal metrics use the corpus’s ASR word timestamps and VAD segments: VAD segments separated by at most 1.0 s are merged, only continuous stretches of at least 12.1 s are kept (following evidence on speaking-rate stabilisation time), and pauses are inter-word gaps of at least 0.2 s. Speech rate is words per minute over total stretch duration, articulation rate excludes pause time, and pause ratio is pause time over total duration; mean pause duration is also reported.
Because the corpus has no speaker-trait metadata, the authors annotate each channel with Vox-Profile predictors (a WavLM-based multitask age/sex model and a dimensional emotion model producing arousal, valence, and dominance), after resampling to 16 kHz and Silero VAD. Model-predicted sex label, model-predicted age bin (18-29 / 30-59 / 60+), arousal, and dominance are used as stratifiers; the paper notes these four gave the most consistent shifts in preliminary screening.
The proposed evaluation protocol (§IV) is: extract the same seven metrics from a dialogue system’s output waveform; select the closest matched human reference stratum (sex label for absolute F0, arousal or dominance sextile for expressivity and rhythm, age bin for timing when available, otherwise the coarsest applicable reference, reported explicitly); convert each metric to a percentile under that stratum; and flag values below the 5th or above the 95th percentile as out-of-regime. The output is a vector of signed percentile deviations rather than a single score.
To check calibration, rows are split into participant-held-out calibration and evaluation sets. 5th-95th percentile intervals are estimated on the calibration split under a pooled reference and under the matched stratum, and the fraction of held-out human rows falling outside each interval is reported; a calibrated reference should flag about 10%.
Key Results
All experiments use human speech only: no spoken dialogue system output is scored in the paper.
- Pooled regimes (Table I). The pooled median trimmed F0 mean is 157.4 Hz (IQR 120.1-198.6), median speech rate 175.9 wpm, articulation rate 237.8 wpm, pause ratio 0.2575, and mean pause duration 0.5845 s.
- Sex-label effects (Table II). Mean F0 separates almost completely by model-predicted sex label (Cliff’s δ = -0.957; medians 121.9 Hz male vs. 200.7 Hz female), with large effects on F0 SD (δ = -0.635) and range (δ = -0.644). Temporal effects are negligible to small (|δ| between 0.066 and 0.180).
- State and age effects (Table III). Arousal correlates with F0 SD (ρ = 0.544), F0 range (ρ = 0.516), speech rate (ρ = 0.187), and pause ratio (ρ = -0.170); dominance shows the same pattern (ρ = 0.463, 0.430, 0.200, -0.194). Age-bin effects are statistically significant but small in effect size (Kruskal-Wallis ε² between 0.00511 and 0.01623).
- Corpus subsets (Table IV). Naturalistic vs. improvised differences are negligible for temporal metrics and small for F0 (maximum |δ| = 0.278), so the two subsets are pooled.
- Held-out calibration (Table V). For F0 expressivity, pooled references over-flag held-out human rows (15.41-21.11% across low/high arousal and dominance groups), while matched references bring flag rates to 8.87-10.16%. For high-arousal speech rate and pause ratio, matching reduces flag rates from 15.76% to 12.50% and from 13.18% to 10.84%, still somewhat above the nominal 10%. For mean F0 the picture differs: pooled flag rates are already close to nominal (9.88% male, 8.63% female) and matching does not bring them closer (12.06% male, 8.35% female). The paper’s argument for sex-label matching on mean F0 rests instead on direction: under the pooled reference, male rows are flagged almost entirely below the interval (9.8% low vs. 0.1% high) and female rows above it (0.0% low vs. 8.6% high) (§III-A).
The calibration test checks that matched intervals behave as nominal percentile intervals on held-out human speech; it does not test whether the flags detect implausible system output or agree with listener judgments.
Novelty Assessment
The individual measurements (trimmed F0 statistics, words-per-minute speech and articulation rate, pause ratio) and the stratifiers (off-the-shelf Vox-Profile predictors) are standard. The contribution is the framing of large-scale conversational distributions as a stratified evaluation reference for dialogue system output, plus the scale of the reference tables and the held-out calibration check showing that pooled references are miscalibrated for state-dependent expressivity. The protocol itself is a simple percentile lookup. The main gap is that the paper never applies the protocol to a real spoken dialogue system, so its usefulness as an evaluation tool is argued rather than demonstrated.
Field Significance
Low. The paper provides an interpretable, stratified human reference for checking the prosody and timing of spoken dialogue output, and evidence on held-out human speech that state-matched references are better calibrated than pooled ones for F0 expressivity and rhythm. Without application to system output, perceptual validation, or a second language or corpus, it remains a proposed plausibility check rather than a validated metric.
Claims
- complicates: Pooled human statistics are a poorly calibrated reference for judging the prosodic expressivity and timing of conversational speech output, because these properties shift systematically with interaction state.
Evidence: On participant-held-out human rows, pooled 5th-95th percentile intervals flag 15.41-21.11% of F0-SD samples across low/high arousal and dominance groups, against a nominal 10%, while arousal- or dominance-matched intervals flag 8.87-10.16%. (§III-A, Table V)
- supports: Conversational pitch expressivity and pacing scale with arousal and dominance, so evaluation of dialogue speech should condition expected F0 range and rate on the interactional state.
Evidence: Across 121,813 speaker channels, F0 SD and range rise monotonically across arousal and dominance sextiles (Spearman ρ up to 0.544), speech rate rises (ρ about 0.19-0.20) and pause ratio falls (ρ about -0.17 to -0.19). (§III-A, §III-B, Table III, Figures 2-4)
- refines: Aggregate out-of-range rates can look calibrated while hiding directional bias, so prosodic evaluation should report the direction of deviation per dimension rather than a single pass rate.
Evidence: Pooled mean-F0 intervals flag 9.88% of held-out male rows and 8.63% of female rows, close to nominal, yet male flags fall almost entirely below the interval (9.8% low vs. 0.1% high) and female flags above it (0.0% vs. 8.6%). Sex-label matching does not lower the aggregate rates (12.06% and 8.35%). (§III-A, Table V)
- complicates: Reference-based prosody and rhythm checks measure behavioural plausibility relative to human distributions, not perceived naturalness or interaction quality.
Evidence: The authors state that deviations are not mapped to perceptual thresholds or user-rated quality, that the reference comes from a single English corpus, and that stratifiers are model predictions rather than ground-truth metadata. (§IV, §Limitations)
Limitations and Open Questions
The protocol is never applied to a spoken dialogue system
All results are computed on human conversational speech. No speech-to-speech system output is scored, and no listening test checks whether out-of-regime flags correspond to perceived unnaturalness, so the evaluation value of the protocol is untested.
- Single English corpus. The references come from Seamless Interaction only; the authors note operating regions may shift with language, domain, recording conditions, and interaction setting.
- Model-predicted stratifiers. Sex label, age bin, arousal, and dominance are Vox-Profile predictions (age-bin accuracy 67.6% as reported by Vox-Profile), so stratum assignment for a synthetic voice depends on how these predictors behave on system output, which is not examined.
- Binary sex label. Predicted from voice; non-binary and self-identified categories are not modelled.
- Pipeline mismatch for system output. The reference temporal metrics use the corpus’s ASR-aligned timestamps, VAD segments, and stretches of at least 12.1 s; how a system’s (often short) dialogue turn is segmented and aligned to obtain comparable metrics is not specified.
- Partial calibration gain for rhythm. Matched high-arousal speech-rate flags (12.50%) remain above nominal, and for mean F0 matching does not improve aggregate calibration.
- Turn-level interaction timing absent. Turn-taking latency, overlap, and backchannels, which are central to dialogue behaviour, are outside the metric set.
Wiki Connections
- Evaluation Metrics — Proposes a stratified percentile-deviation protocol for F0 level, F0 expressivity, speech rate, articulation rate, and pausing, with a held-out calibration check of pooled vs. matched references.
- Vox-Profile — Supplies the WavLM-based age/sex and arousal/dominance predictors used to stratify the human reference distributions and select the matched stratum.