arXiv · 2026 · Preprint

Hao Zhang et al. (Johns Hopkins University / Amazon) · → Paper · Demo: ✗ · Code: ✓

Scores the turn-taking naturalness of a two-channel dialogue as the likelihood of its observed future voice activity under a turn-taking predictor trained on natural conversations, and tests the score on a human-validated benchmark of locally perturbed dialogues.

Problem

Full-duplex spoken dialogue systems must respond, overlap, yield and backchannel at natural moments, but the timing of these behaviours is hard to evaluate automatically. Human listening tests are costly. Existing benchmarks such as Full-Duplex-Bench and its overlap-focused follow-up Full-Duplex-Bench v1.5 score each behaviour (pause handling, backchanneling, interruption) with its own task-specific metric. Talking Turns trains an event-decision judge with event-specific thresholds. None of these places heterogeneous timing failures, such as a late response and an excessive backchannel, on one continuous scale. The paper asks whether a turn-taking prediction model trained only on natural human conversation can provide that single scale.

Method

TurnNat takes a two-channel dialogue and runs voice activity detection on both channels. It cleans the speech islands per speaker: it merges same-speaker islands separated by at most 1.0 s when the other speaker is active for at most 20% of the gap, then drops islands shorter than 200 ms. Each remaining utterance yields two turn-taking boundary units (TBUs), one at its onset and one at its offset. A TBU covers the 2 s of frames before the boundary.

The likelihood source is a causal turn-taking predictor in the Voice Activity Projection (VAP) formulation. At each frame it predicts both speakers’ activity over a 2 s horizon split into four non-uniform bins (0-200, 200-600, 600-1200 and 1200-2000 ms). A bin is active when more than half its frames are voiced. The 8 binary bin states form one of 256 joint states, and a 256-way softmax head predicts them from causal dialogue context. Training uses only natural human-human dialogue and minimises the negative log-likelihood (NLL) of the observed state. A weight α > 1 up-weights frames inside TBUs (α = 1 is the uniform objective).

At evaluation time the frame NLL measures how atypical the observed future activity is. TurnNat averages frame NLL within each TBU and then combines two dialogue-level terms with equal weight (λ = 0.5): the mean TBU NLL and the mean of the top 25% highest-NLL TBUs (tail term). The negated sum is the naturalness score, so higher means more natural. Inference needs no human judgments, perturbation labels or annotated turn-taking events.

The paper instantiates the scorer with two backbones. VAP uses a CPC speech encoder and a causal Transformer; the authors test the released checkpoint (V0) and a fully fine-tuned version (V1). DualTurn uses a frozen Mimi encoder and a Qwen backbone pre-trained with dual-channel generative speech modelling. The authors test its native eight independent Bernoulli future-activity targets (D0 to D2) and a replacement 256-way categorical head (D3, D4). D2 and D4 keep DualTurn’s original turn-taking heads as auxiliary losses. Each model is fine-tuned for up to 5 epochs with AdamW (batch size 8) on one A100 GPU. Parameter counts are not reported.

The benchmark draws on the English naturalistic subset of the Seamless Interaction dataset. It keeps open-ended prompted dyadic conversations and excludes task-driven formats such as storytelling or charades. The speaker-disjoint splits are 250.18 h for training, 20.44 h for development and 129.32 h for testing (Table I). From the test split the authors cut 20-25 s clips around a target event and build a perturbed twin of each clip with one of five edits:

  • Late response: a response is delayed by 1.2-2.0 s.
  • Early entry: a response is advanced by 1.2-2.5 s, which creates premature overlap.
  • Hold instead of shift: the responding turn is removed.
  • Shift instead of hold: a complete turn from the other speaker is inserted.
  • Excessive backchanneling: two or three of the listener’s own backchannels are added.

The benchmark contains 200 pairs per type, 1,000 in total. Three native English-speaking annotators rated a balanced 150-pair subset. They gave an A/B naturalness preference plus five-point naturalness and artifact ratings for each clip.

Key Results

Human validation (Table II). Annotators chose the natural clip in 70.0% of judgments (95% Wilson CI 65.6-74.1) and in 78.0% of pairs after majority voting. Natural clips scored 0.678 points higher on the five-point naturalness scale (natural minus perturbed). Per-type majority-natural rates range from 0.700 (early entry, shift instead) to 0.867 (late response, hold instead). The artifact difference (perturbed minus natural; 0 is ideal) averages 0.289. A participant-level bootstrap puts the naturalness gap 0.389 points above the artifact gap (95% CI 0.138-0.651). The control is uneven across types. For early entry the artifact gap (0.667) is almost as large as the naturalness gap (0.700), and hold instead also shows a sizeable artifact gap (0.500).

Automatic discrimination (Table III). The primary metrics are matched-pair accuracy (natural clip scored above its own perturbed twin) and the C-index (global natural-perturbed separability across all clips). The paper reports these results:

  • D4 with α = 8 (DualTurn, 256-way categorical head plus auxiliary heads, TBU weight 8) reaches 88.0% pair accuracy, the highest of the ten rows. Its C-index of 0.676 ties D4 with α = 3.
  • The baselines reach 80.6% (released VAP, V0), 80.2% (fine-tuned VAP, V1) and 77.5% (released DualTurn with Bernoulli targets, D0).
  • Fully fine-tuning VAP does not help. V1 trails V0 slightly in accuracy and drops on late response from 90.0% to 79.5%.
  • Replacing DualTurn’s Bernoulli targets with the 256-way joint-state head raises accuracy (D1 81.2% to D3 83.3%; D2 81.5% to D4 86.2%).
  • TBU weighting moves D4 from 86.2% (α = 1) to 87.3% (α = 3) and 88.0% (α = 8). On the Bernoulli-plus-auxiliary scorer it gives only 81.5% to 81.7%.

C-index stays between 0.633 and 0.676 for every scorer. Pairs are separated much more reliably than clips drawn from different contexts.

Per perturbation type (Table III). D4 with α = 8 has the highest pair accuracy on late response (95.0%), hold instead (84.5%) and excessive backchanneling (87.0%), where its C-index (0.813) is also the highest. It is not the strongest on early entry: D4 with α = 1 reaches 93.5% accuracy against its 92.5%, and D3 reaches the highest C-index (0.718). On shift instead its accuracy (81.0%) is well below the Bernoulli-plus-auxiliary scorers (92.0% for D2 at α = 1 and α = 3), even though its C-index there (0.628) is the highest. The released VAP checkpoint is weakest on shift instead (66.0%).

Aggregation (Table IV, D4, α = 8). Averaging all frames gives 84.7% accuracy and a C-index of 0.609. TBU mean alone gives 87.4% and 0.662, and TBU tail alone gives 84.5% and 0.674. Combining mean and tail gives the highest values on both metrics, 88.0% and 0.676. The tail term alone yields the largest mean score gap (0.54 ± 0.05) but not the best ranking.

All automatic comparisons are within the authors’ own benchmark. No external turn-taking metric (Full-Duplex-Bench timing metrics, Talking Turns judge) is run on it as a baseline.

Novelty Assessment

The main idea is to turn a VAP-style future voice-activity likelihood into a dialogue-level naturalness score by pooling it over boundary-anchored units with mean and tail terms. VAP’s 256-state target, DualTurn’s backbone and NLL scoring are all existing components. The new parts are the TBU pooling, the TBU-weighted fine-tuning objective and the paired perturbation benchmark. The ablations are clean and support each design choice. Still, the benchmark is built and judged entirely by the authors. The perturbations are large by design (well outside the normal -200 to 400 ms human response window). The metric is never compared with any existing turn-taking evaluation method or applied to a dialogue system.

Field Significance

Low — TurnNat provides a single likelihood-based score that ranks natural above perturbed turn-taking timing across five failure types, together with an English, human-validated perturbation benchmark. Its value for evaluating spoken conversational agents is not yet demonstrated: every test clip is edited human-human speech, and the score is never correlated with human ratings of individual clips or applied to system output.

Claims

  • supports: A turn-taking predictor trained only on natural human conversation can serve as a reference-free scorer that distinguishes natural from disrupted dialogue timing across heterogeneous failure types with one score.

    Evidence: Future two-speaker voice-activity NLL pooled over turn-taking boundary units ranks the natural clip above its perturbed twin in 88.0% of 1,000 pairs covering late responses, early entries, removed or inserted turns and excess backchannels, against 77.5-80.6% for unadapted VAP and DualTurn scorers. (§III-D, §VI-B, Table III)

  • refines: Turn-taking naturalness scores are more discriminative when likelihood is pooled over local regions around utterance boundaries, with a tail term, than when it is averaged over the whole dialogue.

    Evidence: With the same D4 scorer, all-frame averaging gives 84.7% pair accuracy and a C-index of 0.609, while combined TBU mean and top-25% tail pooling gives 88.0% and 0.676. Tail-only pooling has the largest mean score gap but lower pair accuracy (84.5%). (§VI-B, Table IV)

  • complicates: Automatic turn-taking naturalness metrics separate matched natural-perturbed pairs far more reliably than they rank dialogues from different contexts on one scale.

    Evidence: Pair accuracy reaches 77.5-88.0% across scorers, but the global C-index stays between 0.633 and 0.676. The authors attribute the gap to score variation across heterogeneous dialogue contexts. (§VI-B, Table III)

  • complicates: No single future-activity prediction target captures all turn-taking failure types; joint-state and independent per-bin targets are sensitive to different failures.

    Evidence: The 256-way categorical scorer leads on late response, hold-instead and excessive-backchannel pairs, but on shift-instead pairs it reaches 81.0% accuracy against 92.0% for the independent-Bernoulli scorer with auxiliary heads. (§VI-B, Table III)

  • complicates: Perturbation-based benchmarks for dialogue timing can confound turn-taking naturalness with audible editing artifacts, and the confound varies by perturbation type.

    Evidence: Overall, human naturalness preference exceeds the artifact difference by 0.389 points. For early-entry pairs, however, the perturbed-minus-natural artifact difference (0.667) nearly equals the naturalness difference (0.700), and hold-instead pairs show a 0.500 artifact difference. (§VI-A, Table II)

Limitations and Open Questions

Warning

TurnNat is tested only on synthetic edits of English human-human dialogue. It is never run on the output of any spoken dialogue system, and its scores are never correlated with per-clip human naturalness ratings. Its validity as a metric for full-duplex systems therefore remains untested.

  • The perturbations are deliberately large (shifts of 1.2-2.5 s). It is unclear whether the score resolves the subtler timing differences that separate current systems.
  • No existing turn-taking evaluation method is run on the benchmark, so TurnNat’s advantage over behaviour-specific diagnostics is argued, not measured.
  • The human validation covers 150 of the 1,000 pairs with three annotators per pair. Early-entry and hold-instead pairs show substantial artifact differences.
  • The score depends only on voice activity. It cannot capture naturalness that depends on lexical content, discourse intent, prosody or system-side latency (§VIII).
  • Training and evaluation data come from one corpus (Seamless Interaction) in one language. Cross-corpus and cross-language robustness is untested.
  • Model sizes for the VAP and DualTurn scorers are not reported.

Wiki Connections

  • Evaluation Metrics — proposes and validates a reference-free, likelihood-based metric for turn-taking naturalness in two-channel dialogue.
  • DualTurn — the paper fine-tunes DualTurn’s dual-channel turn-taking model with a 256-way categorical head, auxiliary heads and TBU weighting to build its best scorer.
  • Full-Duplex-Bench — cited as the behaviour-specific benchmark whose per-task diagnostics TurnNat aims to complement with a single score.
  • Full-Duplex-Bench v1.5 — cited as the overlap-focused follow-up benchmark; TurnNat targets the same full-duplex timing behaviours.
  • Moshi — cited as an example of the full-duplex speech-to-speech dialogue systems whose turn-taking TurnNat is meant to evaluate.
  • Freeze-Omni — cited as motivation, as one of the real-time speech-to-speech dialogue models the metric targets.
  • WavChat — cited survey of spoken dialogue models that frames the move toward real-time speech-to-speech interaction.