arXiv · 2026 · Preprint
Jia-Hong Huang et al. (University of Amsterdam, Georgia Institute of Technology, Halmstad University) · → Paper · Demo: ? · Code: ?
Frames intra-utterance speaker drift in synthesized speech as a reference-free binary classification task, solved by feeding adjacent-segment speaker-embedding cosine similarities to prompted LLMs, and evaluates it on a small constructed benchmark of drifted and non-drifted utterances.
Problem
Diffusion-based TTS systems produce natural, long-form speech, but the authors argue they can exhibit speaker drift: a gradual change in perceived speaker identity within a single utterance despite conditioning on one fixed speaker embedding or prompt. Existing speaker tasks do not cover this case. Speaker change detection and diarization assume abrupt, discrete turns; speaker verification compares separate utterances against labelled references; and voice cloning consistency is usually judged by human listening or ABX tests rather than an automatic, temporally fine-grained detector (Table 1). The paper reports that no standard protocol, automatic method, or dataset exists for this problem. The arXiv comment states the paper was accepted at ICASSP 2026.
Method
Benchmark construction (§2). The paper does not collect real drift from a TTS model. Instead, it starts from 384 utterances synthesized by an unnamed “high-fidelity, diffusion-based TTS model” and assembles three-segment utterances (9 to 40 seconds, with 500 ms silences between segments) in four subtypes of 32 samples each:
- Non-drift (label 0): three segments from the same speaker.
- Hard negative (label 0): same speaker under altered conditions, such as speaking rate and pitch variation (e.g. speed-up rate 1.05) or added background noise (e.g. -30 dB).
- Abrupt drift (label 1): at least one segment from a different speaker, e.g. [s1, s1, s2].
- Smooth morphing (label 1): a linear time-domain cross-fade between two speakers’ waveforms over a window (e.g. 3 to 6 s), so identity shifts gradually from A to B.
The resulting 128-sample set is label-balanced, and every sample was manually checked by human annotators for clarity and intended speaker characteristics. No listening-test protocol, rater count, or agreement statistic is reported.
Detection pipeline (§3.1 to §3.3). Each utterance is split into its three segments, and a pre-trained encoder (Wav2Vec2 by default; MFCC and Whisper in ablation) produces one embedding per segment. Only two numbers are passed on: the cosine similarities between segments 1 and 2 and between segments 2 and 3. These go into a fixed instruction prompt asking an off-the-shelf LLM (GPT-4o, Gemini-Pro-2.5, Claude-4, DeepSeek-R1, or Qwen-3) to choose “same” or “different” and explain why. Nothing is trained; LLMs are queried in zero-shot and few-shot settings. The authors motivate summarizing embeddings as scalars by the fact that LLMs cannot take high-dimensional embeddings as input directly.
Theory (§3.4). A qualitative proposition argues that a Lipschitz classifier can separate drift from non-drift when embeddings are smooth within a speaker and separated across speakers. The main theorem assumes unit-norm embeddings, same-speaker similarity mean at least µ0, different-speaker mean at most µ′ < µ0, and shared variance bound σ². Under these assumptions, a minimum-similarity threshold classifier has error at most 4·exp(-Δ²/2σ²), where Δ is the margin from the threshold to the nearer class mean. The proof applies Hoeffding’s inequality per segment pair and takes a union bound.
Key Results
All results come from the 128-sample benchmark, reported as F1 (Table 2) and accuracy plus F1 (Table 3).
- Best configuration: GPT-4o with Wav2Vec2 cosine scores reaches 90.7% F1 and 89.5% accuracy. Claude-4 follows at 88.2% F1, then Gemini-Pro-2.5 (82.9%), DeepSeek-R1 (80.0%), and Qwen-3 (72.7%) (Table 2, Table 3).
- Baselines: a fixed threshold of 0.90 on the minimum adjacent similarity scores 61.7% F1. The PCA-based baseline (LLM prompted with PCA-reduced embeddings) scores 71.3% F1 with Wav2Vec2 (Table 2). The authors chose the 0.90 threshold from the empirical distribution and note that the classes overlap: non-drift samples dip as low as 0.77, and drift samples can be as low as 0.80 (§4.1).
- Embedding type: MFCC similarities are much weaker for most LLMs (GPT-4o 39.0% F1). Whisper is close to Wav2Vec2 for GPT-4o (89.41%) and higher for DeepSeek-R1 (90.6% vs. 80.0%). This contradicts the text’s general statement that Wav2Vec2 is best (Table 2).
- Input format: cosine scores beat PCA-reduced embeddings for every LLM. Using 16 PCA dimensions per segment beats 8: GPT-4o goes from 66.7% to 73.6% F1, and DeepSeek-R1 from 71.2% to 74.4% (Table 3).
The parsed Table 3 has misaligned cells for some Gemini, Claude, and Qwen PCA rows, so only clearly aligned values are reported here. Zero-shot and few-shot results are not reported separately.
The comparison leaves an important gap. The LLM decides from two scalars, but the only non-LLM baseline is a single hand-picked threshold. No tuned threshold, logistic regression, or other simple learned classifier on the same two features is reported, so it is unclear how much the LLM adds beyond fitting a decision boundary.
Novelty Assessment
The paper’s main contribution is the framing: it names intra-utterance speaker drift in TTS output as a separate detection problem and builds a small labelled test set for it. The detection method itself is a light combination of known parts (segment-level speaker embeddings, cosine similarity, and an LLM prompt), with no learned components. The error bound is a standard concentration-plus-union-bound argument for a threshold classifier. It does not explain why an LLM should beat that threshold classifier. The benchmark simulates drift through segment concatenation and waveform cross-fading, not through drift produced by a TTS model during generation. This weakens the link between the stated motivation (diffusion TTS drift) and what is actually measured.
Field Significance
Low. The paper introduces speaker drift detection as a task and provides a first, very small benchmark with LLM-based baselines, which can serve as a starting point for automatic intra-utterance speaker-consistency checks in long-form TTS. The evidence is limited by the 128-sample constructed test set, the use of non-speaker-specific encoders, and the lack of learned non-LLM baselines on the same features.
Claims
- supports: Speaker-embedding similarity between adjacent segments of a synthesized utterance provides a usable reference-free signal for detecting within-utterance changes in speaker identity.
Evidence: Adjacent-segment Wav2Vec2 cosine scores passed to GPT-4o reach 90.7% F1 on a balanced 128-sample benchmark covering abrupt speaker swaps and cross-faded morphs against same-speaker and hard-negative utterances. (§3.2, §4.2, Table 2, Table 3)
- complicates: A single fixed threshold on speaker-embedding cosine similarity separates same-speaker from changed-speaker speech poorly, because the two classes’ similarity ranges overlap.
Evidence: A 0.90 minimum-similarity threshold scores only 61.7% F1. Non-drift samples (including speed-perturbed and noise-added hard negatives) fall as low as 0.77, while drift samples reach as low as 0.80. (§4.1, Table 2)
- refines: For automatic speaker-consistency judgments, explicit relational features between segments are more informative than compressed per-segment embeddings.
Evidence: For every LLM tested, pairwise cosine scores outperform 8- or 16-dimensional PCA-reduced embeddings. For GPT-4o, F1 rises from 73.6% (16-dim PCA) to 90.7% (cosine scores). (§4.2, Table 3)
- complicates: The choice of acoustic encoder strongly affects automatic speaker-consistency detection, and no single general-purpose encoder is best for every downstream judge.
Evidence: MFCC-based similarities drop GPT-4o to 39.0% F1. Whisper embeddings match or beat Wav2Vec2 depending on the LLM: 90.6% vs. 80.0% F1 for DeepSeek-R1, but 89.41% vs. 90.70% for GPT-4o. (§4.2, Table 2)
Limitations and Open Questions
Warning
The benchmark contains only 128 constructed utterances. Drift is simulated by concatenating or cross-fading segments from different speakers, not observed in TTS outputs, so the paper does not show that the method detects the gradual drift produced by diffusion TTS models that motivates it.
- The source TTS model, speaker set, and dataset release are not specified, and no code or data availability is stated, which limits reproducibility.
- Human validation is described only as manual inspection, with no rater count, protocol, or agreement measure.
- The speaker representations are general-purpose encoders (Wav2Vec2, Whisper, MFCC). Dedicated speaker-verification embeddings such as x-vectors or ECAPA-style models are mentioned but not tested.
- No tuned or learned non-LLM classifier on the two cosine features is compared, so the value added by LLM reasoning is unclear.
- The detector assumes exactly three equal-length segments with known boundaries. Detecting drift at arbitrary times in free-running TTS output is not addressed.
- Some Table 3 cells are misaligned in the source parse, and zero-shot vs. few-shot results are not broken out.
- Future work proposed by the authors: multilingual and cross-lingual extension, and fine-tuning LLMs for greater sensitivity to subtle identity cues.
Wiki Connections
- Evaluation Metrics — Proposes an automatic, reference-free speaker-consistency check and a labelled benchmark for intra-utterance speaker drift in synthesized speech.
- Self-Supervised Speech — Uses Wav2Vec2 segment embeddings as the default speaker representation underlying the cosine-similarity signal.
- Whisper — Whisper encoder embeddings are an ablation alternative to Wav2Vec2, giving comparable or better F1 depending on the LLM judge.
- NaturalSpeech 2 — Cited as an example of the diffusion-based TTS systems whose long-form output motivates speaker drift detection.
- Grad-TTS — Cited as an example of the diffusion TTS systems the framework is meant to quality-check.
- Continuous-Token Diffusion for Speaker-Referenced TTS — Prior speaker-referenced diffusion TTS work sharing an author, cited as motivation for checking speaker consistency in diffusion TTS.