arXiv · 2026 · Preprint
Michael Kuhlmann et al. (Paderborn University) · → Paper · Demo: ? · Code: ?
Introduces a consistency-regularization technique that stabilizes frame-level speech quality scores from utterance-level-trained SQA predictors, then uses the resulting low-volatility frame scores to localize synthesis artefacts in two state-of-the-art TTS systems (F5-TTS, StyleTTS2), validated with a human listening test confirming detected segments are judged low-quality significantly more often than random controls.
Problem
Automatic speech quality assessment (SQA) models typically summarize an utterance’s quality into a single scalar score, which is useful for tasks like data cleaning or benchmarking generative speech systems but cannot explain what specifically drove a given score, or pinpoint which parts of an utterance are responsible for poor quality. Frame-level scores would offer better interpretability and could localize synthesis artefacts directly, but no large-scale dataset provides frame-level quality labels for training, so any frame-level predictor must be learned indirectly from utterance-level targets alone. Prior work (QualityNet, and the authors’ own earlier study) showed frame-level scores extracted this way can detect artificial distortions, but frame-level scores near a local artefact get “smeared” by the surrounding clean context, since nothing in the utterance-level training objective incentivizes local embeddings to encode only local information, which limits detection precision.
Method
The paper builds on standard encoder-decoder SQA models (a WavLM encoder producing frame embeddings, followed by a decoder, either linear or BLSTM, that produces per-frame scores time-pooled into an utterance-level prediction) and addresses the smearing problem with a consistency-regularization loss adapted from a technique originally used to stabilize neural audio codec token embeddings under varying context. During training, a contiguous slice of the encoder’s local-context feature-extractor output is separately re-encoded by the Transformer encoder, and the mean-squared error between the full-context and slice-only embeddings for that segment is added to the training loss (an embedding-consistency term). For decoders with long-range context (like a BLSTM), a matching frame-score-consistency term (mean absolute error between full-context and slice-only frame scores) is added as well, since embedding consistency alone may not suffice when the decoder itself has long-range dependencies. The combined objective sums the standard utterance-level SQA loss with these consistency penalties, weighted by tunable coefficients. Eight model configurations (varying encoder size, decoder type, and consistency-loss weighting) are trained on the combined BVCC and NISQA corpora and compared on the BVCC test split using both standard utterance/system-level Spearman rank correlation and a novel “frame score volatility” metric (based on the standard deviation of log-returns between consecutive frame scores) that quantifies how erratically the frame-level predictions fluctuate.
The regularized frame scores are then validated on two localization tasks. First, partial-spoof detection: models trained conventionally on utterance-level human MOS labels (with no exposure to spoof-region ground truth) are applied, via a tuned threshold on frame scores, to detect artificially substituted synthetic segments within otherwise bona fide speech, evaluated with an intersection-based detection criterion against a topline model (BAM) that was trained with explicit knowledge of spoof locations. Second, TTS artefact localization: the best-performing configuration is applied to ~10,000 speech samples generated by each of two state-of-the-art zero-shot TTS systems (F5-TTS and StyleTTS2), cloning LibriTTS test-clean utterances, with a per-application detection threshold calibrated so that only the lowest 1% of frame scores from real human reference speech in the target domain (audiobook narration) would be flagged. Detected low-quality segments are filtered for minimum duration and merged via a sliding window, then evaluated in a listening test where speech-technology experts classify each flagged segment (and matched random control segments) as natural human-like speech or not, and, if not, select an artefact type from a provided list.
Key Results
Consistency regularization reduces mean frame-score volatility from 0.51 in the baseline BLSTM-decoder configuration to as low as 0.051-0.098 across regularized variants, while utterance- and system-level correlation with human MOS ratings on BVCC improves or holds steady rather than degrading (utterance SRCC 0.864→0.883, system SRCC 0.904→0.923, comparing the unregularized and best regularized configurations). On PartialSpoof localization, this volatility reduction translates directly into improved detection precision: the best regularized configuration reaches an F1 score of 0.492 (intersection criterion ρ1) versus 0.355 for the unregularized baseline, though a clear precision-recall trade-off emerges, lower-volatility models achieve higher precision but lower recall, since consistent frame scores are less likely to flag brief, high-quality synthetic substitutions that don’t visibly degrade local acoustic quality. Applied to real TTS output, the detector flags 2,454 segments across 1,481 F5-TTS utterances and 911 segments across 555 StyleTTS2 utterances (versus only 857 segments across 407 utterances in the clean LibriTTS reference set at the same false-alarm-rate threshold). In the listening test, every control set (randomly sampled segments) received a higher percentage of “natural human speech” ratings than its corresponding detection set for both TTS systems, confirming the frame-level detector reliably identifies genuinely lower-quality segments rather than producing arbitrary false positives. The two systems showed qualitatively distinct dominant artefact types: StyleTTS2’s flagged segments were most often classified as “non-speech” (complete synthesis failures that should have been filtered earlier but weren’t) or general “low quality” degradation (low SNR, clipping, buzzing), while F5-TTS’s flagged segments were most often classified as “speed” artefacts, unnaturally fast, pause-free speech, which the authors attribute to text-length/reference-audio-length misalignment during voice cloning.
Novelty Assessment
The consistency-regularization technique is a genuine training-methodology contribution: while frame-level SQA prediction and utterance-level-only supervision are both established, the specific adaptation of segment-slice-based consistency regularization (previously used for neural audio codec token stability) to the frame-level speech-quality-prediction setting, with both an embedding-level and a score-level consistency term, is new to this domain and directly and measurably solves the smearing problem prior frame-level SQA work identified but could not resolve without either losing global context (naive chunking) or accepting high volatility. The downstream application, using regularized frame scores to localize real synthesis artefacts in production-grade TTS systems and validating the detections with a genuine human listening test rather than only automatic metrics, demonstrates the technique has practical diagnostic value beyond the PartialSpoof benchmark it is primarily validated on. The finding that different TTS systems produce systematically different dominant artefact types, and that detector sensitivity itself depends on training-data distribution (e.g., under-represented non-verbal vocalizations), is a useful, application-relevant insight for anyone using automatic SQA tools to audit TTS output.
Field Significance
high — this paper solves a genuine, previously identified limitation of frame-level speech quality prediction (context-induced smearing near local artefacts) with a targeted, effective consistency-regularization technique, and is among the few works to validate automatic artefact-localization directly against human judgment on real, state-of-the-art TTS system outputs rather than only on synthetic benchmark distortions, surfacing system-specific artefact-type patterns that have direct diagnostic value for TTS development.
Claims
- supports: A consistency-regularization loss that encourages a frame-level speech-quality predictor to produce the same embeddings and scores for a segment whether or not it is embedded in longer context substantially reduces frame-level score volatility without degrading overall quality-prediction accuracy.
Evidence: Adding embedding- and score-consistency loss terms reduces mean frame-score volatility from 0.51 to as low as 0.051-0.098 across regularized configurations, while utterance- and system-level Spearman correlation with human MOS on BVCC improves or holds steady (0.864→0.883 utterance SRCC, 0.904→0.923 system SRCC). (§3.2, Table 2)
- supports: Reducing frame-level score volatility via consistency regularization directly improves the precision of localizing short, artificially inserted artefacts in audio, even though the model is trained only on utterance-level quality labels with no frame-level ground truth.
Evidence: On PartialSpoof, the best regularized configuration reaches an F1 score of 0.492 (intersection criterion) versus 0.355 for the unregularized baseline, despite no model in the comparison being trained with knowledge of spoofed-region locations. (§3.3, Table 3)
- complicates: Reducing frame-level score volatility to improve artefact-localization precision involves a genuine precision-recall trade-off, since more consistent frame scores tend to miss more true artefact instances even as they produce fewer false detections.
Evidence: Across model configurations, lower volatility positively correlates with higher detection precision on PartialSpoof but is accompanied by decreased recall, and the paper notes many partial-spoof substitutions are themselves high-quality synthesis that frame-level scores are expected to miss regardless of regularization strength. (§3.3)
- supports: A speech-quality-based frame-level detector trained without any TTS-specific supervision can identify audio segments in real state-of-the-art TTS system output that human listeners subsequently judge as containing genuine synthesis artefacts significantly more often than randomly selected control segments from the same output.
Evidence: In a listening test with 9 speech-technology experts, all control sample sets received a higher percentage of “natural human speech” ratings than the corresponding detection sets for both F5-TTS and StyleTTS2, confirming the detector reliably flags genuinely lower-quality segments. (§4.3, Figure 2)
- complicates: The specific types of synthesis artefacts a frame-level speech-quality detector is sensitive to are shaped by its training data distribution and differ systematically across TTS systems, meaning detector outputs require system- and application-specific interpretation rather than a single universal quality reading.
Evidence: StyleTTS2’s flagged detections were most often classified as “non-speech” synthesis failures or general low-quality degradation, while F5-TTS’s flagged detections were most often “speed” artefacts attributed to text/reference-length misalignment during voice cloning, and the detector separately assigns low scores to non-verbal vocalizations like laughter, likely reflecting their under-representation in training data rather than a genuine quality problem. (§4.3, Figure 3; §5)
Limitations and Open Questions
The authors' own discussion confirms that the artefact types the SQA model is sensitive to depend on its training data distribution: an informal listening test found the model assigns low scores to non-verbal vocalizations like laughter or throat-clearing, likely because these were rarely seen during training, which could produce undesired false positives depending on the target application, and the authors explicitly flag future work is needed to align the SQA model's sensitivity with the intended deployment context rather than treating detector output as an application-agnostic quality signal.
The volatility-precision-recall trade-off means the localization approach is not well suited for the partial-spoof scenario specifically, since many spoofed substitutions are themselves high-quality synthesis that low-volatility frame scores are likely to miss; the authors note this scenario is “not a perfect match” for their use case. The TTS artefact listening test evaluated only the 100 lowest-scored segments per system (600 samples total across 2 systems × 3 sets), with 9 raters each providing one rating per sample and 60 of the 600 samples left without any rating due to the experimental design, a relatively small-scale validation for the artefact-type breakdown conclusions drawn from it.
Wiki Connections
- Evaluation Metrics — introduces a consistency-regularization technique for training frame-level speech quality predictors from utterance-level labels alone, plus a frame-score volatility metric, enabling interpretable localization of quality degradation within an utterance.
- Subjective Evaluation — validates automatic artefact detections against real human judgment via a listening test with 9 speech-technology experts classifying detected and control segments as natural speech or specific artefact types.
- F5-TTS — one of two state-of-the-art zero-shot TTS systems whose output is analyzed for synthesis artefacts using the paper’s frame-level detection method, revealing a dominant “speed” artefact pattern.
- UTMOS — cited as a comparable non-intrusive speech quality predictor within the broader SQA methodology landscape this paper’s frame-level approach extends.
- Meta Audiobox Aesthetics — cited as a related unified automatic quality assessment system for speech, music, and sound, part of the broader SQA background this paper builds on.
- LibriTTS — supplies the test-clean split used as the target application domain (audiobook narration) for generating and analyzing TTS outputs from F5-TTS and StyleTTS2.
- The State Of TTS — cited as evidence that F5-TTS and StyleTTS2 are representative state-of-the-art, human-like-sounding TTS systems, motivating their selection for this paper’s artefact analysis.