NLPAICS · 2026 · Workshop

Anatoly Marchenko (Independent Researcher) · → Paper · Demo: ✗ · Code: ✗

Measures jitter, shimmer, and harmonics-to-noise ratio (HNR) on three free commodity TTS engines across 21 languages against a 29-corpus, 24-language natural-speech baseline, and finds that these vocal-perturbation features alone no longer reliably separate a current neural engine (Microsoft Edge TTS) from human speech.

Problem

Jitter (cycle-to-cycle pitch variation), shimmer (cycle-to-cycle amplitude variation), and HNR are cheap, interpretable acoustic measures of vocal fold micro-instability, and earlier work used them (with other prosodic features) to separate natural from synthetic speech. The underlying assumption is that algorithmic synthesis lacks the random fluctuations of a biological larynx. Stochastic neural synthesis may have eroded that assumption, and most detection studies are English-only, so it is unclear how well the perturbation channel holds across engines and languages. The paper frames this as a defender’s question for mass voice-phishing (vishing) with free TTS APIs, explicitly excluding targeted zero-shot voice cloning.

Method

The study is purely measurement-based; no speech model is trained. The natural baseline pools 29 public speech emotion recognition corpora covering 24 languages in 11 language families (35,091 utterances after out-of-range filtering). Emotional speech is used deliberately because vishing relies on emotional manipulation. Three features are extracted per utterance with Parselmouth (Praat): local jitter (%), local shimmer (%), and mean HNR (dB), with a 75 to 600 Hz pitch range.

Synthetic speech comes from three free engines, characterised by the author as distinct technology tiers: Google TTS via gTTS (12 languages, 159 samples), Microsoft Edge TTS (21 languages, 1,056 samples, plain text with no SSML prosody markup), and the macOS system voice (9 languages, 635 samples). Per engine and language, short prompts from a fixed multilingual pool are balanced over four emotion-content categories (neutral, happy, sad, angry), using each engine’s default voice inventory; per-language counts range from 40 to 112.

Detectability is assessed two ways: Mann-Whitney U tests with Bonferroni correction and rank-biserial r as effect size, and binary classification (logistic regression, SVM-RBF, Random Forest with 100 trees) on standardized features. Because the natural pool is far larger than any TTS class, 10 random natural subsamples are matched in size to the TTS class, each with 5-fold cross-validation (50 folds), and mean F1 and AUROC are reported with 95% confidence intervals. A per-feature ablation and a per-language jitter analysis complete the evaluation.

Key Results

Effect sizes (Table 1). All three engines differ from natural speech on all three features at p < 0.001, but magnitudes vary sharply. gTTS is “too clean”: relative to natural means, jitter is 1.58% vs. 2.49%, shimmer 6.50% vs. 11.67%, and HNR 17.52 dB vs. 11.23 dB, with a maximum |r| of 0.75. macOS sits in between (max |r| 0.56). Edge TTS has the smallest effects (max |r| 0.36): its jitter is slightly above natural (2.67% vs. 2.49%), HNR only slightly above (12.09 vs. 11.23 dB, r = 0.10), and shimmer carries most of the remaining signal (8.91% vs. 11.67%, r = 0.36). The paper reports that effect sizes shrink by 4.3x (jitter), 2.1x (shimmer), and 7.4x (HNR) from gTTS to Edge TTS. Bootstrap confidence intervals on these effect sizes are reported as tight and non-overlapping between gTTS and Edge TTS, and re-running against a neutral-only subset of the natural pool preserves the engine ordering (§5).

Classification (Table 2). Against Edge TTS, SVM-RBF gives the highest F1 (0.78, AUROC 0.85) versus 0.74 for both logistic regression and Random Forest. Against gTTS, the three classifiers are close (F1 0.81 to 0.82; SVM-RBF and RF tie at 0.82, with RF reaching the highest AUROC, 0.91). Against macOS, SVM-RBF and RF (0.81 and 0.80 F1) clearly beat logistic regression (0.73). Pooling all engines gives F1 0.73 to 0.77. The author sets F1 >= 0.90 as what a standalone detector would need, and no engine-classifier combination reaches it.

Ablation (Table 3, Random Forest). Against Edge TTS, single features give F1 0.52 (jitter), 0.57 (shimmer), and 0.60 (HNR); jitter plus shimmer gives 0.72 and all three 0.75. Against gTTS, single features already reach 0.66 to 0.73 and all three reach 0.84.

Per language (§4.4). For 14 of 21 languages, Edge TTS jitter is not significantly different from the pooled natural baseline after Bonferroni correction; in the other 7 (German, Estonian, Korean, Persian, Kazakh, Polish, Portuguese), Edge TTS jitter overshoots natural by +0.41 to +1.16 percentage points. Per-language TTS samples are small (40 to 112), so some non-significance may reflect limited power, as the paper notes.

Comparisons across engines are descriptive rather than controlled: engines differ in sample count, language coverage, and voices, and the natural reference is a pooled multi-language emotional baseline rather than matched per-language, per-speaker recordings.

Novelty Assessment

The contribution is empirical. Jitter, shimmer, and HNR are classic Praat measures, the classifiers are standard, and the evaluated engines are black-box commercial services with undisclosed architectures. What is new is the scale of the multilingual natural baseline (29 corpora, 24 languages) combined with a 21-language synthetic set, and the explicit isolation of the perturbation channel from the spectral and learned features that production deepfake detectors also use. The headline negative finding is narrow in scope: it concerns three features in isolation, on clean audio, against three free engines, and the paper does not test modern zero-shot cloning systems or combine perturbation features with spectral or neural detectors.

Field Significance

Low. The evidence comes from a single-author automated study of three commercial engines and three voice-source features, with no human evaluation. The paper does provide a multilingual data point that a current commercial neural TTS engine already reproduces natural-range jitter and HNR closely enough that these perturbation features alone give only near-chance single-feature detection, while an older engine remains clearly separable. For speech generation it characterises how far commodity output has moved toward natural voice-source statistics; for detection it argues that perturbation features should be one input among many rather than a standalone defense.

Claims

  • supports: Current neural TTS output reproduces natural-range voice-source perturbation statistics (jitter, HNR) closely enough that these measures show only small differences from human speech.

    Evidence: Edge TTS jitter (2.67% vs. 2.49% natural) and HNR (12.09 vs. 11.23 dB) yield rank-biserial effect sizes of about 0.14 and 0.10, and a neutral-only natural subset keeps both at or below 0.10. (§4.1, Table 1; §5)

  • refines: Detectability of synthetic speech from hand-crafted perturbation features depends strongly on the generation technology, with older engines being markedly over-regular and newer neural engines much closer to natural variability.

    Evidence: Maximum effect size drops from 0.75 (gTTS) to 0.56 (macOS) to 0.36 (Edge TTS); gTTS shows 44% lower shimmer and 56% higher HNR than natural speech, while Edge TTS differs by 7% (jitter), 24% (shimmer), and 8% (HNR). Engines differ in sample count and language coverage, so the comparison is descriptive. (§4.1, Table 1; §5)

  • complicates: Voice-source perturbation features taken alone are insufficient to separate current commodity neural TTS from natural speech.

    Evidence: Against Edge TTS, the best classifier (SVM-RBF) on all three features reaches F1 0.78 and AUROC 0.85, and in a Random Forest ablation no single feature exceeds F1 0.60 (HNR); the author’s target for a standalone detector is F1 >= 0.90. (§4.2, Table 2; §4.3, Table 3)

  • complicates: Synthetic-speech acoustic artefacts are not uniform across languages, so detection evidence from one language does not transfer automatically.

    Evidence: Edge TTS jitter matches the natural baseline in 14 of 21 languages but significantly exceeds it (+0.41 to +1.16) in 7, including German, Korean, and Polish; per-language samples are 40 to 112, limiting power. (§4.4)

Limitations and Open Questions

Warning

Only three perturbation features are tested, in isolation, on clean audio, against three free black-box engines with unequal sample counts (159, 635, 1,056) and language coverage (12, 9, 21); results say nothing about spectral or neural detectors, telephone-codec conditions, or zero-shot voice-cloning systems.

  • The natural baseline is emotional acted speech pooled across corpora and languages, while TTS prompts were neutral-domain sentences labelled by emotion-content category; per-language natural/synthetic matching is not performed (the per-language test compares each language’s Edge TTS samples with the pooled baseline). The neutral-subset check addresses part of this confound.
  • No human perceptual evaluation or MOS is reported, so whether output that matches natural perturbation statistics also sounds natural, especially in lower-resource languages, is untested.
  • Speaker gender and age are not controlled, although jitter and shimmer differ by sex.
  • Edge TTS was run without SSML, so the effect of explicit prosody markup on these features is unknown.
  • The proposed trend (each engine tier shrinking effect size by about a third) rests on three engines and is presented by the author as speculative.
  • No code or sample release is provided, by design (ethics statement).

Wiki Connections

  • Evaluation Metrics — evaluates whether classic voice-source measures (jitter, shimmer, HNR) still distinguish commodity TTS output from natural speech across 21 languages, finding they no longer do so reliably for a current neural engine.