DialRes Workshop · 2026 · Workshop
Caroline Kleen et al. (Marburg University / Lamarr Institute / University of Bonn) · → Paper · Demo: ? · Code: ?
Tests whether converting German dialect recordings to a single target voice with off-the-shelf Retrieval-based Voice Conversion (RVC) keeps the dialect cues listeners use: 50 lay listeners classify converted speech as accurately as 50 others classify the originals, F0 and articulation rate survive conversion almost unchanged, and RVC-augmented training lifts a neural dialect classifier by 4.5 points.
Problem
Dialect cues and speaker identity live in the same acoustic dimensions, so a dialect classifier trained on few speakers can learn speaker traits instead of dialect. The authors’ earlier work used RVC to map many speakers onto one target voice and found it improved low-resource dialect classification, but that only showed a performance gain. It did not show whether the converted speech still carries the dialect information a human would recognise, i.e. whether RVC is a valid speaker-normalization step for dialect resources or simply a useful perturbation (§1). The paper asks both that question and a second one: whether human and machine dialect recognition respond the same way to the same speaker normalization.
Method
Data (§2.1, Table 1). Recordings come from the REDE corpus: older male speakers (60+) translating the 40 Wenker sentences into their local dialect. Speech is diarized, concatenated per speaker, cut into 10-second segments, and resampled to 16 kHz mono. Nine dialect areas (Frisian excluded) give 5,170 segments from 166 speakers (14.36 h, 91 locations), with a class imbalance from 4 speakers (Brandenburgish) to 34 (Northern Low German).
Voice conversion (§2.2). All segments are converted with RVCv2 to one public pre-trained target voice, a German dubbing-actor mix of a middle-aged adult male. The system is used as is; no conversion model is trained or adapted, and conversion settings (e.g. pitch transposition, index ratio) are not reported. The authors characterise the converted audio by comparing per-segment means of F0, articulation rate (Praat Vocal Toolkit), and long-term average formants F1 to F3 before and after conversion.
Machine experiment (§2.3.1). TRILLsson embeddings feed a two-hidden-layer MLP (297,876 parameters). Training uses speaker-disjoint splits (about one tenth of each dialect’s speakers to validation and to test), repeated over 250 random runs. The augmented condition adds RVC-converted copies of the training segments to the original training data. The paper does not say that test segments are converted, so the machine-side manipulation is training-time augmentation, not classification of converted test speech. A size-matched frequency-masking augmentation controls for the extra data volume (§3.2).
Human experiments (§2.3.2). In a between-subjects design, 50 native German lay listeners (Study-O) classify 100 original 10-second stimuli and 50 others (Study-VC) classify the RVC-converted versions of the same stimuli, one stimulus per speaker, choosing from a map of ten dialect areas plus “don’t know” (Frisian and “don’t know” responses are dropped from accuracy). Analyses compare accuracy (t-test and a TOST equivalence test with ±5-point bounds), confusion-matrix correlation, per-dialect F1, item-level Jensen-Shannon divergence of response distributions, and a mixed-effects model of listener origin × target dialect × condition.
Key Results
What conversion changes acoustically (§2.2, Figure 2). Mean F0 is essentially unchanged (118.55 ± 10.71 Hz original vs. 118.10 ± 10.68 Hz converted, pitch range 59.72 vs. 59.71 Hz), as is articulation rate (4.39 vs. 4.42 syl/s). The spectral envelope moves: F1 rises from 536.20 to 625.22 Hz (about 89 Hz), F2 falls from 1625.51 to 1597.58 Hz, F3 moves from 2616.68 to 2667.59 Hz, and the across-segment standard deviation of F3 halves (92.17 to 41.75 Hz). The authors call these shifts negligible and within the allophonic range, but they also flag the roughly 90 Hz F1 shift in §4 as an open question for dialect vowel contrasts. In this setup, conversion kept each source speaker’s own pitch and timing and changed mainly formant-level timbre, so the “single target speaker” normalizes the spectral envelope but not F0.
Human listeners (§3.1, Figures 3 and 4). Accuracy is 31.66% (SD 13.51) on originals and 31.38% (SD 13.87) on converted speech, t(98) = 0.10, p = .919. The TOST test meets the ±5-point equivalence bound at p = .044. Confusion matrices across conditions correlate at r = 0.96, per-dialect F1 changes range from −0.049 (East Franconian) to +0.102 (Brandenburgish), and item-level Jensen-Shannon divergence is low (M = .091). The listener-origin × target-dialect interaction is significant (F(64, 8603.4) = 4.81, p < .001), and its three-way interaction with condition is not (F = 1.19, p = .143): listeners’ regional bias looks the same with and without conversion.
Classifier (§3.2). RVC augmentation raises accuracy from 52.7% (SD 7.4) to 57.2% (SD 6.7), t(498) = 7.26, p < .001, d = 0.65, with per-dialect F1 gains of +0.019 to +0.073 across all nine classes. Size-matched frequency masking reaches 55.6%, so RVC beats it by 1.6 points (t(498) = 2.28, p = .023, d = 0.20). Most of the gain over baseline (2.9 of 4.5 points) is therefore matched by generic augmentation. Human and model confusion matrices correlate at r = 0.80 (§3.3).
On the framing. The authors call the result a “double dissociation” under “matched conditions”. Only one manipulation is involved: it changes one system and leaves the other unchanged, which is a single dissociation. The two systems are also not exposed to the manipulation in the same way. Listeners classify converted test stimuli, while the classifier is trained with converted copies added and, as far as the paper states, tested on original speech. The human result is direct evidence about the converted audio. The machine result is evidence about converted audio as training data. The authors label d = 0.65 a “large” effect, which is closer to medium on Cohen’s conventional scale.
Novelty Assessment
The voice conversion system, the classifier recipe, and RVC augmentation for dialect classification all come from prior work, including the authors’ own. What this paper adds is a perceptual validation of off-the-shelf RVC output for a property speech-generation papers rarely test: whether dialect identity survives conversion, measured by human dialect classification of converted versus original speech, with an equivalence test and an item-level analysis, plus a simple acoustic before/after characterisation. The human-versus-machine framing is mostly interpretation layered on two differently designed experiments. There is no architectural or training contribution.
Field Significance
low: For speech generation, the useful content is narrow but concrete: in this setup, off-the-shelf RVCv2 conversion of male German dialect speech to a male target voice preserves source F0 and articulation rate, shifts long-term formants (F1 by about 89 Hz), and leaves lay listeners’ dialect classification statistically equivalent to the originals. This supports treating any-to-one conversion as accent- and dialect-preserving when source and target are matched in sex, and as a timbre-only normalizer rather than a full speaker normalizer. It gives no evidence on speaker similarity, naturalness, or cross-sex conversion, and its claims about human-machine representational differences are outside speech generation.
Claims
- supports: Any-to-one voice conversion can change a speaker’s timbre while preserving the regional-dialect cues that human listeners use to classify accent.
Evidence: Fifty lay listeners classified RVC-converted 10-second segments from nine German dialect areas at 31.38% vs. 31.66% for fifty listeners on originals (t(98) = 0.10, p = .919; TOST equivalence at ±5 points, p = .044), with confusion matrices correlated at r = 0.96 and low item-level Jensen-Shannon divergence (M = .091). (§2.3.2, §3.1.1, Figure 3)
- refines: Retrieval-based voice conversion to a same-sex target voice mainly alters the spectral envelope and leaves source pitch and timing largely intact, so it normalizes timbre more than prosody.
Evidence: Mean F0 moved from 118.55 to 118.10 Hz and articulation rate from 4.39 to 4.42 syl/s, while long-term average F1 rose from 536.20 to 625.22 Hz and F3’s across-segment SD fell from 92.17 to 41.75 Hz; source and target were both adult males. (§2.2, Figure 2)
- supports: Voice-converted copies of training speech can improve speaker-independent paralinguistic classifiers in low-speaker-count settings, beyond what generic spectral augmentation provides.
Evidence: RVC augmentation raised a TRILLsson + MLP dialect classifier from 52.7% to 57.2% over 250 speaker-disjoint runs (d = 0.65), with gains in every dialect class, and beat size-matched frequency masking (55.6%) by 1.6 points (p = .023, d = 0.20). (§2.3.1, §3.2)
- complicates: Preservation of perceptual category judgments after voice conversion does not show that the conversion removed speaker identity or kept the speech natural; both need their own measurements.
Evidence: The study uses no speaker-similarity, speaker-verification, naturalness, or intelligibility measure on the converted audio, uses only older male speakers, and its acoustic check shows a roughly 90 Hz F1 shift that the authors leave as an open question for dialect vowel contrasts. (§2.2, §4, §6 Limitations)
Limitations and Open Questions
- Speaker removal is assumed, not measured. The paper calls RVC a speaker-normalization step but never checks how much source identity remains (no speaker-verification or similarity score, no listener speaker-discrimination test). Since F0 and articulation rate are carried over from the source, some speaker information plausibly remains.
- Asymmetric human and machine conditions. Humans hear converted test speech; the classifier sees converted speech only as added training data. The “dissociation” compares a test-time manipulation with a training-time augmentation, and the paper reports no classifier result on converted test audio, which would be the matched comparison.
- Narrow speaker and voice coverage. Only older male speakers, one male target voice, and one conversion configuration are tested (§6), so nothing is shown about cross-sex conversion, where larger F0 and formant changes would be needed.
- Low human accuracy and marginal equivalence. Lay listener accuracy is around 31% across nine classes, and the TOST result (p = .044 at a ±5-point bound) is close to the threshold. A within-subjects replication, which the authors propose, would give more sensitivity.
- No quality evaluation of the converted speech. No MOS, intelligibility, or artifact analysis is reported, so the result cannot tell whether conversion artifacts were simply below the level that affects dialect judgments.
- The vowel-space question the authors raise (whether the F1 shift alters dialect-specific vowel contrasts at the phonological level) remains open.
Wiki Connections
- Voice Conversion — Applies an off-the-shelf any-to-one RVCv2 model and evaluates its output with acoustic before/after measurements and human classification of converted versus original speech, showing dialect cues, F0, and articulation rate are preserved while formants shift.
- Subjective Evaluation — Runs a between-subjects listening experiment (50 + 50 native German listeners, 100 stimuli each) with a TOST equivalence test and item-level response-divergence analysis to assess whether conversion changes perceived dialect.