arXiv · 2026 · Preprint
Venkata Pushpak Teja Menta (Praxel Ventures) · → Paper · Demo: ✗ · Code: ✓
Trains a small projection head on a frozen WavLM-base-plus with a speaker contrastive loss and a gradient-reversal language classifier, yielding a speaker encoder whose same-voice similarity changes little when the voice switches between English, Hindi, Telugu, and Tamil, measured on cross-script pairs synthesised by a commercial multilingual TTS system.
Problem
Cross-script voice cloning and multilingual diarisation both assume that a speaker encoder assigns the same identity to one voice regardless of the language or script being spoken. The paper measures how far off-the-shelf speaker-verification encoders depart from that assumption for Indic scripts. On its Western-accented held-out corpus, WavLM-base-plus-sv drops from a median within-script same-voice cosine of 0.927 to 0.845 cross-script, and ECAPA-TDNN drops from 0.499 to 0.394 (Table I). The paper argues that this leak inflates speaker-change errors at script switches and makes cross-script speaker similarity an unreliable signal for evaluating or conditioning cross-script TTS. Prior cross-language speaker work it cites targets English and European language pairs; no natural same-speaker cross-script Indic corpus exists at scale, because IndicTTS speakers are mostly monolingual and FLEURS splits speakers by language.
Method
LASE (Language-Adversarial Speaker Encoder) is not a speech generator. It is a 256-dimensional speaker embedding built from three parts (§III-A). A frozen WavLM-base-plus backbone produces 768-dimensional frame features at 16 kHz; features from layers 10 to 12 are mean-pooled and passed through a trainable two-layer MLP (768 to 512 to 256, ReLU, dropout 0.1) to give the embedding. The embedding also feeds a gradient-reversal layer and a small MLP that classifies the clip’s language among English, Hindi, Telugu, and Tamil. The total parameter count is not reported.
Training sums two losses (§III-B). A supervised contrastive loss over voice identity (temperature 0.07) treats same-voice clips as positives across scripts, since batches mix languages. A 4-class cross-entropy on the language classifier is back-propagated through the gradient-reversal layer, pushing the projection head to make the embedding language-uninformative; the author’s target is for the language loss to stay near ln 4 (chance). The reversal strength follows a three-phase schedule: 200 steps at zero, a linear ramp to 0.1 over 500 steps, then held (§III-C). Training runs for 1000 steps (batch 16, AdamW, learning rate 1e-4) on one A10G GPU in about 17 minutes.
The training data are TTS-bootstrapped (§IV). Eight ElevenLabs Multilingual voices each synthesise 50 sentences in each of the four languages (1600 nominal clips). Each clip is kept only if its WavLM-base-plus-sv cosine to that voice’s first English sample is at least 0.90, which leaves 1118 training pairs (70% pass rate). A Western-voice held-out set uses the same eight voices with new sentences (1043 pairs, 65% pass rate), so it is content-held-out rather than voice-held-out. A second held-out set uses eight Indian-accented ElevenLabs voices not seen in training (1369 pairs; the paper reports an 86% gate pass rate for Indian-voice pairs in §VI-F).
Evaluation uses a three-distribution framework (§V). For each encoder, 200 random pairs are sampled per bucket: within-script (same voice, same language), cross-script (same voice, different language), and across-speaker (different voices, same language). The paper reports the medians and two summaries: the gap Δ (within minus cross, lower is better) and the margin M (cross minus across-speaker floor, higher is better). Confidence intervals on Δ come from a 1000-iteration bootstrap on the raw cosines. A downstream check builds 50 synthetic code-switching conversations (2 to 4 voices each, 411 segments) from held-out clips, clusters segment embeddings agglomeratively with the true speaker count given and oracle segment boundaries, and scores Adjusted Rand Index (ARI) and cross-script recall (§VI-C).
Key Results
All numbers below were checked against the Table I and Table II images.
- Western-voice held-out (Table I): LASE r1 has Δ = 0.013 [-0.02, 0.05], against 0.083 [0.05, 0.15] for WavLM-base-plus-sv and 0.107 [0.08, 0.14] for ECAPA-TDNN. The prose uses 0.082 and 0.105 for the two baselines (e.g. the 84.3% relative closure, 1 - 0.013/0.082), slightly different from the table values. LASE’s margin is 0.662, against 0.245 and 0.202.
- Indian-accent held-out (Table I): LASE has Δ = 0.026 [-0.04, 0.08]. Here off-the-shelf WavLM-base-plus-sv has the smallest gap of any encoder (0.006 [-0.00, 0.01]), and ECAPA-TDNN has 0.044 [0.02, 0.06]. LASE’s margin (0.344) exceeds WavLM-SV (0.144) and ECAPA-TDNN (0.256), ratios of about 2.4x and 1.3x. The abstract’s “2.4-2.7x over both baselines” therefore holds only against WavLM-SV on this corpus, which §VI-F itself acknowledges.
- Absolute similarity is lower: LASE’s within-script medians (0.757 Western, 0.658 Indian) are well below WavLM-SV’s (0.927, 0.944). The author argues that only relative spacing within each embedding space is comparable.
- GRL ablation (§VI-E, Table I): Training the same recipe on a frozen ECAPA-TDNN backbone cuts ECAPA’s gap from 0.107 to 0.027 (Western) and from 0.044 to 0.037 (Indian). LASE’s Δ is lower than ECAPA+GRL on both corpora, but the CIs overlap, as the paper states. On margin, ECAPA+GRL beats LASE on the Western corpus (0.739 vs 0.662, with an across-speaker floor of -0.052) and trails it on the Indian corpus (0.247 vs 0.344). The comparison isolates the effect of GRL on ECAPA only. No WavLM-plus-head model trained without the adversarial loss is reported, so the GRL contribution on the LASE backbone itself is not separated from the effect of contrastive training on the head.
- Training dynamics (Figure 2): The language loss stays near ln 4 throughout LASE training, while the speaker loss falls from about 2.7 to 0.5 to 1.0. Under the ECAPA backbone, the language loss oscillated between 0.4 and 2.5 (described in text; no figure shown). No post-hoc probe of language information in the final embedding is reported.
- Diarisation (Table II): ECAPA-TDNN leads on mean ARI (0.693 vs 0.640 for LASE) and median ARI (0.793 vs 0.672). Cross-script recall is 0.789 for ECAPA-TDNN and 0.788 for LASE. The paper calls this tie its headline result but gives no confidence interval. Both clearly beat WavLM-base-plus-sv (ARI 0.444, cross-script recall 0.604). The paper does not say which held-out corpus the conversations were drawn from.
- Generalisation (§VI-D): The relative gap closure on the Western held-out set (84.3%) is larger than on training data (78.7%, Δ 0.099 to 0.021 against the training-set WavLM-SV baseline).
Novelty Assessment
The technique is borrowed, and the author says so: gradient-reversal domain-adversarial training (Ganin and Lempitsky) applied to speaker embeddings, as in earlier channel-invariant and multilingual speaker-recognition work. What is new is the application to the Latin/Devanagari/Telugu/Tamil cross-script setting, the TTS-bootstrapped cross-script same-voice corpora, and the within/cross/floor measurement framework, which makes the language-vs-identity leak of a speaker encoder a reportable quantity. The most informative empirical finding is that the leak is accent-conditional: off-the-shelf encoders lose much more same-voice similarity when Western-accented voices speak Indic scripts than when Indian-accented voices do (Δ 0.083 vs 0.006 for WavLM-SV). All of this rests on audio from a single commercial synthesiser, with 200-pair samples per bucket.
Field Significance
Low: the paper offers a cheap, released recipe and a measurement framework for checking whether a speaker-similarity encoder confounds language with identity, which matters for any cross-lingual voice-cloning evaluation that scores speaker similarity with an off-the-shelf verification model. Its evidence is a single-author, synthetic-only, content-held-out study. The encoder does not beat ECAPA-TDNN on general diarisation and has not been validated against human same-voice judgements, so it is better read as an evaluation-methodology caution than as a validated replacement metric.
Claims
- complicates: Speaker-similarity scores from off-the-shelf speaker-verification encoders conflate language or script with speaker identity, so cross-lingual voice-cloning evaluations that use them can understate identity preservation.
Evidence: On same-voice pairs from eight synthetic voices, WavLM-base-plus-sv’s median cosine drops from 0.927 within-script to 0.845 cross-script, and ECAPA-TDNN’s drops from 0.499 to 0.394, among English, Hindi, Telugu, and Tamil. (§VI-A, Table I)
- refines: How much a speaker encoder’s similarity drops across languages depends on the speaker’s accent, and the drop is largest when a voice is rendered in a language whose phonetics it does not natively carry.
Evidence: WavLM-base-plus-sv’s cross-script gap is 0.083 for Western-accented voices but 0.006 for Indian-accented voices speaking the same Indic scripts, and the synthesis quality gate passes 70% of Western-voice pairs vs 86% of Indian-voice pairs. (§VI-F, Table I)
- supports: Adversarial removal of language information from a speaker embedding can reduce cross-lingual identity drift while keeping speakers separable, without large training corpora.
Evidence: A gradient-reversal language classifier plus a supervised contrastive loss on 1118 synthetic pairs brings the Western cross-script gap to 0.013 (CI includes zero) with a 0.662 margin over the across-speaker floor; applying the same objective to an ECAPA-TDNN backbone cuts its gap from 0.107 to 0.027. (§VI-A, §VI-E, Table I)
- complicates: A speaker embedding made language-invariant does not necessarily beat a large-scale speaker-verification encoder on general speaker clustering.
Evidence: In 50 synthetic code-switching conversations, the language-adversarial encoder ties ECAPA-TDNN on cross-script recall (0.788 vs 0.789) but trails it on mean ARI (0.640 vs 0.693) and median ARI (0.672 vs 0.793). (§VI-C, Table II)
Limitations and Open Questions
Warning
All training and evaluation audio is synthesised by ElevenLabs Multilingual, and the training corpus and the Western held-out set share the same eight voices (only the sentences are unseen). No natural human cross-script speech and no human same-voice listening judgements are used, so the paper shows that LASE closes a gap in this synthetic distribution, not that its scores track perceived speaker identity.
- Corpus pairs were kept only if WavLM-base-plus-sv judged them similar (cosine of at least 0.90 to the voice’s first English clip). The held-out sets were therefore filtered with one of the evaluated baselines, which may shape its within- and cross-script distributions. The paper treats the 30% rejection rate as an audit of the synthesiser rather than discussing this selection effect.
- Medians and bootstrap CIs come from 200 sampled pairs per bucket with a single seed. The LASE vs ECAPA+GRL gap difference falls within these CIs on both corpora, and the diarisation tie is reported without any interval.
- The adversarial ablation uses only the ECAPA backbone; a WavLM-plus-head model without the language loss is not reported, so the GRL contribution to LASE itself is inferred, not measured.
- The diarisation benchmark is synthetic, built by concatenating clips with oracle boundaries and a known speaker count, and does not include mixed-script utterances within a single clip.
- Only four languages are covered; Bengali, Kannada, Gujarati, and Malayalam are deferred. Comparisons with gated or unreleased Indic speaker encoders were not run.
- Small numeric mismatches between prose and Table I (baseline gaps 0.082/0.105 in text vs 0.083/0.107 in the table; Indian WavLM-SV margin 0.141 in §VI-F vs 0.144 in the table) were checked against the Table I image. In §VI-E, the margin is cited as a WavLM advantage although ECAPA+GRL has the larger Western margin.
Wiki Connections
- Evaluation Metrics — proposes a cross-script speaker-similarity encoder and a within/cross/floor framework for checking whether a speaker-similarity measure confounds language with identity.
- Disentanglement — removes language information from a speaker embedding with a gradient-reversal language classifier, with an ECAPA+GRL ablation showing the objective reduces the cross-script gap.
- Self-Supervised Speech — builds the encoder as a trained projection head over frozen WavLM-base-plus layers 10 to 12, and compares against the WavLM-base-plus-sv fine-tune.
- Praxy Voice — the companion cross-script Indic TTS system that, per the acknowledgements, consumes LASE’s output in production; LASE itself is not used to evaluate Praxy in this paper.
- PSP — companion accent benchmark by the same author; LASE is framed as measuring the orthogonal speaker-identity axis, with PSP covering accent.
- ECAPA-TDNN — used both as a baseline encoder and as the alternative frozen backbone in the GRL ablation; it ties LASE on cross-script recall and leads on diarisation ARI.
- Quantifying Speaker Embedding Phonological Rule Interactions — cited as the English/British accent counterpart measuring language-induced shift in speaker embeddings, with LASE positioned as the Indic equivalent.
- Pairwise Evaluation of Accent Similarity — cited as an accent-similarity measure based on PPG and formant distances, described as orthogonal to LASE’s identity axis.