arXiv · 2026 · Preprint
Aayush M. Shrestha et al. (IOE, Thapathali Campus) · → Paper · Demo: ? · Code: ?
Adapts the SV2TTS-style speaker-encoder + Tacotron2 + WaveRNN voice-cloning pipeline to Nepali, a low-resource language with no prior multi-speaker voice cloning system, using a newly collected 235-hour speaker-encoder corpus and an 8.67-hour paired text-audio synthesizer corpus.
Problem
Voice cloning research has concentrated almost entirely on high-resource languages, primarily English, leaving low-resource languages like Nepali largely unaddressed despite Nepal’s linguistic and dialectal diversity. Existing Nepali speech-synthesis efforts are limited: some (Shruti, a book reader) provide no multi-speaker or cloning capability and no perceptual evaluation; others (Aawaj) target accessibility but likewise lack multi-speaker modeling and quantitative metrics; and the one prior Nepali voice-cloning study used only a pre-existing OpenSLR corpus and reported MOS values without disclosing evaluation methodology or supporting detail. No existing system combines multi-speaker voice cloning for Nepali with a purpose-built dataset and a rigorously reported, quantitative perceptual evaluation.
Method
The system follows the SV2TTS architecture (speaker-verification-to-TTS transfer learning): a speaker encoder trained independently of the synthesizer, a Tacotron2-based text-to-mel synthesizer conditioned on the resulting speaker embedding, and a neural vocoder converting mel-spectrograms to waveform. The speaker encoder follows a d-vector design: three stacked LSTM layers (256 units each) followed by a fully connected layer producing a 256-dimensional speaker embedding, trained with Generalized End-to-End (GE2E) loss on untranscribed Nepali audio from many speakers, where variation and quantity matter more than transcription accuracy or audio cleanliness. The Tacotron2 synthesizer’s text encoder output is concatenated with the speaker embedding to condition mel-spectrogram generation on speaker identity, trained on a smaller, cleaner corpus of paired Devanagari text and audio where transcription accuracy and low noise are essential. A pretrained WaveRNN vocoder (upsampling network, residual convolutional blocks, recurrent GRU layers) is fine-tuned on the target dataset rather than trained from scratch, given computational constraints, to convert mel-spectrograms into the final waveform.
Two separate datasets were built to serve these different data-quality requirements. The speaker-encoder dataset combines three OpenSLR Nepali corpora with self-collected audio (audiobooks, self-recordings, YouTube interviews/podcasts), totaling 833 speakers (481 male, 352 female), 235 hours, and 6,466 utterances (mean 1,238 utterances per speaker, from 1.6-second chunks with 50% overlap). The synthesizer dataset combines two of the OpenSLR corpora with self-recorded and audiobook audio, totaling 6,046 audio-text pairs (8.67 hours, mean clip duration 5.16s, 79,375 distinct words). Audio preprocessing standardized format, sample rate (16kHz for the encoder, 22.05kHz for the synthesizer), removed excess silence and manually filtered noise/breathing sounds for the synthesizer set specifically. Text preprocessing normalized numerals/dates to full text form, segmented complex sentences, and manually inserted missing sentence-boundary markers. Speaker-encoder quality was validated via UMAP projections of learned embeddings (showing clear per-speaker clustering and a male/female separation) and Equal Error Rate (EER) on a speaker-verification task, which declined from roughly 0.10 to below 0.04 over training.
Key Results
Cosine similarity between original and cloned-voice speaker embeddings averaged 0.904 across ten held-out test speakers, with most speakers exceeding the paper’s own 0.90 “good” cloning-quality threshold and one reaching 0.951 (“excellent”); two speakers fell into a lower “fair” band (0.82-0.83), attributed to less accurate cloning for those specific voices. A perceptual evaluation with 55 raters across ten held-out speakers (five male, five female, three clips each) produced overall MOS ratings of 3.924 for audio quality and 3.87 for speaker similarity, with male voices rated marginally higher than female voices, though individual per-speaker scores varied (range roughly 3.53-4.22 for quality). The authors report their system outperforms the one prior Nepali voice-cloning study (WaveNet-vocoder-based) by 0.02 MOS in quality and 0.67 in naturalness/similarity, while noting that prior study’s evaluation lacked disclosed methodology for independent verification, and that it relied solely on pre-existing OpenSLR data rather than a newly constructed corpus. Neither of the two other existing Nepali TTS systems surveyed (Shruti, Aawaj) reports quantitative perceptual scores or supports multi-speaker synthesis, which the authors position as the main functional and evaluative distinction of their system.
Novelty Assessment
This is an engineering-integration and dataset paper rather than an architectural one: the speaker encoder, Tacotron2 synthesizer, and WaveRNN vocoder are all established components (the same combination underlies the widely used open-source Real-Time Voice Cloning toolkit the authors explicitly adapt), and no new training objective, conditioning mechanism, or model structure is proposed. The contribution is the successful adaptation of this known pipeline to a language with no prior multi-speaker voice-cloning system, backed by two newly constructed, purpose-differentiated Nepali datasets (a large, noise-tolerant speaker-encoder corpus and a smaller, clean synthesizer corpus) and a properly disclosed, quantitative human evaluation (MOS quality/similarity with rater count and protocol specified) — a level of methodological transparency the authors note is largely absent from prior Nepali TTS/voice-cloning work.
Field Significance
moderate — this paper extends a well-established voice-cloning architecture to a previously unaddressed low-resource language with a properly disclosed quantitative evaluation, providing a genuine first data point and reusable dataset for Nepali speech synthesis research, though it does not introduce new architectural or training techniques applicable beyond this specific language-adaptation context.
Claims
- supports: A speaker-encoder-based few-shot voice-cloning architecture (independently trained speaker encoder plus a text-conditioned synthesizer and neural vocoder) can be adapted to a previously unaddressed low-resource language using a modestly sized, purpose-built corpus, producing recognizable and speaker-consistent synthetic speech without training a new architecture from scratch.
Evidence: Fine-tuning the SV2TTS-derived pipeline on 235 hours of Nepali speaker-encoder data (833 speakers) and an 8.67-hour, 6,046-pair synthesizer corpus achieves a mean speaker-embedding cosine similarity of 0.904 across ten held-out test speakers and MOS quality/similarity ratings of 3.92/3.87 from 55 human raters. (§6.1, §6.3, Table 4, Figure 7)
- complicates: Speaker-encoder-based voice cloning systems trained on modestly sized, low-resource-language corpora degrade sharply under noisy input audio or atypical vocal/linguistic variation, sometimes producing an entirely different or unnatural voice rather than a merely degraded version of the target speaker.
Evidence: The authors report inconsistent pronunciation clarity and note that unclear audio, noisy environments, or unusual linguistic variation can cause distorted timbre or unnaturally high pitch in the output, attributed to the dataset’s smaller scale and lower diversity relative to state-of-the-art systems trained on 9,000+ speakers. (§7, Limitations)
- supports: Reporting quantitative perceptual evaluation (MOS quality and similarity, with rater count and protocol disclosed) for a low-resource-language voice cloning system meaningfully differentiates it from prior work in that language, even when architectural novelty is minimal.
Evidence: The authors position their disclosed MOS quality score (3.924) against a prior Nepali voice-cloning study that reported higher values without disclosed evaluation methodology, and note that two other existing Nepali TTS systems report no perceptual evaluation scores at all. (§6.4, Comparative Discussion)
- complicates: A pretrained, off-the-shelf neural vocoder fine-tuned on a small target-language corpus can reach acceptable perceptual quality without training the vocoder from scratch, but this substitution leaves clear headroom relative to more advanced vocoder architectures under greater data or compute budgets.
Evidence: Due to computational constraints, the authors fine-tune a pretrained WaveRNN vocoder rather than training from scratch, reaching MOS quality 3.92, while explicitly noting that HiFi-GAN or Speaker-Conditional WaveRNN would likely offer superior generalization to unseen speakers given sufficient data and compute. (§5.3, §7)
Limitations and Open Questions
The system's failure mode under noisy or atypical input is not a graceful quality degradation: the authors report it can generate a "completely different or unnatural voice" with distorted timbre or unnaturally high pitch, rather than a lower-fidelity version of the intended target speaker, which is a materially different reliability profile than typical TTS degradation.
The authors attribute remaining quality gaps primarily to dataset scale and diversity (fewer than 1,000 speakers for the encoder versus 9,000+ in state-of-the-art systems; roughly 10 hours of transcribed synthesizer data, capped by the time cost of manual audio-transcript alignment) and to constrained compute (training on free or low-resource cloud platforms, which limited model optimization and scaling, and which drove the choice of a fine-tuned WaveRNN vocoder over more data-hungry alternatives like HiFi-GAN). The paper explicitly frames itself as a first step and foundation for Nepali voice cloning rather than a system matching high-resource-language state of the art.
Wiki Connections
- Zero-Shot TTS — adapts the SV2TTS speaker-encoder-conditioned few-shot voice-cloning pattern (independently trained speaker verification network plus a conditioned synthesizer) to synthesize speech for speakers not seen during synthesizer training.
- Speaker Adaptation — conditions Tacotron2 mel-spectrogram generation on speaker embeddings fused with text embeddings, adapting output voice identity per speaker without per-speaker model fine-tuning.
- Tacotron — the synthesizer follows the Tacotron architecture family (specifically Tacotron2), generating mel-spectrograms conditioned on text and, in this system, fused speaker embeddings.