Interspeech · 2026 · Conference
Kirill Borodin et al. (MTUCI, Moscow, Russia) · → Paper · Demo: ? · Code: ✓
Builds an open, fully automatic Russian speech annotation pipeline (semantic segmentation, quality and single-speaker filtering, five-way ASR fusion, punctuation, lexical stress, IPA phonemes) and uses it to release a roughly 5,078-hour corpus, validated by training VITS and a denoiser from scratch on equal 25-hour budgets per corpus.
Problem
Public Russian speech corpora are either audiobook or dictated recordings with rigid, read-style prosody (M-AILABS, RUSLAN, RuLS), manually transcribed far-field data (GOLOS), or large spontaneous collections whose subtitle or ASR transcripts are noisy and lack prosodic markup (OpenSTT, SOVA, DeepSpeech). Almost none carry lexical stress, phonemes, or timestamps, which matters for Russian because of mobile stress, vowel reduction, and the е/ё ambiguity, all of which affect pronunciation and intonation in synthesis. Pipelines built for high-resource languages (WenetSpeech4TTS, LibriTTS-R, TTSOps) do not transfer directly because they omit stress or rely on G2P-inferred stress that fails on Russian homographs. The paper asks whether a scalable, automated, prosody-aware annotation pipeline can produce Russian data that trains better generative models than existing corpora under matched budgets.
Method
Balalaika is a modular, language-specific chain of mostly off-the-shelf components. Raw audio is segmented with SmartTurnV2, a semantic voice activity detector (author term: semantic VAD) that splits at complete utterances instead of fixed windows; segments are kept only if speech occupies at least 70% of the span and no internal silence exceeds 1 s, then grouped into 5 to 15 s chunks (Algorithm 1). Filtering removes clips under 3 s, clips with crest factor above 10 (impulsive energy or clipping), clips with NISQA-S predicted MOS below 4.2, and any clip in which pyannote diarization detects overlap or more than one speaker.
Transcription produces five hypotheses per clip (GigaAM-CTC-v3 with greedy CTC decoding, the same model with an external n-gram LM that also yields word timestamps, GigaAM-RNNT-v3, Vosk, and T-one) and combines them with ROVER consensus voting. The ROVER transcript is the primary text, while the CTC+LM hypothesis and its word-level timestamps are kept as an optional sidecar. The text is then enriched in sequence: RuPunctBig restores punctuation, RuAccent places lexical stress and restores ё, and a small G2P model trained by the authors (a transformer encoder-decoder with 3+3 layers, d_model 128, trained on Wiktextract IPA pronunciations) converts stressed text into IPA while copying punctuation and stress marks through. The G2P model is the only component the authors train themselves; every other stage uses external weights.
The resulting corpus pools processed portions of Yandex Music podcasts, OpenSTT, ESpeech, GOLOS, DeepSpeech, Biggest Russian Books, and other sources (per-source hours in Table 8, several still listed as “TBA”), for a reported total of about 5,078 hours. Audio from sources that cannot be redistributed (Yandex Music and ESpeech podcasts) is released as annotations plus download-and-match scripts.
Validation is by downstream training rather than by any new model. A single VITS model (Adam, lr 1e-4, batch 32, 100k steps, final checkpoint) is trained from scratch on a 25-hour subset of each of 12 corpora, and a SEMamba denoiser (50k steps, MUSAN noise and RIR augmentation) is trained on 25-hour subsets in the same way. Human evaluation uses at least seven native Russian raters per clip from a pool of 68, median-aggregated per clip, on two 0 to 5 rubrics: standard quality MOS and a new intonation-focused IntMOS that rates stress and pause correctness and how conversational versus dictation-like the speech sounds (Appendix C). A manual Text Match Rate (TMR) measures transcript fidelity for dataset comparison.
Key Results
Dataset quality (Table 1). On 200 sampled clips per corpus, Balalaika audio receives the highest human MOS (4.601 ± 0.09, versus 4.049 for RUSLAN and 3.967 for M-AILABS) and the highest NISQA sub-scores and UTMOS (3.019). Its TMR (0.767 ± 0.074) ties M-AILABS (0.767 ± 0.077) and is within rounding of RUSLAN (0.766); confidence intervals for TMR overlap across nearly all corpora, so TMR does not separate the datasets. The NISQA comparison is partly circular because Balalaika itself was filtered with a NISQA-S threshold of 4.2.
TTS (Table 2). VITS trained on Balalaika obtains the best NISQA sub-scores, TTS MOS (3.5703), UTMOS (2.738), and human MOS (3.618 ± 0.083, versus 3.253 for RUSLAN and 2.962 for M-AILABS). It is second on IntMOS (2.532 ± 0.09), well behind single-speaker RUSLAN (3.182 ± 0.094). On CER it ranks fourth (0.1062), behind RUSLAN (0.0496), M-AILABS (0.0908), and RuLS (0.1003); the paper describes this as “on par with the strongest baselines”. GOLOS, OpenSTT, and all three SOVA subsets produce near-unusable VITS models at this budget, with CER between 0.8714 and 1.0. All TTS test texts are drawn from Balalaika’s own held-out split, which may favour the Balalaika-trained model (acknowledged in §4.4).
Annotation ablation (Table 3). With the audio fixed at NISQA MOS > 4.2, plain transcripts give MOS 3.41, IntMOS 2.305, CER 0.1347. Adding stress alone raises MOS to 3.522 and IntMOS to 2.48 (CER 0.1291); punctuation alone gives MOS 3.44, IntMOS 2.448, and a larger CER drop to 0.1123. Both together give the best values on every column (MOS 3.618, IntMOS 2.532, CER 0.1062). Relaxing the quality filter with both annotations degrades results monotonically: threshold 3.5 gives MOS 3.256 and CER 0.171, threshold 3 gives MOS 2.446 and CER 0.2704. Confidence intervals for adjacent rows (for example stress-only versus both, 3.522 ± 0.082 versus 3.618 ± 0.083) overlap, so single-layer differences are suggestive rather than separated.
Denoising (Table 4). SEMamba trained on Balalaika is best on seven of eight metrics (CSIG 3.856, CBAK 3.165, COVL 3.340, PESQ 2.723, ViSQOL 4.036, STOI 0.931, SI-SDR 8.809); RUSLAN is best on UTMOS (2.621 versus 2.614). Margins are small (PESQ 2.723 versus 2.721 for M-AILABS), and the benchmark includes Balalaika’s own held-out clips. An appendix restoration experiment trains SEMamba on the full corpus and reports the highest human MOS among public restoration systems on a SOVA YouTube subset (3.2625 versus 3.0 for MP-SENet, Table 7), while its UTMOS (2.4152) is below the original SEMamba (2.6667) and its accent rate (0.955) is not the best (Table 7). All pipeline stages run faster than real time (Table 6).
Novelty Assessment
The contribution is a dataset and an assembly of existing tools. Semantic VAD, NISQA filtering, pyannote diarization, ROVER fusion of existing Russian ASR models, RuPunct, and RuAccent are all prior components; the only trained component is a small G2P model. What is new for Russian is the combination: an automated pipeline that outputs stress-marked, punctuated, phonemized, and optionally timestamped text over a multi-source, partly spontaneous corpus at about 5k hours, with release of the pipeline and the annotations. The IntMOS rubric is a modest evaluation addition that targets the prosodic effect of the annotation layers. The equal-budget training protocol is a reasonable way to compare data quality, but it tests 25-hour subsets, not the value of the full 5k-hour corpus for large models.
Field Significance
moderate — provides a large, openly annotated Russian TTS corpus with stress, punctuation, and IPA layers, and a reusable pipeline for building more, in a language where such resources were scarce. The controlled VITS ablation adds a small, language-specific data point that explicit stress and punctuation in the input text, together with stricter perceptual-quality filtering, improve rated naturalness of a model trained from scratch.
Claims
- supports: Stricter perceptual-quality filtering of found speech data improves both the naturalness and the intelligibility of TTS models trained on it under a fixed data budget.
Evidence: With stress and punctuation fixed, raising the NISQA MOS threshold from 3 to 3.5 to 4.2 raises VITS human MOS from 2.446 to 3.256 to 3.618 and lowers CER from 0.2704 to 0.171 to 0.1062 on 25-hour training sets. (§4.3, Table 3)
- supports: For languages with mobile lexical stress, adding explicit stress marks and restored punctuation to TTS training transcripts improves perceived intonational naturalness, with the two layers acting complementarily.
Evidence: On identical Russian audio, VITS IntMOS rises from 2.305 (plain) to 2.48 (stress), 2.448 (punctuation), and 2.532 (both), and CER falls from 0.1347 to 0.1062 with both layers, although adjacent rows have overlapping confidence intervals. (§4.3, Table 3)
- complicates: A curated multi-speaker corpus that wins on overall quality ratings can still lose to single-speaker read speech on prosodic naturalness and intelligibility for a small TTS model.
Evidence: VITS trained on Balalaika leads on human MOS (3.618) but trails single-speaker RUSLAN on IntMOS (2.532 versus 3.182) and CER (0.1062 versus 0.0496), and is also behind M-AILABS and RuLS on CER. (§4.3, Table 2)
- complicates: Found or crowd-sourced speech corpora built for recognition, used without perceptual-quality filtering, can be close to unusable for training TTS from scratch at small data budgets.
Evidence: VITS models trained on 25-hour subsets of GOLOS-C, GOLOS-F, OpenSTT, and the three SOVA subsets reach CER between 0.8714 and 1.0 and human MOS of 1.354 or below, against 0.1062 and 3.618 for the filtered Balalaika subset. (§4.3, Table 2)
Limitations and Open Questions
The evaluation favours the proposed corpus in ways that are not fully controlled: TTS test texts come from Balalaika's own held-out split, the denoising benchmark includes Balalaika held-out clips, and the dataset-level NISQA comparison uses the same predictor family (NISQA-S, threshold 4.2) that was used to filter Balalaika. The human MOS gains are the most independent evidence, but the strongest baseline on prosody (RUSLAN) still wins on IntMOS and CER.
- All models are trained on 25-hour subsets for a fixed number of steps and not to convergence (§4.4), so the experiments say nothing about the benefit of the full 5,078-hour corpus or about modern large-scale zero-shot TTS architectures.
- Only one TTS architecture (VITS) is tested, and the paper does not state whether the VITS models were multi-speaker or speaker-conditioned, which matters when comparing multi-speaker corpora against single-speaker RUSLAN.
- The pipeline is Russian-specific (ASR, punctuation, stress, G2P); the appendix on porting (F.2) is a discussion, not an experiment. No accuracy of the stress, punctuation, or G2P layers on Balalaika is reported; G2P accuracy is “not reported”.
- Corpus composition is not final: several per-source hours in Table 8 are “TBA”, and the paper describes itself as work in progress. The §4.1 text refers to Table 8 and to IntMOS for the dataset comparison, but the dataset comparison is Table 1 and has no IntMOS column (checked against table-1.png).
- TMR differences across corpora are within confidence intervals, so it does not support the claim that ROVER fusion yields better transcripts than manually or script-transcribed corpora; a direct WER comparison of the ROVER output against single ASR models is not reported.
Wiki Connections
- GAN Vocoder — trains VITS (a VAE with adversarial waveform decoding) from scratch on each corpus as the sole TTS backbone for comparing data quality and annotation layers.
- Subjective Evaluation — collects MOS and a new intonation-focused IntMOS from at least seven native Russian raters per clip (68 raters total), plus a manual text-match rate for dataset fidelity.
- Combining MLM and Cross-Modal Contrastive Learning for Prosody-Aware TTS — a later paper by the same authors that trains and evaluates its prosody encoder and diffusion TTS systems on the Yandex Podcasts split of the Balalaika corpus.
- Common Voice — the Russian Mozilla Common Voice subset (MCV) is one of the 11 baseline corpora compared on dataset quality, VITS training, and denoiser training.
- Emilia — cited as a general-purpose automated speech-generation data pipeline for high-resource languages, the kind of framework Balalaika adapts to Russian with added stress and phoneme layers.
- Scaling Under-Resourced TTS for Thai — cited as related data-optimized work on building TTS data for an under-resourced language.