arXiv · 2025 · Preprint
Zackary Rackauckas et al. (Columbia University) · → Paper · Demo: ✗ · Code: ✗
Runs the first structured comparative evaluation of VITS and Style-BERT-VITS2 JP Extra (SBV2JE) on in-domain, character-specific Japanese speech, measuring naturalness, intelligibility, and speaker consistency after per-character fine-tuning.
Problem
Japanese “character language” speech, the distinct pitch-accent patterns, speaking rates, and vocabulary tied to a specific fictional character or archetype, is difficult to synthesize because Japanese pitch-accent is highly context-sensitive and character-specific prosody varies widely. Open-source models such as VITS and Style-BERT-VITS2 are already widely used by hobbyist and application communities for character and anime voice synthesis, but the paper argues no formal, controlled comparison of their performance on this specific use case existed prior to this work. The authors motivate the study through a deployed application: character-voiced chatbots for anime-themed Japanese language learning, where naturalness and correctness of synthesized speech are functionally important.
Method
The paper does not propose a new architecture; it fine-tunes and evaluates two existing open-source systems on three character-specific voice datasets.
Systems evaluated. VITS is a single-stage VAE-based TTS model: a posterior encoder extracts acoustic features from ground-truth audio, a prior encoder maps phoneme input through normalizing flows into the same latent space, and a HiFiGAN-style decoder reconstructs the waveform, with a flow-based stochastic duration predictor and adversarial (GAN) training for realism. Style-BERT-VITS2 JP Extra (SBV2JE) builds on this lineage (VITS to VITS2 to BERT-VITS2 to Style-BERT-VITS2) and adds Japanese-specific modifications: an 800-hour Japanese-only pretraining corpus, a WavLM-based discriminator in place of the duration discriminator (to stabilize phoneme spacing), a larger GIN (global conditioning) channel dimension (256 to 512) for greater expressive capacity, style vectors produced by fully connected layers, and manual pitch-accent adjustment controls.
Data. Three character voice datasets were recorded by two professional Japanese voice actors reading scripts generated by GPT-4 and proofread by a native speaker: Hikari (~10 min 17 s), Kitsune (~15 min 3 s), and Saya (~15 min 4 s). Kitsune and Saya share a voice actor but are distinguished by intonation and phoneme-duration patterns.
Fine-tuning. Three VITS models were fine-tuned per character from a pretrained multilingual checkpoint (vits-uma-genshin-honkai) for 200 epochs (batch size 16, Adam, learning rate 0.0002 with decay), using Demucs denoising, mono resampling, silence-based segmentation, and Whisper-large transcription for preprocessing. Three SBV2JE models were separately fine-tuned per character (Hikari: 200 epochs at lr 1e-4; Kitsune and Saya: 400 epochs at lr 5e-5, batch size 4), using two preprocessing pipelines: one producing phoneme-aligned BERT embeddings, and one extracting a neutral style vector per utterance via a pretrained WeSpeaker/Pyannote speaker-embedding model. Because each character’s dataset was only 10-15 minutes, all available audio was used for SBV2JE training, with none held out for evaluation.
Key Results
Three evaluation axes were used, all measured on the same three character voices.
Naturalness (MOS). Eleven crowd-sourced native Japanese raters scored 60 audios each on a 1-5 Likert scale for naturalness, phoneme duration, and pitch-accent correctness. SBV2JE achieved an overall MOS of 4.37 ± 0.74 versus 4.38 ± 0.77 for ground-truth recordings, with a paired two-tailed t-test on aggregate ratings giving p ≈ 0.91 (no significant difference). Per character, SBV2JE underperformed ground truth for Hikari (4.47 vs. 4.76), slightly exceeded it for Kitsune (4.40 vs. 4.14), and matched it almost exactly for Saya (4.24 vs. 4.24) (§V.A, Table I).
Preference (CMOS). Five native Japanese raters compared VITS against SBV2JE on 10 audios per speaker (150 ratings total) on a -3 to 3 scale. SBV2JE was preferred overall (CMOS -0.06), and for every individual character (Hikari -0.12, Saya -0.06, Kitsune -0.03) (§V.B, Table II).
Intelligibility (WER). Using FasterWhisper transcription and Sudachi tokenization with Levenshtein-based alignment to ground-truth text, SBV2JE achieved a lower average WER than VITS (0.04 ± 0.04 vs. 0.05 ± 0.04). Per character, both models were near-identical on Hikari (0.03) and Saya (0.08), while SBV2JE was lower on Kitsune (0.02 vs. 0.03) (§V.D, Table IV).
Speaker consistency. Intra-speaker cosine similarity of Resemblyzer embeddings was high and nearly identical for both models (VITS 96.43% vs. SBV2JE 96.24% overall; per-speaker range 0.96-0.97), with VITS marginally ahead (§V.C, Table III).
Novelty Assessment
The contribution is empirical, not architectural: both VITS and SBV2JE are used as published, fine-tuned per character with existing tooling (vits-fast-finetuning, the official SBV2JE repository). The paper’s own framing is explicit that “our study does not propose novel architectures” and instead “fills an important empirical gap” by contrasting model behavior under controlled, in-domain, character-specific conditions. The value is in the evaluation design (three parallel per-character fine-tunes, four complementary metrics, both automatic and human-rated) applied to a narrow but practically relevant niche (character-driven Japanese TTS for language-learning applications) that had not previously been formally benchmarked.
Field Significance
low — This is a small-scale, narrowly scoped comparative evaluation: two existing open-source systems, three custom character datasets of 10-15 minutes each, and small rater pools (11 for MOS, 5 for CMOS). It provides a useful empirical data point for a specific applied niche (anime/character-voiced Japanese TTS for language learning) but does not introduce new methodology, a new benchmark others can reuse, or evidence intended to generalize beyond this narrow setting.
Claims
- supports: A TTS system fine-tuned from a pretrained checkpoint on a very small (10-15 minute) character-specific voice dataset can reach naturalness ratings statistically indistinguishable from ground-truth human recordings of the same character.
Evidence: SBV2JE overall MOS was 4.37 ± 0.74 versus 4.38 ± 0.77 for ground truth, with a paired t-test giving p ≈ 0.91 across three fine-tuned character voices (§V.A, Table I).
- supports: Replacing a duration discriminator with a self-supervised-representation-based (WavLM) discriminator in an adversarially trained VAE-flow TTS system can improve intelligibility.
Evidence: SBV2JE (which removes the duration discriminator and adds a WavLM-based discriminator) achieved a lower average WER than VITS (0.04 ± 0.04 vs. 0.05 ± 0.04) across the same three fine-tuned characters (§V.D, Table IV; §VI).
- complicates: Automatic intra-speaker consistency metrics based on speaker-embedding cosine similarity may not distinguish between TTS architectures even when naturalness and intelligibility metrics diverge.
Evidence: VITS and SBV2JE produced nearly identical intra-speaker similarity scores (96.43% vs. 96.24% overall, per-speaker range 0.96-0.97), showing no meaningful separation despite SBV2JE’s added discriminator and style-conditioning changes (§V.C, Table III).
- complicates: Comparative naturalness evaluations of fine-tuned character-voice TTS systems can be constrained by evaluating on the same small dataset used for training, which limits how far the naturalness parity result can be generalized.
Evidence: Because each character’s dataset totaled only 10-15 minutes, SBV2JE’s fine-tuning used all available audio, leaving no held-out set; the paper explicitly names small per-character datasets and limited raters as factors that “restrict generalization” (§IV; §VI Discussion).
Limitations and Open Questions
No held-out evaluation split: because each character dataset was only 10-15 minutes, all available audio was used to fine-tune SBV2JE, with none reserved for evaluation. The "ground truth" audio compared against in the MOS study is therefore drawn from the same recordings used in training, which limits how strongly the naturalness-parity result can be read as evidence of generalization to unseen character speech.
Beyond this, the study is limited to three characters voiced by two actors, small rater pools (11 for MOS, 5 for CMOS), and a single language (Japanese) and system pairing (VITS vs. SBV2JE); no comparison is made against more recent codec-based or diffusion/flow-matching TTS architectures. The authors also note that SBV2JE’s higher computational cost (larger GIN channels, WavLM-based discriminator) may constrain deployment on resource-constrained devices, and that stylistic/emotion control capabilities of SBV2JE were left untested in this study, since fine-tuning used a neutral style vector rather than varied emotional conditioning.
Wiki Connections
- Evaluation Metrics — contributes a multi-axis comparative evaluation design (MOS, CMOS, WER, and speaker-embedding consistency) applied consistently across two TTS systems and three character voices.
- Subjective Evaluation — runs genuine human listening tests, an 11-rater MOS study against ground truth and a 5-rater CMOS preference study between VITS and SBV2JE, both on native Japanese speakers.
- Speaker Adaptation — fine-tunes both TTS systems per character from pretrained checkpoints using only 10-15 minutes of character-specific data, a low-resource adaptation setting.
- Self-Supervised Speech — evaluates a system (SBV2JE) whose architecture incorporates a WavLM-based discriminator as a core training component, in place of the original duration discriminator.