arXiv · 2025 · Preprint
Tomoki Koriyama (CyberAgent) · → Paper · Demo: ? · Code: ?
An automatic prosodic label annotation model for Japanese that fuses frozen SSL acoustic features with phoneme-BERT linguistic features to predict pitch accents and phrase break indices at the phoneme level, intended as a labeling tool for training prosody-controllable TTS systems.
Problem
TTS systems that offer explicit prosody control require training data annotated with prosodic labels (pitch accent symbols, break indices), but such labels are expensive to produce because only specialized annotators can assign them reliably. Automatic prosody annotation offers a scalable alternative, but prior work treated it as either a purely acoustic problem (using Whisper or SSL encoders) or a purely linguistic one (using BERT-family encoders), without systematically exploiting both sources. Meanwhile, existing systems that recognize phonemes and accent symbols jointly conflate two tasks of different difficulty: phoneme transcription can be crowd-sourced, while prosodic annotation requires expert knowledge. This paper asks whether the combination of acoustic and linguistic foundation models closes the annotation quality gap.
Method
The proposed system takes two inputs for a speech utterance: the raw waveform and a phoneme sequence (assumed known from transcription or a pronunciation dictionary). The waveform is fed into a frozen speech foundation model, either an SSL encoder (HuBERT, wav2vec 2.0, or WavLM) or a Whisper encoder, to obtain frame-level acoustic features. These frame-level features are averaged over phoneme spans using durations from a VAE-based forced aligner, yielding phoneme-level acoustic vectors. Simultaneously, the phoneme sequence is passed through a frozen phoneme-BERT encoder, either PnG BERT (which inputs a concatenation of phonemes and graphemes) or PL-BERT (which inputs phonemes only and uses graphemes as prediction targets), producing phoneme-level linguistic vectors. Both streams apply a learned weighted sum over encoder hidden layers before the vectors are concatenated and passed into a small six-layer CNN annotation model (hidden size 256, kernel size 5). The annotation head is trained with multi-task cross-entropy losses on four Japanese prosodic label types: accent symbols (ACC), high-low pitch patterns (HL), break indices (BI), and pause presence (PAU). Foundation model parameters are frozen throughout; only the layer weights and the annotation CNN are optimized.
For the phoneme-BERT encoders, the paper pre-trains Japanese-specific variants of PnG BERT and PL-BERT on approximately 630 million sentences (Japanese Wikipedia plus CC-100), using subword grapheme segmentation adapted for Japanese morphology.
Key Results
On the core subset of the Corpus of Spontaneous Japanese (CSJ), the best-performing combination (HuBERT-base with Japanese pre-training + PnG BERT) achieves 89.8% accuracy on accent labels (ACC), 93.2% on high-low labels (HL), and 94.3% on break indices (BI), with pause presence accuracy of 98.7% (Table 1, §5.4). These numbers exceed purely acoustic models (HuBERT alone: 89.0% ACC) and purely linguistic models (PnG BERT alone: 82.5% ACC), confirming complementary contributions from each modality.
Comparisons across acoustic models (Table 2, §5.6) show Japanese-trained SSL models slightly outperform English or multilingual counterparts, though the gap is modest (less than 1 percentage point on most labels). Across Whisper sizes (Table 3, §5.6), performance scales mildly with model size; Whisper-large-v3 achieves 89.7% ACC accuracy, comparable to the best HuBERT-base result. Traditional acoustic features (melspectrogram, F0) are substantially weaker: melspectrogram alone achieves 75.2% ACC accuracy and F0 alone achieves 62.9%, versus 89.8% for HuBERT-base.
Novelty Assessment
The contribution is a systematic engineering integration of well-established components: SSL speech encoders and phoneme-BERT linguistic encoders, combined via a simple concatenation with learned weighted-layer selection and a small CNN head. The individual components (HuBERT, wav2vec 2.0, WavLM, Whisper, PnG BERT, PL-BERT) are all existing models; the paper adapts PnG BERT and PL-BERT to Japanese, which requires re-training due to language-specific morphology. The architectural contribution is minimal. The main value is empirical: a clean ablation over five acoustic inputs and four linguistic inputs that establishes SSL + phoneme-BERT as the practical baseline for Japanese prosody annotation. Notably, the paper does not evaluate whether the predicted labels actually improve downstream TTS quality, which limits the scope of the claimed benefit to annotation accuracy rather than synthesis naturalness.
Field Significance
Low — This SSW13 workshop paper provides a useful ablation of acoustic and linguistic model combinations for Japanese prosody annotation, confirming that SSL features outperform hand-crafted acoustics and that adding phoneme-BERT provides further gains. The contribution is narrow in scope (Japanese only, no TTS quality evaluation), relies on standard concatenation of frozen pre-trained models, and does not introduce novel methodology. It can serve as a practical reference for practitioners building prosody-controllable TTS data pipelines for pitch-accent languages.
Claims
-
supports: Combining acoustic foundation model features with phoneme-level linguistic BERT features improves automatic prosody label prediction over either modality alone.
Evidence: On the CSJ corpus, HuBERT-base + PnG BERT achieves 89.8% ACC accuracy versus 89.0% with HuBERT alone and 82.5% with PnG BERT alone; consistent gains appear on HL and BI labels. (§5.4, Table 1)
-
supports: SSL speech features substantially outperform hand-crafted acoustic features for automatic prosody annotation.
Evidence: HuBERT-base achieves 89.8% ACC accuracy versus 75.2% for melspectrogram and 62.9% for F0 alone on CSJ; the gap widens further when comparing macro F1 scores on rare prosodic classes. (§5.4, Table 1)
-
supports: Language-matched pre-training benefits SSL model utility for prosody prediction in a pitch-accent language.
Evidence: Japanese-trained HuBERT-base and wav2vec2.0-base outperform English and multilingual SSL variants across all four prosodic label types on CSJ, though margins are modest (under 1 percentage point on most labels). (§5.6, Table 2)
-
complicates: Prosodic boundary and pause signals cannot be reliably estimated from text alone; explicit acoustic evidence is required.
Evidence: Without acoustic input, pause presence prediction fails to distinguish short pauses; boundary pitch movement symbols in ACC labels show higher error rates than when HuBERT features are included. (§5.5)
Limitations and Open Questions
Warning
The paper does not evaluate whether the predicted prosodic labels improve downstream TTS naturalness or prosody controllability. All results are prosody annotation accuracy numbers on CSJ; the claimed benefit to TTS training remains unvalidated.
The system is evaluated only on Japanese, a pitch-accent language with specific prosodic structure. Applicability to other languages with different prosodic hierarchies (e.g., stress-timed languages with ToBI conventions) is not tested. The annotation model depends on accurate phoneme alignments from a forced aligner; errors in alignment propagate directly into label prediction. Finally, CSJ is a monologue corpus of academic lectures; generalization to conversational or spontaneous speech styles with different prosodic distributions is an open question.
Wiki Connections
- Prosody Control — the paper’s annotation model directly enables explicit prosody conditioning in TTS by providing pitch accent and break index labels for training prosody-controllable systems.
- Self-Supervised Speech — HuBERT, wav2vec 2.0, and WavLM serve as core acoustic feature extractors in the proposed annotation model, and the paper demonstrates their superiority over hand-crafted features for prosody prediction.
- Evaluation Metrics — the paper provides a systematic cross-ablation of acoustic and linguistic model combinations for prosody annotation, offering a practical benchmark for evaluating annotation quality in TTS data pipelines.