arXiv · 2025 · Preprint
Cheng et al. (Fish Audio; Carnegie Mellon University) · → Paper · Demo: ? · Code: ?
A fully automated multimodal pipeline (MIKU-PAL) for extracting emotionally annotated speech from unlabeled video, achieving inter-rater consistency far exceeding human annotators, together with MIKU-EmoBench, a 131.2-hour dataset covering 26 fine-grained emotion categories.
Problem
Existing emotional speech datasets are small (typically under 300 hours), annotated manually at high cost and low inter-rater consistency, and limited to 5-8 coarse emotion categories derived from Ekman’s basic emotion framework. This restricts emotional TTS to a narrow and psychologically dated taxonomy, while manual annotation cannot scale to the data volumes that modern speech language models require.
Method
MIKU-PAL is a three-stage modular pipeline applied to raw video data. In the audio preprocessing stage, MDX-Net separates vocals from background music, WebRTC VAD filters non-speech segments, and Whisper-large v3 produces transcriptions with timestamps. In the vision preprocessing stage, S3FD or DSFD perform face detection, and TalkNet (which achieves 92% accuracy on the AVA Active Speaker Detection challenge) identifies the active speaker to attribute facial features to the correct utterance. In the emotion analysis stage, Gemini 2.0 Flash receives the aligned video clip and transcription under a structured zero-shot prompt that specifies the task, a textual description of 26 emotion categories derived from Cowen and Keltner’s psychological research, and an output format requiring per-modality reasoning before a final label. The system produces mixed-emotion annotations with intensity scores for each category, enabling continuous-space emotion modelling.
The emotion taxonomy expands the conventional 6-8 category Ekman framework to 26 psychologically validated categories arranged as a continuous gradient space. The pipeline runs at approximately 12x real-time speed on 8× RTX 4090 GPUs, yields a 42% retention rate on randomly collected YouTube video, and costs roughly $0.50 per hour of annotated audio.
MIKU-EmoBench was collected using this pipeline from YouTube, resulting in 131.2 hours, 65,970 utterances (mean duration 7.16 seconds), annotated with all 26 emotion categories plus intensity scores and textual rationales. Downstream validation fine-tunes Fish-Speech with emotion special tokens on MIKU-EmoBench and competing datasets, evaluating on the MELD test set.
Key Results
MIKU-PAL achieves Fleiss kappa of 0.93 on IEMOCAP and 0.95 on MELD across five independent annotation runs, compared to 0.40 and 0.43 for human annotators (Table 1). Accuracy on MELD is 68.5%, consistent with typical human annotator performance, and rises to approximately 75% when categories lacking psychological basis (‘frustration’, ‘neutral’) are excluded. Human rationality annotations rated 83% of MIKU-PAL’s labels as reasonable across 1,000 balanced samples.
For downstream emotional TTS, Fish-Speech fine-tuned on MIKU-EmoBench achieves MOS 4.12 and emotion similarity 0.92 on the MELD test set, compared to MOS 3.91 / emotion similarity 0.88 for the base model and MOS 4.00-4.09 for models fine-tuned on IEMOCAP, MELD, and MSP-Podcast (Table 3). WER remains stable at 2.4%. An ablation on the MELD validation set shows a 25.6% accuracy gain from including the vision preprocessing stage.
Novelty Assessment
The contribution is primarily a dataset and an annotation methodology rather than a new TTS architecture. MIKU-PAL assembles existing components (MDX-Net, Whisper, TalkNet, Gemini 2.0 Flash) into a novel pipeline, with the key innovation being the zero-shot MLLM prompting strategy that incorporates psychologically validated emotion taxonomy descriptions for structured multimodal annotation. The expansion to 26 emotion categories and the demonstration that automated consistency (kappa 0.93+) can substantially exceed human annotators (kappa ~0.40) are the genuine advances. The downstream TTS improvement (+0.08 MOS, +0.04 emotion similarity) is real but modest, and the evaluation uses a single base model (Fish-Speech) with a simple special-token fine-tuning approach.
Field Significance
Moderate — MIKU-PAL and MIKU-EmoBench address a practical bottleneck in emotional TTS: the scarcity of consistently annotated, fine-grained emotional speech data. The pipeline demonstrates that multimodal MLLM-based annotation can exceed human inter-rater agreement at a fraction of the cost, and the 26-category taxonomy provides a more psychologically grounded alternative to the dominant 6-category Ekman framework. The contribution is primarily infrastructural; its value depends on whether the dataset and annotation schema gain adoption in downstream emotional TTS work.
Claims
-
supports: Automated multimodal annotation can match human accuracy while substantially outperforming human annotation in inter-rater consistency for emotional speech labeling.
Evidence: MIKU-PAL achieves Fleiss kappa of 0.93-0.95 on IEMOCAP and MELD, compared to 0.40-0.43 for human annotators, with accuracy of 68.5% on MELD at approximately 10/hour for manual labeling. (§2.3, Table 1)
-
supports: Expanding emotional speech training data beyond conventional 6-category taxonomies to psychologically grounded fine-grained categories improves downstream TTS emotion expressiveness.
Evidence: Fish-Speech fine-tuned on MIKU-EmoBench (26 categories) achieves MOS 4.12 and emotion similarity 0.92, outperforming fine-tuning on IEMOCAP and MELD (MOS 4.01/4.00, emotion similarity 0.89 each). (§3.2, Table 3)
-
complicates: Standard emotional speech benchmark categories introduce annotation ambiguity that inflates apparent system error rates in automatic labeling.
Evidence: MIKU-PAL accuracy on IEMOCAP rises from approximately 58.6% to approximately 75% when ‘frustration’ and ‘neutral’ categories (not recognized as distinct in established psychological classification) are excluded, suggesting benchmark taxonomy is itself a source of apparent error. (§2.3, Figure 4)
-
supports: Visual modality information contributes substantially to speech emotion classification accuracy in multimodal annotation systems.
Evidence: Ablation on MELD shows a 25.6% accuracy improvement when vision preprocessing (face detection and active speaker identification) is included relative to audio-only analysis. (§2.1)
Limitations and Open Questions
Warning
MIKU-PAL’s annotation quality depends on Gemini 2.0 Flash, a proprietary model whose availability, versioning, and cost structure may change; reproducibility over time is uncertain.
The 42% retention rate on raw YouTube video means more than half of collected audio is discarded, limiting efficiency for domain-specific collection. The dataset sources are predominantly English-language content from YouTube, introducing demographic and domain biases. Current SER models are not effective at distinguishing all 26 fine-grained categories, so MIKU-EmoBench’s evaluation potential is limited until recognition models catch up. The downstream TTS validation uses only Fish-Speech with a simple special-token conditioning scheme; generalization to other architectures and emotion conditioning mechanisms is untested.
Wiki Connections
- Emotion Synthesis — MIKU-PAL and MIKU-EmoBench directly address the data scarcity bottleneck for fine-grained emotional TTS, offering a 26-category benchmark intended to replace conventional 6-8 category emotional datasets.
- Evaluation Metrics — MIKU-PAL introduces Fleiss kappa as an annotation consistency measure alongside accuracy, proposing it as a quality criterion for automated emotion labeling systems.
- Subjective Evaluation — The downstream validation uses human-annotated MOS and human rationality ratings to validate both annotation quality and TTS performance improvements.