arXiv · 2025 · Preprint
Qian et al. (Queen Mary University of London) · → Paper · Demo: ✓ · Code: ?
An end-to-end pipeline that takes comic page images as input, detects characters and extracts dialogue via vision models, attributes speech to speakers and infers emotion categories via GPT-4, then synthesizes character-specific emotion-conditioned audio using a reference-conditioned TTS model.
Problem
Generating expressive, character-specific voiceovers for comics requires solving several interdependent sub-problems: detecting characters and their speech bubbles, identifying who speaks each line, inferring the appropriate emotional tone, and synthesizing speech that reflects both the speaker’s voice and that emotion. Prior work has addressed these components in isolation, and no integrated system existed to take raw comic page images and produce complete audio output. Emotion annotation resources for comics, such as KangaiSet, suffer from severe class imbalance, and speaker attribution is especially difficult when dialogue text is spatially separated from the speaking character.
Method
The pipeline has three stages. The first performs image-level analysis using a Faster R-CNN-based scene graph generation model trained on Manga109 bounding-box annotations to detect frames, text regions, and character instances. Dialogue text is extracted with manga-ocr, a Japanese-comics-optimised OCR system. Character identity is determined by a per-comic ResNet-50 classifier fine-tuned on 40 reference crops per character, combined with negative examples from other comics to form a (k+1)-way classification problem. A binary emotion intensity classifier, also a fine-tuned ResNet-50, reformulates the seven-class KangaiSet emotion taxonomy into a single strong/neutral decision to address the severe imbalance (neutral: 35%, disgust: 0.47%); the raw logit serves as a continuous emotion strength score.
The second stage uses GPT-4 as a dialogue attribution and emotion inference engine. For each comic page, the LLM receives the structured visual layout (frame order, detected characters, text segments), the character identity predictions, emotion intensity scores, and a rolling page-level plot summary maintained across pages. GPT-4 predicts the speaker identity and a fine-grained five-way emotion label (neutral, surprise, anger, happiness, sadness) for each dialogue segment, updating the plot summary after each page.
The final stage synthesises speech with a reference-conditioned TTS model (architecture not described in the paper), selecting a distinct voice profile for each character and conditioning on the predicted emotion label to produce expressive output.
Key Results
Evaluated on 20 comic titles from KangaiSet with dialogue annotations from Manga109Dialogue, using the first 10 images per title. Speaker attribution accuracy for the full pipeline (Setting C: predicted character identity and predicted emotion intensity) is 64.8%, compared to 75.7% for the best Manga109Speaker deep-learning baseline. On hard cases where the speaker is not co-located with the dialogue text, accuracy falls to 20.5%, compared to 79.2% on easy cases. The LLM-based method outperforms the zero-shot multimodal baseline (51.8% total) on both easy and hard settings.
Emotion classification (5-way, Setting C) achieves a micro-average F1 of 42.9%. Performance varies widely by category: happiness reaches 52.5% F1, while surprise achieves only 24.2% (high recall, very low precision), with the LLM frequently misclassifying interrogative sentences as surprise. The joint accuracy for both correct speaker and emotion is 20.4%, reflecting compounded error across pipeline stages. Character identification averages 62.9% top-1 accuracy across 20 titles with the 40-reference-image constraint. No standard TTS quality metrics (MOS, WER, UTMOS) are reported.
Novelty Assessment
The contribution is engineering integration: Faster R-CNN for scene graph generation, ResNet-50 classifiers for character recognition and emotion intensity, GPT-4 for multimodal dialogue attribution, and reference-conditioned TTS are all established techniques applied to comic voiceover automation. The two incremental methodological choices are the binary reformulation of emotion intensity classification (to mitigate class imbalance before LLM fine-classification) and the rolling plot summary mechanism maintained across pages in the LLM prompt. Neither is architecturally novel. The TTS component is used as a black box with no described architecture or training details, making the speech synthesis contribution effectively absent. The evaluation does not include any perceptual assessment of speech quality, so the system’s audio output cannot be compared to other TTS systems.
Field Significance
Low — This paper demonstrates the feasibility of fully automated comic voiceover by chaining existing vision, language, and TTS components. It does not advance TTS methodology, propose new evaluation metrics for speech quality, or demonstrate competitive performance on standard speech benchmarks. Its value to the field lies in identifying compounding-error dynamics and data imbalance challenges in multi-stage speech generation pipelines rather than in any core contribution to synthesis techniques.
Claims
-
complicates: End-to-end speech generation pipelines that chain multiple vision and language components suffer from compounding errors that substantially limit overall system accuracy.
Evidence: Speaker attribution accuracy in Setting C is 64.8% and emotion classification micro-F1 is 42.9%, but joint speaker-and-emotion accuracy is only 20.4%, illustrating how independent module errors multiply at the pipeline level. (§3.5)
-
complicates: LLM-based dialogue attribution for TTS struggles in spatially ambiguous layouts where dialogue text is not co-located with the speaking character.
Evidence: Setting C achieves 79.2% speaker accuracy on easy cases but only 20.5% on hard cases (spatially separated dialogue), showing that the LLM falls back to proximity heuristics that fail when layout breaks co-location assumptions. (§3.3, Table 1)
-
complicates: Imbalanced emotion datasets make fine-grained emotion conditioning for TTS unreliable, particularly for the neutral category which dominates real-world speech.
Evidence: KangaiSet’s extreme class imbalance (neutral: 35%, disgust: 0.47%) required binary reformulation of 7-class emotion classification; the neutral class still achieves only 34.6% recall in 5-way LLM-based emotion prediction. (§2.4, §3.4, Table 3)
-
complicates: Standard accuracy metrics for emotion classification may underestimate the perceptual quality of emotion-conditioned speech when ground-truth emotion labels are inherently ambiguous.
Evidence: Manual inspection of Setting C mispredictions reveals that many misclassified emotions still produce perceptually appropriate voice outputs, and the authors attribute this to inherent visual and contextual ambiguity in comic-style emotional expression. (§3.4)
Limitations and Open Questions
Warning
No perceptual evaluation of synthesised speech is conducted. The pipeline’s audio output quality, naturalness, and emotional appropriateness are not assessed with listening tests or any objective metric (MOS, UTMOS, WER). The speech synthesis module cannot be independently compared to other TTS systems.
The TTS component’s architecture and training setup are not described, making the synthesis stage unreproducible. The pipeline is evaluated exclusively on Japanese manga (Manga109/KangaiSet), leaving generalisation to Western comics, webtoons, or other visual styles untested. Character identification at 62.9% accuracy with only 40 reference images per character introduces substantial upstream error that propagates through speaker attribution. Emotion classification accuracy remains low even with LLM-based reasoning, and the 20.4% joint accuracy suggests significant headroom before deployment quality is reached. The evaluation covers only the first 10 pages per comic, which may not capture the full distribution of visual complexity encountered in complete volumes.
Wiki Connections
- Emotion Synthesis — the pipeline applies emotion conditioning to character-specific TTS, identifying imbalanced emotion datasets and label ambiguity as key bottlenecks for automated comic voiceover.
- Zero-Shot TTS — character voice profiles are applied at inference via a reference-conditioned TTS model, enabling synthesis for unseen comic characters without speaker-specific training.
- Speaker Adaptation — per-character voice profiles constructed from small reference image sets are used to maintain consistent speaker identity across a full comic volume.
- Evaluation Metrics — speaker attribution accuracy and 5-way emotion classification F1 serve as task-specific evaluation proxies; no standard TTS quality metrics are reported, leaving speech output quality unquantified.