arXiv · 2025 · Preprint
Zhu et al. (China Telecom) · → Paper · Demo: ? · Code: ✓
M3PDB introduces a 400k-hour multimodal prompt database covering 18 languages and 15k+ speakers, together with a latency-aware two-stage selection strategy, to address the quality mismatch between clean training data and real-world degraded prompts in zero-shot speech generation.
Problem
Zero-shot TTS systems are trained on clean, studio-quality recordings but encounter noisy, incomplete, or out-of-domain audio at inference time. This quality mismatch degrades synthesis quality in three practically common scenarios: low-quality or partial speech prompts (e.g., streaming input with noise), absence of speech prompts (only a facial image or text description of desired voice), and unseen languages that a model cannot generalise to when used directly as prompts. Existing speech datasets are predominantly single-modality (text and audio only) and annotated with a small number of labels, leaving no adequate resource for principled prompt selection across these conditions.
Method
M3PDB is built from four public sources — Emilia, VGGSound, VoxCeleb, and VoxPopuli — supplemented by synthetic speech generated with CosyVoice for speakers with limited samples. The dataset totals approximately 400k hours across 18 languages with around 15k speakers. About 25% of the data is trimodal (video, text, and audio); the remainder is bimodal (text and audio). Ten annotation labels cover age, gender, emotion, language, accent, speaking rate, and speaker identity.
Annotation uses a multi-modal multi-agent framework: a GPT-4o central controller with Retrieval-Augmented Generation (RAG) selects from a registry of specialised agent models (DeepFace for visual age/gender, Emotion2Vec for emotion, Whisper for language and content) and computes weighted aggregations over their outputs. Hierarchical annotation layers convert discrete labels into natural language descriptions and pre-extract acoustic embeddings (emotion vectors, speaker embeddings) to accelerate downstream retrieval.
For unseen languages, the framework identifies the closest language in M3PDB via linguistic family-tree distance, generates candidate cross-lingual synthesis, and filters on language identification probability (LID > 95%) plus downstream objective quality thresholds before committing a sample as a valid candidate prompt.
The latency-aware prompt selection strategy operates in two stages. In an offline registration phase, coarse filtering based on time-invariant attributes (age, gender, text description similarity, facial embedding similarity) narrows the full database to a smaller candidate set. In the online phase, time-varying characteristics — speaking rate, pitch, speaker identity, and emotion — are evaluated in series using a cascade of similarity modules with configurable cascade depth (Top_i thresholds), allowing the selection to be interrupted at any point to trade latency for precision.
Key Results
On age and gender annotation accuracy, the multi-agent system reduces age mean absolute error from 12.54 (SpeechCraft) to 6.22 years, and raises gender accuracy from 80.69% to 91.22% on a diverse VoxCeleb2/VGGFace2 validation set (Table 2). For unseen language synthesis (Italian via XTTS-v2), cross-lingual prompts selected by M3PDB yield CER of 2.78%, compared to 4.32% for native Italian prompts and 5.21% for French prompts selected at random — suggesting that cross-lingual proxy selection can outperform naive use of native-language audio (Table 3).
For visual-prompt-based selection, retrieved M3PDB audio achieves comparable speaker similarity to Imaginary Voice face-to-speech synthesis (15.14% vs 16.18% SS) while substantially improving audio quality (UTMOSv2: 2.69 vs 2.10), highlighting the gap between generative face-to-speech quality and clean audio drawn from a curated database (Table 4).
Online selection on 20 LRS3 speakers under noisy input conditions achieves SS of 44.88% and ES of 72.94%, compared to 39.53% SS and 65.26% ES when using degraded input directly as the prompt, while also improving CER from 4.21% to 1.17% (Table 5).
Note
All evaluations use automatic metrics (SS via cosine speaker embeddings, ES via Emotion2Vec, UTMOSv2 as an automatic quality predictor). No subjective listening tests are reported. Comparisons across experiments use different test sets, so direct cross-table numerical comparison is not meaningful.
Novelty Assessment
The primary contribution is the dataset itself — a resource that is substantially larger and more richly annotated than existing alternatives, and the first to include visual modality alongside speech and text at scale. The multi-agent annotation pipeline is practically valuable but assembles established components (GPT-4o for orchestration, off-the-shelf audio/visual classifiers, RAG lookup). The unseen-language proxy annotation idea is the most conceptually original element, though it is demonstrated only in a single-language ablation (Italian). The latency-aware selection strategy is an engineering design, applying series-cascade retrieval to meet real-time constraints; no novel retrieval technique is introduced.
Field Significance
Moderate — M3PDB addresses a real and underserved gap: while model architectures for zero-shot TTS have matured substantially, the prompt selection problem under degraded or cross-modal input conditions has received little systematic attention. The dataset’s scale and multimodal coverage position it as a practical resource for downstream work on robust prompt retrieval. However, the accompanying selection strategy relies entirely on automatic evaluation, and the paper’s scope remains exploratory — it stops short of a standardised benchmark for prompt selection quality.
Claims
- Quality mismatch between clean training corpora and real-world inference prompts is a meaningful source of degradation in zero-shot speech generation systems. (§1)
- Cross-lingual proxy prompts selected by linguistic proximity can produce lower synthesis error than native-language prompts of lower audio quality. (§3.2, Table 3)
- Multi-modal annotation that fuses visual and acoustic signals yields substantially higher age and gender labelling accuracy than audio-only or text-only annotation strategies. (§5.1, Table 2)
- A curated, high-quality audio database outperforms generative face-to-speech synthesis as a source of speaker prompts when audio quality is the binding constraint. (§5.3, Table 4)
- Cascaded, interruptible similarity retrieval is a viable approach to prompt selection under strict latency constraints without dedicated hardware. (§4.2, §5.4, Table 5)
Limitations and Open Questions
Warning
No subjective listening tests are reported. All quality metrics (UTMOSv2, automatic speaker similarity, Emotion2Vec emotion similarity) are automatic predictors. The relationship between these metrics and perceptual quality in the prompt selection context is not validated.
The evaluation spans only two downstream tasks (TTS and speech translation) and a narrow set of conditions. The claim that M3PDB generalises to other speech generation tasks (interactive systems, voice conversion) is stated as a future direction but not evidenced. The database construction pipeline is described as time-consuming, which limits reproducibility for groups without significant compute. Coverage of 18 languages is broader than most existing resources but still leaves many of the world’s languages out. The unseen-language proxy mechanism is validated only on Italian; its reliability for more distant language pairs is unknown.
Wiki Connections
Core concepts: zero-shot-tts | multilingual-tts | evaluation-metrics | speaker-adaptation | subjective-evaluation | prosody-control
In-corpus references: 2501.15907 (Emilia dataset) | 2409.18512 (EmoPro prompt selection strategy)