ACL · 2025 · Conference
Xiao et al. · → Paper · Demo: ✓ · Code: ✓
PodAgent is a fully automated podcast generation framework that orchestrates a Host-Guest-Writer multi-agent LLM pipeline for content creation, an LLM-driven voice-role matching system, and instruction-following TTS for expressive multi-speaker speech synthesis.
Problem
Generating complete, professionally structured podcast episodes automatically remains unsolved. LLMs can produce text but lack long-form structure and expert-perspective diversity. Zero-shot TTS models synthesise high-quality speech but depend entirely on provided scripts. Text-to-audio models prioritise general audio synthesis over coherent spoken dialogue. Combining these tools naively (as in WavJourney) still yields short, shallow scripts with limited conversational depth. Beyond the pipeline challenge, no standardised evaluation framework exists for assessing podcast-like audio generation across content quality, voice appropriateness, and speech expressiveness simultaneously.
Method
PodAgent integrates three subsystems. For content generation, a Host-Guest-Writer multi-agent system replaces turn-by-turn agent interaction with a more efficient parallel approach: the Host agent formulates guest profiles and a structured interview outline; each Guest agent independently responds from its assigned expert perspective; a Writer agent then synthesises the responses into a coherent, non-redundant conversation script. This avoids the context-management complexity of sequential multi-turn dialogue between agents.
For voice selection, a voice pool is pre-built from LibriTTS-R (English, approximately 350,000 segments from over 2,000 speakers, filtered to 222 unique voices) and AISHELL-3 (Mandarin, 85 speakers). Voice characteristics are extracted and deduplicated using the SpeechCraft dataset’s descriptive labels. A Matching agent then pairs voices from this pool with speaker roles and profiles from the content generation stage.
For speech synthesis, PodAgent uses CosyVoice2 as its TTS backbone. An LLM predicts speaking-style descriptions alongside the conversation script; these descriptions are passed as natural-language instructions to CosyVoice2, which generates speech in the selected reference voice while adhering to the specified style. Sound effects and music follow WavJourney’s pipeline with CosyVoice2 replacing the earlier TTS component.
Evaluation relies on 40 topics derived from a Vicuna-80 subset (Generic, Knowledge, Common-sense, Counterfactual categories). Content quality is assessed with reference-free quantitative metrics (Distinct-N, MATTR, Info-Dens, Semantic-Div) and GPT-4-as-a-Judge comparative scoring; voice matching uses a binary subjective rating by 6 participants; speech expressiveness uses preference tests and CMOS with 9 evaluators.
Key Results
Content generation: the Host-Guest-Writer system outperforms direct GPT-4 generation across all evaluation dimensions and all topic categories. LLM-as-a-Judge overall scores range from +1.40 to +1.75 relative to the GPT-4 baseline (on a -3 to 3 scale), with the strongest gains in informativeness (+1.70 to +2.20) and engagingness (+1.25 to +1.45) (Table 1). Only Semantic-Div in the Generic category shows a marginal decline (-0.005).
Voice-role matching achieves an average pass rate of 87.4%, with per-category rates of 90.0% (Common-sense), 86.7% (Counterfactual), 86.1% (Generic), and 86.7% (Knowledge). More than 90% of sessions have at least two of three speakers successfully matched.
Instruction-following TTS: all CMOS scores comparing LLM-guided synthesis against uninstructed synthesis are positive, ranging from 0.2 to 0.9, and evaluator preference consistently favours the LLM-instructed condition across all topic categories (Figure 7).
Ablation results show that 2-guest setups produce the most content-rich scripts; larger groups introduce redundancy. The full multi-agent framework outperforms a single-agent variant given the same instructions (Table 2), and using a structured interview outline improves all quantitative metrics over an unguided version.
Novelty Assessment
The contribution is primarily engineering integration. All three subsystems rely on established components: GPT-4 for LLM reasoning and role-playing, CosyVoice2 for instruction-following zero-shot TTS, and existing text-to-audio models for sound effects. The Host-Guest-Writer design is a specific instantiation of multi-agent LLM collaboration patterns already studied in the NLP literature; applying it to podcast content generation is new in this domain but not architecturally novel. The voice-role matching pipeline is straightforward: profile a speaker pool, then prompt an LLM to match roles. The evaluation framework is a genuine contribution, as no prior benchmark or protocol covers all three axes (dialogue quality, voice appropriateness, expressive speech) of podcast-like audio generation. The paper also provides the clearest comparison to WavJourney, the most relevant prior system, establishing it as an inadequate baseline for content depth.
Field Significance
Moderate — PodAgent demonstrates that orchestrating existing LLM and TTS capabilities in a structured multi-agent pipeline can address the content-depth shortfall that limits prior audio generation systems. Its main value to the field is establishing the task of podcast generation as a testbed with evaluation criteria, rather than introducing new architectural or training techniques. The result that instruction-following TTS improves subjective preference in the podcast context reinforces the utility of that conditioning paradigm beyond controlled TTS benchmarks.
Claims
-
supports: Parallel multi-agent LLM architectures can produce more informative and engaging conversational scripts than single-prompt generation for long-form spoken content.
Evidence: Host-Guest-Writer system outperforms direct GPT-4 baseline on all five quantitative metrics and all five LLM-as-a-Judge dimensions across four topic categories; the single-agent ablation also shows substantially lower scores on all metrics. (§5.1, §5.3, Table 1, Table 2)
-
supports: LLM-predicted speaking style instructions improve speech expressiveness in zero-shot TTS when used for long-form conversational audio.
Evidence: All CMOS scores (0.2-0.9) are positive and preference tests favour LLM-instructed synthesis over uninstructed synthesis across all four topic categories in a 9-evaluator study. (§5.2, Figure 7)
-
complicates: Automatic voice-role matching from a speaker pool cannot guarantee perceptual alignment between voice characteristics and speaker roles in conversational audio generation.
Evidence: 87.4% average pass rate implies roughly 1 in 8 voice assignments is judged inappropriate by participants; the voice pool is also limited to read-speech data (LibriTTS-R), and the paper notes the need for more conversational-style voices. (§5.2, §7)
-
supports: Increasing the number of participants in multi-agent dialogue generation does not monotonically improve content quality, and smaller groups with more focused interactions outperform larger ensembles.
Evidence: Ablation across 1-5 guest configurations shows that 2-guest setups score highest on Distinct-N, MATTR, and Semantic-Div; quality degrades progressively from 3 to 5 guests. (§5.3, Table 2)
Limitations and Open Questions
Warning
The voice pool is constructed entirely from read-speech corpora (LibriTTS-R, AISHELL-3). Conversational podcasts require naturalistic, spontaneous-sounding voices; the paper acknowledges that current voice quality limitations arise directly from this mismatch, and that long-form generation with CosyVoice2 still produces quality inconsistencies.
Evaluation relies exclusively on GPT-4 for LLM tasks and CosyVoice2 for synthesis, so results reflect the capabilities of those specific systems rather than the framework in general. The content evaluation uses LLM-as-a-Judge (GPT-4 evaluating GPT-4 outputs), which introduces well-documented position and self-preference biases, though the paper mitigates this by averaging forward and reverse comparison orders.
Sound effects and music generation are inherited directly from WavJourney and are explicitly identified as an area needing improvement. No human ground-truth podcasts are evaluated against, making it difficult to assess how close the output is to professional production quality.
Wiki Connections
- Instruction-Conditioned TTS — PodAgent uses LLM-predicted speaking style descriptions as natural language instructions to CosyVoice2, directly applying instruction-following TTS to the podcast generation task.
- Zero-Shot TTS — the framework relies on zero-shot voice cloning (CosyVoice2) to render any script segment in a chosen reference speaker’s voice without speaker-specific fine-tuning.
- Evaluation Metrics — the paper introduces evaluation guidelines for podcast-like audio generation covering dialogue content quality, voice-role matching, and speech expressiveness, filling a gap where no standardised benchmark existed.
- Subjective Evaluation — voice-role matching is assessed via binary subjective judgement by 6 participants across 40 podcast segments, and speech expressiveness via preference tests and CMOS with 9 evaluators.