arXiv · 2026 · Preprint

Yifan Duan et al. (Shanghai Jiao Tong University / Shanghai Innovation Institute / Shanghai AI Laboratory / Xiamen University / State Key Laboratory of Complex & Critical Software Environment) · → Paper · Demo: ✓ · Code: ✓

A training-free multi-agent pipeline that turns a long, free-form audio scene description into a mixed scene of dialogue, sound effects, music and songs, by designing a voice for each character, synthesizing speech first and using the measured utterance durations as fixed anchors for timeline planning; it also releases ASG-Bench, a 601-scene benchmark scored by an audio LLM judge.

Problem

Text-to-speech, text-to-audio and text-to-music models each handle one audio type, while real scenes (radio drama, podcasts, narrated documentaries, field reports) mix dialogue with ambience, effects and music under temporal constraints. Unified generators (e.g. UniAudio, Audiobox, UniSonate) and speech-centric long-form models such as Any2Speech (Borderless Long Speech Synthesis) produce a single mixed stream without separable tracks, which the authors argue limits long-form coherence and post-production. Earlier agent systems (AudioGPT, WavJourney, PodAgent, AudioGenie) are described as relying on predefined voices or voice libraries, planning timelines imprecisely, and lacking an autonomous post-production step (§1, §2.2). There was also no benchmark for generating long-form audio from complex scene descriptions with checkable event and temporal requirements (§5.1).

Method

Audio-Oscar is a chain of LLM agents exchanging structured JSON, with off-the-shelf generators called as tools; nothing is trained (§3, §A). In the reported setup the agents run on DeepSeek-V4-Flash, Qwen3.5-122B-A10B or Qwen3.5-397B-A17B at temperature 0.1 (§5.2).

Speech side. An optional Scene Intake Agent normalizes the input script (the “script mode” used in all evaluations). A Speaker Role Profiling Agent extracts every speaking role and writes a stable voice description limited to persistent attributes (gender, age, pitch, speaking rate, articulation clarity, accent), deliberately excluding scene-specific emotion, plus a short reference sentence (§4.1, §A.1). A voice-design model (Qwen3-TTS-12Hz-1.7B-VoiceDesign) renders that description into a reference clip per character, which serves as the timbre anchor for the character’s lines across the scene. A Speech Event Planning Agent then extracts the dialogue and narration verbatim (it may not add or rewrite lines) and assigns each utterance to a role; a speech generation planner picks a TTS model and parameters per utterance from VoxCPM2 and CosyVoice 3 (§4.2, §5.2). The paper does not report how often each TTS model is chosen or what parameters are set.

Timeline. Because speech duration is hard to predict before synthesis, speech is generated before any scheduling, and each utterance’s measured duration is attached to it. A drafting agent places speech (as protected, fixed-duration elements) and adds non-speech events with type, start time, duration and volume, flagging utterances that need word-level forced alignment (Qwen3-ForcedAligner-0.6B) when nearby effects or overlaps demand it. A timestamp refinement agent then fixes pacing, pauses and overlaps, may add or remove non-speech events only if the scene supports them, and must keep every speech segment exactly once (§4.3, §A.1).

Non-speech, critique and mixing. Separate planning agents write prompts for sound effects (MMAudio, Stable Audio Open), music (Stable Audio Open) and songs (ACE-Step v1.5, with optional Demucs vocal separation) (§4.4, §B.1). Only sound effects pass through an audio critic (Qwen3-Omni-30B-A3B-Instruct, 4-bit AWQ) that scores quality, alignment and aesthetics on 0 to 1; clips below 0.7 are repaired by rewriting the prompt, switching model or changing parameters, up to three retries (§4.4, §B.1). A post-production planning agent attaches sparse per-segment edits (gain, fades, ducking, panning, EQ, compression, reverb), the mixer renders the scene, and a final mix review agent listens to the full mix and issues per-segment volume and edit patches before a remix (§4.4, §4.5). Speech itself is not routed through the critic loop.

Key Results

ASG-Bench (601 scenes: 401 with reference audio derived from Omni-Cloze videos, 200 text-only scenes split evenly between Chinese and English). Event coverage, temporal consistency and the 1 to 5 Quality / Alignment / Aesthetic scores are all produced by Qwen3.5-Omni-Plus acting as judge, not by humans or standard speech metrics (§5.2, §C.4). Baselines are Any2Speech (run through its public web interface with a prepended “Generate vivid scenes, not an audiobook.” instruction) and a WavJourney pipeline that the authors upgraded with CosyVoice 3 for TTS, Stable Audio Open for audio and music, DeepSeek-V4-Flash as planner, and Audio-Oscar’s own voice-design clips as its timbre library (§B.2, §B.3).

SubsetSystemEvent %Temporal %QualityAlignmentAesthetic
Reference-audioReference audio93.6295.264.254.544.26
Reference-audioAny2Speech80.3884.544.173.823.78
Reference-audioWavJourney (DeepSeek)87.2692.093.843.903.57
Reference-audioAudio-Oscar (DeepSeek, no thinking)92.1493.364.134.233.96
Reference-audioAudio-Oscar (DeepSeek, thinking)92.0493.784.034.203.88
Reference-audioAudio-Oscar (Qwen-122B)90.2593.564.184.314.01
Reference-audioAudio-Oscar (Qwen-397B)91.3193.724.114.203.95
Text-onlyAny2Speech51.6541.163.532.632.74
Text-onlyWavJourney (DeepSeek)72.3376.133.663.503.28
Text-onlyAudio-Oscar (DeepSeek, no thinking)84.3482.924.064.123.93
Text-onlyAudio-Oscar (DeepSeek, thinking)81.9481.483.493.743.36
Text-onlyAudio-Oscar (Qwen-122B)79.9879.514.044.123.93
Text-onlyAudio-Oscar (Qwen-397B)83.8081.793.954.113.86

(§5.3, Table 2)

  • Every Audio-Oscar configuration exceeds both baselines on Event, Temporal, Alignment and Aesthetic in both subsets. On judge-rated Quality the picture is mixed: in the reference-audio subset Any2Speech (4.17) scores above three of the four Audio-Oscar settings (4.13, 4.03, 4.11) and below only the Qwen-122B setting (4.18); in the text-only subset the DeepSeek thinking setting (3.49) scores below both WavJourney (3.66) and Any2Speech (3.53).
  • On the reference-audio subset Audio-Oscar’s event and temporal coverage is within roughly 1.5 to 3.5 points of the reference recordings themselves.
  • Enabling DeepSeek’s thinking mode does not help: it is lower on every text-only column and on four of five reference-audio columns, and it is the slowest configuration.

Human listening test. 10 participants rated 20 sampled scenes (10 per subset) on a 1 to 5 scale; Audio-Oscar (DeepSeek, no thinking) is rated above WavJourney and Any2Speech on every dimension in both subsets (reference-audio Overall 4.18 vs 3.70 vs 3.36; text-only Overall 3.94 vs 3.38 vs 2.99) (§5.3, Figure 4). No confidence intervals or significance tests are reported.

Instruction-following text-to-audio (non-speech pathway only, Stable Audio Open as the sole generator; baseline numbers copied from AudioX) (§5.3, Table 1). On T2A-Bench Audio-Oscar has the highest Cnt-acc (22.6 vs 12.4 for AudioX, next best) and Ord-acc (69.8 vs 23.6), but its TS-acc (20.2) is below AudioX (28.2), Stable Audio Open alone (21.8) and MMAudio (21.4). On AudioTime, AudioX is best on all four metrics (Ordering 0.34, Duration 1.30, Frequency 0.74, Timestamp 0.81); Audio-Oscar is second on Duration (1.44) and Frequency (1.08), third on Ordering (0.79, behind AudioX and Make-An-Audio2 at 0.76), and on Timestamp (0.54) is level with AudioGen and AudioLDM-2 and below AudioX and Tango 2 (0.61) and Make-An-Audio2 (0.56). The category subset of T2A-Bench is omitted because the planner parses categories explicitly (§B.4).

Runtime. 156.40 to 329.90 s per sample depending on the LLM, against 95.62 s for WavJourney, on 10 sampled scenes (§5.3, Figure 5).

No speech-specific measurement is reported anywhere: no WER/CER for dialogue intelligibility, no speaker similarity or cross-utterance voice consistency per character, and no evaluation of whether the designed voices match their textual descriptions.

Novelty Assessment

The contribution is system engineering plus a benchmark. Every generator, the critic, the aligner and the planning LLMs are existing models; the novelty is in the workflow. For speech, the two notable design choices are (a) generating a character’s reference voice from a text description instead of drawing from a fixed voice library, and (b) synthesizing all speech before scheduling, so that measured speech durations become hard constraints for the timeline. Neither is isolated by an ablation: the paper does not compare against predicted-duration scheduling, a fixed voice library, or the pipeline without the critic, post-production or final-mix stages, so the contribution of each stage to the Table 2 gains is unknown. The WavJourney comparison is partly controlled (same planning LLM, same voice clips, CosyVoice 3 for its TTS), which narrows but does not remove differences in TTS model choice and non-speech generators. ASG-Bench adds verifiable event and temporal assertions (human-checked after LLM extraction) for multi-type scenes, but its scoring relies on a single audio-LLM judge run once per sample.

Field Significance

Moderate. Audio-Oscar shows that an agent pipeline over off-the-shelf TTS, voice-design and audio generators can produce multi-speaker audio scenes whose judge-rated event and temporal coverage approaches that of real recordings, and that a speech-anchored, multi-stage pipeline outscores both an upgraded WavJourney and a single-model long-form speech system in a small human study. For speech generation the paper is mainly a usage case for description-driven voice design and duration-anchored scheduling; it does not measure the speech itself. ASG-Bench provides an evaluation target for scene-level generation that includes spoken lines as checkable events.

Claims

  • supports: Decomposing a complex scene description into per-character voice design, utterance-level speech synthesis and a timeline anchored on measured speech durations can yield scene audio whose event and temporal coverage approaches that of real recordings.

    Evidence: On the ASG-Bench reference-audio subset Audio-Oscar reaches 90.25 to 92.14% event coverage and 93.36 to 93.78% temporal consistency across four LLM settings, against 93.62% and 95.26% for the reference audio; all scores are from an audio-LLM judge, and no ablation isolates the duration-anchoring step. (§4.2, §4.3, §5.3, Table 2)

  • complicates: Single-model long-form speech generators extended toward acoustic scenes cover far fewer of the requested non-speech events and temporal relations than an orchestrated multi-model pipeline when the scene is specified only by text.

    Evidence: On the text-only subset Any2Speech reaches 51.65% event coverage and 41.16% temporal consistency versus 84.34% and 82.92% for Audio-Oscar (DeepSeek, no thinking); Any2Speech was run through its web interface with an added instruction, so its inputs differ slightly. (§5.3, Table 2, §B.2)

  • complicates: Audio-LLM judge scores of overall audio quality do not necessarily rank systems the way human listeners do in scene-level speech generation.

    Evidence: On the reference-audio subset the judge rates Any2Speech’s quality (4.17) above WavJourney (3.84) and above the Audio-Oscar setting used in the listening test (4.13), while human raters score Any2Speech’s quality lowest of the three (3.52 vs 3.62 vs 3.96); the human study covers only 10 samples per subset, so the sample sets differ. (§5.3, Table 2, Figure 4)

  • complicates: Turning on LLM reasoning in the planning agents of a speech-scene pipeline raises cost without improving output and can degrade judged quality.

    Evidence: With DeepSeek-V4-Flash thinking enabled, text-only Quality drops from 4.06 to 3.49 and event coverage from 84.34% to 81.94%, while average generation time rises from 179.90 s to 329.90 s per sample. (§5.3, Table 2, Figure 5)

  • complicates: Explicit planning improves the count and order of generated events, but precise timestamp placement stays bounded by the underlying generator.

    Evidence: With Stable Audio Open as its only generator, Audio-Oscar raises T2A-Bench Ord-acc to 69.8 (Stable Audio Open alone 6.0) but its TS-acc is 20.2, below Stable Audio Open alone (21.8) and AudioX (28.2); the authors attribute the timestamp gap to the generator. (§5.3, Table 1)

Limitations and Open Questions

Speech quality inside the scenes is not measured

All ASG-Bench automatic scores come from one audio-LLM judge (Qwen3.5-Omni-Plus, one pass per sample), and the human study is 20 samples rated by 10 listeners without significance testing. No intelligibility, speaker-similarity, per-character voice-consistency or voice-description adherence metric is reported, so the paper’s speech-related design choices (voice design, TTS model selection, duration anchoring) are not evaluated on their own terms.

  • The authors note that designed voices may fall short of recorded human voices in authenticity, naturalness and subtle emotional expression; the role profiles deliberately omit scene-level emotion, and no stage injects per-line emotion or performance control into TTS (§Limitations, §A.1).
  • For very long requests the planner may produce one long segment that exceeds the stable range of the underlying generator, and it may assign transient events (door slams, impacts) durations too short to render (§Limitations).
  • No ablation separates the contributions of voice design, duration anchoring, timestamp refinement, critic-guided repair, post-production planning and final mix review.
  • The reference-audio subset’s descriptions were generated by Qwen3.5-Omni-Plus, the same model family used as judge, and its events and assertions were extracted by an LLM before human filtering, which may favour content that this judge detects easily (§C.2.2, §C.4).
  • The pipeline is 1.6 to 3.5 times slower than the adapted WavJourney (Figure 5).

Wiki Connections

  • Evaluation Metrics — Contributes ASG-Bench, which scores scene audio (including spoken lines) by audio-LLM checks of human-verified event and temporal assertions plus judge-rated quality, alignment and aesthetics.
  • Subjective Evaluation — Runs a small human MOS study (10 raters, 20 scenes) whose ranking of the two baselines on quality differs from the audio-LLM judge’s ranking.
  • Borderless Long Speech Synthesis — Any2Speech is the single-model long-form speech baseline, which trails the agent pipeline on event and temporal coverage, sharply so on text-only scenes.
  • Qwen3-TTS — Its VoiceDesign checkpoint renders each character’s textual voice description into the reference clip used for that role.
  • VoxCPM2 — One of the two TTS models the speech generation planner can select per utterance.
  • CosyVoice 3 — The other selectable TTS model, also used as the TTS inside the upgraded WavJourney baseline.
  • VoxCPM — Cited alongside VoxCPM2 as the basis of the selected tokenizer-free TTS model.
  • Qwen3.5-Omni — Serves as the audio-LLM judge for all ASG-Bench automatic scores and as the generator of the reference-audio subset’s descriptions.
  • Qwen3-Omni — A quantized Qwen3-Omni-30B-A3B-Instruct is the audio critic in the sound-effect repair loop.
  • PodAgent — Prior multi-agent podcast generator that the paper positions against for relying on predefined voices and lacking post-production.