Speech Generation Research Wiki

Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models

3 min read

arXiv · 2025 · Preprint

Goel et al. (NVIDIA) · → Paper · Demo: ✓ · Code: ✓

Audio Flamingo 3 (AF3) is a fully open-source large audio-language model that unifies reasoning and understanding across speech, sound, and music through a novel AF-Whisper encoder, four purpose-built training datasets, and a five-stage curriculum training strategy, achieving state-of-the-art results on over 20 audio understanding and reasoning benchmarks.

Citation Stub

This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.

Context in Speech Generation

AF3 introduces AF-Whisper, a unified audio encoder derived from Whisper large-v3 that is fine-tuned via audio captioning on large-scale caption pairs spanning all three modalities (speech, sound, music), enabling a single encoder to replace the separate per-modality encoders common in prior audio-language models. The system architecture couples AF-Whisper with an audio adaptor, a 7B-parameter Qwen-2.5 LLM backbone, and a streaming decoder-only TTS module that enables voice-to-voice interaction with a time-to-first-token of 0.15 seconds. For the speech generation community, AF3 provides a reference open-source system demonstrating how a streaming TTS component can be integrated into a multimodal audio LM, and its AF-Chat fine-tuning stage offers a model capable of multi-turn, multi-audio dialogue relevant to spoken conversational agent research. The paper also introduces four training datasets (AudioSkills-XL at 8M QA pairs, LongAudio-XL at 1.25M long-audio QA pairs, AF-Think at 250k chain-of-thought prefixed pairs, and AF-Chat at 75k multi-turn instances) that provide curated resources for training audio reasoning and dialogue capabilities.

Wiki Connections

  • spoken-language-model — AF3 exemplifies the encoder-decoder LALM paradigm, extending large language models to perceive and reason over diverse audio inputs including speech, sound, and music.
  • streaming-tts — AF3-Chat integrates a streaming decoder-only TTS module that generates audio tokens autoregressively, demonstrating sub-second time-to-first-token in a multimodal LM context.
  • evaluation-metrics — AF3 reports results on MMAU, VoiceBench, OpenAudioBench, LongAudioBench, and the SEED TTS benchmark, providing a broad cross-benchmark evaluation profile for multimodal audio LMs.
  • speech-to-speech — the AF3-Chat variant supports full voice-to-voice interaction by combining an audio-language understanding frontend with a streaming TTS backend.
  • 2410.19168 (MMAU) — AF3 uses MMAU as a primary audio reasoning benchmark, achieving 72.42 accuracy on MMAU-test and surpassing Qwen2.5-Omni.
  • 2503.20215 (Qwen2.5-Omni) — AF3 compares directly against Qwen2.5-Omni across understanding, chat, voice-text, and TTS benchmarks, outperforming it on most tasks while being fully open-source.
  • 2410.17196 (VoiceBench) — AF3-Chat is evaluated on VoiceBench for spoken QA robustness, performing comparably to Qwen2.5-Omni and Qwen2-Audio Chat.
  • 2502.11946 (Step-Audio) — Step-Audio is listed as a baseline in the multi-modal audio LM comparison, situating AF3 within the broader landscape of speech-capable LMs.
  • 2502.17239 (Baichuan-Audio) — Baichuan-Audio is included as a baseline, providing context for the open vs. partially-open LALM capability comparison in Table 1.
  • 2501.07246 (Audio-CoT) — AF3 builds on CoT-style audio reasoning work from Audio-CoT, adopting a lightweight on-demand thinking mechanism via the AF-Think dataset rather than explicit post-training for chain-of-thought.
  • 2410.21276 (GPT-4o System Card) — GPT-4o audio is used as a closed-source baseline for audio understanding and voice interaction benchmarks.
  • 2412.15115 (Qwen2.5 Technical Report) — AF3 uses the Qwen-2.5-7B decoder as its LLM backbone.
  • 2503.01743 (Phi-4-Mini) — Phi-4-mm is compared on ASR benchmarks where it holds prior SOTA on LibriSpeech and several other ASR test sets.
  • 2106.06909 (GigaSpeech) — GigaSpeech is used as both a training data source for AudioSkills-XL speech QA pairs and as an ASR evaluation benchmark.

Backlinks

Graph View