arXiv · 2026 · Preprint
Wenfu Wang et al. (Tencent AI Lab) · → Paper · Demo: ? · Code: ✓
Covo-Audio is a 7B-parameter end-to-end large audio language model (LALM) that consumes continuous audio and generates audio output within a single unified architecture, achieving state-of-the-art or competitive results among comparably sized models across speech-text modeling, spoken dialogue, speech understanding, audio understanding, and full-duplex voice interaction.
Problem
Cascaded speech systems (ASR + LLM + TTS) suffer from information loss and error propagation, and Thinker-Talker architectures (Qwen2.5/3-Omni) that insert a textual reasoning step before acoustic token prediction sacrifice end-to-end speech instruction following and complicate full-duplex behavior. Fully end-to-end LALMs that map audio directly to audio (GPT-4o, GLM-4-Voice, Step-Audio 2) avoid these bottlenecks but expose a separate practical problem: building a production dialogue system with a specific, appealing voice requires collecting large volumes of high-quality dialogue data recorded in that voice, since the model’s dialogue intelligence and its speaker identity are learned jointly and become difficult to disentangle after training. Existing full-duplex approaches (Moshi, Voila, OmniFlatten) also generally require word-level text-speech alignment or multi-stage post-training curricula to acquire duplex behavior, adding engineering complexity.
Method
Covo-Audio is built on Qwen2.5-7B-Base, extended with a Whisper-large-v3 audio encoder (50 Hz output) feeding a three-stage downsampling adapter (two linear layers plus a stride-2 convolution per stage) that reduces the frame rate to 6.25 Hz before entering the LLM. For audio generation, a dedicated speech tokenizer built on WavLM-large with a single Vector Quantization layer (16,384-entry codebook, 25 Hz) produces discrete audio tokens; the tokenizer is trained with a combined ASR loss (semantic grounding), TTS reconstruction loss (acoustic fidelity), and pitch loss (prosody preservation). The model natively processes interleaved continuous acoustic features and text as input, and generates a unified sequence of interleaved text and discrete audio tokens as output. A two-stage speech decoder converts discrete tokens back to waveform: a Flow-Matching-based module first produces continuous acoustic latents, which a BigVGAN vocoder renders into 24 kHz audio.
Pretraining proceeds in two stages over 2T cumulative tokens. Stage 1 freezes the audio encoder and LLM and trains only the adapter on a 200,000-hour multilingual ASR corpus for modality bridging. Stage 2 jointly unfreezes the adapter and LLM and trains on a multi-task mixture (ASR, TTS, audio-only modeling, speech continuation, speech-text interleaving, text-only, and full-duplex pretraining) over roughly 8 million hours of audio and a 3T-token text corpus. The central Stage 2 mechanism is a “Hierarchical Tri-modal Speech-Text Interleaving” strategy that fuses continuous features, discrete tokens, and text using both phrase-level (Sequential Interleaving) and, unlike prior word/character-level interleaving schemes (e.g. GLM-4-Voice), sentence-level (Parallel Integration) patterns intended to preserve long-form semantic coherence alongside fine-grained acoustic alignment.
Post-training builds Covo-Audio-Chat via a mixture of General Intelligence, Spoken Dialogue, Speech Understanding, Speech Generation, and Audio Understanding tasks. Text instruction data (10M samples) is converted to parallel text-to-audio (T2A), audio-to-text (A2T), and audio-to-audio (A2A) training pairs via TTS synthesis across thousands of sampled speaker prompts, with KL-based on-policy distillation from a stronger text LLM used to prevent intelligence degradation. Colloquialism is added by LLM-rewriting formal responses into conversational style before synthesis, and empathy is trained via a 7-category emotion-aware dialogue dataset with emotionally aligned speech.
The paper’s central training-recipe contribution is an “intelligence-speaker decoupling” technique that addresses the cost of collecting dialogue data in a target voice. Thousands of synthetic speakers are randomly assigned during dialogue-data construction so that intellectual quality stays consistent across voices. Because speakers trained purely on TTS-style data transfer poorly to dialogue format, high-quality TTS recordings are reformatted into pseudo-conversations (prompt, speaker, query, response) with the text-response portion of the loss masked out, so only the audio-response loss is backpropagated when using TTS-derived voices.
The full-duplex variant, Covo-Audio-Chat-FD, reformats the audio encoder into chunk-streaming mode with a 1:4 user-to-model chunk-interleaving ratio, and acquires duplex behavior directly during large-scale pretraining (rather than via a separate multi-stage post-training curriculum as in OmniFlatten) followed by a single lightweight supervised fine-tuning step. At inference, the model emits THINK tokens while listening, a SHIFT token to begin speaking, interleaved text-then-speech-token chunks while speaking, and a BREAK token to yield the floor on interruption or turn completion.
Key Results
On pretraining evaluations (Table 3), Covo-Audio-Base reaches performance parity with specialized audio-to-audio (A2A) models on story-continuation (A2A-tSC 83.3) while substantially outperforming them on grammatical (sBLIMP 61.6) and lexical (sWUGGY 74.90) benchmarks, and reaches 99.4 on T2T-tSC, ahead of GLM-4-Voice-Base, SpiritLM, and SIMS despite those baselines sometimes using additional reinforcement learning. On foundational ASR/TTS (Table 4), Covo-Audio-Base reports 1.96% (Aishell-1) and 1.96%/4.55% (LibriSpeech clean/other) recognition error, and 2.44/1.73 on Seed-TTS Test-en/Test-zh, ahead of GLM-4-Voice-Base on most cells.
On spoken dialogue (URO-Bench, Table 5-equivalent), Covo-Audio-Chat leads on four Chinese reasoning tasks (SQuAD 77.34, OpenbookQA 83.60, APE 68.42, MLC 80.69) and two dialogue tasks (AlpacaEval 90.02, Wildchat 90.41), and leads the English track on Gsm8kEval (85.68), against strong baselines including Qwen3-Omni, MiMo-Audio, and Step-Audio 2. On VCB Bench, Covo-Audio-Chat leads instruction following (TIF 93.07, TIF-En 89.94) and all three robustness sub-tasks (Speaker Variation 88.94, Environmental Noise 87.13, Content Variation 90.37), while trailing Qwen3-Omni on general knowledge. On the VStyle empathy benchmark judged by an LLM (Gemini 2.5 Pro), Covo-Audio-Chat attains the top Mandarin scores in anger (4.89), sadness (4.93), and anxiety (5.00). Covo-Audio-Chat-TTS, the variant using speakers transferred directly from TTS data via the decoupling technique, achieves comparable bilingual dialogue scores to the fully-trained Covo-Audio-Chat (e.g. AlpacaEval 89.34 vs. 90.02 Chinese, 84.49 vs. 85.73 English).
Covo-Audio-Chat-FD substantially outperforms Moshi and Freeze-Omni on both URO-Bench dialogue quality and full-duplex interaction metrics, reaching 99.7% turn-taking, 97.6% pause-handling, 96.81% interruption-handling, and 93.89% backchanneling success (Table 9), while retaining dialogue performance close to the half-duplex Covo-Audio-Chat. On audio/speech understanding, the SFT-based model reports an average ASR WER of 4.71 across WenetSpeech, LibriSpeech, Fleurs, and Aishell-1 (Table 10), the best en-zh CoVoST2 translation score among compared systems, and top overall scores on AIR-Bench paralinguistic tasks and MMSU (66.64% average, ahead of Audio Flamingo 3 at 61.40%), while trailing the 30B-parameter Step-Audio 2 on MMAU.
Novelty Assessment
The architectural components (Whisper encoder, WavLM-based tokenizer, Qwen2.5-7B backbone, flow-matching decoder, BigVGAN vocoder, interleaved-stream text-speech decoding) are individually established techniques, so this is primarily an engineering-integration contribution at the system level. Within that integration, two elements are more genuinely novel: the hierarchical (phrase- and sentence-level, rather than word/character-level) tri-modal interleaving pretraining scheme, and the intelligence-speaker decoupling technique that uses masked text-response loss on reformatted TTS data to transfer voice characteristics into a dialogue model without full dialogue-data collection. The latter is a practically motivated training-recipe innovation addressing a real deployment cost problem rather than a new model architecture. The full-duplex training approach (folding duplex behavior into large-scale pretraining in a single step rather than a multi-stage post-training curriculum as in OmniFlatten) is a genuine simplification, though its benefit is demonstrated only against two open-source baselines (Moshi, Freeze-Omni), not the newer parallel-stream full-duplex systems discussed in the paper’s own related work. The comprehensive empirical validation, spanning speech-text modeling, dialogue, empathy, full-duplex behavior, speech understanding, and audio understanding, is broad but relies entirely on existing third-party benchmarks rather than new evaluation contributions.
Field Significance
High significance
High — Covo-Audio is a comprehensively evaluated 7B-scale end-to-end LALM that reports competitive or state-of-the-art results against many of the strongest recently published open and closed systems (Qwen3-Omni, MiMo-Audio, Step-Audio 2, Fun-Audio-Chat, GPT-4o Audio) across an unusually broad task span. Its most transferable contribution is the intelligence-speaker decoupling technique, which offers a concrete, validated recipe for the widely shared industry problem of customizing a dialogue model’s voice without collecting new dialogue-format training data in that voice.
The paper documents both the technique’s benefit (comparable dialogue quality when transferring TTS-only speakers) and its own evaluation’s limits (the LLM-judged empathy scores may not track genuine perceived vocal empathy, by the authors’ own admission), which gives the contribution more evidentiary weight than a purely self-reported leaderboard win. As a technical report rather than a peer-reviewed publication, its comparisons are self-selected and not independently verified.
Claims
- supports: Decoupling dialogue intelligence from speaker identity by reformatting existing TTS recordings into pseudo-conversations with masked text-response loss enables voice customization for a dialogue model without collecting new dialogue-format training data in that voice.
Evidence: Covo-Audio-Chat-TTS, whose dialogue voice is transferred directly from TTS speaker data via masked-loss pseudo-conversation training rather than authentic high-quality dialogue recordings, achieves near-parity bilingual dialogue scores with Covo-Audio-Chat (AlpacaEval 89.34 vs. 90.02 Chinese, 84.49 vs. 85.73 English). (§2.4, §3.2, Table 5)
- supports: Interleaving continuous acoustic features, discrete audio tokens, and text at coarser phrase- and sentence-level granularity during large-scale pretraining, rather than word/character-level alignment, can improve reasoning and grammatical consistency in speech-text foundation models.
Evidence: The hierarchical tri-modal interleaving pretraining yields a foundation model (Covo-Audio-Base) that outperforms character/word-level interleaved and specialized A2A baselines on sBLIMP (61.6 vs. 58.3 for SpiritLM) and sWUGGY (74.90 vs. 69.00 for SpiritLM), and reaches 99.4 T2T-tSC. (§2.2, §3.1, Table 3)
- complicates: Folding full-duplex conversational behavior directly into large-scale pretraining, instead of a dedicated multi-stage post-training curriculum, can preserve most dialogue quality but introduces a distinct failure mode around fine-grained turn boundaries.
Evidence: Covo-Audio-Chat-FD maintains dialogue scores close to the half-duplex Covo-Audio-Chat and reaches 99.7% turn-taking success, but its pause-handling success (97.6%) is lower, traced by the authors to an “early-response” behavior where the model interrupts the user during short intra-turn pauses. (§3.3, Table 9)
- complicates: LLM-as-judge scoring of empathetic speech quality can diverge from human-perceived vocal empathy, since automated judges may weight the semantic content of a response over its acoustic delivery.
Evidence: Covo-Audio-Chat attains the top LLM-judged (Gemini 2.5 Pro) Mandarin empathy scores on the VStyle benchmark, but the authors report that a preliminary subjective listening test found the model still underperforms a leading commercial system (Doubao) on perceived voice empathy, and call for a more equitable evaluation criterion. (§3.2, Table 7)
- supports: A discrete speech tokenizer built on a self-supervised speech representation and trained with a combined recognition, reconstruction, and prosody objective can support both accurate speech understanding and high-fidelity resynthesis within a single LLM token vocabulary.
Evidence: The speech tokenizer (WavLM-large plus a single VQ layer, 16,384-entry codebook, 25 Hz), trained with joint ASR, TTS-reconstruction, and pitch losses, underlies a model that achieves 1.96% Aishell-1 and 1.96%/4.55% LibriSpeech clean/other recognition error alongside 2.44/1.73 Seed-TTS Test-en/Test-zh generation scores from the same token vocabulary. (§2.1, Table 4)
Limitations and Open Questions
Self-reported LLM-judge evaluation gap
The paper’s own empathy evaluation (VStyle, Table 7) uses an LLM judge (Gemini 2.5 Pro) that the authors themselves note is likely to prioritize semantic content over actual vocal quality; a preliminary subjective listening test reportedly found the model underperforming a leading commercial system on real perceived empathy, but no quantitative subjective results are reported.
Full-duplex evaluation is limited to comparison against two open-source baselines (Moshi, Freeze-Omni) rather than the newer parallel-stream or Thinker-Talker full-duplex systems the paper discusses in related work. The knowledge dimension of VCB Bench (general knowledge, dialogue comprehension) is explicitly reported as weaker than Qwen3-Omni, attributed by the authors to a deliberate prioritization of interaction capabilities. As a technical report, all comparisons are self-run and not independently verified, and the pretraining corpus (8 million hours of audio, 3T tokens) is not publicly named or released in detail, limiting reproducibility.
Wiki Connections
- Spoken Language Model — Covo-Audio is an end-to-end LALM in which an external continuous audio signal is consumed by an LLM backbone (Qwen2.5-7B) adapted for direct audio-to-audio spoken dialogue.
- Speech-to-Speech — Covo-Audio-Chat and its full-duplex variant Covo-Audio-Chat-FD are audio-in, audio-out dialogue systems evaluated end-to-end on URO-Bench and VCB Bench without intermediate cascaded ASR/TTS modules.
- Speaker Adaptation — the intelligence-speaker decoupling technique is a dedicated mechanism for adapting a dialogue model’s voice from limited TTS-only speaker data while preserving learned dialogue intelligence.
- Neural Audio Codec — the paper introduces a custom discrete speech tokenizer (WavLM-large plus a single VQ layer, 16,384-entry codebook, 25 Hz) trained with a combined ASR, reconstruction, and pitch objective.
- Multilingual TTS — the model is trained and evaluated bilingually on Chinese and English, with separate reported speech-generation scores on Seed-TTS Test-en and Test-zh.
- Emotion Synthesis — Covo-Audio-Chat is trained on a 7-category emotion-aware dialogue dataset and evaluated on the VStyle empathy benchmark for generating emotionally appropriate spoken responses.
- Qwen3-Omni Technical Report — used as a leading comparison baseline across URO-Bench, VCB Bench, and MMSU, and its Thinker-Talker architecture is discussed as an alternative design point Covo-Audio’s fully unified interleaved-stream approach avoids.
- Fun-Audio-Chat Technical Report — a contemporaneous unified LALM baseline compared against on URO-Bench and VCB Bench.
- CosyVoice 2 — its flow-matching decoding approach is the basis for Covo-Audio’s own flow-matching-based speech decoder, and CosyVoice2 serves as the dedicated TTS baseline on the Seed-TTS benchmark.
- VStyle — provides the empathy benchmark used to evaluate Covo-Audio-Chat’s emotionally appropriate spoken response generation.
- MTR-DuplexBench — its definitions of full-duplex interaction behaviors (turn-taking, pause handling, interruption, backchanneling) are adopted for evaluating Covo-Audio-Chat-FD.