Speech Generation Research Wiki

MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning

9 min read

arXiv · 2026 · Preprint

Chunyu Qiang et al. (Kuaishou Technology) · → Paper · Demo: ? · Code: ✗

Presents a unified flow-matching framework for joint audio-video generation that adds zero-shot voice cloning through a dedicated timbre injection mechanism, achieving voice cloning fidelity comparable to specialized TTS systems while simultaneously generating synchronized video.

Problem

Unified audio-video generation models jointly synthesize visual and acoustic streams to avoid the temporal misalignment that cascaded (audio-conditioned-on-pre-generated-video) pipelines suffer from. However, existing unified models such as Ovi and JavisDiT lack fine-grained control over the acoustic modality, particularly speaker identity: they can produce semantically appropriate sounds but cannot clone a specific speaker’s timbre from a reference clip. This limits their use in applications needing consistent character identity or dubbing. Separately, dedicated zero-shot voice cloning has advanced substantially in the TTS literature, but that capability has not been integrated into a joint audio-video synthesis framework. A further practical obstacle is that standard classifier-free guidance (CFG), which typically uses a null or zero-vector unconditional branch, becomes ambiguous once a model’s acoustic conditioning includes a speaker-identity embedding, since a zero vector may already have a specific meaning during training rather than representing “no information.”

Method

MM-Sonate builds on a Multi-Modal Diffusion Transformer (MM-DiT, following SD3) that treats audio and video as coupled streams in a shared latent space, trained with conditional flow matching along an Optimal Transport path. Text, image, and audio conditions are each processed by dedicated (frozen, pretrained) encoders and projected into a shared embedding space, with absent modalities represented by learnable placeholder tokens; this lets a single model support T2VA, TI2VA, TA2VA, and TIA2VA generation scenarios (text/image/audio-to-video-and-audio, in various combinations) within one framework. The audio latent representation extends the authors’ own prior SecoustiCodec framework as a Mel-VAE codec (encoder, decoder, discriminator) operating on 44.1kHz waveforms; video uses a 3D causal temporal encoder from CogVideoX.

Rather than conditioning on coarse semantic text descriptions alone, MM-Sonate uses a unified instruction-phoneme input: a video instruction description, an audio instruction description (specifying attributes such as speaker gender, age, emotion, style, and accent, or, for music, singer timbre, genre, and instrumentation), and a phoneme sequence produced by a grapheme-to-phoneme model. For zero-shot voice cloning, a pretrained speaker encoder extracts a fixed-length speaker embedding from reference audio and injects it into the phoneme representation via element-wise addition (timbre injection); for multi-speaker dialogue, a learnable per-speaker ID embedding is added alongside the speaker embedding to stabilize identity control across two simultaneous voices. Training uses a stochastic modality-masking strategy, randomly dropping reference image and/or audio inputs (and, with small probability, all conditioning) so a single training pass covers all four input-modality combinations plus the unconditional branch needed for classifier-free guidance.

The framework is a multimodal flow-matching model enabling joint audio-video generation with fine-grained control. The unified instructions combining video/audio captions with phoneme sequences for precise content alignment. The reference audio for zero-shot voice cloning. The first frame image for visual conditioning. A dedicated mechanism injects speaker embeddings (from reference audio) and ID embeddings (for multi-speaker control) directly into the phoneme sequence via element-wise addition. These multimodal features are then fused and processed by the MM-DiT backbone to model the joint distribution of audio and video latents.

To address the CFG ambiguity, the paper identifies that a zero-vector negative speaker embedding is not a neutral “no information” signal: during training, a zero vector specifically indicates the no-timbre-cloning (T2VA) task, so using it as the CFG negative condition merely steers generation away from that mode rather than away from low-quality audio. The proposed fix, a noise-based negative speaker embedding, feeds naturally collected white noise through the pretrained speaker encoder to construct the negative condition instead. The training data combines a large proprietary corpus of roughly 100 million aligned video-audio-caption-transcript quadruplets, spanning speech, sound effects, and music, with a purpose-built high-fidelity synthetic timbre dataset: clean acoustic prompts (denoised speech, separated vocals) condition a zero-shot TTS model (DMP-TTS) to synthesize emotionally neutral speech, which is then filtered by WavLM-based speaker verification to discard samples with low identity similarity to the source prompt.

Key Results

On joint audio-video generation (VerseBench, compared against cascaded systems like AudioLDM+TemoTkn and OpenSora+FoleyGen, and unified systems like JavisDiT, SVG, Universe-1, and Ovi), MM-Sonate reports the lowest Fréchet Distance (1.43) and KL divergence (1.16) on audio fidelity, a WER of 0.020 (roughly an order of magnitude better than OpenSora+See&Hear’s 0.337 and JavisDiT’s 0.256), and the highest SyncNet Confidence lip-sync score (6.51 vs. Ovi’s 4.28). Against dedicated zero-shot TTS systems (MaskGCT, E2-TTS, F5-TTS, ZipVoice, M3-TTS, CosyVoice2) while simultaneously generating synchronized video, MM-Sonate reaches SIM-o speaker similarity of 0.604 (EN) and 0.691 (ZH), trailing the best audio-only baseline MaskGCT (0.713 EN) but matching M3-TTS, with WER of 2.065% (EN) that beats CosyVoice2 (2.57%) and approaches ground truth (2.14%). Against text-to-music baselines (DiffRhythm+, ACE-Step) on SongEval, MM-Sonate scores higher on all five metrics (e.g. Musicality 3.01 vs. ACE-Step’s 2.87). Human preference tests report single-speaker voice-cloning pass rates of 91.3% (ZH) and 85.0% (EN), and dialogue (two-speaker) identity-consistency pass rates of 81.3% (ZH) and 75.0% (EN). Ablations show natural-noise negative conditioning reduces WER from 3.17% (worst setting) to 2.06% relative to zero-vector and Gaussian-noise alternatives, at a small SyncNet Confidence cost (6.12 to 5.94); adding timbre injection causes only a minor quality trade-off on non-cloning tasks (GSB preference scores of 0.89 ZH / 0.83 EN favoring “same or better”).

Novelty Assessment

The core generative backbone (MM-DiT with conditional flow matching) and the audio/video codecs are largely inherited from or extensions of the authors’ own prior work (SecoustiCodec) and established external architectures (SD3’s MM-DiT, CogVideoX’s video encoder). The genuinely new contributions are narrower and more specific: the unified instruction-phoneme input format, the timbre-injection mechanism for extending zero-shot voice cloning into a joint generation setting, the diagnosis and fix for zero-vector CFG ambiguity under speaker conditioning, and the synthetic timbre dataset construction pipeline. Each of these is backed by a targeted ablation rather than asserted, which is a strength. The comparisons against specialized TTS and TTM systems are framed generously (the paper repeatedly notes MM-Sonate is solving “a significantly more complex task” by also generating video), which is a fair point but also means the reported voice-cloning numbers should be read as “competitive despite a harder joint task” rather than “state-of-the-art voice cloning.” The model itself (20B parameters, proprietary training data, closed weights) is not independently reproducible.

Field Significance

Tip

High, this is presented as the first unified audio-video generation model with zero-shot voice-cloning capability, addressing a capability gap that prior unified models (Ovi, JavisDiT, Universe-1) explicitly lack. The demonstrated combination, fine-grained lip synchronization plus TTS-competitive voice cloning within one joint model, is a genuinely new point in the design space, though its industrial scale (20B parameters, proprietary data, closed weights) limits independent verification and reuse.

Claims

  • supports: A unified instruction-phoneme conditioning format, combining natural-language style instructions with explicit phoneme sequences, enforces tighter linguistic and temporal alignment than semantic-description-only conditioning in joint audio-video generation.

    Evidence: MM-Sonate reports a WER of 0.020 on the joint audio-video benchmark, roughly an order of magnitude better than description-only unified baselines such as OpenSora+See&Hear (0.337) and JavisDiT (0.256), alongside the highest lip-synchronization score (SyncNet Confidence 6.51) among compared systems. (§6.4.2, Table 1)

  • supports: A dedicated timbre-injection mechanism that adds a speaker embedding directly into phoneme representations can extend zero-shot voice cloning into a joint audio-video generation framework without substantially degrading general (non-cloning) generation quality.

    Evidence: A side-by-side preference test between the full model and a variant trained without timbre injection on standard non-cloning tasks yields GSB scores of 0.89 (ZH) and 0.83 (EN), indicating most samples are rated as same or better despite the added conditioning complexity. (§6.4.1)

  • complicates: A zero-vector convention for the unconditional branch in classifier-free guidance becomes unreliable once the same zero vector is also used during training to signal a specific conditioning mode, since CFG then steers generation away from that mode rather than away from low-quality output.

    Evidence: Replacing the zero-vector negative speaker embedding with naturally collected noise reduces WER from 3.17% (worst setting) to 2.06%, which the authors attribute to the zero vector being interpreted as the trained no-timbre-cloning (T2VA) signal rather than as a generic negative prior. (§6.4.1, Figure 4)

  • complicates: A model that jointly generates video alongside cloned speech trades off some speaker-similarity fidelity relative to specialized, audio-only zero-shot TTS systems.

    Evidence: MM-Sonate’s speaker similarity (SIM-o 0.604 EN / 0.691 ZH) trails the best audio-only TTS baseline, MaskGCT (0.713 EN), even though MM-Sonate simultaneously generates synchronized video content. (§6.4.3, Table 2)

Limitations and Open Questions

Warning

The model is 20 billion parameters, trained on a proprietary 100-million-pair corpus, and released only under controlled API access rather than as open weights; none of the reported results are independently reproducible.

  • The authors state the model is optimized for 3-15 second clips; maintaining semantic and temporal coherence for longer durations is untested and flagged as future work.
  • Zero-shot voice cloning may struggle to decouple background noise from the reference speaker’s timbre when the reference audio itself is low-quality or noisy.
  • Extreme head poses, rapid camera movement, or heavy facial occlusion can still cause lip-sync desynchronization or visual artifacts.
  • The paper discloses safety mitigations (content classifiers, constraints on high-risk public figures, imperceptible watermarking on all generated output) motivated by the deepfake, impersonation, and non-consensual voice-cloning risks the technology introduces, but these mitigations are described rather than independently evaluated.

Wiki Connections

  • Flow Matching — trains a joint audio-video Diffusion Transformer with a conditional flow matching objective along an Optimal Transport path, extending flow matching from single-modality TTS to a coupled audio-video latent space.
  • Zero-Shot TTS — extends zero-shot voice cloning, established in specialized TTS systems, into a unified joint audio-video generation framework via a dedicated timbre-injection mechanism.
  • Voice Conversion — evaluates zero-shot timbre cloning fidelity with dedicated speaker-similarity (SIM-o) and intelligibility (WER) metrics directly against specialized zero-shot voice cloning and TTS systems.
  • Instruction-Conditioned TTS — accepts natural language audio instructions specifying speaker gender, age, emotion, style, and accent as part of its unified instruction-phoneme conditioning format.
  • Subjective Evaluation — reports human preference (GSB) and pass-rate evaluations for speaker identity preservation in both single-speaker and multi-speaker dialogue scenarios.
  • SecoustiCodec — MM-Sonate’s audio latent codec extends this prior framework’s Mel-VAE architecture rather than introducing a new codec design.
  • DMP-TTS — used as the zero-shot TTS model that synthesizes the neutral-text speech samples underlying the paper’s high-fidelity synthetic timbre training dataset.
  • Flow Matching for Generative Modeling — the paper’s conditional flow matching training objective follows this formulation directly.
  • CosyVoice 2 — a direct TTS baseline in the voice-cloning comparison, which MM-Sonate outperforms on WER (2.065% vs. 2.57% EN) while trading off some speaker similarity.
  • F5-TTS — one of the specialized zero-shot TTS baselines compared against in the voice-cloning evaluation (Table 2).
  • MaskGCT — the strongest speaker-similarity baseline in the voice-cloning comparison (SIM-o 0.713 EN), which MM-Sonate approaches but does not exceed.
  • M3-TTS — a same-research-group prior zero-shot TTS system that MM-Sonate matches in English speaker similarity (SIM-o 0.604).
  • Seed-TTS — the paper’s SIM-o speaker-similarity evaluation protocol (WavLM-large speaker verification) follows the protocol established in this paper.

Backlinks

Graph View