arXiv · 2025 · Preprint
Baher Mohammad et al. (MTS AI, ITMO University) · → Paper · Demo: ? · Code: ?
Introduces a hybrid Mamba-cross-attention autoregressive decoder that replaces Transformer self-attention with a state-space backbone for text-conditioned speech editing and zero-shot TTS, matching or beating a Transformer-based baseline at roughly six times lower memory.
Problem
Text-based speech editing and zero-shot TTS both require an autoregressive decoder to align a short phoneme sequence with a much longer sequence of discrete acoustic tokens (roughly 12 phonemes per second versus 200-400 codec tokens per second). Transformer-based decoders such as VoiceCraft handle this via joint self-attention over concatenated text and audio tokens, but this incurs quadratic complexity in sequence length and, per this paper’s account, forces mitigations like discouraging silence tokens and generating multiple candidates per input and keeping the shortest. Flow-matching and diffusion alternatives (e.g., VoiceBox, UniCATS) improve speed but the paper argues they struggle with temporal coherence and fine-grained prosody, particularly on noisy in-the-wild audio, and existing unified editing/TTS systems either lack rigorous human evaluation (SpeechX) or restrict editing to short spans under two seconds with rule-based prosody markers (UniCATS). Separately, State Space Models (SSMs) like Mamba offer linear-time sequence modeling but, prior to this work, had not been paired with a flexible cross-modal conditioning mechanism that avoids requiring equal-length text and audio sequences (the constraint in prior SSM work such as CM-Mamba).
Method
MAVE (Mamba with Cross-Attention for Voice Editing and Synthesis) is an autoregressive decoder that models discrete acoustic tokens with stacked Mamba (selective SSM) blocks and conditions on text through cross-attention layers interleaved after each block, rather than by concatenating text and audio tokens into one sequence. Audio is tokenized with X-Codec, an RVQ-based neural codec producing 8 parallel codebook streams at 50 Hz (codebook size 1024). For speech editing, one or more spans of audio tokens are masked (Poisson-sampled span count, capped at 3, span lengths up to 600 frames) and rearranged to the end of the sequence following the causal-masking strategy from CM3, so the autoregressive model can condition on both preceding and following unmasked context while still training with a strictly causal objective. A codebook delay pattern is applied across the 8 RVQ levels to speed decoding.
Text conditioning is handled by phonemizing the transcript with the Montreal Forced Aligner and encoding the phoneme sequence with a 4-layer Transformer encoder; the resulting embeddings serve as keys and values for cross-attention modules inserted after every Mamba block, with queries drawn from the Mamba decoder’s hidden state. This design is motivated by an argued asymmetry: Mamba’s compressed selective memory is well suited to audio-token dependencies (where exact low-level detail is less important than high-level acoustic continuity) but is argued to degrade fine-grained retention of distant text tokens if text and audio are simply concatenated into one autoregressive stream, motivating explicit, length-agnostic cross-attention instead. Speaker identity is not modeled via an explicit speaker embedding; instead a short reference utterance is encoded with X-Codec and prepended to the input sequence, and the Mamba recurrence carries this acoustic/prosodic/timbral context through generation for zero-shot TTS (the reference can be omitted for editing, since unmasked surrounding audio already anchors speaker identity). Training minimizes a per-codebook weighted cross-entropy loss over the RVQ streams (higher weight on the earlier, more semantically important codebooks), computed over all valid tokens including masked regions. The released configuration is an 830M-parameter model trained on GigaSpeech (9k hours) for 50k steps on 4 A100 GPUs (~4 days) with the ScaledAdam optimizer and Eden scheduler; inference uses nucleus sampling (p=0.8, temperature 1) with a single generation pass and no post-processing.
Key Results
On the RealEdit speech-editing benchmark (real-world LibriTTS, YouTube, and podcast recordings), MAVE improves WER over VoiceCraft (7.5 vs. 8.4 with Whisper-large) and edges out VoiceCraft on MOS naturalness (3.90 vs. 3.77) and intelligibility (4.25 vs. 4.20), within roughly 0.1 MOS of ground-truth recordings. A pairwise listening study (40 samples, 400 judgments) found 57.2% of listeners rated MAVE-edited audio as perceptually equal to the unedited original, with only 18.0% preferring the edit and 24.8% preferring the original. In a separate 3-way pairwise comparison against VoiceCraft and an enhanced FluentSpeech variant (14 samples provided by the VoiceCraft authors), MAVE was preferred over VoiceCraft in 33.6% of naturalness comparisons (vs. 17.5% the other way) and over FluentSpeech in 62.9% of cases.
For zero-shot TTS on a 372-utterance LibriTTS subset, MAVE was not explicitly trained on this task yet outperforms VoiceCraft on WER (7.4 vs. 9.3), WavLM-based speaker similarity (0.57 vs. 0.55), and MOS naturalness (3.48 vs. 3.22) and intelligibility (4.20 vs. 4.01), though the gap to ground truth remains larger than in the editing setting (e.g., 3.48 vs. 3.90 naturalness) and widens further for longer target transcripts. On efficiency, MAVE uses about 6x less peak GPU memory than VoiceCraft with KV-caching (6.2GB vs. 37.9GB average) on RealEdit-length utterances, though VoiceCraft with caching is faster in wall-clock terms on these short utterances (4m33s vs. 5m17s to process the full test set); the paper argues via a complexity analysis that MAVE’s linear scaling in audio length should overtake Transformer decoders as output length grows. An ablation at matched ~830M parameter budget shows the full cross-attentive Mamba design beats both a Transformer encoder-decoder variant (WER 10.8) and a Mamba-only variant that concatenates text and audio tokens (WER 13.0), isolating the cross-attention mechanism as necessary rather than incidental to the result (WER 7.8 for the full model).
Novelty Assessment
The architectural contribution is genuine: this is, by the authors’ account, the first application of a cross-attentive Mamba backbone to text-conditioned speech editing and zero-shot TTS, and the ablation study (Table 5) provides direct evidence that the specific combination (Mamba for audio dependencies, cross-attention for text conditioning) matters, not just that Mamba is used somewhere in the pipeline. That said, the empirical claims rest on a fairly narrow comparison set: the primary quantitative baseline is a single Transformer AR system (VoiceCraft), and the FluentSpeech comparison relies on only 14 externally provided samples rather than an independently reproduced evaluation. Training data (GigaSpeech, English) and evaluation sets (RealEdit, LibriTTS) are also modest in scale relative to the largest contemporary TTS training runs, and the human evaluation panels, while methodologically detailed, are small (10-20 raters per study). The efficiency argument is well-supported for memory but is explicitly not a wall-clock speed win at the utterance lengths tested; the claimed latency advantage is projected via a theoretical complexity analysis rather than measured on long-form audio.
Field Significance
Moderate — MAVE demonstrates that a selective state-space backbone can be paired with cross-attention to handle length-mismatched text-audio conditioning without the equal-length constraint of prior SSM approaches like CM-Mamba, offering a concrete linear-complexity alternative to Transformer decoders for speech editing and zero-shot TTS. Its contribution is architectural rather than an evaluation-scale or dataset advance, and the supporting comparisons are limited to one primary Transformer baseline and one diffusion baseline evaluated on a small released sample set.
Claims
- supports: Combining a state-space (Mamba) backbone for audio-token modeling with cross-attention for text conditioning outperforms both a pure Transformer encoder-decoder and a same-size Mamba decoder that concatenates text and audio tokens, at matched parameter count.
Evidence: At ~830M parameters, the hybrid MAVE design achieves WER 7.8, MCD 4.29, and PESQ 2.08, versus WER 10.8 for the Transformer encoder-decoder variant and WER 13.0 for the Mamba-only (concatenation) variant on a masked-reconstruction ablation task. (§4.4, Table 5)
- complicates: Cross-attention conditioning is necessary rather than incidental for Mamba-based decoders: naively concatenating text and audio tokens into one autoregressive stream degrades linguistic fidelity as the sequence grows, because the compressed SSM state is argued to lose fine-grained detail about distant tokens.
Evidence: The Mamba-only variant, which concatenates phoneme and audio tokens rather than using cross-attention, more than doubles WER relative to the full cross-attentive model (13.0 vs. 7.8) under an otherwise matched training and parameter setup. (§3.2.1, §4.4, Table 5)
- supports: A prepended audio reference clip, processed through the same recurrent decoder used for generation, can substitute for an explicit speaker embedding in zero-shot TTS while remaining competitive on speaker similarity.
Evidence: Without any explicit speaker embedding, MAVE achieves slightly higher WavLM-based cosine speaker similarity than VoiceCraft (0.57 vs. 0.55) on 372 LibriTTS zero-shot TTS samples using only a prepended ~3-second reference clip for speaker conditioning. (§4.2, Table 2)
- complicates: Linear-complexity SSM decoders trade wall-clock inference speed for memory efficiency relative to Transformer decoders with KV-caching, and the memory advantage does not automatically translate into a latency advantage on short utterances.
Evidence: On the RealEdit test set (mean utterance duration 6.21s), MAVE uses roughly 6x less average GPU memory than VoiceCraft with KV-caching (6.2GB vs. 37.9GB) but takes longer in total wall-clock time to process the dataset (5min17s vs. 4min33s), an effect the paper attributes to autoregressive per-step overhead dominating at short output lengths. (§4.3, Table 4)
- supports: AI-edited speech can be made perceptually close to indistinguishable from the unedited original for a majority of listeners, even on real-world, out-of-domain recordings with background noise.
Evidence: In a pairwise side-by-side study on 40 RealEdit samples (400 judgments across 10 raters), 57.2% of comparisons rated the MAVE-edited and original audio as equally natural, versus 24.8% preferring the original and 18.0% preferring the edit. (§4.1)
Limitations and Open Questions
Warning
The efficiency claims are validated only on short utterances (mean 6.21s on RealEdit); the paper’s own complexity analysis argues MAVE’s advantages should grow with output length, but no direct measurement is provided on longer-form audio, so the projected latency crossover point remains untested.
Beyond that, the evaluation compares against a single Transformer AR baseline (VoiceCraft) with quantitative rigor, while the diffusion-based comparison (FluentSpeech) relies on only 14 externally supplied samples rather than a full reproduced evaluation, and the human listening panels (10-20 raters per study) are modest in scale. Training and evaluation are restricted to English (GigaSpeech, LibriTTS, RealEdit); multilingual behavior is untested. The paper notes that its own performance gap to ground truth widens for longer target transcripts, consistent with GigaSpeech’s moderate utterance-length distribution, and flags future work on training on longer audio sequences to better exploit Mamba’s long-range modeling as an open direction.
Wiki Connections
- Autoregressive Codec TTS — MAVE replaces the Transformer decoder typically used in codec-token autoregressive TTS with a Mamba SSM backbone conditioned via cross-attention, offering a linear-complexity alternative within the same generation paradigm.
- Zero-Shot TTS — the system performs zero-shot TTS via in-context conditioning on a prepended reference clip rather than an explicit speaker embedding, without being explicitly trained on the TTS task.
- Neural Audio Codec — the model tokenizes audio with X-Codec (8-stream RVQ at 50Hz) as its underlying acoustic representation and applies a weighted per-codebook training loss reflecting the codec’s semantic-to-acoustic codebook hierarchy.
- Subjective Evaluation — the paper runs four distinct human listening studies (standalone MOS, pairwise edit-vs-original, three-way pairwise vs. baselines, and length-stratified MOS) to substantiate its editing and TTS quality claims.
- VoiceCraft — serves as the paper’s primary quantitative baseline for both speech editing and zero-shot TTS, compared directly on WER, MOS, speaker similarity, and inference memory/latency.
- VALL-E — cited as the foundational neural codec language model paradigm (transcript-conditioned speech continuation via RVQ tokens) that MAVE’s Transformer-based competitors extend and that MAVE itself departs from by replacing self-attention with a Mamba backbone.
- EnCodec — referenced as background for the RVQ-based neural codec paradigm that discretizes speech into the parallel token streams MAVE’s own codec (X-Codec) also produces.
- Whisper — used as the automatic transcription tool (Whisper-large and Whisper-medium.en) to compute WER for both the speech-editing and zero-shot TTS evaluations.