arXiv · 2026 · Preprint

Sihan Lv et al. (Zhejiang University) · → Paper · Demo: ? · Code: ?

Turns a pre-trained autoregressive plus flow-matching TTS model into a precise text-based speech editor without any training, by inverting the source speech into the decoder’s latent space and stitching preserved and newly synthesized latent segments under an adaptive guidance signal.

Problem

Text-based speech editing must change selected words while leaving the speaker identity, prosody, and timing of the unedited regions intact. Dedicated editors (EditSpeech, CampNet, FluentSpeech, VoiceCraft, SSR-Speech) need task-specific training data, and a fine-tuned TTS editor still handles out-of-distribution input poorly. Applying an off-the-shelf zero-shot TTS model directly gives fluent output but drifts in prosody and duration across unedited regions, so the editing-quality versus consistency trade-off remains. Two further gaps are named: the RealEdit benchmark used by earlier work is no longer publicly available, and utterance-level metrics do not capture local temporal consistency.

Method

AST targets TTS models in the “AM-FM” pattern (a GPT-style autoregressive model producing semantic tokens, followed by a DiT flow-matching decoder that produces a mel-spectrogram). IndexTTS-2 is the backbone, used without parameter updates. The pipeline has three stages. First, the source mel-spectrogram is inverted into the decoder’s prior space with an inverse Euler ODE solver, which assumes the velocity field is approximately constant over a small step. Second, the source and target transcripts are aligned at the word level with a longest-common-subsequence match, and an ASR-based tool maps each word to a mel-frame interval. Latent Recomposition then builds the initial latent by copying inverted source latents for matched words and drawing Gaussian noise for edited words. The semantic condition and a “fact” mel-spectrogram are assembled with the same segment-wise rule. Third, the flow-matching decoder generates the edited mel with Adaptive Weak Fact Guidance (AWFG). A fact-directed velocity is mixed convexly with the model velocity using a per-frame weight that grows with the squared deviation between the current state and the inverted source flow in matched regions, is capped at lambda (0.4 by default), and is zero in edited regions. Guidance is therefore applied only where the trajectory drifts from the source.

Figure 2: Overview of our training-free speech editing framework AST. The pipeline consists of three stages: (1) Input & Inversion, where the source speech is inverted into the latent space via an Inverse ODE Solver, while the target text is processed by a GPT-style autoregressive model to obtain semantic conditions; (2) Alignment & Recomposition, which aligns source and target transcripts at the word level to stitch inverted latents of unchanged regions with target semantics, constructing a masked source speech as a weak fact; and (3) AWFG Generation, where the Flow Matching decoder synthesizes the edited speech by adaptively modulating the velocity field with fact-directed guidance, ensuring natural transitions and faithful preservation of speaker identity in unedited regions.

Because segments are stitched, a style or emotion condition (for example an emotion tag) injected only when generating the target semantic condition stays confined to the edited span; this localized style editing is shown only in a qualitative case study. The paper also introduces LibriSpeech-Edit (2000 edit pairs, 3.6 hours, average edit distance 2.186, target transcripts generated with Qwen3-8B from LibriSpeech test-clean) and Word-level Dynamic Time Warping (WDTW), which forced-aligns source and edited speech, computes DTW over word-duration sequences, and normalizes by total length.

Key Results

On LibriSpeech-Edit (Table 2), AST reaches WER 2.91%, DNSMOS 3.792, speaker similarity 0.986 (WavLM embeddings), and WDTW 0.2025. The unmodified IndexTTS-2 backbone scores WER 2.43%, DNSMOS 3.841, SpkSim 0.971, and WDTW 0.2768, so AST cuts WDTW by about 27% at a small cost in WER and DNSMOS. Against the fine-tuned Step-Audio-EditX (WER 9.58%, WDTW 0.2038) and the task-trained SSR-Speech (WER 3.57%, SpkSim 0.975, WDTW 0.2296), AST has the best speaker similarity and WDTW and a lower WER. The WDTW margin over Step-Audio-EditX is very small (0.0013). Adding AWFG lowers WER from 6.9% to 2.9% and WDTW from 0.226 to 0.203 relative to latent recomposition alone (Figure 6, values as stated in the text). Performance is flat for lambda between 0.2 and 0.9, with a degradation at 0.1 (Figure 5).

Note

Evaluation uses only objective metrics (WER, DNSMOS, SpkSim, WDTW) on the authors’ own benchmark and metric, with no listening test and no statistical significance reported; the original IndexTTS-2 retains the best WER and DNSMOS.

Novelty Assessment

The contribution is a new inference procedure rather than a new model: latent inversion, long known from image-diffusion editing, is adapted to a flow-matching TTS decoder, and the adaptive guidance weighting is the genuinely new piece, because naive stitching fails at boundaries. The method is tied to AM-FM architectures with an invertible-enough decoder and relies on an external aligner to map words to frames. The benchmark and WDTW metric are authored by the same group as the method, and WDTW favors exactly what the method optimizes, so the headline comparison is not fully independent. Only three baselines are compared, and the localized style-editing claim rests on one spectrogram case study.

Field Significance

Moderate — Demonstrates that a frozen zero-shot TTS model can be steered into a competitive speech editor purely at inference time, which lowers the data cost of editing. The released benchmark and temporal-consistency metric are useful if independently adopted, but both are untested beyond this paper.

Claims

  • supports: Inference-time latent manipulation of a pre-trained generative TTS model can yield precise speech editing without task-specific training.

    Evidence: Inverting the source into the flow-matching decoder’s latent space and stitching preserved and new segments gives the best speaker similarity (0.986) and temporal fidelity (WDTW 0.2025) among compared systems, including a task-trained editor and a fine-tuned TTS editor. (§4.2, Table 2)

  • complicates: Strict preservation of unedited regions trades off against unconstrained synthesis quality and intelligibility.

    Evidence: AST’s WER (2.91%) and DNSMOS (3.792) are slightly worse than the unmodified backbone (2.43%, 3.841) while WDTW improves from 0.2768 to 0.2025. (§4.2, Table 2)

  • supports: Adaptive, deviation-dependent guidance suppresses boundary artifacts better than hard latent stitching or uniform guidance.

    Evidence: Removing AWFG raises WER from about 2.9% to 6.9% and WDTW from 0.203 to 0.226, and results are stable for guidance caps from 0.2 to 0.9. (§4.3, §4.4, Figures 5 and 6)

  • supports: Utterance-level metrics are insufficient to assess editing; a word-level alignment-based metric is needed to measure temporal consistency of unedited regions.

    Evidence: Models with comparable WER and DNSMOS differ markedly in WDTW (0.2025 to 0.2768), a distinction the proposed metric exposes. (§4.1.3, §4.2, Table 2)

Limitations and Open Questions

The method needs a flow-matching decoder over mel-spectrograms with a semantic condition and an ASR-based word aligner, so applicability to other TTS families is untested. Evaluation is objective only and limited to English LibriSpeech test-clean material; the edit targets are LLM-generated and filtered by an unspecified quality criterion. The localized style editing (emotion, dialect, timbre) is supported by one qualitative mel-spectrogram example, with dialect and timbre not shown. The inversion relies on a constant-velocity approximation near boundaries that the authors identify as failing, which AWFG compensates for heuristically. Model size, runtime cost of inversion, and code and demo availability are not reported.

Wiki Connections

  • Flow Matching — operates on a flow-matching mel decoder, using ODE inversion and velocity mixing as the editing mechanism.
  • Zero-Shot TTS — repurposes a zero-shot TTS backbone for editing without fine-tuning.
  • Evaluation Metrics — proposes WDTW for temporal consistency of unedited regions alongside WER, DNSMOS, and speaker similarity.
  • IndexTTS2 — serves as the frozen backbone and as the unmodified baseline.
  • Step-Audio-EditX — fine-tuned TTS editing baseline that AST matches on WDTW with much lower WER.
  • VoiceCraft — earlier codec-LM infilling approach to speech editing that requires task-specific training.
  • CosyVoice — representative of the AM-FM architecture family that the method targets.
  • CosyVoice 2 — another AM-FM family member cited as a foundation for latent manipulation.
  • IndexTTS — earlier member of the IndexTTS series underlying the backbone.
  • Qwen3 — Qwen3-8B generates the target transcripts for LibriSpeech-Edit.