arXiv · 2025 · Preprint
Matiyali et al. (IIT Kanpur) · → Paper · Demo: ? · Code: ?
A non-autoregressive, transformer-based model for dynamic-length text-conditioned speech insertion that uses cross-modal attention to transfer speaker style from the surrounding audio context into a synthesised replacement segment.
Problem
Editing recorded speech is error-prone and time-consuming: correcting even a single word typically requires re-recording an entire segment. Prior text-conditioned speech insertion methods either constrain the inserted segment to a fixed, pre-specified duration or rely on a two-stage pipeline that degrades quality in an early reconstruction step. A system that can synthesise a replacement segment of dynamically inferred length while closely matching the surrounding speaker’s voice, prosody, and recording environment would make transcript-driven audio editing practical.
Method
RephraseTTS operates on mel-spectrogram representations and follows the FastSpeech2 non-autoregressive framework. The input consists of two audio streams — the before (B) and after (A) segments flanking the missing region — concatenated into a single mel-spectrogram context X_in, alongside the complete phoneme sequence for the full transcript, including the insertion region (I). Learnable segment embeddings distinguish B, I, and A positions in both the audio and phoneme streams; sinusoidal positional encodings are also added.
A cross-modal attention module forms the core novel contribution: phoneme encodings (from the phoneme encoder) serve as queries, while the audio encoder’s output provides keys and values. This allows the model to extract speaker timbre, prosody, and recording-environment characteristics from the available speech context and inject them into the phoneme representation — without requiring an explicit phoneme-to-audio alignment. A FastSpeech2 variance adaptor then predicts phoneme-level pitch, energy, and duration; a length regulator upsamples phoneme embeddings frame-by-frame. The decoder produces only the mel-spectrogram frames for the inserted segment (Z_I), which are then spliced back into the audio-encoder output before being decoded into the full output spectrogram. A pretrained HiFi-GAN (Kong et al. 2020) vocoder converts the final mel-spectrogram to audio.
Training proceeds in two phases. Phase one uses an L1 reconstruction loss with additional weight on the inserted segment. Phase two adds local and global LSGAN adversarial losses with feature matching, plus a triplet-based style matching loss. The style matching loss trains a jointly-learned ResNet18-based style extractor to pull synthesised windows toward real windows from the same utterance and push them away from other utterances. The local discriminator operates on short windows sampled only from the inserted segment, encouraging fine-grained naturalness in the region of interest. MFA-derived phoneme alignments provide duration supervision during training; at inference, durations are predicted by the variance adaptor, enabling variable-length output. Training runs for 200k iterations on a single GTX 1080 Ti.
Key Results
In a user study of 15 LibriTTS dev-clean utterances, RephraseTTS achieves an overall MOS of 3.93 against MetaStyleSpeech’s 3.03, approaching the ground-truth MOS of 4.32 (Table 1, §4.2). On the short insertion category, the method matches ground-truth MOS exactly at 4.20. Quality degrades gracefully with insertion length — long insertions score 3.97 MOS, versus 2.80 for MetaStyleSpeech and 4.47 for ground truth.
On objective evaluation using Mel-Cepstral Distortion (MCD, lower is better), RephraseTTS achieves 0.579 on dev-clean and 0.624 on dev-other, substantially outperforming MetaStyleSpeech (0.833, 0.918) and the MetaStyleSpeech-Full oracle variant (0.606, 0.633), which has access to the full ground-truth audio context (Table 2, §4.2). The vocoder upper bound (GT-Mel+Vocoder) sits at 0.373/0.376, indicating meaningful remaining headroom.
Ablation experiments (Table 3, §4.3) show that each loss component contributes on dev-clean, though the picture is less consistent on the harder dev-other subset. Removing cross-modal attention and replacing it with a global speaker encoder worsens MCD on dev-clean, confirming that fine-grained contextual style transfer outperforms speaker-level global conditioning.
Novelty Assessment
The primary novel contribution is the cross-modal attention mechanism that uses phoneme queries to attend over audio-context keys and values — a cleaner way to perform style injection than global speaker embeddings in this setting. The rest of the pipeline is a direct adaptation of FastSpeech2, with GAN discriminators following well-established patterns from the vocoder and TTS literature. The triplet-based style matching loss is a distinctive component but is adapted from metric-learning practice rather than being architecturally original. The comparison set is narrow: only MetaStyleSpeech is used as a competitive TTS-based baseline, and SpeechPainter (the closest fixed-duration inpainting work) is not evaluated quantitatively against the proposed method, since the task formulations differ. Code and demos are not released with the preprint, and the model scale is not reported. Overall this is a technically sound but incrementally novel approach to a niche sub-task.
Field Significance
Moderate — RephraseTTS addresses a practically important but underexplored problem in speech editing: replacing arbitrary-length segments while matching the surrounding context. The cross-modal attention formulation is a sensible architectural choice that advances the sub-area beyond global speaker conditioning, and the multi-component training objective improves output naturalness measurably. The work is constrained to a clean, well-curated benchmark and does not address robustness to noisy or spontaneous speech, which would be required for real-world audio editing tools.
Claims
- Cross-modal attention between phoneme and audio representations is more effective than global speaker embeddings for preserving localised prosodic and acoustic context in speech insertion tasks. (§4.3, Table 3)
- Non-autoregressive TTS architectures with duration prediction can infer variable-length insertion segments without explicit phoneme-level alignment at inference time. (§3.1, §3.2)
- Combining local and global adversarial losses with a triplet-based style matching loss reduces perceptual artefacts in synthesised speech segments that L1 reconstruction loss alone cannot eliminate. (§3.3, §4.3)
- Speech insertion quality degrades more gracefully with insertion length when the model conditions directly on local audio context rather than full-utterance speaker embeddings. (§4.2, Table 1)
Limitations and Open Questions
Warning
Code and demos are not publicly released with the preprint, and model size is not reported, making independent evaluation and comparison difficult.
Evaluation is limited to LibriTTS dev-clean and dev-other; both are clean read-speech corpora, so generalisation to noisy or spontaneous speech is untested. The user study comprises only 15 utterances rated by 6 annotators — a small sample that limits statistical power. MCD is used as the primary objective metric, but it is an indirect measure of naturalness quality and is known to correlate imperfectly with perceptual preference. The comparison set is narrow: SpeechPainter is referenced in the related work but not included in the quantitative evaluation because it uses a fixed-duration formulation, leaving the question of how well RephraseTTS would perform relative to more recent neural audio inpainting approaches unanswered. Duration quality of the inserted segment relative to ground truth is not measured directly.
Wiki Connections
Key concept pages informed by this paper:
- transformer-enc-dec-tts — RephraseTTS adapts the FastSpeech2 non-autoregressive framework for speech insertion
- prosody-control — cross-modal attention is the mechanism for extracting and transferring prosodic context
- speaker-adaptation — style transfer from short audio context is central to the insertion task
- disentanglement — the model implicitly separates phoneme content from audio-derived speaker style via cross-modal attention
- subjective-evaluation — small-scale MOS user study design and findings
- zero-shot-tts — the setting (synthesising a new segment from minimal audio context) bears structural similarity to zero-shot speaker adaptation