arXiv · 2026 · Preprint
Hanchen Pei et al. (Microsoft) · → Paper · Demo: ✓ · Code: ✗
Extends a VALL-E-style neural codec language model with a selective attribute-editing mechanism that reproduces a speech prompt’s full acoustic profile by default but overrides only the specific attributes named by an explicit instruction.
Problem
Zero-shot neural codec language models clone a reference speaker’s voice by imitating the reference prompt holistically, folding timbre, prosody, and paralinguistic style into a single opaque conditioning signal. This makes it difficult to change one attribute (e.g. emotion) while leaving everything else, especially speaker identity, untouched. Prior controllable-synthesis approaches address this with text-driven style tags or descriptions (which struggle to reproduce a specific voice), dual-prompt audio-driven control (which introduces attribute leakage when multiple prompts conflict), or explicit disentanglement modules such as gradient-reversal layers or information-bottleneck codebooks (which add architectural complexity and can still leave attributes incompletely separated). The paper frames the gap as a selective attribute editing problem: given a speech prompt, target text, and an edit specification, produce speech that preserves everything the prompt implies except the attributes the edit specification names.
Method
SpeechEdit extends the two-stage (autoregressive + non-autoregressive) VALL-E paradigm for zero-shot TTS with an instruction-guided conditioning interface. Speech is discretized with EnCodec into 8 codebook layers at 6 kbps/24 kHz. The AR stage predicts the first codebook layer autoregressively; the NAR stage predicts the remaining seven layers conditioned on the first, refining acoustic detail. Both stages are conditioned on the same signal: an edit specification C, the prompt transcription, the target text, and the acoustic prompt, concatenated into one token sequence. A modified causal mask allows bidirectional attention over the conditioning prefix while remaining causal over the generated context.
Two attribute types are represented as discrete instruction tokens sharing the same vocabulary and embedding table as BPE text tokens: categorical emotion tags (five classes: Neutral, Happy, Sad, Angry, Surprise) and ordinal prosody tags for pitch, energy, and speaking speed (five levels each, e.g. <pitch-high>, <speed-low>). Speaker identity is instead modeled continuously: a global speaker embedding is extracted from a reference utterance with a pretrained voice-print model and projected into the language-model embedding space via a dedicated layer.
Rather than adding an auxiliary disentanglement module, SpeechEdit relies on a Delta-Pair sampling strategy to teach the model to separate attributes implicitly, from data. In same-speaker delta pairs, two utterances from the same speaker with different emotions are sampled; the model is conditioned on the target utterance’s style tags but the source utterance’s acoustic prompt, forcing the attention mechanism to pull unspecified attributes from the prompt and the specified attribute from the instruction tokens. In cross-speaker delta pairs, source and target utterances come from different speakers; the model conditions on the source acoustic tokens plus target-attribute instructions, with a separate, content-independent reference utterance supplying the target speaker embedding (to avoid content leakage). By deliberately mismatching prompt and target attributes during training, the instruction tokens become the only reliable signal the model can use, which the authors argue induces disentanglement without requiring any auxiliary loss or module. The same conditioning scheme supports zero-shot TTS (all style tokens set to a fill-in placeholder, so the model relies entirely on the acoustic prompt), style editing (specific style tags overridden), and voice conversion (a target speaker embedding substituted, with style tokens optionally mixed).
Training data comes from LibriEdit, a new corpus the authors built by re-segmenting LibriHeavy at breath-group/pause boundaries via Montreal Forced Aligner, labeling emotion with a categorical SER model plus multi-model cross-validation (emotion2Vec-plus-large and Audio Flamingo 3, majority voting), and labeling prosody attributes with signal-processing estimators. The resulting corpus has 2,566 speakers and 708 hours total (129 hours of which carry emotion labels). SpeechEdit (a 12-layer decoder-only Transformer per stage, 16 heads, 1,024-dim embeddings, 4,096-dim FFN) is pretrained on LibriHeavy-large following the VALL-E recipe for 800k updates, then further trained on LibriEdit for another 800k updates with same-speaker and cross-speaker delta pairs each contributing 50% of training pairs.
Key Results
On zero-shot TTS (LibriSpeech test-clean), under a training budget of under 1k hours SpeechEdit reaches 1.3% WER, better than the paper’s VALL-E ablation baselines (2.1% and 2.7% WER at comparable data scales) and Step-Audio-EditX (1.6% WER, 3B params), though speaker similarity (0.48) trails several baselines, which the authors attribute to the expressive prosodic diversity of LibriEdit (Table 1).
On emotion editing, two setups are tested. In the Easy Task (neutral prompts, no attribute conflict, 80 samples), SpeechEdit’s single-stage direct emotion conditioning reaches 63.75% emotion classification accuracy (ECA) at 2.5% WER, already exceeding Step-Audio-EditX’s iterative editing, which raises ECA only from 50% (zero-shot, iteration 0) to 57.5% (iteration 2) with negligible gain after the first edit. In the Hard Task (80% of prompts carry conflicting emotion, 100 samples), SpeechEdit reaches 92% ECA versus 79% for CosyVoice 2 and 73% for IndexTTS 2, with the highest DNSMOS among compared systems, while maintaining a competitive 2.5% WER (IndexTTS 2 is lowest at 2.0%). Speaker similarity is again slightly below the compared systems, which the authors note is expected because SIM is measured against the (now attribute-modified) prompt.
Subjective evaluation of prosody control (CMOS on speed/pitch/energy, 10 listeners) finds over 85% of samples rated consistent with the intended control direction across all three attributes, with SMOS (naturalness) averaging above 4.2. A separate CMOS-style voice-conversion test with four speakers finds only 3.0% of samples rated closer to the source speaker (source-identity leakage), 68.7% rated closer to the target speaker, and 28.3% ambiguous; cross-gender conversions account for the majority of both the strongest target-match and the ambiguous cases.
Novelty Assessment
The architecture itself is a direct extension of the VALL-E AR/NAR codec-LM paradigm; the contribution is not a new backbone but a new conditioning and training recipe layered on top of it: (1) a unified instruction-token interface that lets a single model span zero-shot TTS, voice conversion, and attribute editing without task-specific heads, and (2) Delta-Pair sampling, a data-construction strategy that induces attribute disentanglement through training-pair design rather than through an explicit auxiliary loss, gradient-reversal layer, or information bottleneck. The LibriEdit dataset is a genuine secondary contribution: a speaker-consistent, multi-attribute-labeled corpus built from LibriHeavy audiobook narration, positioned as combining scale (708 hours) with attribute coverage that prior open datasets in this space (e.g. EmoVoice-DB, TextrolSpeech, Expresso) trade off against one another. The paper is honest about the resulting limitations: speaker similarity is measurably lower than baselines, and an ablation shows removing the cross-speaker (voice-conversion) training objective directly improves it, indicating an explicit trade-off rather than a free lunch. Overall this reads as a solid engineering-and-training-recipe contribution built on an established architecture, not a structurally new codec-LM design.
Field Significance
moderate — SpeechEdit demonstrates that implicit, data-driven disentanglement via training-pair construction can match or exceed explicit disentanglement mechanisms on selective attribute editing within a VALL-E-style codec LM, without adding architectural modules. It also contributes a new speaker-consistent, multi-attribute-labeled dataset (LibriEdit) built from an existing large-scale ASR corpus. The evaluation is comparatively narrow (test sets of 80-100 samples per emotion-editing condition, four unseen speakers), and the reported speaker-similarity trade-off is left as an open question rather than resolved.
Claims
- supports: Attribute disentanglement in a codec language model can be induced implicitly through training-pair construction (deliberately mismatching prompt and target attributes) rather than through an explicit auxiliary loss or architectural module.
Evidence: Same-speaker and cross-speaker Delta-Pair sampling, with no auxiliary disentanglement loss, yields 92% emotion classification accuracy on the Hard Task (conflicting prompt/target emotion in 80% of cases), exceeding CosyVoice 2 (79%) and IndexTTS 2 (73%), which use style-tag or dual-prompt conditioning. (§6.1, Table 1)
- supports: Direct single-stage attribute-conditioned generation can express fine-grained control more effectively than iterative post-hoc editing of an already-synthesized utterance.
Evidence: Step-Audio-EditX’s iterative editing raises emotion classification accuracy only from 50% (zero-shot generation) to 56.25% after one edit and 57.5% after a second, with negligible further gain, while SpeechEdit’s single-stage direct emotion conditioning reaches 63.75% ECA at a comparable 2.5% WER. (§6.1, Figure 4)
- complicates: Adding a cross-speaker voice-conversion training objective to a unified selective-editing codec LM trades off speaker-similarity preservation for cross-speaker controllability.
Evidence: A task-ablated variant trained only on same-speaker delta pairs (removing cross-speaker supervision) improves zero-shot TTS speaker similarity from 0.45 to 0.53 relative to the full model, at the cost of losing voice-conversion capability. (§A.4, Task Ablation)
- complicates: Augmenting a training corpus with additional emotional speech from a mismatched distribution (elicited, exaggerated emotion) does not reliably improve emotion-controllable synthesis and can degrade it.
Evidence: Mixing 708 hours of read-audiobook LibriEdit data with 30 hours of internal acted emotional speech and 47 hours of Expresso produced consistent degradation across most objective metrics relative to LibriEdit-only training, attributed to a distribution mismatch between spontaneous and elicited/exaggerated emotional expression. (§A.4, Data Ablation, Table 1)
Limitations and Open Questions
Speaker identity is represented with a single static global embedding, which the authors acknowledge can lose time-varying vocal nuance and idiosyncratic articulation, and the model's disentanglement is entirely implicit (no auxiliary supervision), which the authors note may limit robustness on extreme or rare attribute combinations that the Delta-Pair sampling strategy does not adequately cover.
The controllable attribute space is restricted to emotion, prosody, and speaker identity via a fixed categorical/ordinal instruction vocabulary; it does not support natural-language style instructions or arbitrary multi-attribute specifications. The emotion-editing evaluation sets are small (80-100 samples, four unseen speakers per condition), and no code repository is released alongside the paper, only audio samples.
Wiki Connections
- Zero-Shot TTS — extends the VALL-E zero-shot TTS setting with an instruction interface that leaves the acoustic-prompt-driven generation path intact when no attribute edits are requested.
- Voice Conversion — implements voice conversion as a special case of its unified instruction interface, substituting a target speaker embedding while optionally retaining source prosody, and evaluates it with a dedicated subjective CMOS test across four speakers.
- Emotion Synthesis — introduces categorical emotion instruction tokens and Delta-Pair training specifically to enable single-stage, prompt-independent emotion control, evaluated with objective ECA and subjective classification-confidence analysis.
- Prosody Control — provides explicit ordinal instruction tags for pitch, energy, and speaking speed that independently override the corresponding attribute inferred from the speech prompt.
- Autoregressive Codec TTS — builds directly on the VALL-E two-stage AR/NAR codec-token paradigm, adding a unified instruction-conditioning interface on top of the same backbone.
- VALL-E — SpeechEdit’s AR/NAR backbone and training recipe directly extend this paper’s codec-LM paradigm, and its ablation settings serve as the primary zero-shot TTS baseline.
- Step-Audio-EditX — the paper’s most relevant baseline; SpeechEdit compares its single-stage direct attribute conditioning against Step-Audio-EditX’s iterative post-hoc editing approach.
- CosyVoice — used as a baseline for instruction-based multi-style emotional synthesis on both the Easy and Hard emotion-editing tasks.
- IndexTTS2 — used as a baseline for emotionally expressive, duration-controlled autoregressive zero-shot TTS on both emotion-editing tasks.
- Seed-TTS — cited as an example of holistic, blackbox reference-audio imitation, the limitation SpeechEdit’s selective editing mechanism is designed to address.
- Expresso — used as one of two supplementary datasets in the data-ablation experiment, which shows that mixing in elicited emotional speech degrades performance relative to LibriEdit alone.
- NaturalSpeech 3 — cited as an example of explicit disentanglement via factorized codec representations, contrasted against SpeechEdit’s implicit, data-driven disentanglement approach.
- Vevo — cited as an example of explicit disentanglement via information-bottleneck codebooks, contrasted against SpeechEdit’s implicit, data-driven disentanglement approach.