arXiv · 2025 · Preprint
Yun Chen et al. · → Paper · Demo: ✓ · Code: ?
Introduces ISSE, a ~400-hour, 100,000-pair instruction-guided speech style editing dataset with fine-grained natural language editing instructions, built via an automated TTS+VC+LLM pipeline, and benchmarks it with an autoregressive editing model.
Problem
Speech style editing (changing emotion, speaking rate, or prosodic delivery while preserving linguistic content and speaker identity) has historically relied on explicit categorical labels or reference audio, both of which limit flexibility and scalability. Natural-language-guided approaches promise more flexible control, but the paper argues that available paired datasets for this specific task are inadequate on two fronts: instructions are typically templated and single-attribute (“Change the emotion of speech”), and the datasets themselves are small (ESD: 29 hours; Expresso: 11 hours paired) and stylistically narrow. Description-based expressive TTS datasets (e.g. PromptTTS-style corpora) compound the problem because they only pair text with a style description, with no source-target speech pair to define what an “edit” actually changes, leaving the disentanglement of style from content and timbre ill-posed.
Method
ISSE is constructed through a three-stage automated pipeline. Stage 1 (Paired Speech Synthesis) samples an anchor speech (fixing transcription and speaker identity) from EARS or Expresso, then samples N style-reference speeches with distinct styles; an expressive TTS model (EmoCapTTS) synthesizes a stylized version of the anchor’s transcript in each reference style, after which a voice conversion model (Chatterbox) converts the synthesized speech’s timbre back to match the anchor speaker, isolating style change from speaker change. Stage 2 (Quality Evaluation) filters generated pairs using Whisper-large-v3 for word error rate (content fidelity), emotion2vec for style similarity to the reference, and ECAPA-TDNN for speaker similarity to the anchor, retaining only samples passing all three thresholds (WER < 10, style similarity > 0.5, speaker similarity > 0.5). Stage 3 (Instruction Design) prompts Qwen3-8B to compare the source and target style descriptions and generate diverse, fine-grained natural-language editing instructions (e.g. “Convert the source speech to a whispering style with a high-pitched voice”) rather than templated single-attribute commands.
The resulting corpus contains approximately 382 hours of speech across roughly 100,000 source-target pairs (around 90 hours of real recorded speech and 292 hours of generated speech), spanning 28 distinct styles.
As a benchmark, the authors train LlasaEdit, an instruction-guided autoregressive speech editing model built on the Llasa backbone (a Llama-based AR speech-token model) with the XCodec2 audio codec. Each training triplet consists of source speech tokens, target speech tokens (same speaker and transcript, different style), and a text-tokenized style instruction; the model is trained with teacher forcing to predict target tokens autoregressively conditioned on source tokens and the instruction, minimizing token-level negative log-likelihood. The Llasa-1B backbone is adapted with LoRA (rank 64, alpha 128) applied to attention and MLP layers, trained with AdamW (lr 1e-4) for 5 epochs on 8x A100 GPUs. At inference, the model autoregressively generates edited target tokens from source tokens and the instruction, and XCodec2 decodes the waveform.
Key Results
LlasaEdit trained on ISSE is compared against the same architecture trained on ESD, the closest existing open paired-style dataset. In-domain, the ISSE-trained model outperforms the ESD-trained model on every reported metric: WER 8.06 vs. 10.07, style similarity 0.68 vs. 0.64, speaker similarity 0.58 vs. 0.49, and UTMOS 4.29 vs. 4.01. Under cross-domain evaluation, the ISSE-trained model evaluated on ESD (7.69 WER, 0.59 style-sim, 0.51 speaker-sim, 4.31 UTMOS) actually exceeds the ESD-trained model’s own in-domain numbers on most metrics, while the ESD-trained model evaluated on the ISSE test set collapses to WER 68.17, indicating it fails to generalize to ISSE’s broader, fine-grained instruction space. A secondary experiment fine-tunes Llasa-1B on ISSE for plain TTS and expressive TTS conditioned on style descriptions: TTS WER drops from 5.48 (no fine-tuning) to 3.74, and the style-conditioned variant reaches 0.60 style similarity and 0.96 gender accuracy with comparable naturalness (UTMOS 3.94), demonstrating the corpus is also usable outside the source-target editing setup.
Novelty Assessment
The contribution is primarily a dataset and a data-construction pipeline, not a new model architecture. LlasaEdit reuses the existing Llasa backbone and XCodec2 codec, adapted to the editing task with standard LoRA fine-tuning; the modeling recipe (teacher-forced AR generation over source+instruction-conditioned target tokens) is a direct application of prior instruction-guided and codec-token generation approaches, not a novel training objective. The genuine contribution is the automated construction pipeline that combines expressive TTS, voice conversion, and an LLM to generate fine-grained, multi-attribute editing instructions at a scale (382 hours, ~100K pairs) roughly an order of magnitude larger than the closest prior resource (ESD, 29 hours), while producing free-form rather than templated instructions. The cross-domain generalization result is a genuine (if expected) demonstration that scale and instruction diversity help, but the comparison is against a single small baseline dataset (ESD) rather than a broader set of alternatives, and the benchmark model itself is not compared against other editing architectures.
Field Significance
Moderate — this paper addresses a concrete data bottleneck in instruction-guided speech style editing by releasing a substantially larger and more instruction-diverse open dataset than prior alternatives, and demonstrates with a controlled ESD-vs-ISSE comparison that dataset scale and instruction granularity materially affect both in-domain and cross-domain editing quality. Its scope is limited to English and to a single benchmark architecture, so it functions as a resource and existence proof rather than a new modeling direction.
Claims
- supports: Automated pipelines combining expressive TTS, voice conversion, and LLM-based instruction generation can produce large-scale paired speech style editing data without manual recording or annotation, provided a multi-criterion filtering step is applied.
Evidence: The three-stage pipeline (EmoCapTTS synthesis + Chatterbox voice conversion + Qwen3-8B instruction generation) yields 382 hours and ~100,000 pairs from EARS and Expresso source material, filtered to WER < 10, style similarity > 0.5, and speaker similarity > 0.5 (§2.1, §2.2, Fig. 3).
- supports: Fine-grained, diverse natural-language instructions improve both in-domain accuracy and cross-domain generalization of instruction-guided speech style editing models relative to coarse, templated instruction sets.
Evidence: LlasaEdit trained on ISSE outperforms the same architecture trained on ESD across WER, style similarity, speaker similarity, and UTMOS in-domain (8.06 vs. 10.07 WER; 0.68 vs. 0.64 style-sim), and the ISSE-trained model’s cross-domain performance on ESD exceeds the ESD-trained model’s in-domain performance on several metrics (§4.2, Table 2).
- complicates: Instruction-guided style editing models trained on narrow, templated-instruction datasets fail catastrophically when evaluated on more diverse, fine-grained instruction distributions.
Evidence: The ESD-trained LlasaEdit model, when evaluated on the ISSE test set, produces a WER of 68.17, compared to 10.07 on its own in-domain ESD test set, indicating the model does not generalize beyond the coarse single-attribute instructions it was trained on (§4.2, Table 2).
- complicates: Isolating style variation from speaker identity in synthetically generated paired training data requires an explicit voice-conversion correction step, since expressive TTS models conditioned on style descriptions alone conflate style and timbre changes.
Evidence: EmoCapTTS-generated stylized speech differs from the anchor speech in timbre because the model lacks explicit speaker control; a separate voice conversion stage (Chatterbox) is needed to re-align target timbre to the anchor speaker before the pair can be used to define a style-only edit (§2.1).
Limitations and Open Questions
The dataset and benchmark are limited to English, which the authors explicitly flag as constraining applicability to multilingual editing scenarios. The generated portion of ISSE (292 of 382 hours) is itself the product of a TTS+VC synthesis pipeline rather than real recordings, so any systematic biases or artifacts introduced by EmoCapTTS or Chatterbox could propagate into models trained on it; the quality-filtering thresholds (WER < 10, similarity > 0.5) are relatively loose and their effect on downstream editing fidelity is not separately ablated. The benchmark comparison is against a single alternative dataset (ESD) and a single model architecture (LlasaEdit); no comparison is made against other instruction-guided editing systems such as InstructSpeech, and no ablation isolates the individual contribution of instruction diversity versus raw data scale.
Wiki Connections
- Instruction-Conditioned TTS — ISSE targets natural-language, fine-grained multi-attribute editing instructions rather than the coarse, templated single-attribute commands used in prior instruction-guided speech editing datasets.
- Voice Conversion — the data construction pipeline uses a voice conversion model to strip speaker-identity variation introduced by expressive TTS synthesis, isolating style-only source-target pairs.
- Autoregressive Codec TTS — LlasaEdit is an autoregressive codec-token model (Llasa backbone plus XCodec2) adapted via LoRA to predict edited target speech tokens conditioned on source tokens and a text instruction.
- Neural Audio Codec — both training pairs and model inputs/outputs are represented as XCodec2 discrete tokens, with waveform reconstruction handled by the XCodec2 decoder.
- Prosody Control — the editing task and instruction schema explicitly target prosodic and stylistic attributes (pitch, speaking rate, tone) while holding linguistic content and speaker identity fixed.
- Llasa — LlasaEdit is initialized from and fine-tuned on top of the Llasa autoregressive speech synthesis backbone using LoRA.