arXiv · 2026 · Preprint
Anupam Purwar et al. (Sprinklr AI) · → Paper · Demo: ? · Code: ?
A controlled empirical study of LoRA fine-tuning applied directly to the language-model backbone (Qwen-0.5B) of an autoregressive codec TTS system, showing that perceptual quality gains from LM-backbone adaptation depend on training-data acoustic diversity rather than on loss convergence alone.
Problem
Language-model-based TTS systems collapse linguistic modeling, prosody planning, and long-range acoustic coherence into a single autoregressive backbone, but prior LoRA-based adaptation work for TTS almost exclusively targets downstream synthesis components (acoustic decoders, speaker embeddings, style/emotion modules) while keeping the LM backbone frozen. The paper identifies four open questions this leaves unanswered: how LoRA adaptation of an LM-based TTS backbone interacts with the pretrained acoustic prior, whether validation loss is a reliable proxy for perceptual quality in token-level generative TTS, what role training-data characteristics (acoustic variability, energy spread) play in adaptation success or failure, and how inference-time decoding controls interact with LoRA-adapted backbones.
Method
The authors fine-tune the Qwen2.5-0.5B language-model backbone of NeuTTS, which predicts acoustic token sequences conditioned on text and a speaker prompt, subsequently decoded to waveform by the NeuCodec neural codec. Two fine-tuning regimes are compared: full fine-tuning of all backbone parameters (batch size 2, 5 epochs, Adam, on an NVIDIA L4 24GB GPU), and LoRA fine-tuning (rank 8, alpha 16) applied to the attention projection layers (q_proj, k_proj, v_proj), which permits a larger effective batch size (4 with gradient accumulation 2) and produces a more stable training-loss trajectory. Training data are single-speaker audio-transcript pairs drawn from two sources: HiFi-TTS (speakers 1, 2, 11614) and Libriheavy-HQ (speakers 1401, 1212, 1259), the latter with longer average utterances and higher total duration per speaker. Beyond per-speaker fine-tuning, the paper runs three multi-speaker training configurations (2+2+2, 1+1+1, and Mix, using 2 hours, 1 hour, or roughly 1/9th-2/9th of per-speaker data respectively across all six speakers) to test generalization and data efficiency. Evaluation combines DNS-MOS (OVRL) as the primary perceptual-quality proxy (selected after comparing it against UTMOSv2, WVMOS, and TorchAudio-Squim for consistency and length-bias), WeSpeaker cosine-similarity embeddings for speaker fidelity, and WADA-SNR (blind signal-to-noise estimation, since no clean reference signal is available) for signal-level quality. Inference-time decoding controls (temperature, top-k) and 8-bit GGUF quantization of both the LM backbone and NeuCodec are also evaluated for their effect on perceptual quality and latency.
Key Results
LoRA fine-tuning for 1,000 training steps raises DNS-MOS by up to +0.424 (speaker 2: 3.717 to 4.141) for speakers with high training-data acoustic variability, but degrades DNS-MOS by up to -0.414 (speaker 1212: 3.647 to 3.233) for speakers with low variability (§3, Table 3). Speaker similarity improves for all six evaluated speakers after LoRA fine-tuning, with gains ranging from +1.3% to +24.6% relative to the base model (§3, Table 8). Constrained decoding (T=0.8, k=40) partially recovers DNS-MOS for low-variability speakers while slightly reducing it for the highest-quality speaker (§3.1, Table 4). In the multi-speaker experiments, a model trained only on three HiFi-TTS speakers (2+2+2 FT) achieves a DNS-MOS of 3.806 on three completely unseen Libriheavy-HQ speakers, a +0.293 gain over speaker-specific fine-tuning (3.513) on those same speakers, with substantially lower cross-speaker MOS variance (0.008 vs. 0.052) (§4.6, Table 10). A Mix FT model trained on all six speakers with only 11-22% of per-speaker data reaches speaker similarity within 5-9% of dedicated single-speaker models (§4.6, Table 11). 8-bit GGUF quantization reduces non-streaming generation time by 4.5-6.9x relative to full-precision models, with LoRA-adapted quantized generation within 4% of the base quantized model’s latency (§4.7, Table 7).
Novelty Assessment
The core methodological move, applying LoRA directly to the attention layers of an LM-based TTS backbone rather than to downstream synthesis/style components, is incremental relative to prior LoRA-for-TTS work (the paper cites UtterTune as the only prior LM-backbone LoRA application, and characterizes its own scope as broader). The paper’s more substantive contribution is empirical: a controlled, multi-speaker, multi-dataset analysis that isolates training-data acoustic variability (specifically energy standard deviation) as a predictor of LoRA fine-tuning outcome, and documents a “loss-quality divergence” phenomenon in which monotonically improving training/validation loss does not track DNS-MOS. This is a data-scale and training-recipe study rather than an architectural contribution; the underlying model (Qwen2.5-0.5B + NeuCodec) and adaptation technique (LoRA) are both pre-existing.
Field Significance
moderate — This paper provides a cautionary, practitioner-relevant data point for LM-backbone LoRA adaptation in autoregressive codec TTS: it demonstrates that loss-based checkpoint selection can silently produce perceptual quality regressions, and that acoustic-diversity thresholds in fine-tuning data are a more reliable predictor of outcome than dataset size or absolute energy level. It also offers preliminary evidence that limited-data multi-speaker fine-tuning can generalize zero-shot to unseen speakers, a useful data efficiency observation for practitioners adapting compact LM-based TTS systems under resource constraints.
Claims
- supports: Applying parameter-efficient fine-tuning directly to the language-model backbone of an autoregressive codec TTS system, rather than only to downstream synthesis or style layers, can improve both perceptual quality and speaker similarity for voice cloning.
Evidence: LoRA applied to the q/k/v projection layers of a Qwen2.5-0.5B backbone raised DNS-MOS by up to +0.424 and improved speaker similarity for all six evaluated speakers (e.g., +12.9% for speaker 11614). (§3, Table 3, Table 8)
- complicates: Training-data acoustic homogeneity limits or reverses the benefit of language-model backbone fine-tuning for speech generation, even when the fine-tuning objective continues to improve.
Evidence: Speakers whose training data had energy standard deviation below approximately 10 dB (speakers 1401, 1212, 1259) showed DNS-MOS degradation of up to -0.414 after LoRA fine-tuning, while speakers with energy standard deviation above approximately 13 dB gained up to +0.424. (§4.4, Table 9)
- complicates: Training and validation loss can be an unreliable proxy for perceptual quality in token-likelihood-based (LM-backbone) TTS fine-tuning, so checkpoint selection guided only by loss convergence risks selecting perceptually degraded models.
Evidence: Loss curves were smooth and monotonically decreasing across all six speakers, while DNS-MOS exhibited non-monotonic trajectories, degrading sharply within the first 1,000 steps for low-diversity speakers before partially recovering at later epochs. (§4.2, Figure 2)
- supports: Multi-speaker fine-tuning with limited per-speaker data can generalize to unseen speakers and reduce cross-speaker quality variance relative to dedicated single-speaker fine-tuning.
Evidence: A model trained only on three HiFi-TTS speakers (2+2+2 FT) achieved DNS-MOS 3.806 versus 3.513 for speaker-specific fine-tuning when evaluated zero-shot on three unseen Libriheavy-HQ speakers, and had lower MOS variance across all six speakers (0.008 vs. 0.052) than the speaker-specific baseline. (§4.6, Table 10)
- complicates: Optimal inference-time decoding constraints for LM-based TTS backbones are speaker- and data-dependent rather than universal, interacting with the acoustic diversity of the fine-tuning data.
Evidence: Constrained decoding (temperature 0.8, top-k 40) improved DNS-MOS for low-diversity speakers 1212 and 1401 but produced a slight DNS-MOS reduction for high-diversity speaker 2. (§3.1, Table 4)
Limitations and Open Questions
The study evaluates a single, small (0.5B-parameter) LM backbone across six speakers using synthetic (DNS-MOS) rather than human-rated MOS, and the authors explicitly caution that findings may not transfer to larger or medium-sized language models, or to language-model tasks beyond TTS.
Additional limitations noted by the authors: only the first 5,000 samples per speaker from HiFi-TTS were used even where more data was available, leaving the effect of larger per-speaker audio volume on MOS and similarity unexplored; the acoustic-diversity analysis is limited to frequency and energy statistics rather than spectrogram-based analysis; and reference-audio-length experiments used artificially concatenated or sliced clips that introduced prosodic discontinuities, confounding the relationship between context length and synthesis quality (§4.5).
Wiki Connections
- Autoregressive Codec TTS — studies LoRA fine-tuning of the LM backbone in an autoregressive codec TTS pipeline (Qwen2.5-0.5B predicting acoustic tokens decoded by NeuCodec), extending adaptation beyond the synthesis-layer scope of most prior LoRA-for-TTS work.
- Zero-Shot TTS — the underlying NeuTTS system performs voice cloning from a speaker audio prompt, and the paper’s multi-speaker experiments explicitly test zero-shot generalization of LoRA-adapted backbones to speakers unseen during fine-tuning.
- Speaker Adaptation — is centrally about how LoRA fine-tuning of an LM backbone adapts speaker identity representations, and what training-data properties govern adaptation success versus failure.
- Evaluation Metrics — compares several synthetic MOS estimation tools (UTMOSv2, WVMOS, TorchAudio-Squim, DNS-MOS) for consistency and length bias before selecting DNS-MOS as the primary perceptual-quality metric.
- Neural Audio Codec — evaluates the latency impact of 8-bit GGUF quantization applied to both the LM backbone and the NeuCodec neural codec used for waveform decoding.
- UtterTune — cited as the only prior work injecting LoRA directly into an LM-based TTS backbone, though scoped narrowly to pronunciation and pitch-accent control; this paper positions itself as broadening that scope to perceptual quality, stability, and latency trade-offs.
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — cited as foundational evidence that LM backbones can predict acoustic token sequences conditioned on text and a speaker prompt, the paradigm this paper’s fine-tuning study builds on.
- Qwen2.5 Technical Report — the source of the Qwen2.5-0.5B language model used as the TTS backbone being fine-tuned in this work.