arXiv · 2026 · Preprint

Asaria et al. (Transformer Lab) · → Paper · Demo: ? · Code: ?

Shows that best-of-N sampling with an ASR round-trip check removes catastrophic failures (silence, early stop, repetition, hallucinated content) from four open autoregressive codec TTS models, and that LoRA distillation on the self-verified samples recovers roughly half of that robustness in single-shot decoding on hard inputs.

Problem

Open autoregressive TTS models that generate discrete codec tokens produce good speech on most inputs but fail stochastically and catastrophically on a fraction of utterances: they emit silence, terminate early, loop, or produce unrelated content. On a broad evaluation of Llasa-1B that includes adversarial rare-word and number/date prompts, the authors measure a catastrophic-failure rate of about 0.36, and about 0.27 single-shot on their hand-written hard set (§1). Prior robustness post-training work typically applies one preference method to one model, so it is unclear whether a simple test-time verification baseline, and its distilled form, generalises across models and codecs (§2).

Method

Metric. Each generation is transcribed with Whisper (model size not reported). A generation counts as a catastrophic failure if it is a dropout (fewer than 25 decoded speech tokens, or an ASR transcript of at most one word) or a content failure (WER above 0.5 against the reference text). For best-of-N, a prompt fails only when all N candidates fail, giving a catastrophic-failure rate CFR_N that is non-increasing in N; CFR_1 is the single-shot rate (§3). The authors adopt this dropout/collapse rate after finding that number/date WER is dominated by digit-versus-word formatting noise (§3, §6). Zero-failure cells are reported with a rule-of-three 95% upper bound (3/n), and proportions carrying a claim get Wilson 95% intervals.

Test-time self-verification. Sample N candidates, transcribe each, and return the lowest-WER non-dropout candidate. Because the selector uses the target transcript, best-of-N is a reference-aware oracle: an upper bound on what verification can buy, not a deployable number (§4, §5.1).

Distillation. The model is fine-tuned with LoRA on the best self-verified sample per training prompt so that single-shot decoding imitates the verified behaviour. The authors compare plain supervised distillation (SFT on the chosen sample) with preference optimisation using a rejected sample: DPO (reference policy is the LoRA-disabled base), IPO, a failure-targeted pairing (FTPO, where the rejected sample is specifically a dropout or collapse), and an online iterative variant that re-samples from the current policy, re-scores by ASR and rebuilds pairs for three rounds (§4).

Models: which are trained and which are only sampled. All four systems are LLaMA-style or transformer autoregressive LMs over codec tokens; no codec is trained or modified.

  • Fine-tuned (LoRA distillation): Llasa-1B (XCodec2), the primary model and the only one used for the SFT-versus-preference comparisons; Llasa-3B (XCodec2) and Orpheus-3B (SNAC), which receive a distillation pass evaluated on LibriSpeech only (Table 1).
  • Sampled unchanged: CSM-1B (Sesame, Mimi RVQ codec), used only for base versus best-of-N, because its two-transformer design does not expose the single token stream the distillation path trains on (§5.1).

The three codecs (single-stream XCodec2, flattened-RVQ SNAC, hierarchical-RVQ Mimi) vary together with the model family, so codec choice is not isolated as an experimental variable. Evaluation uses LibriSpeech test-clean (120 transcripts x 3 generations, training prompts from the disjoint dev-clean split) and a hand-written hard set of normal, long and tongue-twister prompts (26 prompts x 6 generations), with rare-word and number/date buckets analysed separately (§3). Total compute is about 45 GPU-hours (§7).

Key Results

Best-of-N on the primary model (Table 2). On the hard set, Llasa-1B falls from 0.269 single-shot (95% CI [0.205, 0.343]) to 0.154 at N=2, 0.038 at N=3, and no observed failures at N=4, 5 and 6 (upper bound 3/156, about 0.019). On LibriSpeech it falls from 0.058 to no observed failures at N=2 and N=3 (upper bound 0.008) (§5.1).

Across four models (Table 1, LibriSpeech test-clean).

ModelCodecBaseBest-of-2Best-of-3Distilled
Llasa-1BXCodec20.0580.0000.0000.058
Orpheus-3BSNAC0.0080.0000.0000.033
CSM-1BMimi0.0170.0000.000not run
Llasa-3BXCodec20.1080.0420.0330.142

Best-of-N reduces failures on all four models; three of four reach no observed failures by N=2, while Llasa-3B plateaus at 0.033 at N=3. The authors caution that the Llasa-3B run used a tighter generation budget (max_new_tokens = 1024) than the 1B run, which can truncate long generations into failures, so they do not read the 1B-versus-3B gap as a scale effect (§5.1).

Distillation (Table 3). On the hard set, single-shot failure drops from 0.199 [0.143, 0.270] to 0.096 [0.059, 0.152] with SFT distillation and 0.083 [0.049, 0.137] with DPO distillation, closing about 52% (SFT) to 58% (DPO) of the failure mass, with residual failures remaining. On LibriSpeech there is no detectable change for Llasa-1B (0.058 to 0.058), and where the base is already near the floor distillation regresses slightly (Orpheus-3B 0.008 to 0.033; Llasa-3B 0.108 to 0.142) (§4, §5.2, Table 1). Because best-of-N is reference-aware, the distilled single-shot column is the deployable number.

Preference optimisation versus SFT (§5.3). In an offline sweep on the hard set (about 108 generations), DPO (0.292), FTPO (0.292) and IPO (0.319) did not improve on one round of SFT-on-best (0.264); the intervals overlap heavily, so the authors read this as “no method beat SFT”, not a ranking. In the matched distillation comparison, SFT (0.096) and DPO (0.083) differ by 0.013, inside the standard error of the difference (about 0.032), which the authors call a tie. The online iterative variant gives the lowest single-shot (trained-model) failure rates in the paper, 0.013 (DPO, 1/78) and 0.026 (SFT, 2/78), but on a smaller evaluation, with 95% intervals (e.g. [0.002, 0.069]) overlapping the single-pass numbers, and as a post-hoc minimum over many conditions; the authors call it promising, not confirmed.

Analysis (§6). Base-generation silence/dropout is about 4.6%. Rare words are a capability ceiling: only 2 of 10 rare-word prompts ever yield a faithful sample within N=3, so neither best-of-N nor distillation can recover them. For numbers/dates, a raw-versus-normalised A/B leaves WER essentially unchanged (0.155 vs. 0.143), which the authors read as artifact-dominated measurement. No human listening test, MOS or speaker-similarity results are reported.

Novelty Assessment

The components are established: ASR-filtered best-of-N sampling, SFT on self-selected samples, and DPO/IPO with ASR-derived pairs. The contribution is empirical: applying one protocol uniformly across four open codec TTS models with three different codecs, reporting zero-failure cells with explicit upper bounds, and running a controlled SFT-versus-preference comparison that comes out as a tie. The failure-rate metric is a thresholded ASR round-trip rather than a new measurement method, though the observation that WER cannot track number/date robustness because of formatting noise is a useful measurement note. The distillation result rests on Llasa-1B for the hard set, and the generalisation study covers the test-time procedure, not distillation, on CSM-1B.

Field Significance

Low. The paper gives a carefully caveated, multi-model data point that catastrophic failures in autoregressive codec TTS are largely a sampling problem that oracle ASR verification removes, and that one supervised distillation pass recovers part of that robustness on hard inputs. Its evidence is a single-language, automated-metric study with single training runs per condition and no human evaluation, so it serves mainly as confirmatory evidence for ASR-guided post-training and as a cautionary data point that offline DPO did not beat SFT at this scale.

Claims

  • supports: Catastrophic failures in autoregressive codec-token TTS are largely stochastic rather than systematic, so selecting among a few samples with an ASR check removes most of them across model families and codec token formats.

    Evidence: Best-of-2 reaches no observed failures on LibriSpeech for Llasa-1B (0.058 base), Orpheus-3B (0.008) and CSM-1B (0.017), and Llasa-1B’s hard-set rate falls from 0.269 to none at N=4; Llasa-3B falls from 0.108 to 0.033 but does not reach zero. (§5.1, Table 1, Table 2)

  • supports: Supervised fine-tuning on self-verified samples transfers part of the robustness of best-of-N selection into single-shot decoding, with the gain concentrated on inputs where the base model fails often.

    Evidence: Hard-set single-shot failure falls from 0.199 to 0.096 (SFT) and 0.083 (DPO) after one distillation pass, while LibriSpeech stays at 0.058 for Llasa-1B and regresses for near-floor models (Orpheus-3B 0.008 to 0.033, Llasa-3B 0.108 to 0.142). (§5.2, Table 1, Table 3)

  • complicates: Offline preference optimisation with ASR-derived chosen/rejected pairs does not necessarily improve over plain supervised distillation on the selected samples for TTS robustness.

    Evidence: Offline DPO (0.292), FTPO (0.292) and IPO (0.319) did not beat SFT-on-best (0.264) on the hard set, and matched SFT (0.096) versus DPO (0.083) differ by 0.013 against a standard error of about 0.032; an online iterative variant reaches 0.013/0.026 but on 78 generations with overlapping intervals. (§5.3)

  • complicates: Sampling-based verification and self-distillation cannot repair failures that reflect a missing capability of the base model.

    Evidence: Only 2 of 10 rare-word prompts yield a faithful sample within N=3, so both best-of-N and distillation plateau high on this bucket. (§6)

  • complicates: WER is an unreliable measure of number and date robustness in TTS because digit-versus-word formatting differences dominate the error.

    Evidence: Number/date WER sits around 0.14 to 0.15 from canonicalisation noise, and a raw-versus-normalised A/B leaves it essentially unchanged (0.155 vs. 0.143), leading the authors to centre the metric on dropout/collapse instead. (§3, §6)

Limitations and Open Questions

Warning

The headline best-of-N numbers are oracle upper bounds: the verifier scores candidates against the reference transcript, which is unavailable at deployment. The deployable result is the distilled single-shot rate, which closes only about 52-58% of hard-set failures and does not improve, or slightly regresses, on LibriSpeech (§5.1, §5.2, Table 1).

  • The claim that distillation gains depend on difficulty rests on a two-regime comparison (hand-written hard set vs. LibriSpeech) that also differs in source, length, word frequency and speaker reference; the within-corpus difficulty sweep that would isolate it was not run (§7).
  • Distillation is evaluated on Llasa-1B/3B and Orpheus-3B only; CSM-1B gets no distillation, and the SFT-versus-preference comparisons are on Llasa-1B only (§5.1, §7).
  • Every distilled number is a single training run; the hard set is small (156 generations) and the online iterative evaluation smaller (78) (§3, §5.3).
  • The metric inherits Whisper’s errors, and its false-positive floor on ground-truth audio was not measured (§7).
  • Single language (English), two corpora; no human listening tests, naturalness or speaker-similarity measures, so possible quality or similarity side effects of distillation are not assessed.
  • The Llasa-3B results carry a generation-budget confound (§5.1).
  • Code and demo availability are not stated.

Wiki Connections

  • Autoregressive Codec TTS — Characterises and reduces stochastic catastrophic failures in four autoregressive codec-token TTS models via ASR-verified sampling and LoRA distillation.
  • RLHF Speech — Trains offline DPO, IPO, a failure-targeted variant and an online iterative DPO on ASR-derived preference pairs and finds offline variants do not beat supervised distillation.
  • Evaluation Metrics — Defines a thresholded ASR round-trip catastrophic-failure rate with rule-of-three bounds and shows WER cannot measure number/date robustness.
  • Llasa — Llasa-1B and Llasa-3B (with the XCodec2 codec) are the primary models, fine-tuned with LoRA for distillation and the preference comparisons.
  • Moshi — Source of the Mimi RVQ codec used by CSM-1B, which is sampled unchanged in the generalisation study.
  • CosyVoice 2 — Cited as prior work applying preference optimisation with automatic rewards to an autoregressive codec TTS model.
  • TTS-1 Technical Report — Cited as prior preference-optimisation work on autoregressive codec TTS with automatic rewards.
  • MegaTTS 3 — Cited as characterising robustness failure modes of autoregressive codec LMs.
  • Metis — Cited as characterising robustness failure modes of autoregressive codec LMs.