ACL 2026 (Findings) · 2026 · Conference

Ho-Lam Chung et al. (National Taiwan University / ASUS Intelligent Cloud Services) · → Paper · → Preprint · Demo: ? · Code: ?

Fine-tunes an existing neural codec with gradients from a frozen LLM (future-token prediction and text-anchored semantic alignment, passed through a hard Gumbel-Softmax bridge), so that a token LM trained on the resulting codes reaches 61.6% on SALMon against 48.3-50.1% for reconstruction-only codecs, while speech reconstruction does not degrade.

Problem

Neural codecs used as speech tokenizers for speech LMs are trained only to reconstruct the waveform, while the LMs that consume their tokens are trained to predict the next token. The authors argue that this objective mismatch makes codecs encode fine acoustic variation (pitch micro-variation, phase, breath, background) as token-level variation that looks like noise to the LM. Two linguistically identical realizations of a word can map to different token sequences, which raises token entropy and LM perplexity. Prior fixes either add semantic encoders or factorized token streams, or change the codec structure. This paper asks whether an existing codec can be retrained, without architecture changes, so that its tokens stay reconstructable but become predictable for an autoregressive LM.

Method

LLM-Codec keeps the codec and the LLM architectures unchanged and adds LM-facing losses to codec fine-tuning. The starting point is the AUV codec at 50 Hz (single codebook of 20,480 entries, 122.63M parameters), and the LLM is Qwen3-4B-Instruct (32 layers, hidden size 2,560), frozen except for 20,480 new audio-token embeddings. Audio-token embeddings are initialized from the LLM’s text embedding space.

Figure 1: Overview of LLM-Codec. Audio is encoded by the codec and passed through a Gumbel bridge to obtain differentiable embeddings; a single LLM forward pass produces hidden states for both future token prediction (K Medusa heads) and semantic alignment with text.

  • Differentiable Gumbel bridge. A linear layer maps the 256-dimensional codec latent to logits over the 20,480 audio tokens. A hard Gumbel-Softmax (temperature annealed from 1.0 to 0.3 over 20k steps) gives discrete tokens in the forward pass and gradients in the backward pass. A cross-entropy term ties the bridge’s logits to the codec quantizer’s original token, so the bridge does not drift from the quantizer.
  • Future Token Prediction (FTP). K Medusa-style linear heads (default K = 5) on the frozen LLM’s hidden states predict the audio token at offsets 1 to K. Each head is initialized from the LLM output projection restricted to the audio-token rows. The heads’ cross-entropy losses use normalized inverse-distance weights (about 0.44, 0.22, 0.15, 0.11, 0.09 for K = 5).
  • Semantic Alignment (SA). For layers 10 to 25 of the LLM, the last-position hidden state of the audio branch is pulled towards the last-position hidden state of the paired transcript (text branch computed without gradients). A cosine loss is combined with a contrastive loss against a FIFO memory bank of 512 text representations (logit scale 5.0, label smoothing 0.1) to avoid collapse.

Training runs for 25k steps on 4-second segments of LibriSpeech train-clean-100 with an effective batch size of 10. Steps 0 to 10k update only the GAN discriminators (multi-period and multi-scale, hinge loss with feature matching, R1 regularization). FTP then ramps in from 10k to 12k and SA from 12k to 14k. Codec encoder and decoder are fine-tuned with SGD at 5e-6 under the full reconstruction loss set (mel, multi-scale mel, multi-resolution STFT, complex STFT with a phase term, adversarial). The auxiliary heads and the bridge are discarded at inference, so the deployed codec has the same structure and cost as AUV.

To measure token learnability, each tokenizer’s LibriSpeech train-clean-100 tokens are used to fine-tune Qwen3-4B-Instruct with LoRA (rank 64 on all linear layers) for 3 epochs of next-token prediction. SALMon scores a pair of coherent and minimally perturbed samples by length-normalized likelihood. For LLM-Codec, LM training starts from the codec-training checkpoint, which already holds trained audio-token embeddings; for baselines, the vocabulary is expanded at the start of LM training.

Key Results

Token learnability (Table 1, Table 7, Table 8, Figure 2). On SALMon, the LM trained on LLM-Codec tokens reaches 61.6% overall (8-category average) against 49.4% for AUV, 49.4% for BigCodec, 50.1% for UniCodec and 48.3% for WavTokenizer-large. LLM-Codec improves over AUV in every category. The largest gains are on background consistency (BG-All 71.5% vs. 49.0%), reverberation (RIR 62.5% vs. 44.0%) and sentiment consistency (64.0% vs. 56.0%). It is not the top system in every category: on BG-Align it scores 48.0% against 51.5% for WavTokenizer-large, and on sentiment alignment it scores 49.5% against 53.0% for BigCodec and 52.5% for UniCodec. Token-LM perplexity falls from 159,768 (AUV) to 4,617, while all baselines stay between 148,122 and 159,768 regardless of codec size (80M to 211M parameters). Replacing the text-initialized audio-token embeddings with random initialization drops SALMon to 50.7% (§5.2).

Reconstruction (Table 2, Tables 9 to 12). On Codec-SUPERB-tiny speech, LLM-Codec has the lowest Mel (0.724) and STFT (1.599) distances of all seven systems, a 5.0% and 3.0% reduction relative to AUV (0.762, 1.648). PESQ and STOI rise slightly over AUV (2.102 vs. 2.094; 0.859 vs. 0.850), but BigCodec keeps the highest perceptual scores (PESQ 2.208, STOI 0.877). Outside speech, LLM-Codec stays close to AUV and does not lead. On music, its Mel distance is the lowest (1.126 vs. 1.129 for AUV) and STOI rises (0.614 vs. 0.609), but AUV keeps higher PESQ (2.195 vs. 2.175) and F0 correlation (0.785 vs. 0.767), and UniCodec has the lowest STFT distance (2.441). On environmental audio, UniCodec leads on Mel (1.337) and STFT (3.336); LLM-Codec improves Mel slightly over AUV (1.836 vs. 1.847) but loses slightly on PESQ (1.562 vs. 1.567). Averaged over all domains, UniCodec has the best spectral metrics (Mel 1.121, STFT 2.534), and LLM-Codec’s F0 correlation (0.630) falls below AUV (0.650) and BigCodec (0.635).

Ablations (Tables 3, 4, 13). FTP alone, SA alone and both together give nearly the same results: Mel 0.725 / 0.723 / 0.724, perplexity 4,631 / 4,616 / 4,617, SALMon 61.8 / 61.3 / 61.6. Varying the number of FTP heads (K = 1, 3, 5, 10) leaves SALMon at 61.4 to 61.6 and perplexity at 4,539 to 4,617. The authors therefore attribute the reconstruction gain to the shared fine-tuning procedure (low-learning-rate fine-tuning with GAN and multi-scale losses), not to FTP or SA. They attribute the learnability gain to the presence of LM-facing gradients, not to the multi-step horizon.

Novelty Assessment

The contribution is a new codec training objective: language-model-facing losses that send gradients from a frozen LLM back into the codec encoder, with the codec and LLM architectures left unchanged. The ingredients are existing techniques (Medusa heads, layer-wise cosine plus memory-bank contrastive alignment, hard Gumbel-Softmax straight-through estimation); what is new is using them as codec-training regularizers. The framing of a reconstruction-versus-prediction mismatch overlaps with prior work on token inconsistency and codec-LM co-design, which the paper cites as complementary. The ablations weaken the paper’s own design rationale. Multi-step prediction adds nothing over K = 1, and either objective alone gives the full gain, so the specific FTP and SA designs are not shown to matter. The SALMon and perplexity gaps are large, but the LM comparison is not matched (see Limitations). The evidence also comes from one base codec, one LLM, one 100-hour read-speech corpus and 3 epochs of LM training.

Field Significance

Moderate — The paper provides evidence that a reconstruction-trained codec can be fine-tuned with gradients from a frozen LLM to give tokens with much lower LM perplexity and better SALMon coherence, with no loss in speech reconstruction and no change to inference cost. It also shows that baseline codecs with quite different reconstruction quality yield near-identical token-LM perplexity and near-chance SALMon scores. Because neither the objective choice nor the prediction horizon changes the outcome, and the LM comparison is not matched, the result is best read as a promising data point on tokenizer learnability rather than a validated recipe.

Claims

  • supports: Fine-tuning a speech codec with language-model-facing objectives, without changing its architecture, can make its tokens much easier for a token LM to model.

    Evidence: LMs trained on LLM-Codec tokens reach 61.6% SALMon accuracy against 49.4% for the unmodified AUV codec, and token-LM perplexity falls from 159,768 to 4,617, with identical codec parameter counts. (§5.2, Table 1, Table 8, Figure 2)

  • supports: Codec reconstruction quality does not predict how learnable its tokens are for an autoregressive LM.

    Evidence: Four baseline codecs whose speech reconstruction ranges from Mel 0.762 to 1.133 and PESQ 1.547 to 2.208 all reach SALMon 48.3% to 50.1% and token-LM perplexity 148,122 to 159,768 under the same LM recipe. (§5.2, §5.4, Table 1, Table 2, Table 8)

  • refines: Learnability gains from LM-facing codec regularization appear to come from the presence of LM gradients rather than from a specific objective or a multi-step prediction horizon.

    Evidence: FTP alone, SA alone and both together give SALMon 61.8 / 61.3 / 61.6 and perplexity 4,631 / 4,616 / 4,617; FTP horizons K = 1 to 10 give SALMon 61.4 to 61.6. (§5.5, §6, Table 3, Table 4)

  • complicates: Reconstruction changes measured after LM-oriented codec fine-tuning can come from the fine-tuning recipe rather than from the LM-facing objectives, and outside speech they are mixed.

    Evidence: All ablation variants improve speech Mel distance by about 5% over AUV and differ from each other by at most 0.002; on music AUV keeps higher PESQ (2.195 vs. 2.175) and F0 correlation (0.785 vs. 0.767), and on environmental audio higher PESQ (1.567 vs. 1.562). (§5.3, §5.5, Table 2, Table 3, Table 13)

  • complicates: Token-LM benchmark gains for a new speech tokenizer depend strongly on how the LM’s speech-token embeddings are initialized.

    Evidence: Initializing LLM-Codec’s audio-token embeddings randomly instead of from the LLM’s text embedding space reduces SALMon accuracy from 61.6% to 50.7%, close to the 48.3% to 50.1% range of the baselines. (§5.2)

Limitations and Open Questions

Warning

The token-LM comparison is not matched across tokenizers. LLM-Codec’s LM starts from a checkpoint whose audio-token embeddings were already trained for 25k steps through the frozen LLM and initialized from text embeddings, while baselines expand the vocabulary only at the start of LM training. Random embedding initialization alone brings LLM-Codec down to 50.7% on SALMon. The SALMon and perplexity gaps may therefore partly reflect embedding pretraining rather than token properties.

  • Objective invariance. FTP-only, SA-only and every K give the same learnability, so the evidence does not show that the specific FTP or SA designs are needed. The Limitations section states that per-component learnability is not ablated, yet Tables 3 and 4 report perplexity and SALMon for each variant; the paper does not reconcile the two.
  • Perplexity scale. Baseline perplexities (148,122 to 159,768) are above the 20,480-entry audio vocabulary of AUV, so after 3 epochs the baseline LMs predict audio tokens worse than a uniform distribution over the audio tokens would. The paper does not discuss why, which makes the 35-fold comparison hard to interpret.
  • No generation evaluation. The paper argues that the mismatch causes repetition, semantic drift and hallucination in generation, but it evaluates only likelihood-based coherence and perplexity. It reports no speech continuation, TTS, WER or listening test.
  • Narrow setup. One base codec (AUV), one frozen LLM (Qwen3-4B-Instruct), 100 hours of read speech (LibriSpeech train-clean-100), and 3 epochs of LM training. The authors note that conversational speech with disfluencies and overlap is untested.
  • Transcript dependence. SA needs paired transcripts, which the authors note limits it for untranscribed audio and non-speech domains.
  • Training cost. Training adds a frozen-LLM forward pass, discriminator updates and auxiliary heads; inference is unchanged.
  • Release. Code and model are announced for later release; availability at the time of writing is not reported.

Wiki Connections

  • Neural Audio Codec — fine-tunes an existing single-codebook codec with LM-facing losses and reports reconstruction on Codec-SUPERB-tiny against BigCodec, UniCodec, WavTokenizer and AUV.
  • Spoken Language Model — measures tokenizer learnability by LoRA-adapting an LLM on codec speech tokens and scoring speech coherence on SALMon.
  • AUV — the base codec that LLM-Codec fine-tunes; it is the main baseline for both reconstruction and token learnability.
  • BigCodec — a 50 Hz baseline codec that keeps the best speech PESQ and STOI but stays near chance on SALMon.
  • UniCodec — a baseline codec with the best overall and environmental-audio spectral metrics and near-chance SALMon accuracy.
  • WavTokenizer — baseline codec family (small, medium, large variants) used for reconstruction and token-LM comparisons.
  • Qwen3 — Qwen3-4B-Instruct is the frozen LLM that supplies FTP and SA gradients and the backbone of the evaluation token LMs.
  • Discrete Speech Token Inconsistency — studies the same token-instability symptom but attributes it to encoder receptive-field leakage; the paper positions its objective-mismatch diagnosis as complementary.
  • Codec-LM Co-design — prior codec-LM co-design work that changes codec structure and training; LLM-Codec instead keeps the architecture and changes only the objectives.
  • SpeechTokenizer — an alternative route to LM-friendly tokens through semantic and acoustic factorization with additional components, which LLM-Codec avoids.