arXiv · 2026 · Preprint

Francesco Paissan et al. (Mila - Québec AI Institute / Université Laval / Concordia University) · → Paper · Demo: ✓ · Code: ?

Replaces the frame-aligned bottleneck of a frozen low-bitrate speech codec with a fixed set of learned, non-temporally aligned discrete slots, then shows that swapping importance-ranked slots between utterances shifts speaker identity and background noise without any task-specific editing model.

Problem

Frame-level codec tokens tie each code to a short temporal window, so global attributes such as speaker identity, background noise, or speaking style are spread across the whole token sequence. Isolating or editing one of these attributes requires finding which frames carry it, which is diffuse and hard. Image tokenizers (TiTok) and one audio tokenizer (ALMTokenizer) have compressed inputs into a small set of learned latent queries, but whether such slots specialize to interpretable global speech factors, and whether that structure can be used for direct intervention, had not been probed. The authors did not compare against ALMTokenizer because its code and checkpoints were not public.

Method

LATTE (Latent Audio Tokenizer for Token-space Editing) is built on top of FocalCodec, whose encoder (producing WavLM layer-6 features, 1024-dimensional, 50 Hz) and waveform decoder stay frozen throughout. Between them, the authors insert a learned compressor-quantizer-decompressor chain:

  • Compressor. L learned latent queries (L = 250 for a 5 s chunk, i.e. 50 tokens per second) are concatenated with the positionally embedded codec feature sequence and passed through a shared self-attention encoder. Only the outputs at the query positions are kept, so every slot can pool information from the whole chunk and all reconstruction information must pass through these slots.
  • Quantizer. Each slot is discretized with Binary Spherical Quantization (BSQ) at d = 13 (8192-entry implicit vocabulary), giving 650 bit/s. BSQ contributes an entropy regularizer that encourages balanced code usage.
  • Decompressor. T learned mask embeddings plus positional embeddings are concatenated with the quantized slots, and the first T outputs are read out as reconstructed codec features, which the frozen FocalCodec decoder turns into a waveform.

The only training objective is mean-squared error in the frozen FocalCodec feature space plus the BSQ regularizer; there is no waveform-domain loss. Two asymmetric variants are trained on LibriTTS (train-clean-100, train-clean-360, and train-other-500) with a lighter encoder than decoder: LATTE BASE (base encoder, large decoder) and LATTE LARGE (large encoder, XL decoder). Parameter counts are not reported. Long utterances are handled at inference by feature-domain overlap-add over 5 s windows with 1 s overlap (§Appendix E).

LATTE turns frame-level codec features into compact latent slots for analysis and editing. A frozen FocalCodec front-end maps speech to acoustic-semantic features; a learned TiTok-style bottleneck keeps only quantized latent slots; importance scores identify slots associated with global factors, which can then be swapped between utterances for zero-shot speaker and noise edits.

Slot-importance scoring and swapping (§4). For a labelled factor (speaker, gender, accent, noise level), utterances are partitioned by label, the mean code vector per slot is computed for each partition, and a slot’s importance is the leading eigenvalue of the between-partition covariance of those means. Editing ranks slots by importance, selects the smallest top-ranked set whose normalized importance mass reaches a threshold gamma, and copies those slots from a target utterance into the source before decoding. No editing model is trained and there is no test-time optimization.

Key Results

Resynthesis (Table 1, all at or below 1.5 kbps; LATTE at 0.65 kbps, same as FocalCodec 50 Hz).

CodecLibriSpeech UTMOSLibriSpeech dWERLibriSpeech SimVoiceBank DNSMOSVoiceBank dWERLibri1Mix DNSMOSLibri1Mix dWERLibri1Mix Sim
FocalCodec4.052.1897.43.168.082.9327.8991.6
BigCodec (1.04 kbps)4.112.5598.53.1920.672.7553.2688.3
Stable Codec (0.70 kbps)4.324.9794.73.3320.322.9143.5290.0
Mimi (0.69 kbps)3.295.7396.03.0128.002.6549.1489.4
LATTE LARGE4.235.8297.43.2916.303.0339.0791.5
LATTE BASE4.207.0796.93.2619.512.9641.4491.7

LATTE LARGE ties FocalCodec on clean speaker similarity (97.4) and has the best Libri1Mix DNSMOS (3.03), but intelligibility is consistently worse than the codec it is built on: dWER is 5.82 vs. 2.18 on LibriSpeech, 16.30 vs. 8.08 on VoiceBank, and 39.07 vs. 27.89 on Libri1Mix. Stable Codec has higher clean UTMOS (4.32) and VoiceBank DNSMOS (3.33); BigCodec has higher clean similarity (98.5) and lower clean dWER (2.55); SpeechTokenizer (5.14) and Stable Codec (4.97) also have lower clean dWER than LATTE LARGE. Note that the reported LATTE LARGE numbers use overlap-add chunking (§6.1), and baselines follow the FocalCodec evaluation protocol rather than being re-run.

Slot structure (Table 2, Figures 2 to 4). Importance profiles are non-uniform and factor-dependent. Noise profiles are the most stable (Spearman 0.735 between white-noise and WHAM! partitions, although top-5 and top-10 Jaccard overlaps are only 0.111 and 0.177). VCTK gender is the most concentrated profile (entropy 4.04, Gini 0.70), LibriTTS speaker identity the most diffuse (Gini 0.13). Speaker-related profiles transfer poorly across datasets (Spearman 0.308 LibriTTS vs. VCTK speaker) and overlap with accent and gender on VCTK. Cumulative-mass curves show no step-like single-slot-per-factor behaviour (Figure 3).

Denoising by noise-slot swap (Table 3). Replacing noise-important slots with those of an unmatched clean utterance raises DNSMOS and lowers dWER relative to no-swap resynthesis: on VoiceBank, DNSMOS goes from 3.29 to 3.50-3.60 and dWER from 16.30 to 9.54-11.61; on Libri1Mix, DNSMOS goes from 3.03 to 3.14-3.41 and dWER from 39.07 to 30.77-36.07. Speaker similarity drops in every swapped setting (e.g. Libri1Mix 91.5 to 86.3 at gamma 0.10 with WHAM! importance). No dedicated enhancement baseline is compared.

Parallel voice conversion by speaker-slot swap (Table 4, VCTK). Swapping 3 to 5 speaker-important slots (gamma 0.05 to 0.10) yields UTMOS 4.14-4.16, dWER 11.24-12.53, and similarity 89.5-90.0. Random and least-important slot controls at the same budget (k = 5) reach only 67.8 and 68.1 similarity, with dWER 16.26 and 12.03. The codec baselines in the same table include higher similarity values than LATTE (WavLM6-KM 92.4, FocalCodec 92.2), while FocalCodec has dWER 21.27 and UTMOS 3.38; the paper does not describe how the plain codec baselines perform conversion in this table. The setup is parallel (same sentence for source and target), which the authors describe as intentionally favourable (§Appendix D). No human listening test is reported anywhere.

Novelty Assessment

The learned-query bottleneck itself is borrowed from TiTok and was already applied to audio by ALMTokenizer; LATTE’s architectural change is narrower: it inserts this bottleneck between a frozen FocalCodec encoder and decoder and trains it with a feature-space reconstruction loss only. The more distinctive contribution is the analysis-plus-intervention protocol: a partition-based slot-importance score and a cumulative-mass slot swap, with random and least-important controls that make the speaker-transfer result interpretable as evidence of slot localization. The reconstruction results show a clear intelligibility cost relative to the underlying codec, and the editing experiments are framed by the authors themselves as controlled probes rather than competitive VC or enhancement systems. Training data is small (LibriTTS only), and the direct ALMTokenizer comparison is absent.

Field Significance

Moderate. The paper provides evidence that a compact, non-temporally aligned discrete bottleneck in a speech codec develops position-specific, factor-dependent structure that can be exploited for training-free speaker and noise edits, at an intelligibility cost relative to a frame-aligned codec at the same bitrate. Its value is as a probe of codec latent structure and a demonstration of slot-level editing, not as a stronger codec or VC system.

Claims

  • supports: Compressing a speech utterance into a small set of learned, non-temporally aligned discrete slots yields positions with factor-dependent roles, so global attributes such as background noise and speaker identity concentrate in identifiable subsets of slots.

    Evidence: Partition-based importance profiles are non-uniform across the 250 LATTE slots, with noise profiles correlating at Spearman 0.735 across white and WHAM! noise and VCTK gender reaching Gini 0.70, whereas the same scoring applied to frame-level FocalCodec tokens gives profiles tied to time rather than fixed roles. (§6.2, Table 2, Figure 2, §Appendix B, Figure 5)

  • supports: Swapping importance-selected discrete codec tokens between utterances can transfer speaker identity or remove background noise without training an editing model.

    Evidence: On parallel VCTK, swapping 5 speaker-important slots gives target similarity 90.0 vs. 67.8 (random) and 68.1 (least-important) at the same budget; noise-slot swaps on VoiceBank lower dWER from 16.30 to as low as 9.54 and raise DNSMOS from 3.29 to up to 3.60. (§6.3, Table 3, Table 4)

  • complicates: Removing temporal alignment from a low-bitrate codec bottleneck trades word-level intelligibility for global structure, even when perceptual quality and speaker similarity are preserved.

    Evidence: At the same 0.65 kbps as FocalCodec, LATTE LARGE has dWER 5.82 vs. 2.18 on LibriSpeech test-clean, 16.30 vs. 8.08 on VoiceBank, and 39.07 vs. 27.89 on Libri1Mix, while matching or exceeding FocalCodec on UTMOS, DNSMOS, and similarity. (§6.1, Table 1)

  • complicates: Slot-level specialization in unsupervised compact codec bottlenecks is partial: correlated speaker attributes stay entangled and their slot assignments do not transfer stably across datasets.

    Evidence: LibriTTS and VCTK speaker profiles correlate at only Spearman 0.308, VCTK speaker importance overlaps with accent (Jaccard@5 0.667) and gender, and cumulative-mass curves never approach a step function. (§6.2, Table 2, §Appendix B, Figure 3)

Limitations and Open Questions

  • The voice-conversion evaluation is parallel only (source and target read the same sentence); non-parallel conversion is untested, and two codec baselines in the same table report higher speaker similarity than the swap results, with the baseline conversion procedure not described.
  • Slot importance requires labelled partitions of a dataset for each factor, and speaker profiles estimated on one dataset transfer poorly to another, so the editing procedure depends on the population used to score slots.
  • The LATTE components are trained on LibriTTS only, far less data than baselines such as WavTokenizer, Stable Codec, and Mimi; the authors note this may affect both reconstruction and how reliably slots specialize (§Limitations).
  • All quality judgments are automatic (UTMOS, DNSMOS, Whisper-based dWER, WavLM speaker verification); there is no human listening test.
  • Denoising is compared only against no-swap resynthesis, not against speech enhancement systems, and every noise swap lowers speaker similarity.
  • Code release is not stated, and model parameter counts are not reported.

Wiki Connections

  • Neural Audio Codec — Proposes a TiTok-style non-temporally aligned latent bottleneck on top of a frozen low-bitrate speech codec and benchmarks it against ten codec baselines.
  • Voice Conversion — Evaluates training-free one-shot parallel VC on VCTK by swapping importance-ranked latent slots, measuring target-speaker similarity against random and least-important controls.
  • Self-Supervised Speech — The frozen FocalCodec front-end supplies WavLM layer-6 features, which define LATTE’s reconstruction target space.
  • FocalCodec — Provides the frozen encoder, decoder, and evaluation protocol; LATTE replaces its frame-level bottleneck and is compared against it at the same bitrate.
  • ALMTokenizer — The closest prior learned-query audio tokenizer; cited as motivation but not compared because no public checkpoint was available.
  • Stable Codec — Low-bitrate baseline that exceeds LATTE on clean UTMOS and VoiceBank DNSMOS.
  • BigCodec — Low-bitrate baseline with higher clean speaker similarity and lower clean dWER than LATTE.
  • Moshi — Source of the Mimi codec used as a resynthesis and VC baseline.
  • WavTokenizer — Single-codebook low-bitrate baseline in the resynthesis and VC comparisons.
  • SpeechTokenizer — Semantic-distilled codec baseline in the resynthesis and VC comparisons.
  • EnCodec — Acoustic RVQ codec baseline at 1.5 kbps.