arXiv · 2023 · Preprint
Nachmani et al. (Google Research) · → Paper · Demo: ✓ · Code: ✗
Spectron adapts a pre-trained LLM to perform spoken question answering and speech continuation by operating directly on mel spectrograms — bypassing discrete token quantisation entirely — and jointly training ASR, text continuation, and speech synthesis objectives in a single end-to-end pass.
Problem
Spoken dialogue systems have historically relied on cascades of independently trained ASR, NLU, and TTS components, each introducing latency and losing speaker and prosodic cues across module boundaries. Discrete-token spoken language models (AudioLM, GSLM, TWIST) address some of this by unifying generation in a single LM, but their quantisation step discards fine-grained acoustic information and requires separate tokeniser and vocoder components. A further gap exists in transferring the factual world knowledge stored in large text-pretrained LMs to the spoken domain: most spoken LMs either train a new LM from scratch or use text LMs purely as a warm-start, without tightly coupling text reasoning to speech synthesis in a single decoding pass.
Method
Spectron connects a pre-trained 600M-parameter Conformer speech encoder (Google USM, trained on 12M hours of web audio) with a pre-trained PaLM-2 prefix decoder LM (350M or 1B parameters). The speech encoder takes a mel spectrogram (128 mel channels, 12.5 ms frame rate) as input and projects its output into the LM’s embedding space via a learned linear layer. No cross-attention between encoder and decoder is used; the encoder’s representations are simply prepended as a prefix to the LM’s input sequence.
At training time, each utterance is split at a 3-second boundary into a prompt segment and a continuation segment. The LM is teacher-forced to produce three things in sequence: the transcript of the prompt, the text continuation of that transcript, and then the spectrogram frames of the continuation speech. Two lightweight MLP modules (pre-net and post-net) project spectrogram frames into and out of the LM’s token dimension, creating a bottleneck that prevents degenerate repetition. The training objective combines cross-entropy losses on the ASR and text continuation outputs with a spectrogram regression loss. The regression loss applies combined L1 and L2 penalties not only to the raw spectrogram but also to its first-order feature-direction deltas and temporal deltas up to order K=3 (the “derivative loss”), encoding longer-range temporal shape information into the supervision signal.
At inference, the model first generates the prompt transcript and text continuation autoregressively in the text token space, then uses those text predictions as an intermediate “scratchpad” before generating spectrogram frames one step at a time. A frozen WaveFit vocoder converts the predicted spectrogram to a waveform.
Key Results
On LibriSpeech test-clean speech continuation (3-second prompt), Spectron (350M) achieves log-perplexity of 126.08, outperforming AudioLM (3-RVQ: 138.96, 12-RVQ: 140.28), GSLM (296.99), TWIST-1.3B (229.53), TWIST-7B (170.81), and SpeechGPT (136.42). Naturalness MOS is 3.68, comparable to AudioLM 3-RVQ (3.61) and below AudioLM 12-RVQ (3.87), but above GSLM (3.13), TWIST (3.03–3.28), and SpeechGPT (3.38). Speaker similarity is 0.42, the highest among all compared systems (AudioLM: 0.35–0.37, GSLM: 0.11, TWIST: 0.23–0.24; SpeechGPT does not target speaker preservation at 0.05).
On spoken QA, Spectron (1B) achieves 22.9% accuracy on LLaMA-Questions and 6.1% on WebQuestions, essentially matching SpeechGPT-7B (21.9% / 6.5%) at one-seventh the LM parameter count — and doing so zero-shot without instruction fine-tuning. TWIST models with 1–7B parameters largely fail at QA (0.5–1%), confirming that warm-starting from an LM alone is insufficient without tight audio-text coupling.
Ablations show that the text cross-entropy loss and the spectrogram derivative loss are the two most important components: removing either causes log-perplexity to increase by more than 580 points. Removing the pre-trained speech encoder or LM individually costs 75–87 perplexity points; removing both costs 118.
Novelty Assessment
The core novelty is the decision to bypass discrete tokenisation entirely and instead regress spectrograms directly from an LM decoder. Prior spoken LMs universally quantise audio into discrete units; Spectron demonstrates that continuous spectrogram frames can be modelled autoregressively within the same LM framework without quantisation artefacts. The derivative loss is a genuinely novel training objective for speech continuation: penalising temporal and frequency-domain derivatives of the spectrogram encourages the model to match longer-range signal structure beyond frame-level accuracy.
The architecture combination (Conformer encoder + prefix LM + MLP projection) is straightforward, and the text-as-scratchpad analogy to chain-of-thought is intuitive rather than deeply new. The model’s throughput is limited by frame-by-frame spectrogram decoding (12.5 ms per frame), which rules out real-time or streaming use. The evaluation is limited to English, read-speech audiobooks, and relatively short continuations; generalisation to conversational or spontaneous speech is untested.
Field Significance
Moderate — Spectron provides early and clean evidence that discrete token quantisation is not a prerequisite for spoken language modelling: a pre-trained LM can generate continuous spectrogram representations directly and match or exceed discrete-token systems on semantic quality and speaker preservation. This contributes a useful data point to the discrete-vs-continuous design debate in audio LMs, though the approach’s high inference latency and frame-level autoregression have limited its direct adoption compared to codec-based successors.
Claims
- Continuous spectrogram representations can support spoken language modelling at quality levels competitive with discrete-token systems, without requiring a separate quantisation stage. (§4.2.1, §4.2.2, Tables 1–3)
- Jointly supervising ASR, text continuation, and speech synthesis within a single decoding pass preserves LLM world knowledge better than separate components or warm-start initialisation alone. (§4.2.3, Table 4)
- Penalising temporal and spectral derivatives of the predicted spectrogram improves semantic coherence in speech continuation more than the basic regression loss alone. (§4.2.4, Table 5)
- A compact LM (1B parameters) that tightly couples audio encoding with text decoding can match a 7B instruction-tuned spoken LM on zero-shot spoken question answering tasks. (§4.2.3, Table 4)
Limitations and Open Questions
Warning
Frame-by-frame spectrogram decoding at 12.5 ms resolution makes generating long utterances computationally prohibitive and precludes streaming inference. The paper explicitly flags parallelised decoding as future work but provides no solution.
The evaluation is restricted to English read-speech from LibriVox, using relatively short 3-second prompts. Performance on conversational, spontaneous, or noisy speech is not assessed. Comparison fairness is complicated by differences in LM size and training data across baselines; the 350M Spectron LM is considerably smaller than TWIST-7B or SpeechGPT-7B, making direct N-MOS comparisons informative but not perfectly controlled. The spoken QA accuracy numbers are low across all systems, suggesting the task formulation (zero-shot continuation without explicit prompt engineering) is extremely challenging for all current spoken LMs.
Wiki Connections
- spoken-language-model — Spectron proposes a continuous-spectrogram alternative to discrete-token spoken LMs, directly competing with AudioLM and GSLM on semantic coherence and speaker preservation benchmarks.
- evaluation-metrics — the paper introduces two spoken QA benchmarks (LLaMA-Questions and the WebQuestions spoken adaptation) and uses log-perplexity, N-MOS, and speaker similarity as a combined evaluation suite for speech continuation.
- speech-to-speech — the system performs end-to-end speech-in, speech-out processing for both continuation and QA tasks, situating it within the broader speech-to-speech and spoken conversational agent landscape.
- 2301.11325 (MusicLM) — cited as a related generative audio LM; Spectron shares the goal of direct audio generation from learned representations but focuses on speech rather than music.
- 2210.13438 (EnCodec) — cited as a representative discrete audio codec approach; Spectron’s continuous spectrogram approach is positioned as an alternative to codec-based tokenisation pipelines.
- 2302.13971 (LLaMA) — used as the basis for SpeechGPT in the baseline comparisons; Spectron uses a PaLM-2 LM and demonstrates comparable QA accuracy at lower parameter count than the LLaMA-7B-based SpeechGPT.