arXiv · 2024 · Preprint
Grattafiori et al. (Meta) · → Paper · Demo: ? · Code: ✓
Llama 3 is a publicly released herd of dense Transformer language models at 8B, 70B, and 405B parameters, pre-trained on 15T+ multilingual tokens and post-trained with SFT and DPO, with the 405B flagship matching GPT-4 quality on standard benchmarks; the paper also describes compositional multimodal extensions that add a speech encoder and adapter for ASR, AST, and spoken dialogue, plus Llama 3 embedding-conditioned text normalization and prosody modeling for streaming TTS.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Llama 3 is an open-weight decoder-only LLM widely used as a language model backbone in TTS and SCA systems. These systems adopt the Llama 3 family (primarily the 8B and 70B instruction-tuned variants) as the language-model component for autoregressive token prediction, instruction-following, and dialogue state tracking, benefiting from its 128K-token context window and strong multilingual capabilities. The paper itself describes a compositional speech integration pathway: a 1B-parameter Conformer speech encoder pre-trained with BEST-RQ self-supervision is connected to the frozen LLM via a 100M-parameter adapter, enabling spoken dialogue and ASR across 34 languages without modifying the LLM weights. Beyond the speech-understanding path, Llama 3 embeddings (drawn from layer 16 of the 8B model) are used to condition streaming text normalization and prosody modeling modules in a proprietary TTS system, demonstrating that intermediate LLM representations carry prosodic and contextual information useful for speech synthesis.
Wiki Connections
spoken-language-model · self-supervised-speech · multilingual-tts
In-corpus papers cited by this work: 2309.16609 · 2312.05187 · 1810.04805 · 2312.11805 · 2001.08361 · 2303.08774 · 2012.03411 · 2402.03300 · 2302.13971 · 2307.09288 · 2007.10310
Cited by: 2504.18425 (Kimi-Audio, Llama 3 as reference foundation LLM for SCA backbone)
2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction