arXiv · 2023 · Preprint
Touvron et al. (Meta AI) · → Paper · Demo: ? · Code: ?
LLaMA introduces a family of decoder-only transformer language models (7B to 65B parameters) trained exclusively on publicly available data, demonstrating that smaller models trained on more tokens can match or exceed the performance of much larger proprietary models.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
LLaMA provides the decoder-only autoregressive transformer backbone that TTS and spoken conversational agent systems adopt when integrating a general-purpose language model into speech generation pipelines. Its architectural choices (RMSNorm pre-normalization, SwiGLU activations, rotary positional embeddings) are explicitly cited by speech LM papers as the configuration they adopt or extend for text and token-sequence modelling. Systems in the spoken conversational agent and codec-language-model family cite LLaMA as the base model they fine-tune or extend to handle interleaved text and speech token sequences. The open release of weights across multiple scales (7B, 13B, 33B, 65B) enables the speech community to run controllable ablations at inference budgets previously inaccessible with proprietary models.
Wiki Connections
autoregressive-codec-tts | spoken-language-model
Cited by in-corpus papers: 2308.16692 (SpeechTokenizer uses LLaMA as background context for codec LM design) · 2305.11000 (SpeechGPT uses LLaMA-7B as its LLM backbone for cross-modal spoken dialogue)
2402.05755 — Spirit LM: Interleaved Spoken and Written Language Model