arXiv · 2023 · Preprint
Touvron et al. (Meta) · → Paper · Demo: ? · Code: ?
Llama 2 is a family of open-weight autoregressive transformer LLMs (7B–70B parameters) pretrained on 2 trillion tokens and then aligned via supervised fine-tuning and RLHF to produce Llama 2-Chat, an open-release dialogue-optimised model competitive with closed-source chat systems.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Llama 2 provides a permissively licensed, openly released decoder-only transformer backbone that text-to-speech and spoken conversational agent systems can adopt as a language model component. Its 7B and 13B variants offer a practical balance between capability and compute cost, making them suitable for integration into hybrid TTS architectures where a large language model drives semantic token generation or instruction following. The Llama 2-Chat variant introduces an RLHF alignment procedure (combining rejection sampling and PPO) that serves as a reference implementation for aligning dialogue systems with human preferences, a pipeline relevant to spoken conversational agent development. The grouped-query attention mechanism introduced at the 34B and 70B scales, combined with a 4k-token context window, enables longer conversational context than its predecessor, supporting multi-turn spoken dialogue generation.
Wiki Connections
- spoken-language-model — Llama 2’s open-weight decoder-only backbone is used directly by speech LM and spoken conversational agent systems as the language model component
- rlhf-speech — the rejection-sampling + PPO alignment pipeline introduced in Llama 2-Chat is a reference implementation that speech preference optimization work adapts
- 2001.08361 — cited as an early scaling study informing Llama 2’s training setup
- 1711.05101 (AdamW) — optimizer used for pretraining
- 2303.08774 (GPT-4) — cited as a closed-source performance ceiling in the chat evaluation
- 2302.13971 (LLaMA 1) — direct predecessor; Llama 2 extends the original Llama architecture with grouped-query attention and RLHF alignment