COLING Industry Track · 2025 · Workshop
Kirmayr et al. (BMW Group Research and Technology / University of Augsburg / Technical University of Munich) · → Paper · Demo: ? · Code: ✓
A category-bounded long-term memory system for LLM-powered in-car voice assistants that restricts preference extraction to predefined hierarchical categories, paired with a synthetic benchmark dataset (CarMem) for evaluating extraction, maintenance, and retrieval components.
Problem
LLM-based voice assistants are stateless by nature, yet useful personalisation requires retaining user preferences across sessions. Prior external memory systems rely on unstructured extraction, storing arbitrary text snippets without user-visible organisation. This creates two industry problems: first, such systems capture irrelevant or non-actionable preferences that exceed what a vehicle’s constrained action space can actually use; second, opaque extraction raises privacy and regulatory concerns under frameworks such as the GDPR and the EU AI Act, which mandate data minimisation and user transparency. Existing dialogue personalisation datasets are either open-domain, limited to single sessions, or focus on the assistant persona rather than user preferences, leaving no suitable benchmark for long-term in-car preference memory systems. (§1, §2)
Method
The system manages user preferences through three stages: extraction, maintenance, and retrieval. Extraction uses LLM function calling (GPT-4o at temperature 0) with a pydantic-defined parameter schema that mirrors a two-level hierarchy of predefined categories, ranging from 4 main categories (such as Points of Interest, Navigation, Vehicle Settings, Entertainment) down to 41 detail categories. Because every parameter in the schema is typed as Optional, the LLM extracts only preferences for which a matching category exists; out-of-category information is discarded. Users may opt out of specific subcategories, which simply removes those parameters from the schema presented to the model.
Maintenance follows the approach of Bae et al. (2022), extended with LLM function calling. Before a new preference is stored, it is compared against existing preferences in the same detail category. Three outcomes are possible: pass (duplicate), update (contradiction or revision), or append (genuinely new). Categories that permit only a single preference disable the append function when one already exists, preventing accumulation of contradictory states.
Retrieval generates embeddings from a concatenation of the detail category name, the category attribute, and the sentence in which the user revealed the preference. At inference time, the system retrieves the top-k preferences by cosine similarity to the current user utterance, where k is set dynamically to the number of preferences stored for that user within the relevant subcategory.
The CarMem dataset provides ground truth for all three components. It contains 1,000 extraction conversations, 1,000 retrieval utterances, and 3,000 maintenance utterances generated by GPT-4-1106-preview (temperature 0.7) using 100 synthetic user profiles with varying age, technological proficiency, location, and conversation style. Profiles were drawn from categories based on real in-car assistant functionality, and each profile is assigned 10 preferences sampled across the detail-category level. (§3, §4)
Key Results
On the 500-entry test split of CarMem, preference extraction achieves F1 of 0.94 at the main-category level (4 categories), 0.90 at subcategory level (11 categories), and 0.78 at detail-category level (41 categories), with 99% valid structured outputs in both In-Schema and Out-of-Schema conditions. When a subcategory is excluded from the schema (simulating user opt-out), the system correctly withholds extraction 75% of the time, demonstrating meaningful category boundedness, though over-extraction still occurs at rates of 12–25% depending on condition. (§5.1, Tables 2–3)
Maintenance reduces redundant preferences by 95% (equal preferences correctly passed or updated) and contradictory preferences by 93% (negated preferences correctly updated). A 2% loss rate remains from incorrect passes in multi-preference categories. (§5.2, Table 5)
Enriched retrieval embeddings (detail category + attribute + sentence) achieve 0.87 top-n accuracy compared to 0.75 for sentence-only embeddings, a 12-point improvement attributable to the structured category context disambiguating semantically vague utterances. (§5.3, Table 6)
Novelty Assessment
The contribution combines two existing techniques — LLM function calling for structured extraction and RAG-style embedding retrieval — in a domain-specific configuration for automotive voice assistants. The maintenance logic adapts Bae et al. (2022) without structural changes. The category hierarchy itself is the primary design decision, and it is domain-specific rather than general. The CarMem dataset is the most concrete contribution: no prior benchmark covers multi-session in-car preference memory with extraction, maintenance, and retrieval splits simultaneously. The evaluation is thorough within its domain but is bounded entirely by a synthetic dataset generated by the same class of model being evaluated, which limits the independence of the benchmark.
Field Significance
Moderate — this paper addresses a practical gap in spoken conversational agent infrastructure: how to retain user preferences across sessions in a privacy-compliant way. Its engineering value lies in combining category-bounded extraction with structured LLM function calling, a pattern that is transferable to other constrained-domain voice assistant settings. The CarMem dataset provides a reusable benchmark, though its synthetic origin and automotive specificity limit how broadly the numbers generalise.
Claims
-
supports: Structured, category-bounded memory extraction reduces both redundant and contradictory preference storage compared to unstructured extraction in conversational agents.
Evidence: The category-bounded system achieves a 95% reduction in redundant preferences and a 92% reduction in contradictory ones on the CarMem maintenance benchmark; hierarchical schema constraints prevent out-of-category storage entirely. (§5.2, Table 5)
-
complicates: LLM-based preference extraction from conversational context degrades significantly as category granularity increases.
Evidence: F1 drops from 0.94 at the 4-category main level to 0.78 at the 41-category detail level, with over-extraction rates of 12–15% in the In-Schema condition driven by confusion among semantically similar categories. (§5.1, Tables 2–3)
-
supports: Enriching retrieval embeddings with structured metadata (category label and attribute) substantially improves preference recall in personalised spoken conversational agents.
Evidence: Concatenating the detail category and attribute to the sentence-of-preference embedding raises top-n retrieval accuracy from 0.75 to 0.87 on CarMem, with the gain explained by category names disambiguating vague user utterances. (§5.3, Table 6)
-
complicates: Synthetic benchmarks evaluated on in-domain synthetic data overestimate system robustness for real-world deployment in spoken conversational agents.
Evidence: The CarMem dataset is generated by GPT-4 and evaluated against GPT-4o, omitting speech recognition errors and multi-preference conversations that are common in real-world in-car interactions; the authors acknowledge this in the limitations. (§7)
Limitations and Open Questions
Warning
The CarMem dataset contains exactly one preference per conversation and is entirely synthetic, generated and evaluated with GPT-class models. No real user data is used at any stage, and no automatic speech recognition errors are simulated, which likely inflates extraction and maintenance accuracy relative to real deployment conditions.
The dataset represents only two interaction sessions per user, preventing any evaluation of long-term preference drift or temporal decay. Category design is specific to automotive functionality; the approach’s generalisability to other domains (smart home, personal assistants) is asserted but untested. Over-extraction at detail level (12–25%) remains a meaningful error rate for a production system where incorrect preference storage is user-visible.
Wiki Connections
- Spoken Language Model — this paper addresses memory and personalisation infrastructure for LLM-based spoken conversational agents, a capability gap orthogonal to speech generation quality.
- Evaluation Metrics — the CarMem benchmark introduces task-specific metrics (extraction F1 by category level, maintenance function accuracy, retrieval top-n accuracy) for evaluating preference memory components in voice assistant systems.