ACL · 2025 · Conference
Sarah E. Finch et al. (Emory University) · → Paper · Demo: ✗ · Code: ✓
A user study comparing text-based and spoken chatbots using African American English (AAE) dialect and accent reveals that while text AAE chatbots underperform SAE baselines across all metrics, spoken chatbots with an African American voice (even with SAE text) significantly improve warmth, similarity, and engagement preference for AAE-speaking users.
Problem
Chatbot personalization through linguistic similarity is poorly understood for dialect variation within a single language. Prior work on LLM dialect generation has used tweet datasets poorly suited to multi-turn dialogue, and accented TTS for demographic groups has not been evaluated with the intended user population. It is unknown whether AAE dialect in text or accent in speech helps or harms chatbot performance with African American users, or how AAE intensity interacts with user experience.
Method
The study develops two types of chatbots using a two-step pipeline:
Text Chatbots: 100 SODA dialogues (5 domains: Customer Assistance, Commerce, Healthcare, Education, Social Companionship) are translated from SAE to three levels of AAE intensity (Low/Medium/High) using three LLMs (Llama3.1-70B, Claude-Sonnet-3.5, GPT-4o) via SAE→AAE translation prompts targeting a middle-aged African American woman persona. This yields 9 AAE Text Chatbots + 1 SAE baseline.
Spoken Chatbots: F5-TTS (a non-autoregressive diffusion-transformer TTS system based on flow-matching) converts dialogue turns to speech. African American accent: a reference clip from CORAAL (file ATL_se0_ag2_f_02_1). SAE accent: a LibriSpeech clip. Each dialogue turn is independently synthesized, then concatenated with brief inter-turn pauses. 4 Spoken Chatbots (SAE/Low/Medium/High AAE dialect) all use the AA accented voice; 1 SAE baseline uses a Standard American voice.
Evaluation: 12 AAE-speaking university evaluators assessed Text Chatbots; 8 assessed Spoken Chatbots on 15 metrics via 5-point Likert scales covering comprehension, warmth, inoffensiveness, trustworthiness, similarity, communication ease, role appropriateness, engagement preference, dialect expression, fidelity, grammaticality, speech naturalness, and clarity.
AAE feature tagging (by Claude-Sonnet-3.5, 91% accuracy on held-out test set) validated that AAE expression scales monotonically with prompt intensity; phonetic features dominate, semantic features are sparse.
Key Results
Text Chatbots: The SAE baseline outperforms all 9 AAE variants on all characteristics. As AAE intensity increases, performance degrades monotonically, with High-AAE showing near-neutral or negative scores for trustworthiness, role appropriateness, and engagement preference. High-AAE also receives the worst inoffensiveness scores — likely due to excessive phonetic modifications producing caricature-like representations rather than authentic AAE.
Spoken Chatbots: The best configuration is AA accent + SAE dialect text, which outperforms the SAE baseline across all evaluated dimensions (comprehension, communication ease, similarity to self, role appropriateness, engagement preference). Low-AAE dialect with AA accent also shows improved warmth and similarity vs. baseline. High-AAE with AA accent degrades similarly to the text condition. Speech clarity decreases as AAE expression increases.
Claude-based chatbots produce the most syntactically authentic AAE; GPT shows the least variation between Low and Medium levels; Llama produces the most phonetic modifications but is least balanced across linguistic categories.
Novelty Assessment
Primarily an HCI/SCA paper that uses TTS as a component. The finding that voice accent improves chatbot reception while text dialect degrades it is counterintuitive and practically important for conversational AI design. The controlled multi-turn evaluation with real AAE speakers fills a gap in prior work. The TTS contribution is limited to deployment of an existing system (F5-TTS) with a reference clip; no synthesis methodology is developed.
Field Significance
Low — This paper provides confirmatory evidence that acoustic persona (voice accent) and lexical dialect have asymmetric effects on user acceptance of spoken chatbots, a finding relevant to spoken conversational agent design but peripheral to core speech synthesis research. The evaluation relies on a small pool of university-student evaluators (8–12 participants) and uses F5-TTS as an off-the-shelf component without modification, so the contribution is primarily observational rather than technical.
Claims
- Incorporating an acoustic accent that matches a user’s linguistic community improves chatbot warmth, similarity, and engagement preference even when the underlying text remains in standard dialect. (§4.4.2, Figure 4)
- Text-based dialect generation at high intensity produces caricature-like representations that degrade perceived inoffensiveness and role appropriateness with speakers of that dialect. (§4.4.1, Figure 3)
- Phonetic modification dominates LLM-generated AAE output, while semantic feature generation remains sparse across all tested models and intensity levels. (§4.3, Figure 2)
- TTS models trained predominantly on standard-dialect data exhibit degraded speech clarity when reproducing minority accent features, limiting the fidelity of accent-conditioned spoken chatbots. (§5)
Limitations and Open Questions
- Small evaluator pool (12 text, 8 spoken) from a single university demographic.
- Static evaluation (third-party observation) may not capture real-time conversational effects.
- F5-TTS was trained predominantly on SAE data; its ability to reproduce AAE phonetics authentically is limited, likely contributing to reduced speech clarity.
- No objective acoustic analysis of the AA accent output quality vs. a ground-truth AA speaker.
- Evaluator overlap between text and spoken conditions was partial; the paper acknowledges this but validates that core findings hold for overlapping participants.
- Open question: does authentic AAE voice (trained on broader AA speech corpora) further improve user acceptance?
Wiki Connections
- spoken-language-model — the paper evaluates spoken chatbot performance in a specific demographic context, providing multi-metric human evaluation evidence for spoken conversational agent design
- F5-TTS — used as the TTS backbone for all spoken chatbot conditions; the paper’s synthesis capability depends entirely on this system
- flow-matching — F5-TTS is a flow-matching diffusion-transformer model; the paper’s spoken chatbot pipeline inherits this architecture
- zero-shot-tts — accent conditioning is achieved via a single reference clip from CORAAL, a direct application of zero-shot speaker adaptation
- subjective-evaluation — the evaluation design (15 Likert-scale metrics, AAE-speaking evaluators, static dialogue assessment) is an example of speaker-centred subjective evaluation methodology
- evaluation-metrics — the 15-metric evaluation framework introduced covers comprehension, warmth, inoffensiveness, trustworthiness, and engagement preference dimensions