arXiv · 2025 · Preprint
Allbert et al. · → Paper · Demo: ? · Code: ?
A large-scale production evaluation of five cascaded STT × LLM × TTS configurations for an AI interview system finds that STT component quality drives the largest performance differences, while automated quality metrics correlate only weakly with actual user satisfaction.
Problem
Cascaded voice AI systems combine STT, LLM, and TTS components sequentially, but which component dominates system quality in real-world, high-stakes deployments remains underexplored. Prior work studies components in isolation or in small controlled experiments, leaving open questions about component interaction effects and whether automated evaluation metrics can adequately substitute for direct user feedback in production conversational systems.
Method
The study evaluates five production configurations of the micro1 Zara AI interview system, varying STT (Google STT or Whisper), LLM (GPT-4.1, GPT-4o, or Groq2), and TTS (OpenAI TTS or Cartesia TTS). The modular architecture allows component swapping without altering others, enabling attribution analysis. Data covers over 5,000 AI-conducted job interviews (20-30 minutes each, drawn from a production system averaging 1,500 interviews per day), partitioned across the five configurations.
Evaluation uses a dual framework: an automated LLM-as-a-Judge component (Claude 3.5 Sonnet) scores interview transcripts on Conversational Quality (dialogue flow, response building, acknowledgement) and Technical Quality (skill alignment, logical progression, question clarity), each on a 1-10 scale; and post-interview candidate satisfaction (1-5 star ratings) provides direct user feedback. Statistical analysis uses Welch’s ANOVA for significance testing, Games-Howell post-hoc tests for pairwise comparison, and Pearson correlation to assess alignment between automated and user-reported scores.
Key Results
The Google + GPT-4.1 + Cartesia configuration (C3) achieves the highest scores across all automated metrics (CQ Overall: 8.78, TQ Overall: 8.57) and the highest user satisfaction (4.53/5). Post-hoc tests confirm it is significantly better than all other configurations on conversational and technical quality (Table 4). Component contribution analysis shows Google STT consistently outperforms Whisper STT by the largest margin across all metrics, indicating that transcription accuracy propagates substantially downstream. GPT-4.1 outperforms GPT-4o on conversational quality (8.66 vs. 8.33) and user satisfaction (4.53 vs. 4.29). Cartesia TTS outperforms OpenAI TTS by a smaller but systematic 0.1-0.2 points across all metrics.
The central cross-cutting finding is a weak correlation between automated LLM-judge scores and user satisfaction: most automated metrics show Pearson correlations below 0.11 with user star ratings. The automated metrics correlate strongly with each other internally but fail to capture the user experience factors that drive post-interview satisfaction. The exception is soft skills assessment, which shows a higher correlation with user ratings (0.53), suggesting perceived interpersonal qualities matter more to candidates than technical accuracy.
Novelty Assessment
The contribution is primarily empirical. The paper evaluates production configurations of an existing commercial system and applies standard statistical methods (Welch’s ANOVA, Games-Howell, Pearson correlation) to attribute component-level effects. The LLM-as-a-Judge evaluation framework applies an established methodology to voice-based interview transcripts rather than introducing a new evaluation approach. The study’s value lies in the production scale (5,000+ interviews), the real-world application domain, and the finding that automated quality metrics diverge substantially from user satisfaction. All components evaluated are proprietary commercial APIs, limiting direct reproducibility.
Field Significance
Low — the paper provides practical evidence that STT quality dominates downstream performance in cascaded voice AI pipelines, and that automated LLM-based quality scores do not reliably predict user satisfaction. These findings are narrowly scoped to a proprietary job interview application with temporal deployment confounds and limited TTS variety, constraining the strength of causal attribution. The work is useful as a cautionary data point for practitioners deploying cascaded STT-LLM-TTS systems who might otherwise rely solely on automated evaluation metrics.
Claims
-
supports: In cascaded STT-LLM-TTS pipelines, transcription quality has a larger effect on downstream conversational performance than LLM or TTS component selection.
Evidence: Switching from Whisper to Google STT produced larger effect sizes on all quality metrics than swapping LLM versions or TTS providers across 5,000+ production interviews in five configurations. (§4.5)
-
complicates: Automated LLM-based quality evaluation scores are insufficient proxies for user satisfaction in production voice-based conversational AI.
Evidence: LLM-as-a-Judge metrics for conversational and technical quality showed Pearson correlations below 0.11 with candidate post-interview star ratings, despite high internal correlations among the automated sub-metrics. (§4.6)
-
supports: TTS component selection has a systematic but secondary effect on user experience in production voice-based conversational AI, smaller than the effects of STT or LLM choices.
Evidence: Cartesia TTS outperformed OpenAI TTS by 0.1-0.2 points across all LLM-judge metrics; the Google+GPT-4.1+Cartesia configuration achieved the highest user satisfaction (4.53/5) and was significantly better than all other configurations in post-hoc tests. (§4.5, Table 3)
-
complicates: User satisfaction in voice-based conversational AI depends on factors beyond measurable conversational and technical quality metrics.
Evidence: The consistently weak correlation (below 0.11) between automated quality scores and user star ratings, combined with a higher correlation for soft skills assessment (0.53), suggests perceived interpersonal qualities and voice naturalness influence satisfaction independently of assessed question quality. (§4.6, §5)
Limitations and Open Questions
Warning
The five configurations were deployed in non-overlapping time windows rather than concurrently, introducing temporal confounds (seasonal variation, candidate pool changes) that prevent clean causal attribution of performance differences to component choice alone.
The study evaluates only two TTS options (OpenAI TTS and Cartesia TTS), and both are mostly paired with the same STT and LLM choices, limiting robust TTS isolation. A full factorial design would be needed for stronger TTS attribution. The findings are specific to job interview contexts; user expectations, interaction patterns, and success criteria differ substantially from customer service, healthcare, or general dialogue domains, so results may not transfer. User satisfaction ratings suffer from potential self-selection bias, since only candidates who chose to submit post-interview feedback are included.
Wiki Connections
- Evaluation Metrics — this paper validates an LLM-as-a-Judge scoring framework for voice-based conversational AI and demonstrates a systematic gap between automated quality scores and user satisfaction.
- Subjective Evaluation — candidate post-interview star ratings provide real human satisfaction data that reveals the limitations of automated evaluation as a substitute for direct user feedback in voice AI.
- Speech-to-Speech — the paper evaluates cascaded STT-LLM-TTS pipelines as end-to-end speech-to-speech conversational systems, attributing performance to individual component contributions.