IWSDS · 2026 · Workshop
Emerson et al. (National Board of Medical Examiners / Ho Chi Minh City University of Foreign Languages) · → Paper · Demo: ? · Code: ?
A pilot study evaluating Amazon Nova Sonic, a commercial speech-to-speech foundation model, as a virtual standardized patient for medical student interview practice, providing the first empirical assessment of a speech-to-speech model in medical education.
Problem
Standardized patient (SP) programs, where trained actors portray clinical scenarios for medical students, are effective for developing clinical interviewing skills but are expensive to scale: recruiting and training human actors is costly, scheduling is inflexible, and maintaining consistent portrayal of a case across many encounters is a quality-assurance burden. Prior attempts to automate virtual patients have relied on text-based chatbots, which strip away the vocal and prosodic cues that make clinical interviews feel realistic and require a separate speech interface layered on top. Speech-to-speech foundation models, which process spoken input and generate spoken output directly rather than through a cascaded ASR-LLM-TTS pipeline, are a candidate for closing this gap, but no prior work had evaluated one in a medical education setting.
Method
The system is a virtual standardized patient (VSP) built directly on Amazon Nova Sonic, a commercial speech-to-speech model accessed through the AWS Bedrock API. The authors treat Nova Sonic as a fixed, closed component: no fine-tuning, architecture modification, or training of any kind is performed. Persona control is implemented entirely through prompting: each of four clinical cases is defined by a shared general system prompt (patient-role guidelines, instructions against ever acting as an assistant or clinician, rules for withholding unsolicited information, and a scripted opening-turn policy) combined with case-specific details (symptoms, history, behavioral traits). Nova Sonic was selected primarily for its reported bidirectional streaming support, which the authors argue improves handling of conversational turn-taking relative to standard request-response voice interfaces, and for its competitive latency and recognition/generation quality as reported in its technical documentation. The application front end is a TypeScript browser client that calls the Nova Sonic API directly; there is no intermediate ASR-then-text-LLM-then-TTS pipeline.
The evaluation is a controlled pilot study rather than a system-development exercise. Twenty third-year medical students (of 24 recruited) each completed pre-encounter diagnostic planning followed by a voice interview with the VSP across four fixed clinical cases (shortness of breath, vomiting, weakness, sleep difficulty), yielding 75 valid recorded interviews after excluding 5 sessions lost to connection errors. Conversation logs (transcripts, turn counts, word counts, durations) were captured automatically, and students completed a post-study survey with 7-point Likert items on realism, naturalness, effectiveness, and engagement; user-experience items on communication ease and error frequency; and comparative and open-ended items contrasting the VSP against human SP encounters.
Key Results
Conversational dynamics resembled real patient-physician interviews: the VSP produced roughly twice as many words per turn as students overall (21.0 vs. 12.2), consistent with patients giving narrative answers to physician questions, and this asymmetry held across all four cases (Table 1). On the 7-point Likert scales (N=20), students rated the system moderately across dimensions, with the highest ratings on engagement (M=4.95, SD=1.75) and effectiveness (M=4.95/4.89 depending on item, SD≈1.5), and somewhat lower ratings on communication ease (M=4.32-4.45, SD≈1.5-1.6) (Table 2). Half of students (50%) reported the VSP produced inappropriate or confusing responses at least “sometimes,” and half also reported forgetting at least once that they were talking to an AI system. When asked to compare directly against human SP encounters, 65% of students rated the VSP experience as less valuable, while only 15% rated it as more valuable; 60% said they would recommend the system to peers, mostly framing it as supplementary low-stakes practice rather than a replacement for human SPs. There is no comparison against a text-based or pipelined (ASR+LLM+TTS) baseline system within this study; the authors position their realism and engagement numbers only informally against ratings reported in separate prior LLM-chatbot virtual-patient studies.
Novelty Assessment
The contribution is entirely an engineering-integration and application study: Nova Sonic is used unmodified as a black box, controlled only through natural-language system prompts, with no new architecture, training procedure, or evaluation metric proposed. The value of the paper lies in being, as far as the authors are aware, the first controlled human-subject evaluation of a commercial speech-to-speech model in a specific applied setting (medical education), and in surfacing concrete, reusable failure modes (persona/role breaks, information over-sharing) that arise specifically from prompt-only control of a speech-to-speech dialogue agent rather than from the underlying model’s speech quality.
Field Significance
low — This paper does not advance speech-to-speech model design; it is a narrow, single-model, 20-participant pilot with no comparison group (text chatbot, cascaded ASR+LLM+TTS, or alternative speech-to-speech model), so it cannot isolate which of its findings are specific to Nova Sonic versus general to prompt-controlled speech-to-speech agents. It does surface two concrete failure modes of prompt-only persona control (character breaks after pauses or brief acknowledgments, over-disclosure of unsolicited information) and shows that a commercial speech-to-speech model’s prosody can be convincing enough that users intermittently forget they are talking to an AI, even while other realism dimensions remain only moderately rated. These observations are plausibly relevant beyond this specific medical-education application, but the study’s narrow scope and lack of any baseline keep its evidentiary weight low.
Claims
-
supports: Speech-to-speech conversational agents, when their turn-generation is shaped only by role-play system prompts, can reproduce realistic turn-length asymmetries characteristic of the target human-human interaction they are simulating.
Evidence: The virtual standardized patient produced nearly twice as many words per turn as students overall (21.0 vs. 12.2 words/turn), mirroring the narrative-response pattern typical of real patient-physician interviews, and this asymmetry held consistently across all four clinical cases. (§4.1, Table 1)
-
complicates: Persona control implemented purely through natural-language system prompting, without any fine-tuning or structural constraint, is vulnerable to role/character breaks in a speech-to-speech dialogue agent during natural conversational pauses or brief user acknowledgments.
Evidence: The most commonly cited limitation across student feedback was the agent breaking patient character and asking about symptoms rather than answering them; the included example transcript shows the agent switching into a diagnostic/clinician register (“Let’s see what we can do to help you feel better… we’ll start with some medications”) after a student’s brief “okay” acknowledgment. (§4.3, §5, Appendix B)
-
complicates: Prompt-only conditioning of a speech-to-speech dialogue agent for a controlled-information-release task can cause the agent to volunteer more information than was explicitly solicited, undermining tasks that depend on the user actively eliciting information.
Evidence: Students identified information over-sharing as a recurring limitation, reporting that the agent provided “pertinent positives and negatives that I didn’t solicit,” which the authors note undermines the diagnostic-questioning pedagogy the interview task is meant to train. (§4.3, §5)
-
supports: Users interacting with a speech-to-speech conversational agent can lose track that they are speaking with an AI system even when their explicit realism ratings for that same system are only moderate.
Evidence: Despite a moderate mean realism rating (M=4.55, SD=1.39) on a 7-point scale, half of the 20 participants (50%) reported forgetting at least once during the study that they were interacting with an AI system, which the authors attribute in part to the naturalness of the agent’s vocal inflection and tone. (§4.2, Table 2)
Limitations and Open Questions
The study is a single pilot with 20 medical students from U.S. medical schools using one commercial model (Nova Sonic) on one application; the authors explicitly did not measure learning outcomes or systematically evaluate the VSP's clinical accuracy or persona consistency, so the reported realism, engagement, and role-confusion findings should not be generalized beyond descriptive, exploratory evidence for this specific system and population.
There is no comparison group using a text-based chatbot, a pipelined ASR+LLM+TTS system, or a different speech-to-speech model, so the paper cannot isolate which of its findings (realism, role confusion, over-disclosure) are specific to Nova Sonic versus general to speech-to-speech or prompt-controlled conversational agents. All four clinical cases were completed in a fixed order, which may confound perception-change results across cases with simple order effects. Nova Sonic’s internal architecture, training data, and model size are not disclosed by the authors and are treated entirely as a black box.
Wiki Connections
- Speech-to-Speech — evaluates a commercial dialogue-oriented speech-to-speech foundation model (Amazon Nova Sonic) in an applied conversational setting, contrasting it against cascaded ASR-then-LLM-then-TTS pipelines for latency and naturalness.
- Subjective Evaluation — collects real human Likert-scale ratings from 20 medical students on realism, naturalness, effectiveness, and engagement, plus open-ended qualitative feedback, as its primary evaluation method.
- Evaluation Metrics — administers a structured post-study survey instrument built on established standardized-patient and conversational-agent assessment frameworks to systematically assess a speech-to-speech dialogue agent, in the absence of any automated quality metric.