IWSDS · 2026 · Workshop
Kenta Yamamoto et al. (SANKEN, University of Osaka) · → Paper · Demo: ? · Code: ?
Identifies which Big Five personality dimensions users can reliably perceive in first-encounter spoken dialogue, and shows that a corpus-derived three-trait compression is expressed more clearly and independently through prosodic speech cues than through text alone.
Problem
Dialogue systems are commonly designed to express a consistent personality, and most prior work adopts the full Big Five model (Extraversion, Openness, Agreeableness, Conscientiousness, Neuroticism) as the target representation because of its psychological validity. However, users do not reliably perceive all five traits in short, first-time interactions: third-party evaluations have repeatedly found that some traits (notably Extraversion) are easy to infer from brief exchanges while others are not. Prior systems that attempt to express all five traits therefore risk targeting dimensions that users cannot actually distinguish, and it has been unclear which traits are worth prioritizing or how the choice of modality (text versus speech) affects perceptibility.
Method
The study proceeds in three stages. First, the authors analyze third-party personality ratings (via the Japanese TIPI-J questionnaire) collected on the Hazumi multimodal Japanese Wizard-of-Oz dialogue corpus, applying principal component analysis (PCA) followed by Varimax-rotated exploratory factor analysis (EFA) to the five Big Five rating dimensions. This surfaces a compressed three-factor structure: EO (Extraversion + Openness), AC (Agreeableness + Conscientiousness), and N (Neuroticism).
Second, in a text-based dialogue experiment, GPT-4 (gpt-4-0613, temperature 0, deterministic) rewrites the system-side turns of ten Hazumi1911 dialogues under prompt-specified personality conditions, injecting trait adjectives into a fixed prompt template while keeping user turns and dialogue history unchanged. Experiment 1 controls each of the five Big Five traits individually (high/low); Experiment 2 controls the three composite traits (EO, AC, N) instead. Crowd-sourced workers then rate the resulting dialogues on the seven-point TIPI-J scale.
Third, in a speech-based dialogue experiment (Experiment 3), the same linguistic content is synthesized with the commercial VoiceText text-to-speech engine (female voice), and personality is additionally expressed through explicit prosodic/paralinguistic controls layered on top of the synthesized audio: backchannel insertion timing (high EO), turn-taking pause duration extended to 3.0s or shortened to 0.5s from a 1.0s corpus baseline (high/low AC), and filler insertion (“e-”, “sono-”) combined with alternating speech rate at 80%/110% of the corpus default (high/low N). Evaluators rate approximately 1.5-minute dialogue videos (audio plus two static icons, to avoid visual bias) using the same TIPI-J-derived EO/AC/N scores.
Key Results
PCA on third-party Big Five ratings shows the first three principal components explain 86% of total variance (49%, 22%, 15% respectively), and the resulting EFA loadings cleanly separate into the EO, AC, and N factors, with Extraversion-Openness correlating most strongly (r = 0.68).
In the text-based individual-trait experiment (Experiment 1), all five traits show statistically significant high/low separation on their own scale, but strong off-diagonal cross-trait correlations appear: manipulating Extraversion or Agreeableness also shifts Openness and Conscientiousness ratings. In the three-composite-trait text experiment (Experiment 2), EO and AC still separate significantly and show some mutual overlap, while N produces smaller, less consistent differences, indicating Neuroticism is hard to convey through text alone.
In the speech-based experiment (Experiment 3), each controlled trait (EO, AC, N) shows a significant high/low difference concentrated mainly on its own rated dimension, with markedly reduced cross-trait leakage compared to text, and Neuroticism in particular becomes clearly separable (control-rating correlation of 0.28 for N in speech versus weaker text-based correspondence), attributed to fillers, backchannels, and speech-rate variation.
Novelty Assessment
The paper’s contribution is empirical and conceptual rather than architectural: it does not propose a new TTS or dialogue-generation model, instead reusing an off-the-shelf LLM (GPT-4 prompting) and a commercial TTS engine. The genuine contribution is the empirically-grounded compression of the Big Five personality model into three perceptually salient composite dimensions for first-encounter spoken dialogue, and the systematic cross-modal comparison (text-only prompting versus prosody-augmented speech) showing that the harder-to-convey trait (Neuroticism) becomes reliably perceptible once paralinguistic cues are added. This reframes personality design for dialogue systems around what users can actually perceive rather than around the full trait taxonomy, and is a useful negative-to-positive contrast: text-only control is shown to leak across traits, while prosody-augmented speech control is shown to reduce that leakage for the hardest trait.
Field Significance
moderate — the paper provides an empirically-grounded simplification of personality representation for spoken dialogue systems and a controlled demonstration that specific prosodic manipulations (speech rate, pause duration, fillers, backchannels) can express affective/personality dimensions that resist text-only control. Its direct scope is narrow: a single language (Japanese), a single commercial TTS engine, and a small corpus-derived stimulus set, so the paper itself does not establish generality beyond this setting.
Claims
- supports: Perceived personality/style traits in short, first-encounter spoken dialogue are not uniformly distinguishable; a subset of composite dimensions captures most of the perceptual variance that a larger nominal trait taxonomy targets.
Evidence: PCA on third-party Big Five ratings of the Hazumi corpus found the first three principal components explain 86% of variance, and Varimax-rotated EFA produced three interpretable factors (EO, AC, N) rather than five independent ones (§3.2, Table 1, Table 2).
- supports: Prosodic and paralinguistic speech cues can convey affective/personality information that text-only generation cannot reliably express.
Evidence: In the speech condition, Neuroticism became significantly and independently perceptible via fillers, alternating speech rate, and pause manipulation, whereas the same trait showed weak, inconsistent separation in the text-only condition (§4.5, §5.2-5.3, Table 10, Table 11).
- complicates: Independently controlling multiple style or persona dimensions through prompt-based text generation is prone to cross-dimension interference, complicating claims of independent controllability for trait-conditioned generation.
Evidence: In Experiment 1, manipulating a single Big Five trait via GPT-4 prompting significantly shifted ratings on unintended traits (e.g., controlling Extraversion or Agreeableness also raised Openness and Conscientiousness ratings), visible in the off-diagonal correlations of Table 6 (§4.3, Table 6).
- refines: Reducing a high-dimensional control target to an empirically-derived compressed set of dimensions improves perceptual controllability and consistency compared with attempting to control every dimension of the original taxonomy independently.
Evidence: The three-trait (EO, AC, N) conditions in Experiments 2 and 3 produced significant high/low separations with lower cross-trait correlation than the five-trait condition in Experiment 1, in both text and speech modalities (§4.4-4.5, §5.2-5.3, Table 7, Table 8, Table 10, Table 11).
Limitations and Open Questions
The study is confined to a single language (Japanese), a single dialogue corpus (Hazumi), and a single commercial TTS engine (VoiceText); the authors themselves note that personality impressions may be influenced by the specific TTS system used and call for validation with different TTS engines in future work (§5.3). Personality control in the text condition relies on prompting a single proprietary LLM (GPT-4, version gpt-4-0613) without fine-tuning, so findings about text-based controllability are tied to that model’s prompt-following behavior and may not generalize to other LLMs. The speech-based prosodic manipulations (backchannel timing, pause duration, filler insertion, speech-rate scaling) are binary high/low settings drawn from a prior study rather than a learned or continuously tunable control mechanism, and the stimulus set is small (ten dialogues, six controlled speech variations). The study also evaluates perception in a single first-encounter context and does not examine longer-term or repeated interactions.
Wiki Connections
- Emotion Synthesis — investigates how paralinguistic and prosodic cues (speech rate, fillers, backchannels, pause timing) shape listener perception of affective/personality qualities in synthesized dialogue speech, a closely related control target to expressive/emotional speech synthesis.
- Prosody Control — manipulates speech rate, turn-taking pause duration, filler placement, and backchannel timing as an explicit mechanism to convey personality traits independently of the underlying linguistic content and TTS voice.
- Subjective Evaluation — relies entirely on crowd-sourced human listener and viewer ratings using the validated TIPI-J instrument to assess perceived personality across both text-only and speech-based dialogue conditions.