arXiv · 2026 · Preprint
Timo K. Koch et al. (University of St. Gallen / LMU Munich / University of Mannheim / Charlotte Fresenius Hochschule) · → Paper · Demo: ✗ · Code: ✓
A content-controlled, privacy-first smartphone protocol for collecting prosodic speech data in everyday life, using scripted valence-balanced read-aloud sentences to decouple prosody from semantics and on-device feature extraction with immediate raw-audio deletion, empirically evaluated on a 560-participant, 9,877-recording deployment.
Problem
Naturalistic, in-the-wild speech data is valuable for prosodic research (e.g., inferring affect or speaker traits from vocal delivery) because it captures real variation that controlled lab recordings cannot. But unconstrained field speech confounds two things that lab protocols traditionally separate: prosody (how something is said) and semantics (what is said), since what a person happens to say varies just as much as their vocal delivery. Prior lab-based solutions to this confound (fixed reading passages, phonetically balanced sentence lists, acted/read-speech corpora) have rarely been adapted for smartphone-based field collection with explicit control of lexical valence. Separately, in-the-wild speech collection has typically relied on storing raw audio, which is privacy-sensitive and creates legal exposure (e.g., under GDPR) at scale. No prior smartphone protocol combined content control for prosody-semantics decoupling with privacy-preserving, audio-free data transmission.
Method
The protocol is implemented as a module inside the PhoneStudy app’s ecological momentary assessment (EMA) system, which prompts participants multiple times per day. At each prompt, participants are shown three short sentences drawn (with replacement) from a validated bank of 54 German sentences with balanced emotional valence (positive, negative, neutral), read aloud with the order of the three valence conditions randomized per assessment. Recordings are bounded between 4 and 12 seconds, calibrated to the time needed to read three sentences at extreme fast/slow paces.
On the device, a four-step pipeline runs after each recording: local audio capture as uncompressed 16-bit/44.1kHz WAV, on-device acoustic feature extraction with OpenSMILE (an Android-native build) using both the 88-dimension eGeMAPS set and the 6,373-dimension ComParE 2016 set, immediate deletion of the raw WAV and intermediate CSV files, and SSL-encrypted batch transmission of only the feature vectors to a remote server when the device is idle and on WiFi. No raw audio or intermediate artifact persists on the device or leaves it. This design is deliberately not a generative or synthesis system: it is a measurement pipeline that trades representational richness (raw waveform, learned embeddings) for privacy guarantees and deployability.
For validation, the authors ran two diagnostic downstream prediction tasks on the extracted feature vectors, not to build a deployable classifier but to characterize how much signal the protocol’s features retain: a random forest classifier (1000 trees, mtry = floor(sqrt(p)), min.node.size = 1) predicting self-reported speaker sex, and a random forest regressor (same tree count, min.node.size = 5) predicting concurrently reported momentary valence and arousal (single-item 6-point Likert EMA prompts), both evaluated with participant-blocked 10-fold cross-validation to prevent within-speaker leakage across folds.
Key Results
The protocol was deployed across two two-week EMA phases with N = 850 recruited participants (quota-matched to the German population), yielding 11,217 raw voice recordings from 3,813 EMA instances. Compliance was 67.8% of prompts initiated (3,813 of 5,627), and 96.8% of initiated sessions completed all three valence-condition recordings. After feature-based quality filtering (dropping non-speech clips via voicing-probability/segment heuristics, and low-quality clips via harmonic-to-noise ratio ≤ 0 dB), the final analyzed set was 9,877 recordings from 560 participants (46% female).
Speaker-sex classification from the extracted prosodic features was strong and nearly identical across feature sets: balanced accuracy Md = 92.3% (SD = 6.74% across CV folds) with eGeMAPS, and Md = 92.1% (SD = 8.21%) with ComParE, despite ComParE having roughly 72x more features. Momentary affect prediction was weak: arousal ρ Md = 0.11 (eGeMAPS) and 0.15 (ComParE); valence ρ Md = 0.06 for both feature sets, with no reliable difference across sentence-valence conditions (paired Friedman test, p > 0.05 for both outcomes). Linear mixed-effects models on four acoustic diagnostics (Table 1) found small but statistically reliable condition effects for HNR and voiced-segments-per-second (|β| ≈ 0.06-0.13 SD, p < .001) and loudness (p ≈ .02-.03), but no reliable F0-variability difference across valence conditions (p ≥ .35); between-participant variance (ICC) ranged 32.5%-69.3% across the four metrics, indicating substantial and consistent within-speaker prosodic signatures.
Novelty Assessment
The contribution is a field protocol and its empirical characterization, not a new modeling method: OpenSMILE feature extraction, GDPR-motivated on-device privacy design, and random forest diagnostics are all established techniques. What is new is the combination of (1) explicit lexical-valence-balanced content control transferred from lab methodology into a smartphone EMA deployment, and (2) a fully audio-free transmission pipeline validated at a meaningful scale (560 participants, ~10K recordings) rather than a small pilot. The prediction tasks are framed by the authors as diagnostic checks on protocol signal quality, not as a claimed state-of-the-art sex- or affect-classification system, and the paper is explicit that its engineered acoustic features may underperform contemporary self-supervised audio embeddings for the affect-prediction task, positioning the affect results as a signal-ceiling characterization rather than a competitive benchmark result.
Field Significance
low — this paper’s primary relevance to speech generation research is indirect: it demonstrates a reproducible protocol for producing prosody-labeled, content-controlled speech data at scale without raw-audio retention, which could plausibly serve as a template for building future prosody-controlled training or evaluation corpora that need content/prosody disentanglement or privacy-constrained collection. It does not itself propose, train, or evaluate a speech synthesis, voice conversion, or spoken dialogue system, and its diagnostic prediction tasks (sex classification, momentary-affect regression) are a data-quality validation exercise for the collection protocol rather than a contribution to generation methodology.
Claims
- supports: A content-controlled elicitation design, using valence-balanced scripted sentences with randomized condition order, can decouple prosodic variation from semantic content in field speech collection while preserving substantial between-speaker prosodic signal.
Evidence: Speaker-sex classification from extracted prosodic features reached balanced accuracy Md = 92.1-92.3% across two independent feature sets (eGeMAPS, ComParE), and between-participant intraclass correlation accounted for 32.5%-69.3% of variance across four acoustic diagnostics. (§3.4, Table 1)
- complicates: Engineered low-dimensional acoustic feature sets extracted from short scripted utterances carry only weak signal for momentary affective state, even when speaker-identity signal in the same features is strong.
Evidence: Momentary valence prediction from prosodic features reached only ρ Md = 0.06 (both eGeMAPS and ComParE), and arousal reached ρ Md = 0.11-0.15, despite the same feature pipeline yielding >92% balanced accuracy for speaker-sex classification. (§3.4)
- complicates: On-device, audio-free data collection pipelines that delete raw waveforms immediately after feature extraction cannot support post hoc verification of lexical or acoustic fidelity, limiting the ability to audit compliance or re-derive alternative representations.
Evidence: The authors note that because raw audio was not retained, they could not verify post hoc whether participants read the displayed sentences verbatim (paraphrasing or disfluencies could reintroduce semantic variance), and could not retrospectively compute alternative feature representations such as self-supervised embeddings. (§4.2)
Limitations and Open Questions
Warning
Because raw audio is deleted immediately on-device as a core design requirement, the protocol cannot verify lexical fidelity (whether participants actually read the prompted sentences verbatim) or retrospectively re-extract features with alternative representations. Occasional paraphrasing or disfluencies could have reintroduced semantic variance and altered the prosodic signal in ways the authors cannot detect or correct for.
The paper relies on engineered acoustic features (eGeMAPS, ComParE) chosen for comparability and transparency rather than maximal representational capacity, and the authors explicitly suggest that self-supervised audio embeddings computed on-device could improve downstream prediction, particularly for the weak affect-prediction results. The single-item Likert affect measures (a common but coarse EMA practice) may themselves introduce measurement error that caps attainable prediction performance independent of the acoustic feature quality. The protocol’s generalizability beyond German-language, Android-based, 18-65-year-old participants in a single national deployment is untested.
Wiki Connections
No qualifying concept connections: this paper’s contribution is a data-collection protocol and empirical validation study for prosodic speech data, not a speech generation, conditioning, or evaluation-metric mechanism meeting any of this wiki’s tracked concept definitions. No in-corpus paper is cited by this work; all 24 references are external prior work in psychology, paralinguistics, and speech-privacy research.