SIGDIAL 2025 · 2025 · Workshop
Manaseryan et al. · → Paper · Demo: ✓ · Code: ?
rrSDS 2.0 is an updated toolkit for building incremental, multimodal, distributed spoken dialogue systems for robotic platforms, extending the Retico-based rrSDS framework with new perception, ASR, simulation, and visualization modules.
Problem
Spoken dialogue systems for robotic platforms must balance real-time responsiveness across heterogeneous modalities, including vision, speech, and physical actuation, within a single coherent framework. The original rrSDS addressed this through five design principles (modularity, multimodality, distributiveness, incrementality, and alignment), but required expansion to support newer ASR models, richer vision modules, robot simulation environments, and developer tooling for debugging and benchmarking.
Method
rrSDS 2.0 is built on the Retico framework, which implements the Incremental Unit (IU) Framework of Schlangen and Skantze (2009). All data between modules is passed as typed incremental units, allowing modules to be independently developed, swapped, and composed into pipelines.
The system is organized into perception, speech, language, robot, and simulation layers. On the perception side, rrSDS 2.0 adds YOLOv11, YOLOv12, RT-DETR, SAM (Segment Anything Model), and DINOv2 for object detection and feature extraction, as well as MediaPipe-based hand and body pose tracking. For speech, two local ASR modules are introduced: Whisper (encoder-decoder transformer trained on 680k hours of multilingual audio) and Wav2vec 2.0 (trained on 960 hours of unlabeled audio using raw waveform input rather than spectrograms). Language understanding and generation uses the Retico-HuggingFace module, which wraps HuggingFace’s pipeline API for real-time text generation, together with RASA 3.0 for natural language understanding and dialogue management.
Robot integration is provided through Misty II modules (camera input, movement, and speech output) and a suite of CoppeliaSim modules for virtual robot simulation, including joint manipulation, camera feeds, and state updates for a custom Cozmo simulation. Object permanence for the Cozmo robot is supported via SLAM through the Cozmo SDK’s Navigation Memory Map.
Developer tooling includes a visualization tool that streams real-time module connection graphs and IU data flows to a web interface, and a SimpleLogger module that records all IU payloads, update types, and provenance to JSON for analysis.
Key Results
This is a demo/system description paper. No quantitative evaluation metrics are reported. The paper demonstrates three example system implementations: a verbal joystick controller for a virtual Cozmo robot in CoppeliaSim, a Nim game with a Misty II robot using reinforcement learning and real-time object detection, and a math tutoring system in which Misty II uses vision and speech jointly to verify a child’s arithmetic answers. These demonstrations illustrate the toolkit’s capacity for multimodal, real-time pipelines across physical and simulated robots.
Novelty Assessment
The contribution is engineering integration. rrSDS 2.0 assembles a collection of individually established components (Whisper, Wav2vec 2.0, YOLO variants, SAM, DINOv2, MediaPipe, RASA 3.0, HuggingFace pipelines, CoppeliaSim) under the existing Retico/IU framework. No new model architecture, training method, or evaluation methodology is introduced. The primary additions are module coverage, developer tooling (visualization and logging), and a robot simulation interface. The value is primarily infrastructural: lowering the barrier for researchers building multimodal dialogue systems for robotic settings.
Field Significance
Low — rrSDS 2.0 provides useful open-source infrastructure for researchers in robotic spoken dialogue, extending a Retico-based toolkit with updated ASR, vision, and simulation modules. Its contribution is tooling and integration rather than a speech synthesis or understanding advance, and no quantitative evaluation is offered to situate it relative to other dialogue systems.
Claims
-
supports: Modular, incremental architectures built on the IU framework enable flexible composition of heterogeneous speech and perception components for real-time spoken dialogue.
Evidence: rrSDS 2.0 supports seamless swapping of ASR modules (Whisper vs. Wav2vec) and vision modules (YOLOv11, YOLOv12, RT-DETR, SAM, DINOv2) through a shared IU interface, demonstrated in three multi-component robot pipelines. (§2, §3)
-
complicates: Real-time multimodal spoken dialogue for physical and simulated robotic platforms introduces synchronization challenges not present in text-only or speech-only systems.
Evidence: The authors report facing challenges balancing real-time, multimodal interaction when integrating rrSDS 2.0 across physical and simulated robots, without providing quantitative results characterizing the trade-offs. (§4)
-
supports: Open-source, framework-level integration of state-of-the-art ASR and vision components reduces the engineering overhead for building multimodal robotic dialogue systems.
Evidence: rrSDS 2.0 wraps Whisper, Wav2vec 2.0, YOLO variants, SAM, DINOv2, MediaPipe, RASA 3.0, and HuggingFace text generation behind a common IU interface, with pypi distribution and improved documentation to lower the setup barrier. (§1, §2)
Limitations and Open Questions
No quantitative evaluation is reported, making it impossible to compare rrSDS 2.0 against other dialogue system frameworks on latency, accuracy, or robustness. The demo paper format means claims about incremental processing quality and real-time performance rest on qualitative demonstration rather than systematic measurement. Integration with additional benchmarks beyond ALFRED is noted as future work, as is compatibility with the Remdis framework for LLM-driven incremental dialogue.
Wiki Connections
- Streaming TTS — rrSDS 2.0 is built on the Incremental Unit framework, where partial ASR and NLU outputs are passed between modules in real-time, directly embodying the incremental processing paradigm central to streaming spoken interaction.
- Evaluation Metrics — the absence of quantitative benchmarking in this demo highlights the ongoing difficulty of evaluating real-time, multimodal spoken dialogue systems across heterogeneous robotic platforms.
- Self-Supervised Speech — rrSDS 2.0 adds a Wav2vec 2.0 ASR module as an optional component, providing a self-supervised alternative to the Whisper (supervised) ASR option for low-resource or offline deployment scenarios.