Project overview
This project asks whether synchronized RGB recordings can supervise event-based human perception without any real-event training labels. RGB videos are annotated, converted into simulated events with v2e, and used to train detection and segmentation models. The models are then tested on real event streams from the same multimodal platform under same-scenario, cross-scenario, and combined-domain conditions.
Core contributions
- A synchronized RGB-event dataset for human detection and instance segmentation under occlusion, crowding, clutter, ego-motion, and changing illumination.
- A controlled pipeline that trains exclusively on simulated events and directly measures performance on real event-camera data.
- Same-scenario, cross-scenario, and combined-domain evaluation across distinct Industry and Hospital environments.
- Bounding-box and pixel-level person annotations transferred from RGB to the spatially aligned event domain.
Current evidence
YOLO-m provides the strongest combined real-event detection result, while YOLO-l reaches 0.3718 mask mAP@0.50:0.95 for segmentation. Combined synthetic training is more robust than single-environment training, and the Hospital setting shows the larger domain gap.
Dataset and scenarios
- Approximately 80 minutes of synchronized RGB-event recordings at 1280×720 resolution.
- Roughly 219,000 annotated frames with up to nine people visible simultaneously.
- Industry and Hospital environments, static and moving robot conditions, high and low illumination, and different occlusion levels.
- Event frames accumulated over 30-60 ms windows for model input.
Relevance to human-in-the-loop robotics
Robots operating near people must detect complete and partially occluded humans while the robot itself is moving. This benchmark exposes where simulator-trained perception fails and identifies training strategies that improve robustness without requiring real-event supervision.