Project overview
This project treats video-to-event conversion as a core methodology rather than a temporary workaround for missing datasets. It reviews how converters model threshold crossings, temporal interpolation, sensor noise, bandwidth, optical effects, and hardware constraints, then connects these choices to downstream sim-to-real performance.
Core contributions
- A five-family taxonomy covering frame-difference emulators, renderer-driven simulators, physics and circuit-level models, learning-based generators, and real-time or commercial tools.
- A comparison of sensor fidelity, temporal super-resolution, computational cost, real-time capability, and openness across major tools.
- A task-oriented synthesis of transfer evidence in reconstruction, optical flow, odometry, detection, segmentation, human pose, and computational photography.
- An analysis of the remaining domain gap and a research agenda for calibration, differentiable simulation, hardware-specific modeling, and standardized fidelity benchmarks.
Current evidence
The analysis argues that simulator choice should follow the target task’s sensitivity rather than popularity. Statistics-matching calibration can be as valuable as photorealistic rendering, while the strongest transfer pipelines usually combine synthetic pretraining with randomization, calibration, or limited real-data adaptation.
Representative converter landscape
- Model-based tools such as ESIM, Vid2E, and v2e.
- Physics and circuit-level tools including DVS-Voltmeter, ADV2E, and PECS.
- Learning-based approaches such as EventGAN and V2CE.
- Real-time and platform-integrated options including Raw2Event, CARLA DVS, Gazebo/AirSim plugins, VISTA 2.0, and industrial SDKs.
Why it matters for robotics
Robotics needs large, labeled event datasets across environments that are difficult to capture safely or repeatedly. Better converters let researchers reuse RGB datasets and simulation worlds, but only if the generated events preserve the sensor effects that matter to the intended robot task.