Visual-Inertial Navigation System (VINS) using a Rolling-Shutter Camera

UMN Researchers

Chao Guo, Dimitrios Kottas, Ryan DuToit, Ahmed Ahmed, Ruipeng Li, and Stergios Roumeliotis (PI)

Motivation

In order to develop Vision-aided Inertial Navigation Systems (VINS) on mobile devices, such as cell phones and tablets, one needs to consider two important issues, both due to the commercial-grade underlying hardware:

  • The rolling-shutter effect caused by CMOS sensors
  • The unknown and varying time offset between the camera and IMU clocks

Without appropriately modelling their effect and compensating for them online, the navigation accuracy will significantly degrade.

Stack of overlapping Golden Gate Bridge photos, with warped, slanted edges on the front images
Stack of overlapping photos of the Golden Gate Bridge
A comparison between global-shutter and rolling-shutter camera images(from https://graphics.stanford.edu/papers/stabilization/karpenko_gyro.pdf)
Timing diagram: a global camera captures one image at an instant, while a rolling-shutter camera reads rows 1 to n in sequence
Rolling-shutter and time synchronization cause a time-varying offset between the IMU and camera measurements' arrival time

Summary

In this work, we present a high-precision VINS that explicitly considers and accounts for both the rolling-shutter and time synchronization issues of an IMU-camera system. In particular, our main contributions are:

  • We introduce an interpolation model for expressing the camera pose of each visual measurement as a function of neighboring IMU poses, which introduces a significant speedup as compared to alternative constant-velocity-based models
  • We analytically determine the system’s unobservable directions when applying our interpolation measurement model, and use them to improve the VINS consistency and accuracy by employing the observability-constrained extended Kalman filter (OC-EKF)

Experimental validation has been done using a wide range of mobile devices, such as the Galaxy S4 cell phone, the Google Glass, and the Bebop quadrotor.

Videos

 

 

 



Relevant Publications

C1. C.X. Guo, D.G. Kottas, R.C. DuToit, A. Ahmed, R. Li, and S.I. Roumeliotis, "Efficient Visual-Inertial Navigation using a Rolling-Shutter Camera with Inaccurate Timestamps," Robotics: Science and Systems Conference (RSS), Berkeley, CA, Jul. 12-16, 2014 (pdf).