Self-Supervised 3D Keypoint Learning for Ego-Motion Estimation
Authors
Jiexiong Tang (KTH Royal Institute of Technology)*; Rareș Ambruș (Toyota Research Institute); Vitor Guizilini (Toyota Research Institute); Sudeep Pillai (Toyota Research Institute); Hanme Kim (Toyota Research Institute); Patric Jensfelt (Royal Institute of Technology); Adrien Gaidon (Toyota Research Institute)
Interactive Session
2020-11-16, 11:50 - 12:20 PST | PheedLoop Session
Abstract
Detecting and matching robust viewpoint-invariant keypoints is critical for visual SLAM and Structure-from-Motion. State-of-the-art learning-based methods generate training samples via homography adaptation to create 2D synthetic views with known keypoint matches from a single image. This approach does not, however, generalize to non-planar 3D scenes with illumination variations commonly seen in real-world videos. In this work, we propose self-supervised learning depth-aware keypoints from unlabeled videos directly. We jointly learn keypoint and depth estimation networks by combining appearance and geometric matching via a differentiable structure-from-motion module based on Procrustean residual pose correction. We show how our self-supervised keypoints can be trivially incorporated into state-of-the-art visual odometry frameworks for robust and accurate ego-motion estimation of autonomous vehicles in real-world conditions.