Link Search Menu Expand Document

Modeling Long-horizon Tasks as Sequential Interaction Landscapes

Paper PDF


Soeren Pirk (Google)*; Karol Hausman (Google Brain); Alexander Toshev (Google); Mohi Khansari (X, The Moonshot Factory)

Interactive Session

2020-11-16, 12:30 - 13:00 PST | PheedLoop Session


Task planning over long-time horizons is a challenging and open problem in robotics and its complexity grows exponentially with an increasing number of subtasks. In this paper we present a deep neural network that learns dependencies and transitions across subtasks solely from a set of demonstration videos. We represent each subtasks as action symbols (e.g. move cup), and show that these symbols can be learned and predicted directly from image observations. Learning symbol sequences provides the network with additional information about the most frequent transitions and relevant dependencies between subtasks and thereby structures tasks over long-time horizons. Learning from images, on the other hand, allows the network to continuously monitor the task progress and thus to interactively adapt to changes in the environment. We evaluate our framework on two long horizon tasks: (1) block stacking of puzzle pieces being executed by humans, and (2) a robot manipulation task involving pick and place of objects and sliding a cabinet door with a 7-DoF robot arm. We show that complex plans can be carried out when executing the robotic task and the robot can interactively adapt to changes in the environment and recover from failure cases.


Reviews and Rebuttal

Reviews & Rebuttal

Conference on Robot Learning 2020