Inverting the Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting

Abstract

Many autonomous systems must forecast the future in order to make decisions. For example, autonomous driving and robotic manipulation systems often forecast future object poses by leveraging supervised learning techniques. A significant downside to these approaches is the labeling bottleneck – they require labeled sequences of agent poses, which are costly to obtain, especially in 3D space. Can we circumvent this labeling bottleneck? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with 100k points) and then detect and track objects on the predicted point cloud sequences to obtain future pose trajectories, i.e., a forecast-then-detect pipeline. Part of this work’s focus is on the challenging first step – Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To evaluate our proposed forecast-then-detect pipeline and compare with the detect-then-forecast pipeline, we also propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two real autonomous driving datasets, we show that SPFNet is effective for the SPF task and that our forecast-then-detect pipeline outperforms the conventional pipeline.