Multi-Stage Fusion for Multi-Class 3D Lidar Detection

Zejie Wang, Zhen Zhao, Zhao Jin, Zhengping Che, Jian Tang, Chaomin Shen, Yaxin Peng; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3120-3128

In autonomous driving, the robust and accurate perceptions of the environment is a fundamental and challenging task. Resorting to the advancing of different sensors such as LiDAR and Camera, the autonomous systems are able to capture and process complementary perceptual information for better detection and classifying objects. In this paper, we propose a LiDAR-Camera fusion method for multi-class 3D object detection. The proposed method makes the utmost use of data from the two sensors by multiple fusion stages, and can be learned in an end-to-end manner. First, we apply a multi-level gated adaptive fusion mechanism with the feature extraction backbone. This point-wise fusion stage assiduously exploits the image and point cloud inputs, and obtains joint semantic representations of the scene. Next, given the regions of interest (RoIs) proposed based on the LiDAR features, the corresponding Camera features are selected by RoI-based feature pooling. These features are used to enrich the LiDAR features in local regions and enhance the proposal refinement. Moreover, we introduce a multi-label classification task as an auxiliary regularization to the object detection network. Without relying on extra labels, it helps the model better mine the extracted features and discover hard object instances. The experiments conducted on the KITTI dataset have proved all our fusion strategies are effective.