Improvements to Image Reconstruction-Based Performance Prediction for Semantic Segmentation in Highly Automated Driving

Andreas Bär, Daniel Kusuma, Tim Fingscheidt; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 219-229

The performance of deep neural networks is typically measured with ground truth data which is expensive and not available during operation. At the same time, safety-critical applications, such as highly automated driving, require an awareness of the current performance, especially during operation with distorted inputs. Recently, performance prediction for semantic segmentation by an image reconstruction decoder was proposed. In this work, we investigate three approaches to improve its predictive power: Parameter initialization, parameter sharing, and inter-decoder lateral connections. Our best setup establishes a new state of the art in performance prediction with image-only inputs on Cityscapes and KITTI and even excels a method exploiting both point cloud and image inputs on Cityscapes. Further, our investigations reveal that the best Pearson correlation between the segmentation quality and the reconstruction quality does not always lead to the best predictive power. Code is available at https://github.com/ifnspaml/PerfPredRecV2.