Sensor Equivariance: A Framework for Semantic Segmentation with Diverse Camera Models

Hannes Reichert, Manuel Hetzel, Andreas Hubert, Konrad Doll, Bernhard Sick; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 1254-1261

Objects are represented differently in projection-based sensors such as cameras depending on sensor resolution field of view and distortion leading to distorted physical and geometric properties. As a result sensor data processing depend on these properties. With the large variations of sensors on the market an equivariant representation and suitable processing are necessary to become independent of the sensor used. In this work we propose an extension of conventional image data by an additional channel in which the associated projection properties are encoded. Furthermore we introduce a SensorConv layer as an extension to the conventional convolution layer. SensorConv enable using projection properties in convolutional neural networks. To that end we propose an architecture for using the SensorConv layer in the Detectron2 framework. We collected a dataset of equirectangular images for our experiments with the CARLA simulator. To analyze multiple sensor models (i.e. sensor intrinsic) we created an augmentation method to emulate a high variability of sensors from the collected equirectangular panoramas. In our experiment we show that our method can generalize better across different camera sensors.