Towards Engineered Safe AI with Modular Concept Models

Lena Heidemann, Iwo Kurzidem, Maureen Monnet, Karsten Roscher, Stephan Günnemann; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 3564-3573

The inherent complexity and uncertainty of Machine Learning (ML) makes it difficult for ML-based Computer Vision (CV) approaches to become prevalent in safety-critical domains like autonomous driving despite their high performance. A crucial challenge in these domains is the safety assurance of ML-based systems. To address this recent safety standardization in the automotive domain has introduced an ML safety lifecycle following an iterative development process. While this approach facilitates safety assurance its iterative nature requires frequent adaptation and optimization of the ML function which might include costly retraining of the ML model and is not guaranteed to converge to a safe AI solution. In this paper we propose a modular ML approach which allows for more efficient and targeted measures to each of the modules and process steps. Each module of the modular concept model represents one visual concept and is aggregated with the other modules' outputs into a task output. The design choices of a modular concept model can be categorized into the selection of the concept modules the aggregation of their output and the training of the concept modules. Using the example of traffic sign classification we present each step of the involved design choices and the corresponding targeted measures to take in an iterative development process for engineering safe AI.