Towards a Robust Differentiable Architecture Search Under Label Noise

Christian Simon, Piotr Koniusz, Lars Petersson, Yan Han, Mehrtash Harandi; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 3256-3266

We all have experienced the difficulty of designing appropriate neural architectures due to the lack of general principles and best practices. The game changer might be touse Neural Architecture Search (NAS) where a machine does all the hard work for us based on the data at its disposal. Invarious problems and in particular in classification, architectures designed by NAS outperform or compete with the best manual network designs in terms of accuracy, size, memory footprint and FLOPs. That said, previous studies focus ondeveloping NAS algorithms for clean high quality data, a restrictive and somewhat unrealistic assumption. In this paper, focusing on the differentiable NAS algorithms, we show that vanilla NAS algorithms suffer from a performance loss if class labels are noisy. To combat this issue, we propose tomake use of the principle of information bottleneck as a regularizer. This leads us to develop a noise injecting operation that is included during the learning process, preventing the network from learning from noisy samples. Our empirical evaluations show that the noise injecting operation does not degrade the performance of the NAS algorithm if the data is indeed clean. In contrast, if the data is noisy, the architecture learned by our algorithm comfortably outperforms algorithms specifically equipped with sophisticated mechanisms to learn in the presence of label noise. In contrast to many algorithms designed to work in the presence of noisylabels, prior knowledge about the properties of the noise and its characteristics are not required for our algorithm.