The Devil is in Discretization Discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol

Konstanty Subbotko, Wojciech Jablonski, Piotr Bilinski; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 1665-1674

Neural Architecture Search (NAS) has been widely adopted to design neural networks for various computer vision tasks. One of its most promising subdomains is differentiable NAS (DNAS) where the optimal architecture is found in a differentiable manner. However gradient-based methods suffer from the discretization error which can severely damage the process of obtaining the final architecture. In our work we first study the risk of discretization error and show how it affects an unregularized supernet. Then we present that penalizing high entropy a common technique of architecture regularization can hinder the supernet's performance. Therefore to robustify the DNAS framework we introduce a novel single-stage searching protocol which is not reliant on decoding a continuous architecture. Our results demonstrate that this approach outperforms other DNAS methods by achieving 75.3% in the searching stage on the Cityscapes validation dataset and attains performance 1.1% higher than the optimal network of DCNAS on the non-dense search space comprising short connections. The entire training process takes only 5.5 GPU days due to the weight reuse and yields a computationally efficient architecture. Additionally we propose a new dataset split procedure which substantially improves results and prevents architecture degeneration in DARTS.