Debiased One-Shot NAS Via Density-Aware Sampling

Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, 2026, pp. 2357-2366

One-shot Neural Architecture Search (NAS) is based on training a supernet, a single model from which many different architectures with shared weights are sampled and updated during training. Training the shared weights of the supernet is much more computationally efficient than training each architecture independently. However, during the supernet training, architectures with similar gradients (dense regions in the gradient space) would cooperate with each other increasing their training, while architectures in sparse(low-density) regions would not receive as much training benefit from others. This does not allow all architectures to be trained with the same amount of effective updates, producing a training bias that favors architectures in denser regions. As a consequence, the correlation between the supernet estimations and the actual performance of models independently trained is reduced. This negatively affects the supernet's ability to select good architectures once trained. In this paper, we propose two computationally feasible ways for different computational budgets and search spaces to approximate the architecture densities and implement a density-aware debiasing mechanism for supernet training. We propose a fine-grained density calculation for numerable smaller search spaces and an online density approximation based on density prototypes from a clustering algorithm for larger spaces. We validate our method on CIFAR10, CIFAR100 and ImageNet datasets using various search strategies and show consistently improved results compared to several single-path one-shot supernet training methods.