Dataset Efficient Training With Model Ensembling

Yeonju Ro, Cong Xu, Agnieszka Ciborowska, Suparna Bhattacharya, Frankie Li, Martin Foltin; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 4700-4704

We propose a dataset-efficient deep learning training method by ensembling different models trained on different subsets. The ensembling method leverages the difficulty level of data samples to select subsets that are representative and diverse. The approach involves building a common base model with a random subset of data and then allotting different subsets to different models in an ensemble. The models are trained with their own subsets and then merged into a single model. We then propose an iterative multi-phase ensemble training that aggregates models in the ensemble more frequently and prevents divergence. The experiments on ResNet18 and ImageNet show that ensembling outperforms the no-ensemble case and achieves 64.8% accuracy with only 30% of dataset, saving 20 hours of training time in a single V100 GPU training experiment.