Data Distillation: Towards Omni-Supervised Learning

Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, Kaiming He; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4119-4128

Abstract


We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lower-bounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging real-world data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.

Related Material


[pdf] [arXiv]
[bibtex]
@InProceedings{Radosavovic_2018_CVPR,
author = {Radosavovic, Ilija and Dollár, Piotr and Girshick, Ross and Gkioxari, Georgia and He, Kaiming},
title = {Data Distillation: Towards Omni-Supervised Learning},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}