Progressive Target Refinement by Self-Distillation for Human Pose Estimation

Jingtian Li, Lin Fang, Yi Wu, Shangfei Wang; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3934-3946

The handcrafted heatmap target can be improved and one way is knowledge distillation, which takes the predicted heatmaps from another model as auxiliary supervision. However, previous pose distillation methods are training inefficient, requiring either an extra training stage or complex network architecture modification. In this paper, we propose a novel Self-Distillation for Human Pose Estimation (SDP) method for better distillation efficiency. Specifically, a student pose estimator distills the soft targets from itself with the backup information of a previous batch, where the targets are progressively refined through model updating. The main advantage of our method is that we achieve efficient training and simple implementation simultaneously. Existing pose estimation networks can benefit from the proposed method effortlessly. A stepping strategy, that widens the distillation distance with the decaying of the learning rate, is further proposed. It ensures the difference between teacher and student in a low learning rate condition. Experimental results on two widely-used benchmark datasets, MPII and COCO, illustrate the effectiveness of the proposed approach.