Gift from nature: Potential Energy Minimization for explainable dataset distillation

Zijia Wang, Wenbin Yang, Zhisong Liu, Qiang Chen, Jiacheng Ni, Zhen Jia; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2022, pp. 235-250

Dataset distillation aims to reduce the dataset size by capturing important information from original dataset. It can significantly improve the feature extraction effectiveness, storage efficiency and training robustness. Furthermore, we study the features from the data distillation and found unique discriminative properties that can be exploited. Therefore, based on Potential Energy Minimization, we propose a generalized and explainable dataset distillation algorithm, called Potential Energy Minimization Dataset Distillation(PEMDD). The motivation is that when the distribution for each class is regular (that is, almost a compact high-dimensional ball in the feature space) and has minimal potential energy in its location, the mixed-distributions of all classes should be stable. In this stable state, Unscented Transform (UT) can be implemented to distill the data and reconstruct the stable distribution using these distilled data. Moreover, a simple but efficient framework of using the distilled data to fuse different datasets is proposed, where only a lightweight finetune is required. To demonstrate the superior performance over other works, we first visualize the classification results in terms of storage cost and performance. We then report quantitative improvement by comparing our proposed method with other state-of-the-art methods on several datasets. Finally, we conduct experiments on few-shot learning, and show the efficiency of our proposed methods with significant improvement in terms of the storage size requirement.