DynaDistill: Leveraging Real-Time Feedback for Effective Dataset Distillation

Zongxiong Chen, Derui Zhu, Jiahui Geng, Sonja Schimmler, Manfred Hauswirth; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8154-8158

Dataset Distillation (DD) aims to compress the knowledge contained in a large-scale dataset into a substantially smaller synthetic dataset. While this synthetic dataset is meticulously crafted to mirror the performance of the original it poses significant challenges in training efficiency and data utility. This singular focus especially the selection of a sole expert trajectory in MTT or a single model in IDC inadvertently undermines the potential performance of the distilled synthetic dataset at certain intervals within the whole distillation process. This inefficiency necessitates a protracted series of training iterations to culminate in an improved performance outcome. In this paper we hypothesize that there exists an optimized training routine across the entire optimization phase specifically for synthetic dataset training through gradient or trajectory matching. To address these challenges this paper introduces a novel methodology namely DynaDistill which is designed to expedite the distillation process by dramatically decreasing the required number of distillation steps in current state-of-the-art methods without compromising their performance. Our empirical results demonstrate that our method achieves comparable performance on par with state-of-the-art methods. Moreover the design of our method allows it to integrate as a plug-and-play module into existing distillation techniques seamlessly.