SieveNet: A Unified Framework for Robust Image-Based Virtual Try-On

Surgan Jandial, Ayush Chopra, Kumar Ayush, Mayur Hemani, Balaji Krishnamurthy, Abhijeet Halwai; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 2182-2190

Image-based virtual try-on for fashion has attracted considerable attention recently. The task requires trying on the desired clothing item on a target model. An efficient framework for this is composed of 2 stages: (1) warping (transforming) the try-on cloth to align with the pose and shape of the target model, and (2) a texture transfer module to seamlessly integrate the warped try-on cloth onto the target model image. Existing methods suffer from artifacts and distortions in their try-on output. In this work, we present SieveNet, a framework for robust image-based virtual try-on. Firstly, we introduce a multi-stage coarse-to-fine warping network to better model fine-grained intricacies in try-on clothing item and train it with a novel perceptual geometric matching loss. Next, we introduce a try-on cloth conditioned segmentation mask prior to improve the texture transfer network. Finally, we also introduce a dueling triplet strategy for training the texture transfer network which further improves the quality of the generated try-on result. We present extensive qualitative and quantitative evaluations on each component of the proposed pipeline and show significant performance improvements against existing state-of-the-art methods.