Person in Place: Generating Associative Skeleton-Guidance Maps for Human-Object Interaction Image Editing

ChangHee Yang, ChanHee Kang, Kyeongbo Kong, Hanni Oh, Suk-Ju Kang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 8164-8175

Recently there were remarkable advances in image editing tasks in various ways. Nevertheless existing image editing models are not designed for Human-Object Interaction (HOI) image editing. One of these approaches (e.g. ControlNet) employs the skeleton guidance to offer precise representations of human showing better results in HOI image editing. However using conventional methods manually creating HOI skeleton guidance is necessary. This paper proposes the object interactive diffuser with associative attention that considers both the interaction with objects and the joint graph structure automating the generation of HOI skeleton guidance. Additionally we propose the HOI loss with novel scaling parameter demonstrating its effectiveness in generating skeletons that interact better. To evaluate generated object-interactive skeletons we propose two metrics top-N accuracy and skeleton probabilistic distance. Our framework integrates object interactive diffuser that generates object-interactive skeletons with previous methods demonstrating the outstanding results in HOI image editing. Finally we present potentials of our framework beyond HOI image editing as applications to human-to-human interaction skeleton editing and 3D mesh optimization. The code is available at https://github.com/YangChangHee/CVPR2024_Person-In-Place_RELEASE