EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting

Min-Hui Lin, Mahesh Reddy, Guillaume Berger, Michel Sarkis, Fatih Porikli, Ning Bi; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 831-840

In this paper we present EdgeRelight360 an approach for real-time video portrait relighting on mobile devices utilizing text-conditioned generation of 360-degree high dynamic range image (HDRI) maps. Our method proposes a diffusion-based text-to-360-degree image generation in the HDR domain taking advantage of the HDR10 standard. This technique facilitates the generation of high-quality realistic lighting conditions from textual descriptions offering flexibility and control in portrait video relighting task. Unlike the previous relighting frameworks our proposed system performs video relighting directly on-device enabling real-time inference with real 360-degree HDRI maps. This on-device processing ensures both privacy and guarantees low runtime providing an immediate response to changes in lighting conditions or user inputs. Our approach paves the way for new possibilities in real-time video applications including video conferencing gaming and augmented reality by allowing dynamic text-based control of lighting conditions.