Magic Layouts: Structural Prior for Component Detection in User Interface Designs

Dipu Manandhar, Hailin Jin, John Collomosse; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15809-15818

Abstract

We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.

Related Material

[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Manandhar_2021_CVPR, author = {Manandhar, Dipu and Jin, Hailin and Collomosse, John}, title = {Magic Layouts: Structural Prior for Component Detection in User Interface Designs}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2021}, pages = {15809-15818} }