Sketching With Style: Visual Search With Sketches and Aesthetic Context

John Collomosse, Tu Bui, Michael J. Wilber, Chen Fang, Hailin Jin; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2660-2668

Abstract


We propose a novel measure of visual similarity for image retrieval that incorporates both structural and aesthetic (style) constraints. Our algorithm accepts a query as sketched shape, and a set of one or more contextual images specifying the desired visual aesthetic. A triplet network is used to learn a feature embedding capable of measuring style similarity independent of structure, delivering significant gains over previous networks for style discrimination. We incorporate this model within a hierarchical triplet network to unify and learn a joint space from two discriminatively trained streams for style and structure. We demonstrate that this space enables, for the first time, style-constrained sketch search over a diverse domain of digital artwork comprising graphics, paintings and drawings. We also briefly explore alternative query modalities.

Related Material


[pdf] [supp]
[bibtex]
@InProceedings{Collomosse_2017_ICCV,
author = {Collomosse, John and Bui, Tu and Wilber, Michael J. and Fang, Chen and Jin, Hailin},
title = {Sketching With Style: Visual Search With Sketches and Aesthetic Context},
booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
month = {Oct},
year = {2017}
}