MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval
- Mark Hamilton ,
- Stephanie Fu ,
- Mindren Lu ,
- Johnny Bui ,
- Darius Bopp ,
- Zhenbang Chen ,
- Felix Tran ,
- Margaret Wang ,
- Marina Rogers ,
- Lei Zhang ,
- Chris Hoder ,
- William T. Freeman
NeurIPS 2020 Competition and Demonstration Track |
We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia. To create this application, we introduce Conditional Image Retrieval (CIR) which combines visual similarity search with user supplied filters or “conditions”. This technique allows one to find pairs of similar images that span distinct subsets of the image corpus. We provide a generic way to adapt existing image retrieval data-structures to this new domain and provide theoretical bounds on our approach’s efficiency. To quantify the performance of CIR systems, we introduce new datasets for evaluating CIR methods and show that CIR performs non-parametric style transfer. Finally, we demonstrate that our CIR data-structures can identify “blind spots” in Generative Adversarial Networks (GAN) where they fail to properly model the true data distribution.
NeurIPs 2020: MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval
We introduce MosAIc, an interactive website that allows users to discover hidden connections between works of art across culture, media, artists, and time. MosAIc finds "visual analogies", or works of art with the same semantic structure but very different cultural and artistic context, within the combined works of the Metropolitan Museum of Art and the Rijksmuseum. Users can take any work from the collection and find analogous works in particular genres, cultures, or media of art. Our approach finds visual analogies that mirror larger scale cultural trends, such as the flows of artistic techniques across the globe due to trade routes. Our approach is based on generalizing deep image retrieval methods to flexibly adapt to logical filters and predicates. This allows image retrieval methods to…
Discovering hidden connections in art with deep, interpretable visual analogies
Image retrieval systems allow individuals to find images that are semantically similar to a query image. This serves as the backbone of reverse image search engines and many product recommendation engines. Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world. In this webinar, Microsoft Research Development Engineer Mark Hamilton presents a novel method for specializing image retrieval systems called conditional image retrieval. When applied over large art datasets in particular, conditional image retrieval provides visual analogies that bring to light hidden connections among different artists, cultures, and media. This aims to encourage a new level of engagement with creative artifacts and inspire people to imagine new works of art. Hamilton will demonstrate how conditional image retrieval…