Embeddings and Metric Learning

Understanding similarities between images is a key problem in computer vision. To measure the similarity between images, they are typically embedded in a featurevector space, in which their distance preserve the relative dissimilarity. These vector space representations are commonly used in applications such as image retrieval, classification or visualizations.

In our group, we are interested in many aspects of similarity learning. On one hand, we work on fundamental methods to learn similarity spaces. One the other hand, we work on applications of visual similarities in areas such as fine-grained categorization and recommendation systems.

Works

  • Conditional Similarity Networks What makes images similar? To measure the similarity between images, they are typically embedded in a featurevector space, in which their distance preserve the relative dissimilarity. However, when learning such similarity embeddings the simplifying assumption is commonly made that images are only compared to one unique measure of similarity. A main reason for this is ...
  • Fine-Grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop Existing fine-grained visual categorization methods often suffer from three challenges: lack of training data, large number of fine-grained categories, and high intra-class vs. low inter-class variance. In this work we propose a generic iterative framework for fine-grained categorization and dataset bootstrapping that handles these three challenges. Using deep metric learning with humans in the loop, ...
  • Learning Localized Perceptual Similarity Metrics for Interactive Categorization Current similarity-based approaches to interactive finegrained categorization rely on learning metrics from holistic perceptual measurements of similarity between objects or images. However, making a single judgment of similarity at the object level can be a difficult or overwhelming task for the human user to perform. Secondly, a single general metric of similarity may not be ...
  • Similarity Comparisons for Interactive Fine-Grained Categorization Current human-in-the-loop fine-grained visual categorization systems depend on a predefined vocabulary of attributes and parts, usually determined by experts. In this work, we move away from that expert-driven and attributecentric paradigm and present a novel interactive classifi- cation system that incorporates computer vision and perceptual similarity metrics in a unified framework. At test time, users ...
  • Cost-effective HITs for Relative Similarity Comparisons Recently in machine learning, there has been a growing interest in collecting human similarity comparisons of the form “Is a more similar to b than to c?” Each answer provides a constraint, $d(a, b) < d(a, c)$, for some perceptual distance function $d$. In computer vision, perceptually-based embeddings can be constructed from many of these ...
  • Concept Embeddings with SNaCK This paper presents our work on “SNaCK,” a low-dimensional concept embedding algorithm that combines human expertise with automatic machine similarity kernels. Both parts are complimentary: human insight can capture relationships that are not apparent from the object’s visual similarity and the machine can help relieve the human from having to exhaustively specify many constraints.
  • Learning Visual Clothing Style with Heterogeneous Dyads ‘What outfit goes well with this pair of shoes?’ To answer this type of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer this type of questions. Paper Abstract With the rapid proliferation of smart mobile devices, ...
  • Perception of Reflectance We design and implement a comprehensive study of the perception of gloss. This is the largest study of its kind to date, and the first to use real material measurements. In addition, we develop a novel Multi-Dimensional Scaling (MDS) algorithm for analyzing pairwise comparisons. The data from the psychophysics study and the MDS algorithm is ...

People