Concept Embeddings with SNaCK

This paper presents our work on “SNaCK,” a low-dimensional concept embedding algorithm that combines human expertise with automatic machine similarity kernels. Both parts are complimentary: human insight can capture relationships that are not apparent from the object’s visual similarity and the machine can help relieve the human from having to exhaustively specify many constraints.

As input, our SNaCK algorithm takes two sources:

  • Several “relative similarity comparisons.” Each triplet has the form \((a,b,c)\), meaning that in the lower-dimension embedding \(Y\), \(Y_a\) should be closer to \(Y_b\) than it is to \(Y_c\). Experts can generate many of these constraints using crowdsourcing.
  • Feature vector representations of each point. For instance, such features could come from HOG, SIFT, a deep-learned CNN, word embeddings, and so on.

SNaCK then generates an embedding that satisfies both classes of constraints.



Kevin Musgrave Serge Belongie, Ser-Nam Lim

A Metric Learning Reality Check

European Conference on Computer Vision (ECCV), Glasgow, Scotland, 2020.



Collaborative Metric Learning

Hsieh, Cheng-Kang; Yang, Longqi; Cui, Yin; Lin, Tsung-Yi; Belongie, Serge; Estrin, Deborah

Collaborative Metric Learning

International Conference on World Wide Web (WWW), Perth, 2017.

(Links | BibTeX)


Bayesian representation learning with oracle constraints

Karaletsos, Theofanis; Belongie, Serge; Rätsch, Gunnar

Bayesian representation learning with oracle constraints

International Conference on Learning Representations (ICLR), San Juan, PR, 2016.

(Links | BibTeX)


Learning Concept Embeddings with Combined Human-Machine Expertise

Wilber, Michael; Kwak, Iljung; Kriegman, David; Belongie, Serge

Learning Concept Embeddings with Combined Human-Machine Expertise

International Conference on Computer Vision (ICCV), 2015.

(Links | BibTeX)


A Python implementation of SNaCK is freely available. View SNaCK code and documentation on Github. If you are using Anaconda an Linux or Mac OS X, SNaCK is easy to install. Run $ conda install snack
Otherwise, please follow the instructions in the README file on Github.

Click the button below to launch an IPython Notebook that contains SNaCK and Food-10k:
Launch on Binder


Download the Food-10k dataset here: Food-10k.tar.xz. This dataset includes 10,000 Yummly IDs of foods and 958,479 triplet constraints collected using crowdsourcing.

The CU-Birds 200 dataset can be found at the Caltech-UCSD Birds 200-2011 webpage. In our experiments, we used the “Birdlets” subset, consisting of the following 14 classes:

  • 192.Downy_Woodpecker
  • 155.Warbling_Vireo
  • 036.Northern_Flicker
  • 156.White_eyed_Vireo
  • 157.Yellow_throated_Vireo
  • 151.Black_capped_Vireo
  • 152.Blue_headed_Vireo
  • 153.Philadelphia_Vireo
  • 154.Red_eyed_Vireo
  • 187.American_Three_toed_Woodpecker
  • 188.Pileated_Woodpecker
  • 189.Red_bellied_Woodpecker
  • 190.Red_cockaded_Woodpecker
  • 191.Red_headed_Woodpecker

60-second Video Spotlight