This document discusses building a semantic image search engine through deep representation learning. It describes using a pre-trained convolutional neural network to extract image embeddings, which are then used to build an index for fast proximity search of similar images. The model is further trained to predict word embeddings for images, allowing image search to be guided by text queries beyond the initial training classes. This provides a way to build a joint image-text embedding space and enable generalized image search using only a small dataset for training.

1. Deep Representation: Building
a Semantic Image Search
Engine
Emmanuel Ameisen

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
semantic-search-engine

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. PINTEREST SEARCH

5. IMAGE SEARCH ENGINE

6. IMAGE TAGGING
thenextweb.com

7. BACKGROUND
▰ Why am I speaking about this?

8. ABOUT INSIGHT
DATA SCIENCE
DATA ENGINEERING
HEALTH DATA
ARTIFICIAL INTELLIGENCE
7-Week Fellowship in
+ REMOTE
SILICON VALLEY & SAN
FRANCISCO
NEW YORK
BOSTON
SEATTLE
PRODUCT MANAGEMENT
TORONTO
DEVOPS
www.insightdata.ai

9. INSIGHT DATA – FELLOW PROJECTS
FASHION CLASSIFIER AUTOMATIC REVIEW GENERATION READING TEXT IN VIDEOS
HEART SEGMENTATION SUPPORT REQUEST CLASSIFICATION SPEECH UNSAMPLING

10. 1,600+
INSIGHT ALUMNI

11. INSIGHT FELLOWS ARE DATA SCIENTISTS AND DATA ENGINEERS
EVERYWHERE
400+
COMPANIES

12. ON THE MENU
▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both

13. ON THE MENU
▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both

14. ▰ Massive models
▻ Dataset of 1M+images
▻ For multiple days
▰ Automates feature engineering
▰ Use cases
▻ Fashion
▻ Security
▻ Medicine
▻ …
CONVOLUTIONAL NEURAL NETWORKS
(CNN)

15. ▰ Incorporates local and global information
▰ Use cases
▻ Medical
▻ Security
▻ Autonomous Vehicles
EXTRACTING INFORMATION
@arthur_ouaknine

16. ▰ Pose Estimation
▰ Scene Parsing
▰ 3D Point cloud estimation
ADVANCED APPLICATIONS
Insight Fellow Project with Piccolo
Felipe Mejia

17. ▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both
ON THE MENU

18. ▰ Traditional NLP tasks
▻ Classification (sentiment analysis, spam detection, code classification)
▰ Extracting Information
▻ Named Entity Recognition, Information extraction
▰ Advanced applications
▻ Translation, sequence to sequence learning
NLP

19. ▰ Sequence to sequence models are still often too
rough to be deployed, even with sizable datasets
▻ Recognized Tosh as a swear word
▰ They can be used efficiently for data augmentation
▻ Paired with other latent approaches
SENTENCE PARAPHRASING
Victor Suthichai

20. ▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both
ON THE MENU

21. ▰ Prime language model with features
extracted from CNN
▰ Feed to an NLP language model
▰ End-to-end
▻ Elegant
▻ Hard to debug and validate
▻ Hard to productionize
IMAGE CAPTIONING
A horse is standing in a field with a fence in the background.

22. CODE GENERATION
Ashwin Kumar
§ Harder problem for humans
- Anyone can describe an image
- Coding takes specific training
§ We can solve it using a similar model
§ The trick is in getting the data!

23. ▰ These methods mix and match different architectures
▰ The combined representation is often learned implicitly
▻ Hard to cache and optimize to re-use across services
▻ Hard to validate and do QA on
▰ The models are entangled
▻ What if we want to learn a simple joint representation?
BUT DOES IT SCALE?

24. Image Search

25. Goals
§ Searching for similar images to an input image
- Computer Vision: (Image → Image)
§ Searching for images using text & generating tags for images
- Computer Vision + Natural Language Processing: (Image ↔ Text)
§ Bonus: finding similar words to an input word
- Natural Language Processing: (Text → Text)

26. ▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both
ON THE MENU

27. Image Based Search
Let’s build this!

28. Dataset
Credit to Cyrus Rashtchian, PeterYoung, Micah Hodosh, and Julia Hockenmaier for the dataset.
§ 1000 images
- 20 classes, 50 images per class
§ 3 orders of magnitude smaller than usual deep
learning datasets
§ Noisy

29. WHICH CLASS?

30. DATA PROBLEMS
Bottle L

31. A FEW APPROACHES
§ Ways to think about searching for similar images

32. IF WE HAD INFINITE DATA
§ Train on all images
§ Pros:
- One Forward Pass (fast inference)
§ Cons:
- Hard too optimize
- Poor scaling
- Frequent Retraining

33. SIMILARITY MODEL
§ Train on each image pair
§ Pros:
- Scales to large datasets
§ Cons:
- Slow
- Does not work for text
- Needs good examples

34. EMBEDDING MODEL
§ Find embedding for each image
§ Calculate ahead of time
§ Pros:
- Scalable
- Fast
§ Cons:
- Simple representations

35. Mikolov et Al. 2013
WORD EMBEDDINGS

36. LEVERAGING A PRE-TRAINED MODEL

37. HOW AN EMBEDDING LOOKS

38. PROXIMITY SEARCH IS FAST
How do you find the 5 most similar images to a given one when you
have over a million users?
▰ Fast index search
▰ Spotify uses annoy (we will as well)
▰ Flickr uses LOPQ
▰ Nmslib is also very fast
▰ Some rely on making the queries approximate in order to make them
fast

39. PRETTY IMPRESSIVE!
IN OUT

40. FOCUSING OUR SEARCH
§ Sometimes we are only interested in part of the image.
§ For example, given an image of a cat and a bottle, we might be only interested in similar cats, not similar bottles.
§ How do we incorporate this information

41. IMPROVING RESULTS: STILL NO TRAINING
§ Computationally expensive approach:
- Object detection model first
- (We don’t do this)
- Image search on a cropped image
- (We don’t do this)
§ Semi-Supervised approach:
- Hacky, but efficient!
- re-weighing the activations
- Only use the class of interest to re-
weigh embeddings

42. EVEN BETTER
IN OUT

43. ▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both
ON THE MENU

44. GENERALIZING
§ We have added some ability to guide the search, but it is limited to classes our model was initially trained on
§ We would like to be able to use any word
§ How do we combine words and images?

45. Mikolov et Al. 2013
WORD EMBEDDINGS

46. SEMANTIC TEXT!
§ Load a set of pre-trained vectors (GloVe)
- Wikipedia data
- Semantic relationships
§ One big issue:
- The embeddings for images are of size 4096
- While those for words are of size 300
- And both models trained in a different fashion
§ What we need: Joint model!

47. ▰ A quick overview of Computer Vision (CV) tasks and challenges
▰ Natural Language Processing (NLP) tasks and challenges
▰ Challenges in combining both
▰ Representations learning in CV
▰ Representation learning in NLP
▰ Combining both
ON THE MENU

48. Inspiration

49. TIME TO TRAIN
Image à Image Image à Text

50. IMAGE à TEXT
§ Re-train model to predict the word vector
- i.e. 300-length vector associated with cat
§ Training
- Takes more time per example than image à class
- But much faster than on Imagenet (7 hours, no GPU)
§ Important to note
- Training data can be very small: ~1000 images
- Miniscule compared to Imagenet (1+ Million images)
§ Once model is trained
- Build a new fast index of images
- Save to disk
How do you
think this
model will
perform?

51. IMAGE à TEXT

52. GENERALIZED IMAGE SEARCH WITH MINIMAL
DATA
IN:“DOG” OUT

53. SEARCH FOR WORD NOT IN DATASET
IN:“OCEAN” OUT

54. SEARCH FOR WORD NOT IN DATASET
IN:“STREET” OUT

55. MULTIPLE WORDS!

56. MULTIPLE WORDS!
IN:“CAT SOFA” OUT

57. Learn More:
Find the repo on Github!

58. Next steps
§ Incorporating user feedback
- Most real world image search systems use user clicks as a signal
§ Capturing domain specific aspects
- Often times, users have different meanings for similarity
§ Keep the conversation going
- Reach me on Twitter @EmmanuelAmeisen

59. www.insightdata.ai/apply
EMMANUEL AMEISEN
Head of AI, ML Engineer
@emmanuelameisen
emmanuel@insightdata.ai
bit.ly/imagefromscratch

60. CV Approaches
White-box Algorithms Black-Box Algorithms
@Andrey Nikishaev

61. ▰ NLP Classification is generally more shallow
▻ Logistic Regression/Naïve Bayes
▻ Two layer CNN
▰ This is starting to change
▻ The triumph of pre-training and transfer learning
CLASSIFICATION

62. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
semantic-search-engine