For the last 3 decades, Microsoft has been powered by Machine Learning. Come to this session for a first time ever, under the hood look at how we use ML to improve every product and business at Microsoft. Then, see how that same technology is available to you in Azure.

3. Agenda
Scaled out AI Platform with Search (Bing and
Ads)
AI Platform is now used across Microsoft
Microsoft needs and drives Open and
Interoperable AI
Azure AI Services

4. Several Billion dollar AI businesses at
Microsoft
Learned a ton as we built them
Bing Ads Story
My AI Story

5. Step 3: Profit
Eric joins
MSFT

6. How

7. Bing Ads Execution
 Shipped once every 6 months
 3 experiments / month
 Big bets that didn’t work.
 Created dev teams with core metrics
 Pushed teams to move their metric
 Developed AI Infrastructure to do
rapid experimentation.
 Over a few years made >300x
improvement (1000/mo expr)

8. Experiment – 90% of ideas fail
Iterate – faster you try things, the more successful ideas you have
Build Infrastructure to enable fast iteration of experiments

10. RPM Gains

12. Search Scenario Evolution
Keyword Based Search
Natural Language Search
Voice, Vision, Context-Based Search

13. How to ensure best results?

14.  Vector-based search:
 Word is represented as vector.
 Vector captures word meaning – its semantics.
 Words with similar meanings get similar vectors.
 In Bing, we trained a GloVe model over
Bing’s query logs and generated 300-
dimensional vector, enabled by deep
neural network.
Deep Learning: Semantic Understanding

15. Vector-based Retrieval
 DL-generated vectors semantically represent queries, documents and passages
 Doc retrieval based on query-doc-passage semantic similarity (vector distance)
AND
long
does
…
canned
how
Query: {how long
does a canned
soda last}
canned
does
how
long
…
Posting 1
Posting 2
Posting 3
Posting 4
…
Matching
BM25
Perfect Match
Sem. Similarity
Vector
Recall
…
(0.78, 0.8, 0.4, 0.3, …)
(0.75, 0.6, 0.1, 0.8, …)
…
Approximate Nearest Neighbor
Search (ANN)
RANKING STACK

17. Bing AI Platform – Bing QnA
DATA PREP BUILD TRAIN DEPLOY INTELLIGENT APPS
ACTIONINTELLIGENCEDATA
Stored on
Cosmos
Æther with
TensorFlow
Philly
Ultra-fast Inferencing
using FPGAs

18. Youngji Kim
Principal Program Manager
Bing Relevance and AI

20. AI Expands Beyond Bing

21. AI Transforms all businesses across Microsoft
 AI is now done in almost every team
 Products that don’t seem to obviously use AI are powered by it
 Using the same tools and infrastructure to accelerate new teams

22. Took some time to warm up to the ideas…
To: Eric Boyd
From: xxxxx
Subject: ABC Experiment
…we have decided to ship the
XXX feature without running
the experiment first. We’re
quite confident in the
feature.
Our teams have a rich history
of shipping features without
any experimentation, something
that may be alien for folks
from Bing.
To: xxxxx
From: Eric Boyd
Subject: Re: ABC Experiment
I have a rich history of
driving my car with my eyes
closed. I’m quite confident I
know the way to work and have
only crashed a few times.

25. 0
5k
10k
15k
Dailykeptcount
Rule-based Machine learning
Models trained on
exploration data
Filter using user
interactions
Offline model
Machine Learning for SmartArt Suggestions
0
5
10
15
20
25
30
35
40
45
1-Apr 15-Apr 29-Apr 13-May 27-May 10-Jun 24-Jun 8-Jul 22-Jul 5-Aug 19-Aug 2-Sep 16-Sep 30-Sep 14-Oct 28-Oct 11-Nov 25-Nov
Slidekeptrate

26. Machine Learning for SmartArt Suggestions
0
50k
100k
150k
Dailykeptcount
Rule-based Machine learning
0
5
10
15
20
25
30
35
40
45
1-Apr 15-Apr 29-Apr 13-May 27-May 10-Jun 24-Jun 8-Jul 22-Jul 5-Aug 19-Aug 2-Sep 16-Sep 30-Sep 14-Oct 28-Oct 11-Nov 25-Nov
Slidekeptrate
More SmartArts,
trained on all data
Offline model
Filter using user
interactions
Models trained on
exploration data

27. Office – PowerPoint Designer
DATA PREP BUILD & TRAIN DEPLOY INTELLIGENT APPS
ACTIONINTELLIGENCEDATA
Stored on
Cosmos
Æther with
TLC (ML.NET)
Power Point
User Behavior
Data prep using
Azure Databricks
osi

28. Anand Balachandran
Principal Program Manager
Office Intelligent Services

31. Every business at Microsoft investing in AI
+ more!
 1000s of Data Scientists & AI Developers
 Classical ML & Deep Learning
 Compliant & non-compliant data
 Internal & OSS frameworks
 Internal & OSS tools & languages
 Deployments to Azure, AP, Windows, Mobile, Edge

32. Building your own AI models for Transforming Data into Intelligence
Prepare Data Build & Train Deploy

33. 8 Exabytes
30M events/sec
From 1B+ devices
Cosmos
Step 1: Prepare Data
GDPR compliant cloud data processing
system for near real-time ingestion &
processing

34. System for managing ML pipelines
Optimized for rapid prototyping, data reuse,
and collaboration
>20M Pipelines
>1.75M active Datasources (>2.7 total)
Æther
Step 2: Build and Train

35. DLIS
Run multiple models in parallel
ML, DL, Transforms & Featurizers
Abstracting platform details
(CPU/GPU/FPGA)
600K req/sec
<35ms latency
Æther
Step 3: Deploy

36. Hyperparameter Tuning
 As the models get more complex, the
space of hyperparameters to explore
increases exponentially!
 Grid or Random Search becomes too
costly
 Bayesian Optimization is used to optimize
Acquisition Functions

37. AI Platform
investments
Built an exabyte-scale data lake for everyone
to put their data of all types (structured and
unstructured)
Built AI tools approachable by any developer
Built machine learning tools for collaborating
across large experiment models
Summary

39. Project Kinect for Azure

40. How do we get from raw point clouds to cloud intelligence?
Wall
Floor
Ceiling
Table
Chair
Background
Project Kinect for Azure

41. Depth and Deep Learning
Using the power of our AI tools and infrastructure, we can take that raw output and train a
model capable of high fidelity environment perception.
Input: Project
Kinect for Azure
Raw Depth +
Active Brightness data Labelling Tool
Labelled
“Ground Truth”
Training Set Test Set
Philly
CNTK model
Analytics Client
ONNX model

42. Michelle Brook
Program Manager II
AI Perception and Mixed Reality

43. Labelled “Ground
Truth”
Raw Depth + Active
Brightness data
Labelling Tool Labelled “Ground
Truth”
Building the Training Data – How do we scale?
TABLE

46. Caffe2 | TensorFlow | PyTorch | CNTK Framework Backends
Hardware-Backed Libraries
e.g. CPU, GPU, FPGA

49. Open
ecosystem
for interoperable
AI models
Tools Should
Work Together.

50. ONNX enables models to be trained in one framework and transferred to another for inference.
CPUGPU
ML HW
DSPFPGA
High level API &
Framework Frontends
Hardware Vendor
Libraries & Devices
Any tools exporting ONNX models can benefit ONNX-compatible runtimes and libraries designed
to maximize performance on some of the best hardware in the industry.
Seamless Interoperability
ONNX.ai

51. Partners

52. Contribute
Get Involved
github.com/onnx
ONNX is a community project. We
encourage you to join the effort
and contribute feedback, ideas,
and code. Join us on Github.
Use ONNX
ONNX.ai
Start experimenting today. Check
out our Getting Started Guide,
Supported Tools, and Tutorials.
Follow Us
Stay up to date with the latest
ONNX news.
onnxai
onnxai

54. Empowering Physicians with Medical ImagingAI

55. Data Scientists Developers

57. Intelligent Disease Prediction
Data Prep Build Train Deploy Intelligent Apps
Azure Machine
Learning
IoT Hub
WindowsML
IOT Edge
Model:
DenseNet-121
Code:
Keras +
TensorFlow
National Institute
of Health
Chest Xray Data
112,120 images
14 pathology labels
30,805 unique patients
Visual Studio
Tools for AI
Azure
Deep Learning GPU VM
VSTS +
CI/CD
CosmosDB +
Azure Functions
NuGet

58. Chris Lauren
Principal Program Manager
AI Platform

60. With , you don’t have to be the
size of Bing to solve the problem

61. Bringing the best of AI to Azure and the best of Azure to AI
Pre-Build AI
Azure Cognitive Services
Conversational AI
Azure Bot Services
Custom AI
Azure Machine Learning

62. Integrated with Azure Machine Learning
Create new deep learning projects easily
Scale Out with Azure Batch AI
Monitor model training progress & GPU utilization
Visualize your model processing with integrated open
tools like TensorBoard
Get started quickly with the Samples Gallery
Productive AI developer tools to train
models and infuse AI into your apps

63. The Machine Learning framework made by and for .NET developers
Proven & Extensible
Open Source
Supported on Windows, Linux, and macOS
Developer Focused
Join at github.com/dotnet/machinelearning
Customizable
ML.NET Preview
Cross-platform open Source Machine learning framework for .NET
Extensively used across Microsoft: Windows, Bing, Azure
High productivity for complete workflow
Extensible to other frameworks (TensorFlow, CNTK…)

64. Microsoft scale Tools/Services available/coming to you
 Cognitive Toolkit (CNTK)
 ML.Net
 ONNX
 Azure Batch AI
 Hyper Parameter Tuning in Azure ML
 Project BrainWave (FPGA model acceleration) in Azure ML
 Visual Studio Tools for AI
And more coming soon…

67. SeeingAI.com
seeingai@microsoft.com

71.  Cloud data processing system
 Near real time ingestion
 Near real time processing
 Fully GDPR compliant
 8 Exabytes
 30M events/sec
 From 1B+ devices
COSMOS Æther
 System for managing ML
pipelines
 Optimized for rapid
prototyping, data reuse, and
collaboration
• >20M Pipelines
• >1.75M active Datasources
(>2.7 total)
Data Prep -> Build/Train -> Deploy
DLIS
 Run multiple models in
parallel
 ML, DL, Transforms &
Featurizers
 Abstracting platform details
(CPU/GPU/FPGA)
 600K req/sec
 <35ms latency