3. “But Watson couldn’t distinguish between polite language and profanity —
which the Urban Dictionary is full of”
- Eric Brown (IBM)
4. Subverting Machine Learning
for Fun And Profit
Ram Shankar Siva Kumar, John Walton
Email: Ram.Shankar@Microsoft.com; JoWalt@Microsoft.com
5. Goals
• This talk:
• Is a primer on Adversarial Machine Learning
• Will show, through a sampling, how ML algorithms are vulnerable
• Illustrates how to defend against such attacks
• This talk IS NOT
• An exhaustive review of all algorithms
• End goal: Gain an intuitive understanding of ML algorithms and how
to attack them
6. Agenda
• Motivation to Attack ML systems
• Practical Attacks and Defenses
• Best Practices
7. ML is everywhere…
“Machine Learning is shifting from an academic discipline to an
industrial tool” – John Langford
8. In Security…!!
“The only effective approach to
defending against today’s ever-increasing
volume and diversity of
attacks is to shift to fully
automated systems capable of
discovering and neutralizing
attacks instantly.”
- Mike Walker (on DARPA
Cyber Grand Challenge)
9. Computer
System
Data
Program
Output
Computer
System
Data
Output
Program
Traditional Programming
Machine Learning
Source: Lectures by Pedro Domingos
10. Things to note about
• For the program to be functional, input data must be functional
• What does a program/model look like?
• Literally, bunch of numbers and data points
• The output model can be expressed in terms of parameters:
Linear Regression
y = 225x + 875
3,500
3,000
2,500
2,000
1,500
1,000
500
0 R² = 0.574
1 2 3 4 5 6 7 8
Number of Logons
Time
Non-Linear
y = 982.23e0.1305x
R² = 0.6624
3,500
3,000
2,500
2,000
1,500
1,000
500
0
1 2 3 4 5 6 7 8
Number of Logons = 225 * Time + 875 Number of Logons = 982*e
0.1305* (Time)
11. Malicious Mindset
• Data and parameters define the model
• By controlling the data or parameters, you can change the model
• Where do you find them?
• Data
• At the source
• Collected in a big data store
• Stored in the cloud (MLaaS)
• Parameters:
• Code repository
12. The mother lode
Data is collected
Data is within
anomaly detector’s
purview
Anomaly is
significant for
detector
Anomaly is
surfaced!
Source: Arun Viswanathan, Kymie Tan, and Clifford Neuman, Deconstructing the Assessment of Anomaly-based Intrusion Detectors, RAID 2013.
13.
14. Putting it all together
• Opportunity = ML is/will be everywhere
• Prevalence = ML is/will be widely used in security
• Ease = (most) ML algorithms can be easily subverted by controlling
data/parameters
• High rate of return = Once subverted, you can evade or even control
the system
Opportunity * Prevalence * Ease * High Rate of Return =
15. Agenda
• Motivation to Attack ML systems
• Practical Attacks and Defenses
• Intuitive understanding of the algorithm
• How the system looks before the attack?
• How the system looks after the attack?
• How to defend from these attacks?
• Takeaway – From Evasion to total control of the system
• Best Practices
16. About the dataset
• Used Enron Spam Dataset
• Came out of the Federal investigation of Enron corporation
• Real world corpus of spam and ham messages.
• 619,446 email messages belonging to 158 users. After cleaning it up
(removing duplicate messages, discussion threads), you end up with
200,399 messages.
17.
18.
19. Word P(Word|Spam) P(Word|Ham)
Assets 0/3 2/3
Assignment 0/3 2/3
Cialis 3/3 0/3
Group 0/3 2/3
Viagra 1/3 0/3
Vallium 2/3 0/3
Naïve Bayes Algorithm
Choose whichever probability is higher:
푃 푆푝푎푚 푀 ∝ 푃 푆푝푎푚 ∗ 푃(W|Spam)
푃 퐻푎푚 푀 ∝ 푃 퐻푎푚 ∗ 푃(W|Ham)
P(Spam|M) = 0.5*(0/3)*(0/3)*(0/3) = 0
P(Ham|M) = 0.5*(2/3)*(2/3)*(2/3) = 0.14
Since 0.14 > 0 => Message is more likely to be
Ham
20. Before Attack
• Built a vanilla Naïve Bayes classifier on Enron email dataset (with
some normalizations)
• Goal: Given a new subject, can I predict if it is spam or ham?
• Testing on 20% of data, you get test accuracy of 62%
21. After the attack
• Good Word Attack: Introduce innocuous words in the message
E.g: Gas Meeting Expense Report Payroll
-> Test Accuracy dropped to 52.8%
100
80
60
40
20
0
0 10 20 30
False Positive Rate
Number of Benign words added
22. Takeaway
• How to use in real-world: Spear phishing
• By manipulating the input to the algorithm, we can increase the false
positive rate
• Make the system unusable!
23. Support Vector Machines – The Ferrari of ML
• Immensely popular
• Quite fast
• Deliver a solid performance
• Widely used in classification setting
In Security setting, beginning to gain
popularity in the Malware community.
• Goal: Given a piece of code, is it Malicious
or benign?
25. SVM Intuition
Choose the hyperplane, that maximizes the
margin between the positive and negative
examples!
Those examples on the boundary are called
support vectors!
26. Facts about SVMs
• Output of SVM = a set of weights + Support vectors
• Once you have the support vectors {special points in the
training data}, rest of the training data can be thrown away
• Takeaway: A good part of the model, is determined by
support vectors
• Intuition: Controlling the support vectors, should help us to
control the model
30. Attack Intuition
Center
Before Attack
After Attack
Attack Point
to be included
Source:Laskov, Pavel, and Marius Kloft. "A framework for quantitative security analysis of machine learning." Proceedings of the 2nd ACM workshop on Security and artificial intelligence. ACM, 2009.
31.
32. Takeaway
• In order to attack the algorithm, we don’t change the parameter
(centroid) -> Simply send in data as part of “normal” traffic
• Increased the false negative rate
33. Summary of Attacks
Algorithm Result of Attack What does this mean?
Naïve Bayes Increased false positive rate You can make the system unusable
K-means
clustering
Increased false negative rate You can evade detection
SVM Control of the decision boundary You have full control of what gets alerted
and what doesn’t
34. Ensembling – You can’t fool ‘em all
- Build separate models to detect
malicious activity
- The models are chosen so that they are
orthogonal
- Each model independently assess for
maliciousness
- Results are combining using a separate
function
35. • Used Gaussian Naïve Bayes, linear SVM in addition to Naïve Bayes
• Used a simple majority voting method, to combine the three outputs.
36. Using Robust Learning Methods
• Intuition: Treat the tainted data points
as outliers (presumably because of
noise)
Outlier?
37. Instead of Consider
Vanilla Naïve Bayes Multinomial Model (even better
than multivariate Bernoulli model)
SVM Robust SVM (feature noise, and
label noise)
K-means with finite window K-means with infinite window
Logistic Regression Robust Logistic Regression using
Shift Parameters
Vanilla PCA Robust PCA with Laplcian
Threshold (Antidote)
38. Caution!
• Pros: Well studied field with a gamut of choices
• Optimization perspective
• Game Theoretic perspective
• Statistical perspective
• Cons:
• Some of these algorithms have higher computational complexity than standard
algorithms
Standard SVM: 10 minutes Robust SVM: 1 hr and 8 mins
(Single node implementation, 50k data points, 20% test, no kernel )
• Requires a lot more tuning and babysitting
39. Agenda
• Motivation to Attack ML systems
• Practical Attacks and Defenses
• Best Practices
40. Threat Modeling
• Adversary Goal - Evasion? Poisoning? Deletion?
• Adversary’s knowledge – Perfect Knowledge? Limited Knowledge?
• Training set or part of it
• Feature representation of each sample
• Type of a learning algorithm and the form of its decision function
• Parameters and hyper-parameters of the learned model
• Feedback from the classifier; e.g., classifier labels for samples chosen by the
adversary.
• Attacker’s capability
• Ability to modify – Complete or partial?
Source:Biggio, Battista, Blaine Nelson, and Pavel Laskov. "Poisoning attacks against support vector machines." arXiv preprint arXiv:1206.6389 (2012).
41. Tablestakes
• Secure log sources
• Secure your storage space
• Monitor data quality
• Treat parameters and features as secrets
• Don’t use publically available datasets to train your system
• When designing the system, avoid interactive feedback
42. 3 Key Takeaways
1) Naïve implementation of machine Learning Algorithms are
vulnerable to attacks.
2) Attackers can evade detections, cause the system to be unusable or
even control it.
3) Trustworthy results depend on trustworthy data.
43. Thank you!
- TwC: Tim Burell
- Azure Security: Ross Snider, Shrikant
Adhirkala, Sacha Faust Bourque,
Bryan Smith, Marcin Olszewski,
Ashish Kurmi, Lars Mohr, Ben
Ridgway
- O365 Security: Dave Hull, Chetan
Bhat, Jerry Cochran
- MSR: Jay Stokes, Gang Wang
(intern)
- LCA: Matt Sommer
Source: http://www.lecun.org/gallery/libpro/20011121-allyourbayes/dsc01228-02-h.jpg
Notas do Editor
NELL or Never Ending Language Learning is a research project at CMU that learns the web. started by Tom Mitchell.
Uses innovative semi-supervised learning techniques, wherein it learns most of the facts on its own, but there is minimal human involvement.
For instance, it automatically learned that broiled chicken is a type of meat. You can even follow it on Twitter, and rate its confidence
One day, it learned that Donald Trump is a type of wig.
The same problem even plagued Watson. The researchers wanted to teach the couldn’t distinguish between polite language and profanity — which the Urban Dictionary is full of. Watson picked up some bad habits from reading Wikipedia as well. In tests it even used the word “bullshit” in an answer to a researcher’s query
By 2016, 25 Percent of Large Global Companies Will Have Adopted Big Data Analytics For At Least One Security or Fraud Detection Use Case
- Defenders are key;
Some things to note since the last slide,
-> For the end program to be useful, the input data must be functional
-> So what does this program, or in machine learning speak model actually look like? Actually, it is not any fancy equation or math. It is literally a bunch of numbers/data points.
To illustrate this we modeled the number of logons at various time. Our input to the system is a time series of logons. In case of a linear regression, you get a linear relationship. The red numbers, which are the parameters is what gets stored.
When we did a non-linear regression, only the numbers change.
So, what are the takeaways? Data (like the time series of logon) and parameters (the end numbers) define the model
By controlling one or both of them, you can control the model.
Quick digression – where do you find the data/parameters:
So, now that we know that an attacker can control the model or parameters, what can he do with it? For this we will walk through how Anomaly Detection system works.
Walkthrough
As you can see, once the data is corrupted, the end anomalies that get surfaced is also corrupted.
In fact, things get really bad. You can increase the false negative rate (and evade) detection.
You can increase the false positive rate and frustrate the Incident responder (or even take complete control of the system.
Here is the data set: when you are before attack.
One of the four datasets used in Spam -- CSDMC2010, SpamAs- sassin, LingSpam, and Enron-Spam.
I am going to Built a vanilla Naïve Bayes classifier on Enron email dataset (with some normalizations)
619,446 email messages belonging to 158 users. After cleaning it up (removing duplicate messages, discussion threads), you end up with 200,399 messages.
Which is Spam, and which is ham?
Intuition behind Naïve Bayes – Conditional independence – that is the naïve part. Also thought of as Bag of Words model.
Talk about Bag of Words assumption
I want to walk
What is normalization – You don’t estimate MLE; you estimate the log of it (makes calculation easy);
If it is marked out as spam from Outlook.
Spear phishing avoiding – The attackers can get succesfful – Drive by downloads.
They control the input and that is how they use the center point. I control the input, which influences the parameters.
A. Globerson and S. T. Roweis. Nightmare at test time: robust learning by feature deletion.
In William W. Cohen and Andrew Moore, eds, ICML, vol. 148 of ACM Int’l Conf. Proc.
Series, pp. 353–360, 2006.
Typically seek a Nash equilibrium: neither player has an incentive to
change his strategy
I Does not mean that either player’s payoff is maximized
Combine
i.e., how real objects such as emails, network packets are mapped into the classifier’s feature space;