machine learning, features engineering

1. Lecture 01
Introduction to Machine Learning

2. What is Machine Learning?
 All useful Programs “Learn” something.
 Early definition of Machine Learning
“ Field of study that gives computers the ability to learn without
being explicitly programmed” Arthur Samuel (1959).
 Computer pioneer who wrote first self-learning program,
which played checkers- learned from experience.
 Tom Mitchell(1997)- “Said to learn from experience with
respect to some class of tasks, and a performance measure P,
if[ the learner’s] performance at tasks in the class as measured
by P, improves with experience”

3. Contd…
Traditional Programming:
Machine Learning:
Machine Learning is the central part of some other
courses like Natural Language Processing, Computational
Biology, Computer Vision, robotics etc.
Computer
Data
Output
Program
Computer
Data
Program
Output

4. How are things learned?
 Memorization:
 Accumulation of individual facts
Limited by-
Time to observe facts
Memory to store facts
 Generalization:
 Deduce new facts from old facts
Limited by accuracy of deduction process
 Essentially a predictive activity
 Assumes that past predicts the future
 Interested in extending to programs that can infer useful
information from implicit patterns in data.
Declarative Knowledge
Imperative Knowledge

5. Basic Paradigm:
 Observe set of examples: Training data
e.g., Football Players, labeled by position with height and
weight data.
 Infer something about process that generated that data
 Find canonical model of position by statistics
 Use interface to make predictions about previously
unseen data: Test data
e.g., predict position of new players.

6. Variations of Paradigm:
 Supervised Learning
 Learn an input to output map
• Classification: categorical output
• Regression: continuous output
Unsupervised Learning:
 Discover patterns in the data
• Clustering: cohesive grouping
• Association: Frequent cooccurance
 Reinforcement Learning
 Learning control

7. Machine Learning Tasks:
Task Measure
 Classification Classification error
 Regression Prediction error
 Clustering scatter/purity
 Associations Support/confidence
Challenges:
 How good is a model?
 How do I choose a model?
 Is the data of sufficient quality?
 Errors in data. E.g., Age=225; noise in low resolution images
 Missing vales

8. Contd…
 How confident can I be of the results?
 Am I describing the data correctly?
 Are age and income enough? Should I look at Gender also?
 How should I represent age? As a number or as young,
middle age, old?

9. Supervised Learning:
Labeled Training Data
Classification Possible
Classifiers

10. Supervised Learning:

11. Inductive Bias:
 Need to generalize
Assumption about lines
Encoding/Normalization of data
In general Inductive Bias: X1=
<0.15,0.25>,Y1= -1
 Language Bias and Search Bias X2=
<0.4,0.45>,Y2= +1
Process of supervised learning:
Training data X1=<30000,25>,Y1= doesnotbuycomputer
X2=<80000,45>,Y2= buycomputer
X1, Y1
X2, Y2
X3, Y3
-------
-------
Training Algorithm
X1, Y1
X2, Y2
X3, Y3
Testing
data
Classifier
Validiation

12. Supervised Learning:
Applications:
1. Credit Card Fraud detection (valid transaction or
not)
2. Sentiment Analysis (Opinion mining, buzz
analysis etc)
3. Churn Prediction (Potential Churner or not?)
4. Medical Diagnoses (Risk Analysis)

13. Prediction or Regression:

14. Regression:
Applications:
1.Time series Predictions
Rain Fall in a certain region, spend on
voice call
2.Classifications
3. Data Reduction
4. Trend Analysis
5. Risk Factor Analysis

15. Examples of Classifying and Clustering:

16. Unlabeled Data:

17. Clustering examples into group:

18. Similarity based on weight:

19. Similarity based on Height:

20. Cluster into two groups using both attributes:

21. Labeled Data:

22. Finding Classifier surface:
 Given Labeled groups into feature space, want to
find a surface
in that space that separates the groups.
 Subject to constraints on the complexity of
subsurface
 In this example, have 2 D space, so find line(or
connected set of line segments) that best
separates the two groups.
 When the examples get separated, it is straight
forward.
 When examples in labeled groups overlap, may
have to trade of false positives and false negative.

23. Labeled Data:

24. Adding some new data:
 Suppose we have learned to separate receivers
versus linemen
 Now we are given some running backs, and want to
use model to decide if they are more like receiver or
linemen.
Blount = [‘blount’, 75, 250]
white = [ ‘white’, 70, 205]

25. Clustering using Unlabeled data:

26. Classified using Labeled data:

27. Machine Learning Methods:

28. Requirements for Methods:
 Choosing training data and evaluation method.
 Representation of the features
 Distance matrix for feature vectors
 Objective function and constraints
 Optimization method for learning the model

29. Feature Representation:

30. An Example:

31. Contd…

34. Need to measure distance between Features
Feature Engineering:
 Deciding which features to include and which
are merely adding noise to classifier.
 Defining how to measure distances between
training examples (and ultimately between
classifiers and new instances)
 Deciding how to weight relative importance of
different dimensions of features vector, which
impacts defination of distance.

35. Measuring distance between animals:
 We can think of our animal examples as consisting
of four binary features and one integer features.
 One way to learn to separate Reptiles from non-
reptiles is to measure the distance between pairs of
examples and use that:
 To cluster nearby examples into a common class
(Unlabeled) data or
 To find a classifier surface in space of examples
that optimally separates different (Labeled)
collections of examples from ther collections.

36. Minkowski Matric:

37. Euclidean Distance between animals:

38. Euclidean Distance between animals:
Using Euclidean Distance rattlesnake and boa
constrictor are much closer to each other , than they are
to the dart frog.

39. Add an Alligator
Alligator is closer to dart frog than snake-why?
 Alligator differs from frog in 3 features, from boa in
only 2 features.
 But scale on “legs” is from 0 to 4, on other features
is 0 to 1
 “legs” dimension is disproportionately large

40. Using Binary Features:
Now alligator is closer to snakes than it is to
frog
- Make sense

41. Clustering Approach:
 Suppose we know that there are k different groups
in our training data, but we don’t know labels.
 Pick K samples (at random?) as exemplars
 Cluster remaining samples by minimizing distance
between samples in same cluster(objective function)
put samples in group with closest exemplars.
 Find median example in each cluster as new
exemplars
 Repeat until no change
Issues:
 How do we decide on the best number of clusters?
 How do we select the best features, the best
distance metric?

42. Classification Approach:
 Want to find boundaries in feature space that
separate different classes of labeled examples.
 Look for simple surface (e.g., best line or plane)
that separates classes.
 Look for more complex surface(subject to
constraints) that separate classes.
 Use voting schemes.
use k nearest training examples, use majority vote
to select label.
Issues:
 How do we avoid over-fitting to data?
 How do we measure performance?
 How do we select best features?

43. Classification:
 Attempt to minimize error on training data
 Similar to fitting a curve to data
Evaluate on training data.

44. Randomly Divide Data into Training and Test set:

45. Two possible models for a Training Set

46. Confusion Matrices (Training Error)

47. Training Accuracy of Model:
 0.7 for both models?
Which is better?
 Can we find a model with less training error? Yes.

48. Applying Model to test Data:

49. Other statistical Measures:

50. Summery:
 Machine learning methods provide a way of
building models of processes from data sets
 Supervised Learning uses labeled and create
classifiers that optimally separate data into known
classes.
 Unsupervised Learning tries to infer latent
variables by clustering training examples into
nearby groups.
 Choice of features influences results.
 Choice of distance measurement between
examples influences results.
 We will see some clustering methods like k-
means.