1. An Introduction to RandomForests™
Salford Systems
http://www.salford-systems.com
golomi@salford-systems.com
Dan Steinberg, Mikhail Golovnya, N. Scott Cardell

2.  New approach for many data analytical tasks developed by
Leo Breiman of University of California, Berkeley
◦ Co-author of CART® with Friedman, Olshen, and Stone
◦ Author of Bagging and Arcing approaches to combining trees
 Good for classification and regression problems
◦ Also for clustering, density estimation
◦ Outlier and anomaly detection
◦ Explicit missing value imputation
 Builds on the notions of committees of experts but is
substantially different in key implementation details

3.  The term usually refers to pattern discovery in large data bases
 Initially appeared in the late twentieth century and directly
associated with the PC boom
◦ Spread of data collection devices
◦ Dramatically increased data storage capacity
◦ Exponential growth in computational power of CPUs
 The necessity to go way beyond standard statistical techniques
in data analysis
◦ Dealing with extremely large numbers of variables
◦ Dealing with highly non-linear dependency structures
◦ Dealing with missing values and dirty data

4.  The following major classes of problems are
usually considered:
◦ Supervised Learning (interested in predicting some
outcome variable based on observed predictors)
 Regression (quantitative outcome)
 Classification (nominal or categorical outcome)
◦ Unsupervised Learning (no single target variable
available- interested in partitioning data into cluster,
finding association rules, etc.)

5.  Relating gene expressions to the presence of a
certain decease based upon microarray data
 Indentifying potential fraud cases in credit card
transactions (binary target)
 Predicting level of user satisfaction as poor, average,
good, excellent (4-level target)
 Optical Digit Recognition (10-level target)
 Predicting consumer preferences towards different
kinds of vehicles (could be as many as several
hundred level target)

6.  Predicting efficacy of a drug based upon demographic factors
 Predicting the amount of sales (target) based on current
observed conditions
 Predicting user energy consumption (target) depending on
the season, business type, location, etc.
 Predicting medium house value (target) based on the crime
rate, pollution level, proximity, age, industrialization level,
etc.

7.  DNA Microarray Data- which samples cluster together? Which
genes cluster together?
 Market Basket Analysis- which products do customers tend to
buy together?
 Clustering For Classification- Handwritten zip code problem:
can we find prototype digits for 1,2, etc. to use for
classification?

8.  The answer usually has two sides:
◦ Understanding the relationship
◦ Predictive accuracy
 Some algorithms dominate one side (understanding)
◦ Classical methods
◦ Single trees
◦ Nearest neighbor
◦ MARS
 Others dominate the other side (predicting)
◦ Neural nets
◦ TreeNet
◦ Random Forests

9.  Leo Breiman says:
◦ Framing the question as the choice between accuracy
and interpretability is an incorrect interpretation of what
the goal of a statistical analysis is
 The goal is NOT interpretability, but accurate information
 Nature’s mechanisms are generally complex and cannot be
summarized by a relatively simple stochastic model, even as
a first approximation
 The better the model fits the data, the more sound the
inferences about the phenomenon are

10.  The only way to attain the best predictive accuracy o
real life data is to build a complex model
 Analyzing this model will also provide the most
accurate insight!
 At the same time, the model complexity makes it far
more difficult to analyze it
◦ A random forest may contain 3,000 trees jointly
contributing to the overall prediction
◦ There could be 5,000 association rules found in a typical
unsupervised learning algorithm

11.  (Insert table)
 Example of a classification tree for UCSD
heart decease study

12.  Relatively fast
 Requires minimal supervision by analyst
 Produces easy to understand models
 Conducts automatic variable selection
 Handles missing values via surrogate splits
 Invariant to monotonic transformations of predictors
 Impervious to outliers

13.  Piece-wise constant models
 “Sharp” decision boundaries
 Exponential data exhaustion
 Difficulties capturing global linear patterns
 Models tend to evolve around the strongest effects
 Not the best predictive accuracy

14.  A random forest is a collection of single trees grown in a
special way
 The overall prediction is determined by voting (in
classification) or averaging (in regression)
 The law of Large Numbers ensures convergence
 The key to accuracy is low correlation and bias
 To keep bias low, trees are grown to maximum depth

15.  Each tree is grown on a bootstrap sample from the learning
set
 A number R us specified (square root by default) such that it
is noticeably smaller than the total number of available
predictors
 During tree growing phase, at each node only R predictors are
randomly selected and tried

16.  All major advantages of a single tree are automatically
preserved
 Since each tree is grown on a bootstrap sample, one can
◦ Use out of bag samples to compute an unbiased estimate of
the accuracy
◦ Use out of bag samples to determine variable importances
 There is no overfitting as the number of trees increases

17.  It is possible to compute generalized proximity between any pair
of cases
 Based on proximities one can
◦ Proceed with a well-defined clustering solution
◦ Detect outliers
◦ Generate informative data views/projections using scaling
coordinates
◦ Do missing value imputation
 Easy expansion into the unsupervised learning domain

18.  High levels of predictive accuracy delivered automatically
◦ Only a few control parameters to experiment with
◦ Strong for both regression and classification
 Resistant to overtraining (overfitting)- generalizes well to new data
 Trains rapidly even with thousands of potential predictors
◦ No need for prior feature (variable) selection
 Diagnostic pinpoint multivariate outliers
 Offers a revolutionary new approach to clustering using tree-based
between-record distance measures
 Built on CART® inspired trees and thus
◦ Results invariant to monotone transformations of variables

19.  Method intended to generate a large number of substantially
different models
◦ Randomness introduced in two simultaneous ways
◦ By row: records selected for training at random with replacement (as in
bootstrap resampling of the bagger)
◦ By column: candidate predictors at any node are chosen at random and
best splitter selected from the random subset
 Each tree is grown out to maximal size and left unpruned
◦ Trees are deliberately overfit, becoming a form of nearest neighbor
predictor
◦ Experiments convincingly show that pruning these trees hurt performance
◦ Overfit individual trees combine to yield properly fit ensembles

20.  Self-testing possible even if all data is used for training
◦ Only 63% of available training data will be used to grow any one
tree
◦ A 37% portion of training data always unused
 The unused portion of the training data is known as Out-Of-Bag (OOB)
data and can be used to provide an ongoing dynamic assessment of
model performance
◦ Allows fitting to small data sets without explicitly holding back
any data for testing
◦ All training data is used cumulatively in training, but only a 63%
portion used at any one time
 Similar to cross-validation but unstructured

21.  Intensive post processing of data to extract more
insight into data
◦ Most important is introduction of distance metric
between any two data records
◦ The more similar two records are the more often they
will land in same terminal node of a tree
◦ With a large number of different trees simply count the
number of times they co-locate in same leaf nodes
◦ Distance metric can be used to construct dissimilarity
matrix input into hierarchical clustering

22.  Ultimately in modeling our goal is to produce a single
score, prediction, forecast, or class assignment
 The motivation generating multiple models is the
hope that by somehow combining models results will
be better than if we relied on a single model
 When multiple models are generated they are
normally combined by
◦ Voting in classification problems, perhaps weighted
◦ Averaging in regression problems, perhaps weighted

23.  Combining trees via averaging or voting will only be
beneficial if the trees are different from each other
 In original bootstrap aggregation paper Breiman noted
bagging worked best for high variance (unstable)
techniques
◦ If results of each model are near identical little to be
gained by averaging
 Resampling of the bagger from the training data
intended to induce differences in trees
◦ Accomplished essentially varying the weight on any
data record

24.  Bootstrap sample is fairly similar to taking a 65% sample from
the original training data
 If you grow many trees each based on a different 65% random
sample of your data you expect some variation in the trees
produced
 Bootstrap sample goes a bit further in ensuring that the new
sample is of the same size as the original by allowing some
records to be selected multiple times
 In practice the different samples induce different trees but
trees are not that different

25.  The bagger was limited by the fact that even with resampling
trees are likely to be somewhat similar to each other,
particularly with strong data structure
 Random Forests induces vastly more between tree differences
by forcing splits to be based on different predictors
◦ Accomplished by introducing randomness into split
selection

26.  Breiman points out tradeoff:
◦ As R increases strength of individual tree should increase
◦ However, correlation between trees also increases reducing advantage of
combining
 Want to select R to optimally balance the two effects
◦ Can only be determined via experimentation
 Breiman has suggested three values to test:
◦ R= 1/2sqrt(M)
◦ R= sqrt(M)
◦ R= 2sqrt(M)
◦ For M= 100 test values for R: 5,10,20
◦ For M= 400 test values for R: 10, 20, 40

27.  Random Forests machinery unlike CART in that
◦ Only one splitting rule: Gini
◦ Class weight concept but no explicit priors or costs
◦ No surrogates: Missing values imputed for data first automatically
 Default fast imputation just uses means
 Compute intensive method uses tree-based nearest neighbors to base
imputation on (discussed later)
◦ None of the display and reporting machinery are tree refinement
services of CART
 Does follow CART in that all splits are binary

28.  Trees combined via voting (classification) or averaging
(regression)
 Classification trees “vote”
◦ Recall that classification trees classify
 Assign each case to ONE class only
◦ With 50 trees, 50 class assignments for each case
◦ Winner is the class with the most votes
◦ Votes could be weighted- say by accuracy of individual trees
 Regression trees assign a real predicted value for each case
◦ Predictions are combined via averaging
◦ Results will be much smoother than from a single tree

29.  Probability of being omitted in a single draw is (1-1/n)
 Probability of being omitted in all n draws is (1-1/n)n
 Limit of series as n increases is (1/e)= 0.368
◦ Approximately 36.8% sample excluded 0% of resample
◦ 36.8% sample included once 36.8% of resample
◦ 18.4% sample included twice thus represent…36.8% of resample
◦ 6.1% sample included three times…18.4% of resample
◦ 1.9% sample included four or more times…8% if resample 100%
◦ Example: distribution of weights in a 2,000 record resample:
◦ (insert table)

30.  Want to use mass spectrometer data to classify
different types of prostate cancer
◦ 772 observations available
 398- healthy samples
 178- 1st type of cancer samples
 196- 2nd type of cancer samples
◦ 111 mass spectra measurements are recorded for each
sample

31.  (insert table)
 The above table shows cross-validated prediction success
results of a single CART tree for the prostate data
 The run was conducted under PRIORS DATA to facilitate
comparisons with subsequent RF run
◦ The relative error corresponds to the absolute error of
30.4%

32.  Topic discussed by several Machine Learning researchers
 Possibilities:
◦ Select splitter, split point, or both at random
◦ Choose splitter at random from the top K splitters
 Random Forests: Suppose we have M available predictors
◦ Select R eligible splitters at random and let best split node
◦ If R=1 this is just random splitter selection
◦ If R=M this becomes Brieman’s bagger
◦ If R<< M then we get Breian’s Random Forests
 Breiman suggests R=sqrt(M) as a good rule of thumb

33.  A performance of a single tree will be somewhat driven by the
number of candidate predictors allowed at each node
 Consider R=1: the splitter is always chosen at random +
performance could be quite weak
 As relevant splitters get into tree and tree is allowed to grow
massively, single tree can be predictive even if R=1
 As R is allowed to increase quality of splits can improve as
there will be better (and more relevant) splitters

34.  (insert graph)
 In this experiment, we ran RF with 100 trees on the
prostate data using different values for the number
of variables Nvars searched at each split

35.  RF clearly outperforms single tree for any number of Nvars
◦ We saw above that a properly pruned tree gives cross-validated absolute
error of 30.4% (the very right end of the red curve)
 The performance of a single tree tends to deviate substantially
with the number of predictors allowed to be searched (a single
tree is a high variance object)
 The RF reaches the nearly stable error rate of about 20% when
only 10 variables are searched in each node (marked by the blue
color)
 Discounting the minor fluctuations, the error rate also remains
stable for Nvars above 10
◦ This generally agrees with Breiman’s suggestion to use square root N=111
as a rough estimate of the optimal value for Nvars
 The performance for small Nvars can be usually further improved
by increasing the number of runs

36.  (insert graph)

37.  (insert table)
 The above results correspond to a standard RF run
with 500 trees, Nvars=15, and unit class weights
 Note that the overall error rate is 19.4% which is
2/3 of the baseline CART error of 30.4%

38.  RF does not use a test dataset to report accuracy
 For every tree grown, about 30% of data are left out-of-bag
(OOB)
 This means that these cases can be safely used in place of the
test data to evaluate the performance of the current tree
 For any tree in RF, its own OOB sample is used- hence no bias is
ever introduced into the estimates
 The final OOB estimate for the entire RF can be simply obtained
by averaging individual OOB estimates
 Consequently, this estimate is unbiased and behaves as if we had
an independent test sample of the same size as the learn sample

39.  (insert table)

40.  The prostate dataset is somewhat partially unbalanced- class 1
contains fewer records than the remaining classes
 Under the default RF settings, the minority classes will have
higher misclassification rates than the dominant classes
 Misbalance in the individual class error rates may also be caused
by other data specific issues
 Class weights are used in RF to boost the accuracy of the
specified classes
 General Rule of Thumb: to increase accuracy in the given class,
one should increase the corresponding class weight
 In many ways this is similar to the PRIORS control used in CART
for the same purpose

41.  Our next run sets the weight for class one to
2
 As a result, class 1 is classified with a much
better accuracy at the cost of slightly reduced
accuracy in the remaining classes

42.  At the end of an RF run, the proportion of votes for
each class is recorded
 We can define Margin of a case simply as the
proportion of votes for the true class minus the
maximum proportion of votes for the other classes
 The larger the margin, the higher the confidence of
classification

43.  (insert table)
 This extract shows percent votes for the top 30
records in the dataset along with the
corresponding margins
 The green lines have high margins and therefore
high confidence of predictions
 The pink lines have negative margins, which means
that these observations are not classified correctly

44.  The concept of margin allows new “unbiased” definition of variable
importance
 To estimate the importance of the mth variable:
◦ Take the OOB cases for the ldh tree, assume that we already know the margin for
those cases M
◦ Randomly permute all values of the variable m
◦ Apply the ldh tree to the OOB cases with the permuted values
◦ Compute the new margin M
◦ Compute the difference M-M
 The variable importance is defined as the average lowering of the margin
across all OOB cases and all trees in the RF
 This procedure is fundamentally different from the intrinsic variable
importance scored computed by CART- the latter are always based on
the LEARN data and are subject to the overfitting issues

45.  The top portion of the variable importance list for the
data is shown here
 Analysis of the complete list reveals that all 111
variables are nearly equally strongly contributing to
the model predictions
 This is in a striking contrast with the single CART tree
that has no choice but to use a limited subset of
variables by tree’s construction
 The above explains why the RF model has a
significantly lower error rate (20%) when compared to
a single CART tree (30%)

46.  RF introduces a novel way to define proximity between two
observations
◦ Initialize proximities to zeroes
◦ For any given tree, apply the tree to all cases
◦ If case I and j both end up in the same node, increase proximity prox(ij)
between I and j by one
◦ Accumulate over all trees in RF and normalize by twice the number of trees
in RF
 The resulting matrix of size NxN provides intrinsic measure of
proximity
◦ The measure is invariant to monotone transformations
◦ The measure is clearly defined for any type of independent variables,
including categorical

47.  (insert graph)
 The above extract shows the proximity matrix for the
top 10 records of the prostate dataset
◦ Note ones on the main diagonal- any case has
“perfect” proximity to itself
◦ Observations that are “alike” will have proximities
close to one
 these cells have green background
◦ The closer proximity to 0, the more dissimilar cases i
and j are
 These cells have pink B

48.  Having the full intrinsic proximity matrix opens new horizons
◦ Informative data views using metric scaling
◦ Missing value imputation
◦ Outlier detection
 Unfortunately, things get out of control when dataset size
exceeds 5,000 observations (25,000,000+ cells are needed)
 RF switches to “compressed” form of the proximity matrix to
handle large datasets- for any case, only M closest cases are
recorded. M is usually less than 100.

49.  The values 1-prox(ij) can be treated as Euclidean distances
in a high dimensional space
 The theory of metric scaling solves the problem of finding
the most representative projections of the underlying data
“cloud” onto low dimensional space using the data
proximities
◦ The theory is similar in spirit to the principal components analysis
and discriminant analysis
 The solution is given in the form of ordered “scaling
coordinates”
 Looking at the scatter plots of the top scaling coordinates
provides informative views of the data

50.  (insert graph)
 This extract shows five initial scaling coordinates for
the top 30 records of the prostate data
 We will look at the scatter plots among the first,
second, and third scaling coordinates
 The following color codes will be used for the target
classes:
◦ Green- class 0
◦ Red- class 1
◦ Blue- class 2

51.  (insert graphs)
 A nearly perfect separation of all three classes is clearly seen
 From this we conclude that the outcome variable admits clear
prediction using RF model which utilizes 111 original
predictors
 The residual error is mostly due to the presence of the “focal”
point where all the three rays meet

52.  (insert graph)

53.  (insert graphs)
 Again, three distinct target classes show up as
separate clusters
 The “focal” point represents a cluster of records
that can’t be distinguished from each other

54.  Outliers are defined as cases having small proximities to
all other cases belonging to the same target class
 The following algorithm is used:
◦ For a case n, compute the sum of the squares of prox(nk) for all k
in the same class as n
◦ Take the inverse- it will be large if the case is “far away” from the
rest
◦ Standardize using the median and standard deviation
◦
◦ Look at the cases with the largest values- those are potential
outliers
 Generally, a value above 10 is reason to suspect the case
of being an outlier

55.  This extract shows top 30 records of the prostate
dataset sorted descending by the outlier measure
 Clearly the top 6 cases (class 2 with IDs: 771, 683,
539, and class 0 with IDs 127, 281, 282) are
suspicious
 All of these seem to be located at the “focal point”
on the corresponding scaling coordinate plots

56.  (insert graph)

57.  RF offers two ways of missing value imputation
 The Cheap Way- conventional median imputation for continuous
variables and mode imputation for categorical variables
 The Right Way:
◦ Suppose case n has x coordinate missing
◦ Do the Cheap Way imputation for starters
◦ Grow a full size RF
◦ We can now re-estimate the missing value by a weighted average
◦ over all cases k with non-missing x using weights prox(nk)
◦ Repeat steps 2 and 3 several times to ensure convergence

58.  An alternative display to view how the target classes are
different with respect to the individual predictors
◦ Recall, at the end of an RF run all cases in the dataset, obtain K
separate votes for the class membership (assuming K target
classes)
◦ Take any target class and sort all observations by the count of
votes for this class descending
◦ Take the top 50 observations and the bottom 50 observations,
those are correspondingly the most likely and the least likely
members of the given target class
◦ Parallel coordinate plots report uniformly (0,1) scaled values of all
predictors for the top 50 and bottom 50 sorted records, along
with the 25th, 50th and j percentiles within each predictor

59.  (insert graph)
 This is a detailed display of the normalized values
of the initial 20 predictors for the top voted 50
records in each target class (this gives 50x3=150
graphs)
 Class 0 generally has normalized values of the
initial 20 predictors close to 0 (left side 0tt, lw, y,
o, ragg, wp) except perhaps M9X11

60.  (insert graph)
 It is easier to see this when looking at the quartile
plots only
 Note that class 2 tends to have the largest values
of the corresponding predictors
 The graph can be scrolled forward to view all of the
111 predictors

61.  (insert graph)
 The least likely plots roughly result to the similar
conclusions: small predictor values are the least
likely for class 2, etc.

62.  RF admits an interesting possibility to solve unsupervised learning
problems, in particular, clustering problems and missing value
imputation in the general sense
 Recall that in the unsupervised learning the concept of target is not
defined
 RF generates a synthetic target variable in order to proceed with a
regular run:
◦ Give class label 1 to the original data
◦ Create a copy of the data such that each variable is sampled independently from the
values available in the original dataset
◦ Give class label 2 to the copy of the data
◦ Note that the second copy has marginal distributions identical to the first copy,
whereas the possible dependency among predictors is completely destroyed
◦
◦ A necessary drawback is that the resulting dataset is twice as large as the original

63.  We now have a clear binary supervised learning problem
 Running an RF on this dataset may provide the following
insights:
◦ When the resulting misclassification error is high (above 50%), the
variables are basically independent- no interesting structure exists
◦ Otherwise, the dependency structure can be further studied by looking at
the scaling coordinates and exploiting the proximity matrix in other ways
◦ For instance, the resulting proximity matrix can be used as an important
starting point for the subsequent hierarchical clustering analysis
 Recall that the proximity measures are invariant to monotone
transformations and naturally support categorical variables
 The same missing value imputation procedure as before can now
be employed
 These techniques work extremely well for small datasets

64.  We generated a synthetic dataset based on the
prostate data
 The resulting dataset still has 111 predictors but
twice the number of records- the first half being
the exact replica of the original data
 The final error is only 0.2% which is an indication of
a very strong dependency among the predictors

65.  (insert graph)
 The resulting plots resemble what we had before
 However, this distance is in terms of how
dependent the predictors are, whereas previously it
was in terms of having the same target class
 In view of this, the non cancerous tissue (green)
appears to stand apart from the cancerous

66.  + Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123-140.
 + Breiman, L. (1996). Arcing classifiers (Technical Report). Berkeley: Statistics
Department, University of California.
 + Buntine, W. (1991). Learning classification trees. In D.J. Hand, ed., Artificial
Intelligence Frontiers in Statistics, Chapman and Hall: London, 182-201.
 + Dietterich, T. (1998). An experimental comparison of three methods for
constructing ensembles of decision trees: Bagging, Boosting, and Randomization.
Machine Learning, 40, 139-158.
 + Freund, Y. & Schapire, R. E. (1996). Experiments with a new boosting algorithm.
In L. Saitta, ed., Machine Learning: Proceedings of the Thirteenth National
Conference, Morgan Kaufmann, pp. 148-156.
 + Friedman, J.H. (1999). RandomForests. Stanford: Statistics Department, Stanford
University.
 + Friedman, J.H. (1999). Greedy function approximation: a gradient boosting
machine. Stanford: Statistics Department, Stanford University.
 + Heath, D., Kasif, S., and Salzberg, S. (1993) k-dt: A multi-tree learning method.
Proceedings of the Second International Workshop on Multistrategy Learning,
1002-1007, Morgan Kaufman: Chambery, France.
 + Kwok, S., and Carter, C. (1990). Multiple decision trees. In Shachter, R., Levitt,
T., Kanal, L., and Lemmer, J., eds. Uncertainty in Artificial Intelligence 4, North-
Holland, 327-335.