2. Outline
Introduction of data mining and caret
before model training
building model
advance topic
exercise
·
·
visualization
pre-processing
Data slitting
-
-
-
·
Model training and Tuning
Model performance
variable importance
-
-
-
·
feature selection
parallel processing
-
-
·
/
4. Introduction of caret
The caret package (short for Classification And REgression Training) is a set of functions that
attempt to streamline the process for creating predictive models. The package contains tools for:
data splitting
pre-processing
feature selection
model tuning using resampling
variable importance estimation
·
·
·
·
·
/
7. visualizations
The featurePlot function is a wrapper for different lattice plots to visualize the data.
Scatterplot Matrix
boxplot
featurePlot(x=iris[,1:4],
y=iris$Species,
plot="pairs",
##Addakeyatthetop
auto.key=list(columns=3))
featurePlot(x=iris[,1:4],
y=iris$Species,
plot="box",
##Addakeyatthetop
auto.key=list(columns=3))
/
15. data splitting
create balanced splits of the data
set.seed(1)
trainIndex<-createDataPartition(iris$Species,p=0.8,list=FALSE, times=1)
head(trainIndex)
irisTrain<-iris[trainIndex,]
irisTest<-iris[-trainIndex,]
summary(irisTest$Species)
createResample can be used to make simple bootstrap samples
createFolds can be used to generate balanced cross–validation groupings from a set of data.
·
·
/
16. Model Training and Parameter Tuning
The train function can be used to
evaluate, using resampling, the effect of model tuning parameters on performance
choose the "optimal" model across these parameters
estimate model performance from a training set
·
·
·
/
20. Model Training and Parameter Tuning
define the type of resampling method
k-fold cross-validation (once or repeated)
leave-one-out cross-validation
bootstrap (simple estimation or the 632 rule)
·
·
·
fitControl<-trainControl(method="repeatedcv",
#10-foldcrossvalidation
number=10,
#repeated3times
repeats=3)
/
23. The trainControl Function
method: The resampling method
number and repeats: number controls with the number of folds in K-fold cross-validation or
number of resampling iterations for bootstrapping and leave-group-out cross-validation.
verboseIter: A logical for printing a training log.
returnData: A logical for saving the data into a slot called trainingData.
classProbs: a logical value determining whether class probabilities should be computed for held-
out samples during resample.
summaryFunction: a function to compute alternate performance summaries.
selectionFunction: a function to choose the optimal tuning parameters.
returnResamp: a character string containing one of the following values: "all", "final" or "none".
This specifies how much of the resampled performance measures to save.
·
·
·
·
·
·
·
·
/
24. Alternate Performance Metrics
Performance Metrics:
Another built-in function, twoClassSummary, will compute the sensitivity, specificity and area under
the ROC curve
regression: RMSE and R2
classification: accuracy and Kappa
·
·
fitControl<-trainControl(method="repeatedcv",
number=10,
repeats=3,
classProbs=TRUE,
summaryFunction=twoClassSummary)
treemodel<-train(x=trainData,
y=trainClass,
method='rpart',
trControl=fitControl,
tuneGrid=tunedf,
metric="ROC")
treemodel
/
26. Evaluating Test Sets
caret also contains several functions that can be used to describe the performance of classification
models
testPred<-predict(treemodel,testData)
testPred.prob<-predict(treemodel,testData,type='prob')
postResample(testPred,testClass)
confusionMatrix(testPred,testClass)
/
29. Exploring and Comparing Resampling
Distributions
Given these models, can we make statistical statements about their performance differences? To do
this, we first collect the resampling results using resamples.
We can compute the differences, then use a simple t-test to evaluate the null hypothesis that there is
no difference between models.
resamps<-resamples(list(tree=treemodel,
nnet=nnetmodel))
bwplot(resamps)
densityplot(resamps,metric='ROC')
difValues<-diff(resamps)
summary(difValues)
/
30. Variable importance evaluation
Variable importance evaluation functions can be separated into two groups:
model-based approach
Model Independent approach
·
·
For classification, ROC curve analysis is conducted on each predictor.
For regression, the relationship between each predictor and the outcome is evaluated
-
-
#model-basedapproach
treeimp<-varImp(treemodel)
plot(treeimp)
#ModelIndependentapproach
RocImp<-varImp(treemodel,useModel=FALSE)
plot(RocImp)
#or
RocImp<-filterVarImp(x=trainData,y=trainClass)
plot(RocImp)
/
31. feature selection
Many models do not necessarily use all the predictors
Feature Selection Using Search Algorithms("wrapper" approach)
Feature Selection Using Univariate Filters('filter' approach)
·
·
·
/
33. feature selection: wrapper approach
feature selection based on random forest model
pre-defined sets of functions: linear regression(lmFuncs), random forests (rfFuncs), naive Bayes
(nbFuncs), bagged trees (treebagFuncs)
ctrl<-rfeControl(functions=rfFuncs,
method="repeatedcv",
number=10,
repeats=3,
verbose=FALSE,
returnResamp="final")
Profile<-rfe(x=trainData,
y=trainClass,
sizes=1:8,
rfeControl=ctrl)
Profile
/