Introduction to Mahout with HDInsight

MAKING BUSINESS INTELLIGENT
www.pragmaticworks.com
Introduction to Mahout
with HDInsight (Hadoop)
Chris Price
Senior BI Consultant
@BluewaterSQL

Intro
Chris Price
Senior BI Consultant with Pragmatic Works
Author
Regular Speaker
Data Geek & Super Dad!
@BluewaterSQL
http://bluewatersql.wordpress.com/
cprice@pragmaticworks.com

Survey
 Whose currently using Machine Learning?
 Google
 Facebook
 LinkedIn
 Twitter
 Amazon
 Wal-Mart

Outline
 Mahout Introduction
 The Algorithms
 Hands On:
 A recommendation engine

Riding the Elephant
 Born out of the Apache Lucene project
 Top-level Apache project
 A scalable machine learning library
 Fast, Efficient & Pragmatic
 Many of the algorithms can be run on Hadoop

Algorithms
 Collaborative Filtering
 Item/User Recommenders
 Clustering
 Grouping movies by type
 Classification
 Categorizing documents
 Frequent Itemset
 Market basket analysis

Collaborative Filtering
 Find subset of users who have similar
taste/preferences to target user and use this
subset for recommendations
 Types:
 User-Based
 Item-Based
 Examples:
 Amazon

Clustering
 Group similar objects
 Examples:
 News Aggregator
 Customer Grouping

Clustering
 Algorithms:
 K-Means
 Fuzzy K-Means
 Mean Shift
 Canopy
 Dirichlet
 Similarity Distance:
 Euclidean
 Squared Euclidean
 Cosine
 Tanimoto
 Manhattan
** Also weighted measures

Clustering

Classification
 Using a pre-determined set of groups:
 Predict the type of a new object based on its
features
 Classifiable Data
 Continuous – Quantitative Value (i.e. Stock Price)
 Categorical – Small known set (i.e. Colors)
 Word-Like – Large unknown set
 Text-Like – Many word-like that are unordered
 Examples:
 Spam Identification
 Photo Facial Recognition

Frequent Itemset
 Examples:
 Product Placement
 Market Basket Analysis
 Query Recommendations

Mahout on HDInsight
 Installation
 Download
 http://www.apache.org/dyn/closer.cgi/mahout/
 Unpack
 Add to Path (Environment Variable)

Recommendation Engine
 Define the Business Objective
 Metrics
 Context
 Identify Data
 Sources
 Normalization
 Data Shift
 Which Algorithm?
 Integration?

Business Objective
Navigational
Inefficiency
Cross-Sell
Up-Sell
Increase #
of Orders
Increase Items
per Order
Increase
Average
Item Price
Website
Increase
Revenue

Handling Context
???
January
20 degrees & Snowing…..

Data Acquisition
 Sources of Data for Recommendation
 Implicit
 Ratings
 Feedback
 Demographics
 Pyschographics (Personality/Lifestyle/Attitude),
 Ephemeral Need (Need for a moment)
 Explicit
 Purchase History
 Click/Browse History
 Product/Item
 Taxonomy
 Attributes
 Descriptions

Data Preparation
 Preparation
 Remove Outliers (Z-Score)
 Remove frequent buyers (Skew)
 Normalize Data (Unity-Based)
 Beware of Data Shift

Algorithms
 Collaborative Filtering (Mahout)
 User-Based
 Item-Based
 Content-Based (Mahout Clustering)
 Data Mining (SSAS)
 Association
 Clustering

CF Recommendations
 Neighborhood Formation
 Similarity Metrics
 Pearson Correlation
 Euclidean Distance
 Spearman Correlation
 Cosine
 Tanimoto Coefficient
 Log-Likelihood

CF Pseudo-Code
for each item i that u has no preference
for each user v that has a preference for i
compute similarity s between u and v
calculate running average of v‘s
preference for i, weighted by s
return top ranked (weighted average) i
Restrict to Neighborhood

Testing
 Smell Test
 Built-In (Requires Java Coding)
 Root Mean Squared Error (RMSE)
 Average Absolute Difference
RandomUtils.useTestSeed()
Evaluator.evaluate(builder,null,0.7,1.0)

Recommendation Engine Steps
 1 – Generate List of ItemIDs
 2 – Create Preference Vector
 3 – Count Unique Users
 4 – Transpose Preference Vectors
 5 – Row Similarity
 Compute Weights
 Computer Similarities
 Similarity Matrix
 6 – Pre-Partial Multiply, Similarity Matrix
 7 – Pre-Partial Multiply, Preferences
 8 – Partial Multiple (Steps 6 & 7)
 9 – Filter Items
 10 – Aggregate & Recommend

Batch Integration
 ETL Data to HDFS
 SSIS
 Map/Reduce
 Process with Mahout
 ETL Results
 Map/Reduce
 Hive/Sqoop

Hands-On Demo

Resources
 Mahout in Action
Sean Owen, Robin Anil,
Ted Dunning, Ellen Friedman
 Hadoop: The Definitive Guide
 Tom White

www.pragmaticworks.comMAKING BUSINESS INTELLIGENT
Thank you!
@BluewaterSQL
http://bluewatersql.wordpress.com/
cprice@pragmaticworks.com

Introduction to Mahout with HDInsight

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (9)

Destaque

Destaque (6)

Semelhante a Introduction to Mahout with HDInsight

Semelhante a Introduction to Mahout with HDInsight (20)

Último

Último (20)

Introduction to Mahout with HDInsight

Notas do Editor