1.
1
Advanced Analytics with Big Data
Thomas W. Dinsmore

2.
Advanced Analytics with Big Data
•What do we mean by “Big Data”?
•Do we need to use all of the data?
•What analytics can run inside Big Data platforms?
2

3.
Big Data
•Data that cannot be efficiently handled in a relational database
•The three Vs:
•Volume
•Variety
•Velocity
3

4.
Big Data Platforms
•Hadoop ecosystem: MapReduce, Hive, Impala, Spark etc
•Appliances: Teradata, IBM PureData, Pivotal, Oracle BDA, Vertica, Par Accel/Redshift etc etc
•NoSQL/NewSQL: Cassandra, Mongo, MemSQL
•Streaming engines: Infosphere Streams
4
Convergence: Federated SQL engines (e.g.) Pivotal Hawq

5.
6
Analytics Platform
For aggregate models, you can simply sample the data and work offline.

6.
7
Anomaly
Detection
Affinity
Analysis
Microsegmentation
Social Network Analysis
Collaborative
Filtering
However, for some use cases you may need to use all of the data.

7.
8
Catastrophic Risk Modeling
Modeling with Fine-grained
Behavioral Data
For other use cases, using all of the data is worth extra time and effort.

8.
9
HDFS
HDFS
HDFS
HDFS
HDFS
HDFS
Data
Most legacy analytic packages can read HDFS files.

9.
10
HDFS
HDFS
HDFS
HDFS
HDFS
HDFS
MapReduce
Data
Some tools also provide pass-through capabilities.

10.
11
HDFS
HDFS
HDFS
HDFS
HDFS
HDFS
MapReduce
Advantages
•Co-exists w/ other applications
•Integrated workload management
•Simplified administration
Disdvantages
•MapReduce latency
Several tools translate user requests to MapReduce. This eliminates data movement and co-exists well with other applications.

11.
12
YARN
HDFS
Map Reduce
HDFS
Map
Reduce
HDFS
Map Reduce
HDFS
Map
Reduce
HDFS
Map Reduce
HDFS
Map
Reduce
Advantages
•Easy to adapt legacy apps
•Isolates analytic workload
Disdvantages
•Data moves within the cluster
•Requires YARN
YARN (*) makes it possible to bypass MapReduce and run analytics in memory on dedicated nodes.
(*) Yet Another Resource Negotiater

12.
13
HDFS
Map
Reduce
YARN
HDFS
Map Reduce
HDFS
Map Reduce
HDFS
Map
Reduce
HDFS
Map Reduce
HDFS
Map
Reduce
HDFS
Map
Reduce
Advantages
•Lowest latency
Disdvantages
•Upgrade every node
•Requires YARN
Distributing in-memory analytics across the Hadoop cluster minimizes internal data movement.

13.
14
Open Source Projects

14.
Apache Mahout
•Apache incubator project (2007)
•Machine learning library
•Included in most distributions
•Thin acceptance, few contributors
•Diverse architecture
•Single-node
•MapReduce
•New algos run on Spark
•Recently cleaned up
15

15.
Apache Giraph
•Apache top-level project
•Runs in MapReduce
•Dedicated graph engine
•Used by Facebook, few others
•Dead in the water
•No presence in leading distros
•No significant commercial support
•No releases in 13 months
•No recent code commits on Git
16

16.
GraphLab
•Carnegie Mellon project (2009)
•Distributed in-memory engine:
•Primarily graph analysis
•Selected machine learning algos
•Interface from Java, JavaScript, Python
•GraphLab Inc provides commercial support (2013, $6.75MM)
•Independent distribution, or through Pivotal
•Minimal development effort past six months
17

17.
0xdata H2O
•Vendor-driven open source project
•0xdata sells support, customization
•Distributed in-memory prediction engine
•Multiple deployment options:
•Standalone (with HDFS)
•Over YARN
•In MapReduce
•Claims 2,000+ users
•4 public references
•Used by a leading P&C insurer
•Java, R, Python and Scala interfaces
18

18.
Apache Spark
•Top-level Apache project (2/14)
•Release 1.02 (8/14)
•Distributed in-memory analytics
•Machine learning
•Graph analytics
•Streaming analytics
•Fast SQL
•Compatible with Hadoop storage
•Integrated with YARN
•Scala, Python, Java interfaces (+SparkR)
•Growing ecosystem
•Supported in leading Hadoop distributions
19

19.
Analytic Features
22
0xdata H2O 2.2
Apache Giraph 1.1
Apache Mahout 0.9
Apache Spark 1.02
GraphLab 2.2
Prediction
+++
+
+++
Dimension Reduction
+
+++
+
+
Clustering
+
+++
+
+++
Collaborative Filtering
+++
+
+++
Text Analytics
+++
+++
Matrix Operations
+
+++
+
Graph Analysis
+
+
+++

20.
Summary: Open Source
•Giraph appears to be dead in the water
•Mahout may be recovering from roadkill status
•GraphLab outperforms Spark GraphX today in graph analytics
•0xdata H2O currently has more machine learning features than Spark MLLib and a better R interface
•Spark catching up fast
•More resources and distribution
•Integrated platform for ML and graph analysis
23

21.
24
Commercial Software

22.
Alpine
•Business user interface
•Collaboration environment
•Broad library of techniques
•Strong cloud offering
•Leverages Hadoop (multiple distros), Hawq or Pivotal Greenplum
•Push-down MapReduce
•Certified on Spark
•Small but growing customer base
25

23.
IBM SPSS Analytics Server
•Introduced 2013
•Serves as “back end” for SPSS Modeler
•Uses push-down MR
•Limited analytic feature set
•IBM supports on multiple Hadoop distros
•Customer acceptance unknown
26

24.
Revolution Analytics ScaleR
•ScaleR library of distributed statistics, machine learning functions
•Tools to distribute arbitrary R functions
•Runs in Cloudera, Hortonworks, Teradata, LSF clusters, MS HPC
•Hadoop edition uses MR push-down
•Tools simplify installation in large clusters
•R interface
•Partnerships with Alteryx, Qlik, MicroStrategy, Tableau provide business interfaces
27

25.
Skytree Server
•Georgia Tech’s FastLab project, repurposed as commercial software
•Distributed machine learning platform
•Very opaque about technical details
•User interface is an API
•Co-located in Hadoop under YARN
•Just certified by Hortonworks
•Customer acceptance unknown
•No new public references in a year
•Used by leading credit card company
28

26.
SAS High-Performance Analytics
•Distributed in-memory analytics
•Designed to run in special-purpose appliances (2011)
•Repurposed to run in Hadoop (2013)
•Co-exists poorly — cannot run SAS and MapReduce at the same time
•Reads entire dataset into memory
•Uses MPI to communicate among nodes
•Requires upgrades from standard Hadoop infrastructure
•Customer acceptance unknown
•No public references
•Generic success stories missing from Strata presos
29

27.
SAS LASR Server
•SAS’ “other” distributed in-memory platform
•Back end for several end-user products
•SAS Visual Analytics (2012)
•SAS Visual Statistics (New)
•SAS In-Memory Statistics for Hadoop (New)
•Recently added statistics and machine learning
•Does not read raw HDFS; must be transformed to proprietary SASHDAT
•Like HPA, reads entire dataset into memory.
•16 Core 256GB node can load 75GB table
•Runs DS2 programs, not Legacy SAS programs
•Fast, but with limited feature set
•SAS claims 1,400 “sites” for Visual Analytics
•Many of those are standalone boxes
30

28.
Summary: Commercial
•Alpine’s interface is compelling to business user
•IBM Analytics Server is a good first release
•RRE ScaleR appeals to R users, plays well in Hadoop sandbox
•Skytree Server: strong in prediction
•SAS: why two competing memory-centric architectures?
31

29.
Progress
•Spark: blindingly fast maturity
•Rapidly expanding library of analytic features
•Growing developer community, ecosystem
•Commercial: from zero to many
32

30.
Interesting Questions
•Will Mahout get a second wind?
•Will Spark MLLib displace 0xdata?
•Will Spark GraphX catch up to GraphLab?
•Can Spark Streaming compete with Storm and commercial entrants?
•How quickly will customers adopt memory-centric architecture for analytics?
•What will Alpine and MicroStrategy do with Spark?
•When will SAS announce a reference customer for HPA/LASR in Hadoop?
33

31.
Questions
34

32.
Thank You
35

33.
36
Advanced Analytics with Big Data
Thomas W. Dinsmore