Big Data Analytics and Cloud Computing Insights

Big Data Analytics
Jon Einkauf
Senior Product Manager, Amazon Elastic MapReduce

1. Introducing Big Data
2. From data to actionable information
3. Analytics and Cloud Computing
Overview

Generation
Collection & storage
Analytics & computation
Collaboration & sharing

The cost of data generation
is falling

Generation
Lower cost,
higher throughput

Generation
Lower cost,
higher throughput
Highly
constrained

Generated data
Available for analysis
Data volume
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares

Elastic and highly scalable
No upfront capital expense
Only pay for what you use
+
+
Available on-demand
+
=
Remove
constraints

Generation
Accelerated

Technologies and techniques for
working productively with data,
at any scale.
Big Data

From data to
actionable information
2

3.5 billion records
13 TB of click stream logs
71 million unique cookies
Per day:

500% return on ad spend
17,000% reduction in procurement time
Results:

“Who is using our
service?”

Identified early mobile usage
Invested heavily in mobile development
Finding signal in the noise of logs

9,432,061 unique mobile devices
used the Yelp mobile app.
4 million+ calls. 5 million+ directions.
In January 2013

Open web index.
3.4 billion records.
Available to all.

Full parse for impact of
social networks
300 lines of Ruby code.
14 hours.
$100.

You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011
Tweeting about Flu

Analytics and
Cloud Computing
3

Generation
S3, Glacier,
Storage Gateway,
DynamoDB,
Redshift, RDS,
HBase

Generation
EC2 &
Elastic MapReduce

Generation
EC2 & S3,
CloudFormation,
Elastic MapReduce,
RDS, DynamoDB, Redshift

Generation
EC2 & S3,
CloudFormation,
Elastic MapReduce,
RDS, DynamoDB, Redshift
EC2 &
Elastic MapReduce
S3, Glacier,
Storage Gateway,
DynamoDB,
Redshift, RDS,
HBase
AWS Data Pipeline

How does it work?
EMR
EMR ClusterS3
1. Put the data
into S3 (or HDFS)
3. Get the
results
2. Launch your cluster.
Choose:
• Hadoop distribution
• How many nodes
• Node type (hi-CPU,
hi-memory, etc.)
• Hadoop apps (Hive,
Pig, HBase)

EMR
EMR Cluster
How does it work?
S3
You can
easily resize
the cluster

EMR
EMR Cluster
How does it work?
S3
Use Spot
nodes to
save time
and money

EMR
EMR Cluster
How does it work?
S3
Launch parallel clusters
against the same data
source (tune for the
workload)

How does it work?
EMR ClusterS3
When the work is complete,
you can terminate the cluster
(and stop paying)

EMR Cluster
How does it work?
You can store
everything in HDFS
(local disk)
High Storage nodes
= 48 TB/node

EMR Cluster
How does it work?
Launch in a Virtual
Private Cloud for
extra security

Thousands of Customers, 5+ Million Clusters

Give it a try.
Cost to run a 100-node EMR cluster:
£4.90 / hour

AWS Data Pipeline
Data-intensive orchestration and automation
Reliable and scheduled
Easy to use, drag and drop
Execution and retry logic
Map data dependencies
Create and manage temporary compute
resources

Additional checks and notifications

Thanks.
jeinkauf@amazon.com
To Learn More:
aws.amazon.com/elasticmapreduce
aws.amazon.com/datapipeline
aws.amazon.com/big-data

Back to the Future
Big Data at Channel 4
Bob Harris
Chief Technology Officer – Channel 4 Television
April 2013

The Disclaimer
<IMHO>
blah blah blah…..
</IMHO>

C4 in the Cloud
• 2008 – Started investigations into Cloud Computing
• 2008 – Launched our first applications on AWS
• 2009 – Entered into an Enterprise Agreement with Amazon for AWS
Rapid growth of AWS based offerings during 2009/2010
• 2011 – AWS established as the default platform of choice for new websites

C4 in the Cloud
• 2008 – Started investigations into Cloud Computing
• 2008 – Launched our first applications on AWS
• 2009 – Entered into an Enterprise Agreement with Amazon for AWS
Rapid growth of AWS based offerings during 2009/2010
• 2011 – AWS established as the default platform of choice for new websites
• 2012 – Adopted cloud-based analytics
• 2013 – Investigating cloud-based back-up and archiving

Business Intelligence at C4
• Well established Business Intelligence capability
• Based on industry standard proprietary products
• Real-time data warehousing
• Comprehensive business reporting
• Excellent internal skills
• Good external skills availability

Big Data at C4
2011
• Embarked on Big Data initiative in 2011
• Ran in-house and cloud-based PoCs
• Selected AWS Elastic Map Reduce
2012
• Ran EMR in parallel with conventional BI stack
• Hive deployed to Data Analysts in 2012
• EMR workflows deployed to production in 2012
2013
• EMR confirmed as primary Big Data platform
• EMR usage growing, focus on automation
• Experimenting with R and Mahout

Big Data at C4 – Elastic MapReduce
• AWS EMR established as our Big Data platform of choice
• Friendly front-end developed to allow Data Analysts to
start/stop clusters and submit/track queries.

Big Data at C4 – Big Data Control Panel

Big Data at C4 – Elastic MapReduce
• AWS EMR established as our Big Data platform of choice
• Friendly front-end developed to allow Data Analysts to
start/stop clusters and submit/track queries.
• Production workflows written predominantly in Python and
Pig
• Fully integrated with our conventional BI stack making
EMR outputs available for reporting
• Experimenting with ADP (AWS Data Pipeline)
• Next steps – MapR and HBase

Personalising the viewer experience
Most popular dramas
Drama
collections
US drama
Single view of the viewer
recognising them across devices
and serving relevant content
Big Data – Improving Viewer Experience

Myths or Truths? – It’s all about Perspective!
• Nothing that can’t be done with an RDBMS
• It’s a completely different approach
• It’s really difficult
• It’s immature and lacks good tools
• It’s totally incompatible with you current BI platform
and tools
• It’s difficult to find skilled and experienced staff
Image by Tayrawr Fortune
Elastic MapReduce has provided a cost effective
approach to establishing our Big Data platform

That’s all folks…
bharris@channel4.co.uk
@bobharrisuk
uk.linkedin.com/in/bobharrisuk01

Alan Priestley
EMEA Enterprise Marketing
Intel Corporation

Analysis of Data Can Transform Society
Create new business
models and improve
organizational
processes.
Enhance scientific
understanding, drive
innovation, and
accelerate medical cures.
Increase public safety
and improve
energy efficiency with
smart grids.

Democratizing Analytics gets Value out of Big Data
Unlock Value in
Silicon
Support Open
Platforms
Deliver Software Value

Intel at the Intersection of Big Data
Enabling exascale
computing on massive
data sets
Helping enterprises
build open
interoperable clouds
Contributing code
and fostering
ecosystem
HPC Cloud Open Source

Intel at the Heart of the Cloud
Server
Storage
Network

Scale-Out Platform Optimizations for Big Data
Cost-effective performance
•Intel® Advanced Vector Extension Technology
•Intel® Turbo Boost Technology 2.0
•Intel® Advanced Encryption Standard New
Instructions Technology

66
Intel® Advanced Vector Extensions Technology
• Newest in a long line of
processor instruction
innovations
• Increases floating point
operations per clock up to
2X1 performance
1 : Performance comparison using Linpack benchmark. See backup for configuration details.
For more legal information on performance forecasts go to http://www.intel.com/performance
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are
measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other
information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Intel® Turbo Boost Technology 2.0
More Performance
Higher turbo speeds maximize
performance for single and
multi-threaded applications

Intel® Advanced Encryption
Standard New Instructions
• Processor assistance for
performing AES encryption
7 new instructions
• Makes enabled encryption
software faster and stronger

Power of the Platform built by Intel
Richer
user
experiences
4HRS
50%
Reduction
10MIN
80%
Reduction 50%
Reduction 40%
Reduction
TeraSort for
1TB sort
Intel®
Xeon®
Processor
E5 2600
Solid-State
Drive 10G
Ethernet Intel® Apache
Hadoop
Previous
Intel®
Xeon®
Processor

Cloud
Intelligent Systems
Clients
Virtuous Cycle of Data-Driven Experience

Get 600 Hours of free supercomputing
time!
www.powerof60.com

Big Data Analytics and Cloud Computing Insights

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Big Data Analytics and Cloud Computing Insights

Semelhante a Big Data Analytics and Cloud Computing Insights (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Último

Último (20)

Big Data Analytics and Cloud Computing Insights