This document discusses Big Data and provides definitions and examples. It defines Big Data as very large and loosely structured data sets that are difficult to process using traditional database and software techniques. Examples of Big Data sources include social networks and machine-to-machine data. The document also discusses Hadoop and NoSQL databases as tools for managing and analyzing Big Data, and provides examples of companies using these technologies.

1. Cloud Computing
BigData
2011.12
- - 2.0
.

3. babokim@gmail.com
) (www.gruter.com)
SDS, NHN
www.jaso.co.kr
www.cloudata.org
www.cloumon.org
www.twitter.com/babokim
www.facebook.com/babokim

4. BigData Definition(1)
Big Data(BD) / /
, ,
DB (McKinsey, 2011)
What - SW , ,
is
DB (IDC, 2011)
BigData? - Big Data
( ) , ,
, ,
Gartner McKinsey
Economist (2011.03) (2011.05)
(2010.05)
21 /
SNS M2M ,
/
, , , , 5
Information silo 6
: Big Data, (KT )

5. BigData Definition(2)
Very large, distributed aggregations of loosely
structured data
Petabytes/exabytes of data,
Millions/billions of people,
Billions/trillions of records,
Loosely-structured and often distributed data,
Flat schemas with few complex interrelationships,
Often involving time-stamped events,
Often made up of incomplete data,
Often including connections between data elements that must
be probabilistically inferred,
Applications that involved Big-data can be
Transactional (e.g., Facebook, PhotoBox), or,
Analytic (e.g., ClickFox, Merced Applications).
http://wikibon.org/wiki/v/Enterprise_Big-data

6. Big-data Analytics Complements Data Warehouse
Traditional Data Warehouse
- Complete record from transactional system
- All data centralized
- Analytics designed against stable environment
- Many reports run on a production basis
Big-data Analytic Environment
- Data from many sources inside and outside of organization
(including traditional DW)
- Data often physically distributed
- Need to iteration solution to test/improve models
- Large-memory analytics also part of iteration
- Every iteration usually requires complete reload of information
http://wikibon.org/wiki/v/Enterprise_Big-data

7. Facebook Social plug-in
Transactional
process over 20 billion events per
day (200,000 events per second)
with a lag of less than 30 seconds.
Feedback
Analytic

8. BigData
Collecting Store Analysis Reporting/Searching
, SNS
/
Senti-
Cluster- Classifi-
mental Indexing
ing cation
Analysis
( )
/
Repository/
User Define
Query Script
Robot
RSS Reader (DBMS,
ETL
NoSQL)
OpenAPI
Index
Data Aggregator

9. Twitter : backtype
Workers choose queue to enqueue All updates for same URL
to using hash/mod of URL guaranteed to go to same worker
Workers share the load of
schemifying tweets
Distribute tweets randomly Workers schemify tweets Workers update statistics on URLs by
on multiple queues and append to Hadoop incrementing counters in Cassandra

10. BigData
Architectural Requirements ?
Scalability
- Scale-out -
- Elasticity -
Reliability -
- - Hadoop
-
Flexibility Component ,
- Easy for adding Analysis Rule
- Support various data format
Latency
- Real time, Near Real time, Batch
High Throughput IBM, HP, Oracle
- Global web scale traffic
- ~ /sec
-
- BI/DW

11. BigData
Flume, Scribe, Chukwa
Hadoop FileSystem
MogileFS
, NoSQL(Cloudata, HBase,
Cassandra)
Katta, ElasticSearch
count, sum aggregation S4, Storm
Hadoop MapReduce(Hive,
,
Pig)
Giraph, GoldenOrb
/ Cluster, Classification
Mahout, R
ZooKeeper, HUE, Cloumon
Serialization Thrift, Avro, ProtoBuf

12. Hadoop Echo System
http://indoos.wordpress.com/2010/08/16/hadoop-ecosystem-world-map/

13. Software Stack
Interface
Rule Management
Web Phone Pad
Data Visualization
(Near)Real-time Batch Analysis
Analysis
Management
Analysis Job
Monitoring
(cloumon)
Analysis Job Mining Lib Statistics Lib
Script Language(Hive, Pig) (Mahout) (R)
Real-time Analysis
Platform Job Workflow Engine(oozie, cascade)
CEP Engine
Data Analysis Platform(hadoop)
(Esper)
Management
(ZooKeeper)
Cluster
Aggregator Data Store
Collector File System NoSQL Search
(flume, scribe, chukwa) (HadoopFS) (Cloudata, HBase, Cassandra) (ElasticSearch)

14. Application
Application Server Collector #1
Server Log4j
Centralized
Storage
Agent
log (HDFS)
(local)
Collector #2
Temp Log
Chukwa(Yahoo)
Hadoop FileSystem
HDFS
MapReduce ( )
Scribe(Facebook)
(thrift)
Hadoop JNI
Flume(Cloudera)
, ,
Hadoop, HBase, Search Engine

15. - Esper Event
- Gruter ClouStream, Yahoo S4, Twitter Storm, Facebook Puma
ClouStream
Puma

16. : Hadoop File System
BigData Defacto Standard
x86
/
NameNode SPOF(Single Point Of Failure)

17. : MapReduce

18. : Hadoop MapReduce
MapReduce ,
MapReduce
MapReduce
Hadoop FileSystem
/
DB, FTP Server
FIFO, Fair, Capacity
/
MapReduce ,
(streaming)

19. : Script Language
Hive
Hive> CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING);
hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE invites;
hive> SELECT a.foo FROM invites a WHERE a.ds='2008-08-15';
hive> FROM pokes t1 JOIN invites t2 ON (t1.bar = t2.bar) INSERT OVERWRITE TABLE events SEL
ECT t1.bar, t1.foo, t2.foo;
Pig
Visits = load /data/visits as (user, url, time);
Visits = foreach Visits generate user, Canonicalize(url), time;
Pages = load /data/pages as (url, pagerank);
VP = join Visits by url, Pages by url;
UserVisits = group VP by user;
UserPageranks = foreach UserVisits generate user,
AVG(VP.pagerank) as avgpr;
GoodUsers = filter UserPageranks by avgpr > 0.5 ;
store GoodUsers into '/data/good_users';

20. Next Generation Hadoop(0.23)
YARN
(Next MapReduce Framework)
HDFS Federation

21. NoSQL
, , Scale-out
,
Key/value, Document , Simple Column
Schema Free
Big Data
x86
CAP(Brewers Conjecture)
Eventually consistent / BASE (not ACID)
Simple API
Twitter: Cassandra, HBase, Hadoop, Scribe,
FlockDB, Redis
Facebook: Cassandra, HBase, Hadoop, Scribe,
Hive
Netflix: Amazon SimpleDB, Cassandra
Digg: Cassandra
SimpleGeo: Cassandra
StumbleUpon: HBase, OpenTSDB
Yahoo!: Hadoop, HBase, PNUTS
Rackspace: Cassandra
DAUM: MongoDB
NCSoft: Cassandra

22. NoSQL: Cloudata/HBase
Distributed Data Storage Create, drop, modify table schema
semi-structured data store(not file system)
Single row operation
/ Multi row operation: like, between
Google Bigtable clone
Data Model, Architecture, Features Scanner, Direct Uploader, MapReduce Adapter
Open source
http://www.cloudata.org Automatic table split & re-assignment
Goal (Hadoop)
500 nodes Failover
300 GB /node, Peta bytes ~

24. seenal.com

27. 10.29

29. BigData .
, BigData
.
, , , BigData
.
BigData , Data
.
,
. .
.
(6 ~ 1 )
.
.
.
.

30. .
Facebook:
babokim@gruter.com
www.jaso.co.kr