Recomendados
Mais conteúdo relacionado
Último
Último (20)
Destaque
Destaque (20)
Big data
- 3. 25 20 P1 P2 P3 15 P4 P5 P6 10 P7 P8 P9 P10 5 P11 0 1-Nov-11 2-Nov-11 3-Nov-11 4-Nov-11 5-Nov-11 6-Nov-11 7-Nov-11 8-Nov-11 9-Nov-11 10-Nov-11
- 4. Data BIG
- 5. • Value • Inter disciplinary • Lots of technical challenges • Better prospects
- 6. 1024 MEGABYTES = 1 GIGABYTE 1024 GIGABYTES = 1 TERABYTE …………………………………………………… …………………………………………………… …………………………………………………… …. 1024 TERABYTES = 1 PETABYTE …………………………………………………… …………………………………………………… …………………………………………………… ….
- 7. 1 13.3 YEARS Petabyte OF HD-TV VIDEO 1.5 SIZE OF 10 BILLION PHOTOS ON Petabytes FACEBOOK 20 AMOUNT OF DATA PROCESSED Petabytes BY GOOGLE PER DAY 50 ENTIRE WRITTEN WORKS OF Petabytes MANKIND IN ALL LANGUAGES
- 19. Look ma, I have a supercomputer! • Amazon Web Services is ranked 102 in Top 500 • AWS offers large instance for $ 0.24 / hour • 1000 instances = $ 240 • Data transfer costs $ 0.12 / GB • 1 TB = $ 123 • Total cost = $ 363 ( ~ INR 20691)
- 20. Christmas sale Max capacity
- 21. Christmas sale Max capacity Under provision Demand
- 22. • Elastic • Pay as you go • Disaster management • Replication and fault tolerance
- 24. TOOLS
- 26. MR • MR is a parallel programming model and associated infra-structure • introduced by Google in 2004: • Assumes large numbers of cheap, commodity machines. • Failure is a part of life. • Tailored for dealing with Big Data • Simple • Scales well
- 27. MR • Who uses it? • Google (more than 1 million cores, rumours have it) • Yahoo! (more than 100K cores) • Facebook (8.8k cores, 12 PB storage) • Twitter • IBM • Amazon Web services • Edinburgh University • Many many small start-ups • http://wiki.apache.org/hadoop/PoweredBy
- 28. MR • Googlers' hammer for 80% of our data crunching • Large-scale web search indexing • Clustering problems for Google News • Produce reports for popular queries, e.g. Google Trend • Processing of satellite imagery data • Language model processing for statistical machine • translation • Large-scale machine learning problems • Just a plain tool to reliably spawn large number of tasks • e.g. parallel data backup and restore • The other 20%? e.g. Pregel Source: Zhao et al, Sigmetrics 09
- 29. Google trends Hadoop and Scala trends over the years - www.google.com/trends/
- 31. Example: Word Count Input sentences: • the cat • the dog Mapper output Key Value the 1 cat 1 the 1 dog 1
- 32. Example: Word Count Reducer 1 input Reducer 1 output the, 1 the, 2 the, 1 dog, 1 dog, 1 Reducer 2 input Reducer 2 output cat, 1 cat, 1
- 33. E0F