2. ... - 2003
2 days in 2011
10 minutes in 2013
5 billion GB
Live stats
2016
3. Where those data comes from?
Activity Listening music, reading a book, searching, shopping, etc.
Our conversations in social media are now digitally recorded.Conversation
We upload and share 100s of thousands of them on social media
sites every second.
Photo and Video
We are increasingly surrounded by sensors that collect and share
data.
Sensor
We now have smart TVs that are able to collect and process data.The Internet of Things
4. The basic idea behind the phrase
'Big Data' is that everything we
do is increasingly leaving a digital
trace (or data), which we (and
others) can use and analyse
5. Big data :
means really a big data, it is
a collection of large
datasets that cannot be
processed using traditional
computing techniques.
6. Big Data includes huge volume, high velocity,
and extensible variety of data.
Structured
Item 2
Semi Structured Unstructured
● Database
● Census records
● Economic data
● Phone numbers
● JSON
● XML
● Word
● PDF
● Text
● Media Logs
7. Benefits of Big Data
https://www.youtube.com/watch?v=HqsBensINkE
8. Big Data Technologies
Operational Big Data
This include systems like MongoDB that
provide operational capabilities for real-
time, interactive workloads where data is
primarily captured and stored.
NoSQL Big Data systems are designed to
allow massive computations to be run
inexpensively and efficiently. This makes
operational big data workloads much
easier to manage, cheaper, and faster to
implement.
Analytical Big Data
This includes systems like Massively
Parallel Processing (MPP) database
systems and MapReduce that provide
analytical capabilities for retrospective
and complex analysis.
A system based on MapReduce can be
scaled up from single servers to
thousands of high and low end machines.
10. Traditional Approach
In this approach, an enterprise will have a
computer to store and process big data. Here
data will be stored in an RDBMS, process the
required data and present it to the users for
analysis purpose. tutorialspoint.com
11. Google’s
Solution
Google solved this problem using an
algorithm called MapReduce. This
algorithm divides the task into small
parts and assigns those parts to
many computers connected over
the network, and collects the results
to form the final result dataset.
tutorialspoint.com
12. Hadoop
Hadoop runs applications using the
MapReduce algorithm, where the
data is processed in parallel on
different CPU nodes. In short,
Hadoop framework is capable
enough to develop applications,
capable of running on clusters of
computers and they could perform
complete statistical analysis for a
huge amounts of data.
tutorialspoint.com
15. MapReduce
Data
Map
Converts data into another set of
data. Elements are broken down
into tuples (key/value pairs).
Reduce
Shuffle stage and the Reduce
stage that produces a new set
of output, which will be stored
in the HDFS.
1 2 3
18. HDFS
● Fault detection and recovery : HDFS
should have mechanisms for quick and
automatic fault detection and recovery.
● Huge datasets : HDFS should have
hundreds of nodes per cluster to manage
the applications having huge data sets.
● Hardware at data : A requested task can
be done efficiently.
tutorialspoint.com
Perkembangan teknologi, alat, dan media komunikasi yang semakin pesat, berbanding lurus dengan jumlah data yang dihasilkan oleh umat manusia. Dari awal terbentuknya bumi sampai 2003, ketika bilik-bilik warnet masih sepi, dan internet masih benda asing, data yang dihasilkan umat manusia itu sebanyak 5 milliar GB. Kemudian di tahun-tahun berikutnya, muncul friendster, facebook, twitter, pun perangkat baru mulai bermunculan seperti ipod, nokia yang dibekali dengan gprs sehingga umat manusia mulai menggunakan internet.
Delapan tahun berlalu, blackberry mulai booming, disertai dengan whatsapp, twitter, dan dalam 2 hari mampu memproduksi 5 milyar GB meskipun untuk paketan internet saat itu masih eman-eman. Android pun mulai menjamur beberapa tahun sesudahnya, pengguna pun mulai banyak, umat manusia sudah mulai terbiasa dengan paketan internet dan akhirnya data sebanyak 5 milyar GB dapat diproduksi dalam eaktu 10 menit.
Simple activities like listening to music or reading a book are now generating data. Digital music players and eBooks collect data on our activities. Your smart phone collects data on how you use it and your web browser collects information on what you are searching for. Your credit card company collects data on where you shop and your shop collects data on what you buy. It is hard to imagine any activity that does not generate data.
Our conversations are now digitally recorded. It all started with emails but nowadays most of our conversations leave a digital trail. Just think of all the conversations we have on social media sites like Facebook or Twitter. Even many of our phone conversations are now digitally recorded.
Just think about all the pictures we take on our smart phones or digital cameras. We upload and share 100s of thousands of them on social media sites every second. The increasing amounts of CCTV cameras take video images and we up-load hundreds of hours of video images to YouTube and other sites every minute .
We are increasingly surrounded by sensors that collect and share data. Take your smart phone, it contains a global positioning sensor to track exactly where you are every second of the day, it includes an accelometer to track the speed and direction at which you are travelling. We now have sensors in many devices and products.
We now have smart TVs that are able to collect and process data, we have smart watches, smart fridges, and smart alarms. The Internet of Things, or Internet of Everything connects these devices so that e.g. the traffic sensors on the road send data to your alarm clock which will wake you up earlier than planned because the blocked road means you have to leave earlier to make your 9am meeting…
Volume refers to the vast amounts of data generated every second. We are not talking Terabytes but Zettabytes or Brontobytes. If we take all the data generated in the world between the beginning of time and 2008, the same amount of data will soon be generated every minute. New big data tools use distributed systems so that we can store and analyse data across databases that are dotted around anywhere in the world.
Velocity refers to the speed at which new data is generated and the speed at which data moves around. Just think of social media messages going viral in seconds. Technology allows us now to analyse the data while it is being generated (sometimes referred to as in-memory analytics), without ever putting it into databases.
Variety refers to the different types of data we can now use. In the past we only focused on structured data that neatly fitted into tables or relational databases, such as financial data. In fact, 80% of the world’s data is unstructured (text, images, video, voice, etc.) With big data technology we can now analyse and bring together data of different types such as messages, social media conversations, photos, sensor data, video or voice recordings.
Limitation :
This approach works well where we have less volume of data that can be accommodated by standard database servers, or up to the limit of the processor which is processing the data. But when it comes to dealing with huge amounts of data, it is really a tedious task to process such data through a traditional database server.
