NoSQL Type, Bigdata, and Analytics

1. Technology Company Open Sourced On
Cassandra DataStax Apache Cassandra
used by Facebook , Linkedin ,
Twitter
BigTable Google Google BigTable
Apache HBase Apache HBase ( used by many
companies most popular)
MongoDB MongoDB Inc. Apache (written on C++,Erlang,C)
Couchbase CouchBase Inc Apache (written on Erlang)

2. Category No SQL database
Column
Oriented
Accumulo, Cassandra, Hbase.
Document Clusterpoint,Couchdb, Couchbase, MarkLogic, MongoDB
Key-Value Dynamo, FoundationDB, MemcacheDB, Redis, Riak, FairCom c-
treeACE
Graph Allegro, Neo4J, OrientDB, Virtuoso, Stardog
- Column Oriented DB store database store Values in Column By Column
rather in other RDBMS row by row.
- It leads to better Compression Of data and hence less space required to
store DB.
- There are Still higher Compression can be achieved when used Probabilistic
Databases.
- Similarly Document oriented Store and arrange data in form of documents.
- Key-Value store Data in form of collection of Key-value pairs. Allowing add,
insert, delete to key-value pairs.
- Graph Databases: Every Element is direct pointer to its adjacent hence no-
lookup required.

3.  Go through the link below:
http://sandyclassic.wordpress.com/2013/07
/02/data-warehousing-business-intelligence-
and-cloud-computing

4. The Term Bigdata stems from Characterisized
by 5V:
Volume: Large Volume of data
Velocity: amount of data per seconds
Variability: level of unintentional modification
affecting data Quality throughout lifecycle of
data.
Value: Value derived from data.
Variety: large range of data which is received
from video , audio, text, image.

5.  Sources Example by 5V.
 Volume: Youtube, large volume of video
feeds received and maintained at many video
sites like youtube, vimeo etc…
 Variety: Large variety of data text, audio,
video, images, received in sites like
facebook, twitter, other social media
platforms.
 Velocity: Speed at which data is received in
sites like twitter, facebook (1 billion people
all feeding there data on one site)

6.  Batch Processing Vs Real Time processing
Batch Jobs run at particular time of day like
Nightly jobs or morning jobs which depends on
slack time When server has less load.
But people now want to see the Status like in
transportation when bus is arriving on
particular stand in real time. Or in Retail as
soon they update there status the require real
time advertisements. This is shaping move
towards Big data.

7. Problems differentiated by 5V.
Velocity: With large volume of data received and quick turn
around latency required to reflect the data fed at facebook
then Can it be managed by regular DBMS?
DBMS- maintains ACID properties & have lots of constraints like
primary, foreign keys, check constraints etc.. with quick
turnaround or short latency required these constraints add up
processing time and volume required for storage. So all of
these sites have there own File based storage DBMS like
systems with does not have these constraints. All data is
maintained in files, id assigned to files are indexed and
regularly moved (these are publically know open sourced
databases like Cassandra developed by facebook, BigTable by
Google, etc…)
Most of this databases are popularly Categorized as NoSQL
databases.

8.  As we know now Bigdata is solving problems of 5V like
the huge (V)olume of storage required for video sites
like youtube. Etc.
 It’s changing how We perceive and Visualize or Analyze
data like HBase used for data storage, Mahout of used
to run analytics and find patterns. These databases
have variety of data which require different kind of
processing cannot be achieved by traditional RDBMS
based products. Example link below:
 http://sandyclassic.wordpress.com/2013/06/18/gini-
coefficient-of-economics-and-roc-curve-machine-
learning/

9.  Map-Reduce Algorithm was starting point of
All we see in BigData created by Google
researcher.
 Mapper divides work into multiple parallel
task, sorts within queue and filters into
queue of say 1 queue for each name.
 Reducer Component Aggregates data or
summarizes from multiple units.

11.  So Since data is mostly unstructured the best
way to analyze unstructured data is using
Analytics here Comes New Career Called
Data Scientist.
 Skill Set Required for Data Scientist:
Mathematics (mostly statistics), Computer
Science, Domain like Sociology (like Social
Media Analysis),

14.  One application of Bigdata has been to
gather feedback about product from social
media.
 Here is Sample project Report below How
and what tools can be used to Analyze social
media.
 http://www.slideshare.net/SandeepSharma6
5/social-media-analysis-project

15.  Hadoop allows to distribute load among many
clusters.
 There can be Database clusters, OS clusters,
Application Web server level clustering But
here we are dealing with OS like Distributed
File System(DFS). Hadoop DFS (HDFS) File
system developed by yahoo Competes with
BigTable of Google providing quick storage
and retrieval of data in form of files used by
many social media platforms.

16.  ‘R’ was open source Statistical Analysis
language having Statistical Constructs
available used for Analysis of data.
 Java data mining API, .Net data mining API ,
python libraries are used to mine and
understand trends in Data.
 PIG is another Apache Hadoop based system
used provide High level language for
analyzing large data sets.

17.  Data Science
http://thedatascience.wordpress.com/
 Big Data
:http://thebigdatatrends.wordpress.com
 Data Science Blog2:
http://thedatascientistview.blogspot.ie/

18.  Retail generates huge amount of data for
product positioned on different shelf at store,
replenishment level, reorder level,
merchandising, assortment planning all this data
most of it usually structured Since lots of system
is Automated but there are lots of forms,
customer feedback, planning data analysis of
mails other chat platforms.
 Large Warehouses of Retail store needs plan
positioning and containers in Aisle.
 Analyze trends from social media to find
customer preferences for products and offers.
 Retail Innovation read:
http://sandyclassic.wordpress.com/2013/10/26/
retail-sector-innovations/

19.  Retail uses lots of Sensors for tracking items
with warehouse and inside Store. The Huge
real time data (video , text and other forms)
generated every milli-second from Sensors
embedded across every store and warehouse
Cannot be analyzed by any other medium
better than in Hadoop or Bigdata based
System.

20.  Finance being Game of numbers huge data
from Book of accounts, P&L, Balance sheets
of etc accumulates of different business over
a period of time But most books are
Structured and hence the data. But Hadoop
offers huge scalable clusters to quickly
analyze structured data as well.
 Lots of social media data about interest for
share or any instrument does get reflected in
numbers.
 Spreadsheets are popular medium of analysis
and other textual forms can be better
analyzed if available over Hadoop like
clusters for a kind of semi-structured data