NoSQL! is it for you?

NoSQL
What it is and is it for you?

Iraj Islam
Rubayeet Islam
Nurul Ferdous

NewsCred

Thursday, February 3, 2011

Agenda NewsCred

• Part 1. Why NoSQL?

• Part 2. NoSQL Use Cases

• Part 3. Choosing a NoSQL Solution

• Part 4. Understanding MongoDB

• Part 5. Building a MongoDB App

• Part 6. Scaling MongoDB

• Questions


Who We Are NewsCred

Iraj Islam
CTO/Co-founder, NewsCred

Rubayeet Islam
Senior Software Engineer, NewsCred

Nurul Ferdous
Senior Software Engineer, NewsCred


Our Story NewsCred

Launched 2008
Founded by two Bangladeshis 2008

Funded By Investors of Twitter
Floodgate Ventures (twitter), Bessemer Cap. (LinkedIn)

Top-tier Clients
Yahoo! Orange Telecom, Harvard U, The Daily Star etc.


What We Do NewsCred

Domain Expertise
• Big Data

• Information Retrieval

• Machine Learning

• Semantic Web

Technologies
• Apache Solr

• MySQL/MongoDB

• Python/Java


Part 1
Why NoSQL?

NewsCred


What’s NoSQL? NewsCred

NoSQL
What’s with the weird name?


What’s NoSQL? NewsCred

NoSQL
Non-relational, web-scale database.


Why NoSQL? NewsCred

Web 1.0
The read intensive web
Publishing Model


Why NoSQL? NewsCred

Web 1.0
Publishing Model

Textual Content


Why NoSQL? NewsCred

Web 1.0
Publishing Model Small Data

Textual Content


Why NoSQL? NewsCred

Web 1.0
Publishing Model Browsing Small Data

Textual Content


Why NoSQL? NewsCred

Web 1.0

Textual Content Search


Why NoSQL? NewsCred

Web 1.0

Textual Content Personal Computer Search


Why NoSQL? NewsCred

The Age of Big Data
Exabytes (1018) of data stored per year

1000

750

500

250
2006
2007
2008 0
2009
2010


Why NoSQL? NewsCred

Web 2.0+
The write intensive web


Why NoSQL? NewsCred

Web 2.0+

User-generated Content


Why NoSQL? NewsCred

Web 2.0+
Big Data



Why NoSQL? NewsCred

Web 2.0+
Semi-structured Data Big Data



Why NoSQL? NewsCred

Web 2.0+
Semi-structured Data Big Data

Semantic Web User-generated Content


Why NoSQL? NewsCred

Web 2.0+
Semi-structured Data Real-time Big Data

Semantic Web User-generated Content


Why NoSQL? NewsCred

Web 2.0+
Semi-structured Data Real-time Big Data

Semantic Web Ubiquity User-generated Content
Any device. Anywhere.


Why NoSQL? NewsCred

The MySQL Problem
1. Default

Application

Data
Source
Writing

MySQL

User Reading


Why NoSQL? NewsCred

The MySQL Problem
1. Default

Application
Bottleneck, too much load!

Data
Source
Writing

MySQL

User Reading


Why NoSQL? NewsCred

The MySQL Problem
2. Replication

Application

Data
Source
Writing MySQL
Master

User Reading MySQL
Slaves


Why NoSQL? NewsCred

The MySQL Problem
2. Replication

Application

Data
Source
Writing MySQL
Master

User Reading MySQL
Slaves

Scalable Reads!


Why NoSQL? NewsCred

The MySQL Problem
2. Replication
Bottleneck, writes won’t scale!
Application

Data
Source
Writing MySQL
Master

User Reading MySQL
Slaves

Scalable Reads!


Why NoSQL? NewsCred

The MySQL Problem
3. Sharding

Application

Data
Source
Writing S

MySQL

User Reading S


Why NoSQL? NewsCred

The MySQL Problem
3. Sharding

Application
Great, scalable writes!

Data
Source
Writing S

MySQL

User Reading S


Why NoSQL? NewsCred

The MySQL Problem
3. Sharding

Application
Great, scalable writes!

Data
Source
Writing S

MySQL

User Reading S
Development and maintenance
costs just skyrocketed!


Why NoSQL? NewsCred

The NoSQL Solution
Design Goals

Semi-structure >> Schema-free


Why NoSQL? NewsCred

The NoSQL Solution
Design Goals


Big Data >> Scalable reads/writes


Why NoSQL? NewsCred

The NoSQL Solution
Design Goals



Real-time >> High-performance


Why NoSQL? NewsCred

The NoSQL Solution
Design Goals



Real-time >> High-performance

Ubiquity >> High-availability


NoSQL vs RDMS NewsCred

NoSQL RDBMS
• Schema-free • Relational schema
• Scalable writes/reads • Scalable reads
vs
• Auto high-availability • Custom high-availability


NoSQL vs RDMS NewsCred

NoSQL RDBMS
vs
• Limited queries • Flexible queries
• Eventual Consistency * • Consistency
• BASE • ACID
* Applies to most NoSQL systems


Is NoSQL For You? NewsCred

NoSQL RDBMS
vs
• Limited queries • Flexible queries
• Eventual Consistency * • Consistency
• BASE • ACID
* Applies to most NoSQL systems


Part 2
NoSQL Use Cases

NewsCred


Who’s Using NoSQL? NewsCred


NoSQL Use Cases NewsCred

• Consumer Use Cases
• Facebook
• Twitter
• NetFlix

• Enterprise Use Cases
• Rackspace
• TrendMicro
• NewsCred



• Facebook
• Hbase - Facebook messages
• Scribe - Real-time click logs
• Hive - SQL queries -> MapReduce jobs
• Hadoop
• Web analytics warehouse
• Distributed datastore
• MySQL backups



• Twitter
• Hadoop - Analytics
• Hbase - People search
• Scribe - Log collection framework
• FlockDB - Social graph analysis



• Rackspace
• Cassandra – stat collection, mail and apps

• TrendMicro
• Hbase & Hadoop – reputation databases

• NewsCred
• MongoDB
• API usage analytics
• Pixel tracking analytics
• Entity metadata storage


Demo
NewsCred API Analytics

NewsCred


Part 3
Choosing a NoSQL Solution

NewsCred


Choosing a NoSQL Solution NewsCred

Availability
Each:client:can:always:read:and:write

A
RDBMSs Cassandra
MySQL: Voldemort
PostgreSQL CouchDB
Aster:Data CA AP Dynamo
GreenPlum SimpleDB
Vertica Tokyo:Cabinet
Riak

C P PartitionDtolerance:
Consistency CP
All:clients:have:the:same:view:of: The:system:works:well:despite:
the:data BigTable Scalaris physical:network:partitions

HyperTable Berkeley:DB
Hbase Memcache:DB
MongoDB Redis


Consistent, Available (CA) NewsCred

CA-systems have trouble with partitions and
deal with it with replication.

• Examples
• MySQL (relational)
• Aster Data (relational)
• Greenplum (relational)
• Vertica (column)


Availability, Partition-Tolerant (AP) NewsCred

AP-systems have trouble with consistency, achieve
“eventual consistency” through replication.

• Examples
• Cassandra (column/tabular)
• Dynamo (key-value)
• Voldemort (key-value)
• Tokyo Cabinet (key-value)
• CouchDB (document)
• SimpleDB (document)
• Riak (document)


Consistent, Partition-Tolerant (CP) NewsCred

CP-systems have trouble with availability while
keeping data consistent across partitioned nodes.

• Examples
• MongoDB (document)
• BigTable (column/tabular)
• HyperTable (column/tabular)
• Hbase (column/tabular)
• Redis (key-value)
• Scalaris (key-value)
• MemcacheDB (key-value)


Hbase NewsCred

Selling point: A
Billions of rows, millions of columns

Use when you need:
Random, real-time access to Big Data
C P

Written in: Java
License: Apache
Type: Column/Tabular
Protocol: HTTP/REST/Thrift Users:
Community Support: Good Yahoo!, Facebook, Microsoft, Adobe,
Learning Curve: High StumbleUpon etc.


Cassandra NewsCred

Selling point: A
Best of Google BigTable and Amazon Dynamo

Use when you need:
To write more than you read (logging)
C P

Written in: Java
License: Apache
Type: Column/Tabular
Protocol: Custom, binary (Thrift) Users:
Community Support: Great Facebook, Twitter, Digg, Reddit,
Learning Curve: Medium Rackspace, Cisco, SimpleGeo, Cloudkick etc.


Redis NewsCred

Selling point: A
Blazing fast, in-memory like memcached

Use when you need:
To manage rapidly changing data
C P

Written in: C/C++
License: BSD
Type: Key-value
Protocol: Telnet-like Users:
Community Support: Good Github, Craigslist, Stackoverflow,
Learning Curve: Low Disqus, The Guardian Uk etc.


MongoDB NewsCred

Selling point: A
Best of NoSQL and RDBMS

Use when you need:
Dynamic queries and indexing on a Big DB
C P

Written in: C++
License: AGPL
Type: Document
Protocol: Custom, binary (BSON) Users:
Community Support: Great NewsCred, Foursquare, Github, Sourceforge,
Learning Curve: Low The New York Times, Etsy, Shutterfly etc.


Part 4
Understanding MongoDB

NewsCred


Understanding MongoDB NewsCred

• Database == Database
• Table == Collection
• Row == Document



• Mongo Shell



• INSERT



• SELECT

SELECT * FROM users WHERE X = 3 AND Y = 'abc';

db.users.find({X:3, Y: ”abc”})

SELECT * FROM users WHERE X = 3 AND Y = 'abc' ORDER BY X ASC;

db.users.find({X:3, Y: ”abc”}).sort({X:1})

SELECT username, email FROM users WHERE X = 3 AND Y = 'abc';

db.users.find({X:3, Y: ”abc”}, {username:true, email:true})



• UPDATE
db.collection.update(criteria, modiﬁer, upsert, multi)

criteria : Query which selects the record(s) to update
modiﬁer : $set, $inc, $unset, $push, $pop...
upsert : Insert if not exists, update otherwise
multi : Update multiple docs matching the criteria

UPDATE users SET X = 4, Y = 'abc' WHERE username = 'joegunchy';

db.users.update({username:”joegunchy”}, {$set: {X:4, Y:'abc'}}, true, true)



• DELETE
db.articles.remove({}) /*remove all*/

db.articles.remove({tag:'sql'}) /*remove all articles with tag = 'sql'*/

db.articles.remove({tag:'sql'}) /*block other ops while removing*/



• AGGREGATION
> db.users.count()
42

> db.addresses.distinct('zipcode', {'city':'Dhaka'})
[1000, 1100, 1204, 1205....]



• Map/Reduce
• Algorithm introduced by Google for processing large
datasets on clusters

• MongoDB uses it for:
• Aggregation (Group By, Avg, Sum etc.)
• Batch processing jobs



• Map/Reduce



• Map/Reduce Example

Document

We want to do something like...




Map

Reduce




Execute




Result


Part 5
Building a MongoDB App

NewsCred


Part 6
Scaling with MongoDB

NewsCred


Scaling with MongoDB NewsCred

• Scaling is a challenge

• No silver bullet

• Strategies
• Replication
• Replica Sets
• Auto-sharding



Replication

Master

Slave Slave Slave



Replica Sets

Secondary

User
Passive

Primary



Replica Sets: Election
Synced,3ms,ago

C
Priority,1
A

Synced,1ms,ago

E
Priority,1
Priority 1

B

D
Priority,0



• Replica Sets: Network Partition
• Election Process initiated
• When a node can’t reach primary
• When primary can’t reach majority of nodes in set

• New primary is elected by majority of nodes in set

• Node with the most recent data gets priority

• Arbiter node used to break ties



• Auto-sharding
• Cluster handles sharding data and rebalancing
automatically

• No administrative headaches of manual sharding

• Application is oblivious to existence of shards



Auto-sharding

Big$Collection



Auto-sharding

User

Router)



Auto-sharding

• Connect to a single server
• db = connect(‘localhost:27017’)

• Connect to a router
• db = connect(‘localhost:27017’)

User

Mongo)DB



• When to shard?
• Running out of disk space
• Write intensive
• Need to keep large chunk of data in memory

• Don’t start out with a sharded collection!

• Shard “if and when” you need to



• Choosing a Shard Key
• Incremental
• Example: timestamps i.e. ‘created_at’
• Queries on shard key is highly efficient

• Random
• Example: ‘username’
• Writes are distributed across multiple shards



Sharding + Replica Sets

User

Router

P P

S S S S


Questions? NewsCred

Iraj Islam
iraj@newscred.com, @irajislam

Rubayeet Islam
rubayeet@newscred.com, @rubayeet

Nurul Ferdous
nurul@newscred.com, @ferdous


NoSQL! is it for you?

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a NoSQL! is it for you?

Semelhante a NoSQL! is it for you? (20)

Último

Último (20)

NoSQL! is it for you?