The document discusses how apps and growth have changed in today's digital landscape. It notes the massive growth in internet users and smartphone adoption. It then provides examples of how apps like Instagram and Draw Something experienced explosive overnight growth when expanding to new platforms or going viral. The document emphasizes that for apps to handle unpredictable growth, they need scalable data architectures that can dynamically expand without disruptions. It reviews limitations of traditional SQL/MySQL approaches and outlines advantages of NoSQL databases in providing scalability and high availability. Finally, it provides a taxonomy of different NoSQL database categories like key-value stores, document databases and their features.

1. NoSQL
Landscape
&
Grid
Compu7ng
Chris
Anderson
@jchris
1
Saturday, October 6, 12

2. Growth
is
the
New
Reality
• 2.2
Billion
internet
users
• 50%
Americans
use
smartphones
• Your
app
can
grow
overnight
• Are
you
ready?
2
Saturday, October 6, 12

3. Instagrowth:
Android
Launch
E xa mple
• Instagram
gained
nearly
1
million
users
overnight
when
they
expanded
to
Android
3
Saturday, October 6, 12

4. Instagrowth:
Android
Launch
E xa mple 1
Instagram
=
7.5M
MAU*
4
Saturday, October 6, 12

5. Draw
Something
-­‐
Social
Game
35 million monthly active users in 1 month
about 5 Instagrams
(Instagram today is waaaay more than 1 Instagram)
5
Saturday, October 6, 12

6. Goes
Viral
3
Weeks
aOer
Launch
Draw
Something
by
OMGPOP
Daily
Ac)ve
Users
(millions)
16
14
35+M
MAU
12 at
peak
10
8
6
4
2
2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21
6
Saturday, October 6, 12

7. By
Contrast,
at
1/2
an
Instagram
The
Simpson’s:
Tapped
Out
Daily
Ac)ve
Users
(millions)
7
Saturday, October 6, 12

8. GETTING
IT
RIGHT
8
Saturday, October 6, 12

9. Scalable
Data
Layer
●On-­‐demand
cluster
sizing
● Grow
or
shrink
with
workload
●Easy
node
provisioning
● All
nodes
are
the
same
●MulA-­‐master
Cross-­‐Datacenter
ReplicaAon
● For
a
fast
and
reliable
user
experience
worldwide
●EffecAve
Auto-­‐sharding
● Should
avoid
cluster
hot
spots
9
Saturday, October 6, 12

10. Old
School
Hits
a
Scale
Wall
Application Scales Out
Just add more commodity web servers
Database Scales Up
Get a bigger, more complex server
Expensive & disruptive sharding, doesn’t perform at web scale
10
Saturday, October 6, 12

11. Tradi^onal
MySQL
+
Memcached
Architecture
● Run as many MySQL machines as
you need
www.example.com
● Data sharded evenly across the
machines using client code App
Servers
● Memcached used to provide
faster response time for users
Memcached
Tier
and reduce load on the database
MySQL
Tier
11
Saturday, October 6, 12

12. Limita^ons
of
MySQL
+
Memcached
● To scale you need to start using
MySQL more simply
● Scale by hand
● Replication / Sharding is a black art
● Code overhead to manage keeping
memcache and mysql in sync
● Lots of components to deploy
Learn
From
Others
-­‐
This
Scenario
Costs
Time
and
Money.
Scaling
SQL
is
poten^ally
disastrous
when
going
Viral:
very
risky
^me
for
major
code
changes
and
migra^ons...
you
have
no
Time
when
skyrocke^ng
up.
12
Saturday, October 6, 12

13. NoSQL
Architectural
Promise
• High Performance data access
• Scale Up/Down Horizontally
www.example.com
• 24x7x365 Always-On Availability
• Flexible Schema Document
Model App
Servers
Couchbase
Database
Servers
13
Saturday, October 6, 12

14. NOSQL
TAXONOMY
14
Saturday, October 6, 12

15. The Key-Value Store – the foundation of NoSQL
Key
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
Opaque
101100101000100010011101
101100101000100010011101
Binary
101100101000100010011101
101100101000100010011101
Value
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
15
Saturday, October 6, 12

16. Memcached – the NoSQL precursor
Key
101100101000100010011101 memcached
101100101000100010011101
101100101000100010011101
101100101000100010011101 In-­‐memory
only
101100101000100010011101 Limited
set
of
opera^ons
Opaque
101100101000100010011101 Blob
Storage:
Set,
Add,
Replace,
CAS
101100101000100010011101
Binary
101100101000100010011101
Retrieval:
Get
101100101000100010011101 Structured
Data:
Append,
Increment
Value
101100101000100010011101
101100101000100010011101 “Simple
and
fast.”
101100101000100010011101
101100101000100010011101 Challenges:
cold
cache,
disrup^ve
elas^city
101100101000100010011101
101100101000100010011101
16
Saturday, October 6, 12

17. Redis
–
More
“Structured
Data”
commands
Key
101100101000100010011101 redis
101100101000100010011101
101100101000100010011101
101100101000100010011101 In-­‐memory
only
“Data
Structures”
101100101000100010011101 Vast
set
of
opera^ons
Blob
101100101000100010011101 Blob
Storage:
Set,
Add,
Replace,
CAS
101100101000100010011101
List
101100101000100010011101
Retrieval:
Get,
Pub-­‐Sub
101100101000100010011101 Structured
Data:
Strings,
Hashes,
Lists,
Sets,
Set
101100101000100010011101 Sorted
lists
Hash
101100101000100010011101
101100101000100010011101
…
101100101000100010011101 Example
opera7ons
for
a
Set
101100101000100010011101 Add,
count,
subtract
sets,
intersec^on,
is
101100101000100010011101 member?,
atomic
move
from
one
set
to
another
17
Saturday, October 6, 12

18. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
18
Saturday, October 6, 12

19. Membase
–
From
key-­‐value
cache
to
database
Key
101100101000100010011101 membase
101100101000100010011101
101100101000100010011101
101100101000100010011101 Disk-­‐based
with
built-­‐in
memcached
cache
101100101000100010011101 Cache
refill
on
restart
Opaque
101100101000100010011101 Memcached
compa^ble
(drop
in
replacement)
101100101000100010011101 Highly-­‐available
(data
replica^on)
Binary
101100101000100010011101
Add
or
remove
capacity
to
live
cluster
101100101000100010011101
Value
101100101000100010011101
101100101000100010011101 “Simple,
fast,
elas^c.”
101100101000100010011101
101100101000100010011101
101100101000100010011101
101100101000100010011101
19
Saturday, October 6, 12

20. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
(memory/disk)
membase
Database
20
Saturday, October 6, 12

21. Couchbase
–
document-­‐oriented
database
Key
Couchbase
{
“string”
:
“string”,
“string”
:
value,
Auto-­‐sharding
“string”
:
Disk-­‐based
with
built-­‐in
memcached
cache
{
“string”
:
“string”,
JSON Cache
refill
on
restart
“string”
:
value
},
“string”
:OBJECT
Memcached
compa^ble
(drop
in
replace)
[
array
]
Highly-­‐available
(data
replica^on)
} (“DOCUMENT”) Add
or
remove
capacity
to
live
cluster
When
values
are
JSON
objects
(“documents”):
Create
indices,
views
and
query
against
the
views
21
Saturday, October 6, 12

22. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
(memory/disk)
membase couchbase
Database
22
Saturday, October 6, 12

23. MongoDB
–
Document-­‐oriented
database
Key
MongoDB
{
“string”
:
“string”,
“string”
:
value,
Disk-­‐based
with
in-­‐memory
“caching”
“string”
:
BSON BSON
(“binary
JSON”)
format
and
wire
protocol
{
“string”
:
“string”,
Master-­‐slave
replica^on
OBJECT
“string”
:
value
},
Auto-­‐sharding
“string”
:
[
array
]
} (“DOCUMENT”) Values
are
BSON
objects
Supports
ad
hoc
queries
–
best
when
indexed
23
Saturday, October 6, 12

24. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
(memory/disk)
membase couchbase
Database
mongoDB
24
Saturday, October 6, 12

25. Cassandra
–
Column
overlays
Cassandra
Column
1
Disk-­‐based
system
Clustered
Column
2 External
caching
required
for
low-­‐latency
reads
“Columns”
are
overlaid
on
the
data
Not
all
rows
must
have
all
columns
Column
3
Supports
efficient
queries
on
columns
(not
present)
Restart
required
when
adding
columns
Good
cross-­‐datacenter
support
25
Saturday, October 6, 12

26. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
(memory/disk)
membase couchbase cassandra
Database
mongoDB
26
Saturday, October 6, 12

27. Neo4j
–
Graph
database
Neo4j
Disk-­‐based
system
External
caching
required
for
low-­‐latency
reads
Nodes,
rela^onships
and
paths
Proper^es
on
nodes
Delete,
Insert,
Traverse,
etc.
27
Saturday, October 6, 12

28. NoSQL
catalog
Key-­‐Value Data
Structure Document Column Graph
(memory
only)
Cache
memcached redis
(memory/disk)
membase couchbase cassandra Neo4j
Database
mongoDB
28
Saturday, October 6, 12

29. The
Landscape
S3
HBase
Scale Riak Cassandra
Couchbase
SimpleDB
RDBMS
Datomic
MongoDB
memcached
CouchDB
Neo4j Redis
Speed
29
Saturday, October 6, 12

30. Datomic
-­‐
immutable
func^onal
data
30
Saturday, October 6, 12

31. Hello
Couchbase
Server
2.0
31
Saturday, October 6, 12

32. Couchbase
Server
2.0
beta
32
Saturday, October 6, 12

33. Couchbase
handles
real
world
scale
33
Saturday, October 6, 12

34. (Really)
High
Performance
Cisco and Solarflare benchmark of Couchbase Server
Latency Throughput
less than 1/2 ms grows linearly with cluster size
5 Nodes -- 1.75M operations per second
34
Saturday, October 6, 12

35. How
fast?
hrp://www.slideshare.net/renatko/couchbase-­‐performance-­‐benchmarking
35
Saturday, October 6, 12

36. Latency
Saturday, October 6, 12

37. Latency
Saturday, October 6, 12

38. COMPLEXITY
IS
THE
ENEMY
37
Saturday, October 6, 12

39. Couchbase
Server
Basic
Opera^on
APP
SERVER
1 APP
SERVER
2
§Docs
distributed
evenly
across
servers
in
the
cluster
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY §Each
server
stores
both
ac)ve
&
CLUSTER
MAP CLUSTER
MAP
replica
docs
§ Only
one
server
ac^ve
at
a
^me
§Client
library
provides
app
with
simple
interface
to
database
§Cluster
map
provides
map
to
which
server
doc
is
on
§ App
never
needs
to
know
SERVER
1 SERVER
2 SERVER
3 § App
reads,
writes,
updates
docs
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs § Mul^ple
App
Servers
can
access
same
document
at
same
^me
Doc
5 DOC Doc
4 DOC Doc
1 DOC
Doc
2 DOC Doc
7 DOC Doc
3 DOC
Doc
9 DOC Doc
8 DOC Doc
6 DOC
COUCHBASE
SERVER
CLUSTER
38
Saturday, October 6, 12

40. Couchbase
Server
Basic
Opera^on
APP
SERVER
1 APP
SERVER
2
§Docs
distributed
evenly
across
servers
in
the
cluster
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY §Each
server
stores
both
ac)ve
&
CLUSTER
MAP CLUSTER
MAP
replica
docs
§ Only
one
server
ac^ve
at
a
^me
§Client
library
provides
app
with
simple
interface
to
database
§Cluster
map
provides
map
to
which
server
doc
is
on
§ App
never
needs
to
know
SERVER
1 SERVER
2 SERVER
3 § App
reads,
writes,
updates
docs
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs § Mul^ple
App
Servers
can
access
same
document
at
same
^me
Doc
5 DOC Doc
4 DOC Doc
1 DOC
Doc
2 DOC Doc
7 DOC Doc
3 DOC
Doc
9 DOC Doc
8 DOC Doc
6 DOC
Replica
Docs Replica
Docs Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC
Doc
1 DOC Doc
3 DOC Doc
9 DOC
Doc
8 DOC Doc
2 DOC Doc
5 DOC
COUCHBASE
SERVER
CLUSTER
38
Saturday, October 6, 12

41. Couchbase
Server
Basic
Opera^on
APP
SERVER
1 APP
SERVER
2
§Docs
distributed
evenly
across
servers
in
the
cluster
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY §Each
server
stores
both
ac)ve
&
CLUSTER
MAP CLUSTER
MAP
replica
docs
§ Only
one
server
ac^ve
at
a
^me
§Client
library
provides
app
with
simple
interface
to
database
Read/Write/Update Read/Write/Update
§Cluster
map
provides
map
to
which
server
doc
is
on
§ App
never
needs
to
know
SERVER
1 SERVER
2 SERVER
3 § App
reads,
writes,
updates
docs
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs § Mul^ple
App
Servers
can
access
same
document
at
same
^me
Doc
5 DOC Doc
4 DOC Doc
1 DOC
Doc
2 DOC Doc
7 DOC Doc
3 DOC
Doc
9 DOC Doc
8 DOC Doc
6 DOC
Replica
Docs Replica
Docs Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC
Doc
1 DOC Doc
3 DOC Doc
9 DOC
Doc
8 DOC Doc
2 DOC Doc
5 DOC
COUCHBASE
SERVER
CLUSTER
38
Saturday, October 6, 12

42. Add
Nodes
to
the
Cluster
APP
SERVER
1 APP
SERVER
2
§ Two
servers
added
to
cluster
§ One-­‐click
opera^on
§ Docs
automa^cally
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY rebalanced
across
cluster
CLUSTER
MAP CLUSTER
MAP § Even
distribu^on
of
docs
§ Minimum
doc
movement
§ Cluster
map
updated
§ App
database
calls
now
distributed
over
larger
#
of
servers
SERVER
1 SERVER
2 SERVER
3
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC
Doc
6
Doc
9 DOC Doc
8 DOC Doc
6 DOC
Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC
Doc
7
Doc
1 DOC Doc
3 DOC Doc
9 DOC
Doc
9
Doc
8 DOC Doc
2 DOC Doc
5 DOC
COUCHBASE
SERVER
CLUSTER
39
Saturday, October 6, 12

43. Add
Nodes
to
the
Cluster
APP
SERVER
1 APP
SERVER
2
§ Two
servers
added
to
cluster
§ One-­‐click
opera^on
§ Docs
automa^cally
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY rebalanced
across
cluster
CLUSTER
MAP CLUSTER
MAP § Even
distribu^on
of
docs
§ Minimum
doc
movement
§ Cluster
map
updated
§ App
database
calls
now
distributed
over
larger
#
of
servers
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC
Doc
6
Doc
9 DOC Doc
8 DOC Doc
6 DOC
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC
Doc
7
Doc
1 DOC Doc
3 DOC Doc
9 DOC
Doc
9
Doc
8 DOC Doc
2 DOC Doc
5 DOC
COUCHBASE
SERVER
CLUSTER
39
Saturday, October 6, 12

44. Add
Nodes
to
the
Cluster
APP
SERVER
1 APP
SERVER
2
§ Two
servers
added
to
cluster
§ One-­‐click
opera^on
§ Docs
automa^cally
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY rebalanced
across
cluster
CLUSTER
MAP CLUSTER
MAP § Even
distribu^on
of
docs
§ Minimum
doc
movement
§ Cluster
map
updated
§ App
database
calls
now
distributed
over
larger
#
of
servers
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC
Doc
6
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 DOC Doc
8 DOC
Doc
7
Doc
1 DOC Doc
3 DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
39
Saturday, October 6, 12

45. Add
Nodes
to
the
Cluster
APP
SERVER
1 APP
SERVER
2
§ Two
servers
added
to
cluster
§ One-­‐click
opera^on
§ Docs
automa^cally
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY rebalanced
across
cluster
CLUSTER
MAP CLUSTER
MAP § Even
distribu^on
of
docs
§ Minimum
doc
movement
§ Cluster
map
updated
§ App
database
calls
now
distributed
over
larger
#
of
servers
Read/Write/Update Read/Write/Update
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC
Doc
6
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 DOC Doc
8 DOC
Doc
7
Doc
1 DOC Doc
3 DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
39
Saturday, October 6, 12

46. Fail
Over
Node
§ App
servers
happily
accessing
docs
on
Server
3
§ Server
fails
§ App
server
requests
to
server
3
fail
§ Cluster
detects
server
has
failed
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY
CLUSTER
MAP CLUSTER
MAP § Promotes
replicas
of
docs
to
ac)ve
§ Updates
cluster
map
§ App
server
requests
for
docs
now
go
to
appropriate
server
§ Typically
rebalance
would
follow
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC
Doc
6
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 DOC Doc
8 DOC
Doc
7
Doc
1 DOC Doc
3 DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
40
Saturday, October 6, 12

47. Fail
Over
Node
§ App
servers
happily
accessing
docs
on
Server
3
§ Server
fails
§ App
server
requests
to
server
3
fail
§ Cluster
detects
server
has
failed
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY
CLUSTER
MAP CLUSTER
MAP § Promotes
replicas
of
docs
to
ac)ve
§ Updates
cluster
map
§ App
server
requests
for
docs
now
go
to
appropriate
server
§ Typically
rebalance
would
follow
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC DOC
Doc
6
Doc
1 Doc
3
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 Doc
8 DOC
Doc
7
DOC DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
40
Saturday, October 6, 12

48. Fail
Over
Node
§ App
servers
happily
accessing
docs
on
Server
3
§ Server
fails
§ App
server
requests
to
server
3
fail
§ Cluster
detects
server
has
failed
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY
CLUSTER
MAP CLUSTER
MAP § Promotes
replicas
of
docs
to
ac)ve
§ Updates
cluster
map
§ App
server
requests
for
docs
now
go
to
appropriate
server
§ Typically
rebalance
would
follow
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC DOC
Doc
6
Doc
1 Doc
3
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 Doc
8 DOC
Doc
7
DOC DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
40
Saturday, October 6, 12

49. Fail
Over
Node
§ App
servers
happily
accessing
docs
on
Server
3
§ Server
fails
§ App
server
requests
to
server
3
fail
§ Cluster
detects
server
has
failed
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY
CLUSTER
MAP CLUSTER
MAP § Promotes
replicas
of
docs
to
ac)ve
§ Updates
cluster
map
§ App
server
requests
for
docs
now
go
to
appropriate
server
§ Typically
rebalance
would
follow
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
1 DOC Doc
9 DOC Doc
6 DOC
Doc
3
Doc
2 DOC Doc
7 DOC Doc
3 DOC Doc
8 DOC DOC
Doc
6
Doc
1 Doc
3
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
7 DOC Doc
5 Doc
8 DOC
Doc
7
DOC DOC Doc
9 DOC Doc
2 DOC
Doc
9
COUCHBASE
SERVER
CLUSTER
40
Saturday, October 6, 12

50. Fail
Over
Node
§ App
servers
happily
accessing
docs
on
Server
3
§ Server
fails
§ App
server
requests
to
server
3
fail
§ Cluster
detects
server
has
failed
COUCHBASE
CLIENT
LIBRARY COUCHBASE
CLIENT
LIBRARY
CLUSTER
MAP CLUSTER
MAP § Promotes
replicas
of
docs
to
ac)ve
§ Updates
cluster
map
§ App
server
requests
for
docs
now
go
to
appropriate
server
§ Typically
rebalance
would
follow
SERVER
1 SERVER
2 SERVER
3 SERVER
4 SERVER
5
Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs Ac^ve
Docs
Doc
5 DOC Doc
4 DOC Doc
9 DOC Doc
6 DOC
Doc
2 DOC Doc
7 DOC Doc
8 DOC DOC
Doc
1 Doc
3
Replica
Docs Replica
Docs Replica
Docs Replica
Docs Replica
Docs
Doc
4 DOC Doc
6 DOC Doc
5 Doc
8 DOC
DOC DOC Doc
2 DOC
COUCHBASE
SERVER
CLUSTER
40
Saturday, October 6, 12

51. War
Story:
EBS
Outage
●Suddenly,disk writes all began to time out
●Many services experienced outages:
● FourSquare, Reddit, Quora, among others
●With memory buffered writes, a scalable
data layer keeps working
● When EBS came back online, Couchbase wrote all
the updated data to disk without missing a beat.
41
Saturday, October 6, 12

52. Cross
Data
Center
Replica^on
US
DATA
CENTER EUROPE
DATA
CENTER ASIA
DATA
CENTER
Replica7on Replica7on
Replica7on
§Data
close
to
users
§Mul^ple
loca^ons
for
disaster
recovery
§Independently
managed
clusters
serving
local
data
42
Saturday, October 6, 12

53. Built
for
Produc^on
43
Saturday, October 6, 12

54. JSON
DOCUMENT
DATABASE
44
Saturday, October 6, 12

55. Document
Database
as
Aggregate
Database
hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html
45
Saturday, October 6, 12

56. Document
Database
o::1001
{
uid:
ji22jd,
customer:
Ann,
line_items:
[
{
sku:
0321293533,
quan:
3,
unit_price:
48.0
},
{
sku:
0321601912,
quan:
1,
unit_price:
39.0
},
{
sku:
0131495054,
quan:
1,
unit_price:
51.0
}
],
payment:
{
type:
Amex,
expiry:
04/2001,
last5:
12345
}
}
This synergy between the programming model and the distribution model is
very valuable. It allows the database to use its knowledge of how the
application programmer clusters the data to help performance across the
cluster.
hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html 46
Saturday, October 6, 12

57. Developers
<3
JSON
47
Saturday, October 6, 12

58. LET’S
GET
POST-­‐RELATIONAL!
4
Saturday, October 6, 12

59. JSON
Documents
• Maps
more
closely
to
external
API
• CRUD
Opera^ons,
lightweight
schema
{
“fields” : [“with basic types”, 3.14159, true],
“like” : “your favorite language”
}
• Stored
under
an
iden^fier
key
client.set(“mydocumentid”, myDocument);
mySavedDocument = client.get(“mydocumentid”);
49
Saturday, October 6, 12

60. Meta
+
Document
Body
{ Document
"brewery":
"New
Belgium
Brewing", user data,
"name":
"1554
Enlightened
Black
Ale", can be anything
"abv":
5.5,
"descrip7on":
"Born
of
a
flood...",
"category":
"Belgian
and
French
Ale",
"style":
"Other
Belgian-­‐Style
Ales",
"updated":
"2010-­‐07-­‐22
20:00:20"
}
unique ID “vintage” date format from an SQL dump
{ >_<
"id"
:
"beer_Enlightened_Black_Ale”, Metadata
... identifier,
{ expiration, etc
50
Saturday, October 6, 12

61. {
Add
comments
to
the
beer
"brewery":
"New
Belgium
"name":
"1554
Enlightene
"abv":
5.5,
{
"descrip7on":
"Born
of
a
fl
"type":
"comment",
"category":
"Belgian
and
F
"about_id":
"beer_Enlightened_Black_Ale",
link to
"style":
"Other
Belgian-­‐Sty
beer
"user_id":
525,
"updated":
"2010-­‐07-­‐22
2
"text":
"tastes
like
college!",
“ra7ngs”
:
{
"updated":
"2010-­‐07-­‐22
20:00:20"
“525”
:
5,
}
“30”
:
4,
{
“1044”
:
2
"id":
"f1e62"
},
}
“comment_ids”
:
[
“f1e62”,
link to
“6ad8c”
comments
]
}
Saturday, October 6, 12

62. How
to:
look
up
comments
from
a
beer
• SERIALIZED
LOOP
beer = client.get(“beer:A_cold_one”);
beer.comment_ids.each { |id|
comments.push(client.get(id));
}
• FAST
MULTI-­‐KEY
LOOKUP
beer = client.get(“beer:A_cold_one”);
comments = client.multiGet(beer.comment_ids)
• ASYNC
VIEW
QUERY
comments = client.query(“myapp”,“by_comment_on”,
{:key => “beer:A_cold_one”});
figure
hrp://www.ibm.com/developerworks/webservices/library/ws-­‐sdoarch/ 52
Saturday, October 6, 12

63. Emergent
Schema
• The
database
can
handle
it
• Your
app
controls
the
schema
"Capture
the
user's
intent"
JSON.org
Github
API
Twiqer
API
53
Saturday, October 6, 12

64. Audience
par^cipa^on!
*that
means
you
npm install twitterfight
npm start twitterfight
54
Saturday, October 6, 12

65. Thank
You!
Chris
Anderson
@jchris
hrp://www.couchbase.com/
55
Saturday, October 6, 12

66. INCREMENTAL
MAP-­‐REDUCE
FOR
REALTIME
ANALYTICS
5
Saturday, October 6, 12

67. What
do
you
mean
“Incremental?”
like:
CREATE INDEX city ON brewery city;
57
Saturday, October 6, 12

68. QUERY
PATTERN:
FIND
BY
ATTRIBUTE
5
Saturday, October 6, 12

69. Find
documents
by
a
specific
arribute
• Lets
find
beers
by
brewery_id!
59
Saturday, October 6, 12

70. The
index
defini^on
60
Saturday, October 6, 12

71. The
result
set:
beers
keyed
by
brewery_id
61
Saturday, October 6, 12

72. QUERY
PATTERN:
BASIC
AGGREGATIONS
6
Saturday, October 6, 12

73. Use
a
built-­‐in
reduce
func^on
with
a
group
query
• Lets
find
average
abv
for
each
brewery!
63
Saturday, October 6, 12

74. We
are
reducing
doc.abv
with
_stats
64
Saturday, October 6, 12

75. Group
reduce
(reduce
by
unique
key)
65
Saturday, October 6, 12

76. QUERY
PATTERN:
TIME-­‐BASED
ROLLUPS
WITH
KEY
GROUPING
6
Saturday, October 6, 12

77. group_level=3
-­‐
daily
results
-­‐
great
for
graphing
• Daily,
hourly,
minute
or
second
rollup
all
possible
with
the
same
index.
• hrp://crate.im/posts/couchbase-­‐views-­‐reddit-­‐data/
67
Saturday, October 6, 12

78. GEO
INDEX
&
FULL
TEXT
INTEGRATION
6
Saturday, October 6, 12

79. GeoCouch
R-­‐Tree
Index
• Op^mized
for
bulk
loading
of
large
data
sets
• Simplified
query
model
(bounding
box,
nearest
neighbor)
6
Saturday, October 6, 12

80. Elas^c
Search
Adapter
• Elas^c
Search
is
good
for
ad-­‐hoc
queries
and
faceted
browsing
• Our
adapter
is
aware
of
changing
Couchbase
topology
• Indexed
by
Elas^c
Search
aOer
stored
to
disk
in
Couchbase
70
Saturday, October 6, 12

81. Thank
You!
Chris
Anderson
@jchris
hrp://www.couchbase.com/
71
Saturday, October 6, 12