My presentation at NoSQL Now 2013

1. How
Companies
use
NoSQL
and
Couchbase
Dip7
Borkar
Director,
Product
Management

2. Couchbase
Server
NoSQL
Document
Database

3. Couchbase
Open
Source
Project
• Leading
NoSQL
database
project
focused
on
distributed
database
technology
and
surrounding
ecosystem
• Supports
both
key-­‐value
and
document-­‐oriented
use
cases
• All
components
are
available
under
the
Apache
2.0
Public
License
• Obtained
as
packaged
so?ware
in
both
enterprise
and
community
ediAons.
Couchbase
Open Source Project

4. OLTP
Analy7cs
2
kinds
of
database
management
system

5. NoSQL
+
Big
Data
Map-­‐reduce
against
huge
datasets
to
analyze
and
find
insights
and
answers
Opera7onal
database
for
web
and
mobile
apps
with
high
performance
at
scale

6. Common
Use
Cases
Social
Gaming
• Couchbase
stores
player
and
game
data
• Examples
customers
include:
Zynga
• Tapjoy,
Ubiso?,
Tencent
Mobile
Apps
• Couchbase
stores
user
info
and
app
content
• Examples
customers
include:
Kobo,
PlayAka
Ad
Targe7ng
• Couchbase
stores
user
informaAon
for
fast
access
• Examples
customers
include:
AOL,
Mediamind,
Convertro
Session
store
• Couchbase
Server
as
a
key-­‐
value
store
• Examples
customers
include:
Concur,
Sabre
User
Profile
Store
• Couchbase
Server
as
a
key-­‐value
store
• Examples
customers
include:
Tunewiki
High
availability
cache
• Couchbase
Server
used
as
a
cache
Aer
replacement
• Examples
customers
include:
Orbitz
Content
&
Metadata
Store
• Couchbase
document
store
with
ElasAc
Search
• Examples
customers
include:
McGraw
Hill
3rd
party
data
aggrega7on
• Couchbase
stores
social
media
and
data
feeds
• Examples
customers
include:
LivePerson

7. • Applica7on
objects
• Popular
search
query
results
• Session
informa7on
• Heavily
accessed
web
landing
pages
High
availability
caching
Use
Case:
High-­‐Availability
Caching
• Speed
up
RDBMS
• Consistently
low
response
7mes
for
document
/
key
lookups
• High-­‐availability
24x7x365
• Replacement
for
en7re
caching
7er
• Low
latency
in
sub-­‐milliseconds
with
consistently
high
read
/
write
throughput
using
built-­‐in
cache
• Always-­‐on
opera7ons
even
for
database
upgrades
and
maintenance
with
zero
down
7me
• memcached
compa7bility
for
easy
migra7on
to
Couchbase
without
any
applica7on
changes
• High
availability
and
disaster
replica7on
with
intra-­‐cluster
and
cross-­‐cluster
replica7on
(XDCR)
Data
Cached
in
Couchbase
Applica7on
Requirements
Why
NoSQL
and
Couchbase
RDBMS
Applica7on
Layer
User
Requests
Cache
Misses
and
Write
Requests
Read-­‐Write
Requests
Couchbase
Distributed
Cache
Key
Ver7cals
-­‐ E-­‐commerce
-­‐
Travel
-­‐ High-­‐tech
-­‐
HR
(ADP)

8. • Session
values
or
Cookies
(stored
as
key-­‐value
pairs)
• Examples
include:
items
in
a
shopping
cart,
flights
selected,
search
results,
etc.
Session
Store
Use
Case:
Session
Store
• Extremely
fast
access
to
session
data
using
unique
session
ID
• Easy
scalability
to
handle
fast
growing
number
of
users
and
user-­‐generated
data
• Always-­‐on
func7onality
for
global
user
base
• Low
latency
in
sub-­‐milliseconds
with
consistently
high
read
/
write
throughput
for
session
data
via
the
built-­‐in
object-­‐level
cache
• Linear
throughput
scalability
to
grow
the
database
as
user
and
data
volume
grow
• Always-­‐on
opera7ons
even
par7cularly
high
availability
using
Couchbase
replica7on
and
failover
• Intra
cluster
and
cross
cluster
(XDCR)
replica7on
for
globally
distributed
ac7ve-­‐ac7ve
plaeorm
Data
Stored
in
Couchbase
Applica7on
Requirements
Why
NoSQL
and
Couchbase
Key
Ver7cals
• Ad
Targe7ng
• Travel
• E-­‐commerce

9. User
ID
/
Profile
Store
Use
Case:
Globally
Distributed
User
Profile
Store
• Extremely
fast
access
to
individual
profiles
• Always
online
system
as
mul7ple
applica7ons
access
user
profiles
• Flexibility
to
add
and
update
user
agributes
• Easy
scalability
to
handle
fast
growing
number
of
users
• Low
latency
and
high
throughput
for
very
quick
lookups
for
millions
of
concurrent
users
using
built-­‐in
cache
• Intra
cluster
and
cross
cluster
(XDCR)
replica7on
for
high
availability
and
disaster
recovery
• Ac7ve-­‐ac7ve
geo-­‐distributed
system
to
handle
globally
distributed
user
base
• Online
admin
opera7ons
eliminate
system
down7me
Data
Stored
in
Couchbase
Applica7on
Requirements
Why
NoSQL
and
Couchbase
• User
profile
with
unique
ID
• User
sehng
/
preferences
• User’s
network
• User
applica7on
state
Key
Ver7cals
-­‐ E-­‐commerce
-­‐
Social
gaming
-­‐ Sojware
as
a
service

10. • Content
metadata
• Content:
Ar7cles,
text
• Landing
pages
for
website
• Digital
content:
eBooks,
magazine,
research
material
Content
and
Metadata
Store
Use
Case:
Content
and
Metadata
Store
• Flexibility
to
store
any
kind
of
content
• Fast
access
to
content
metadata
(most
accessed
objects)
and
content
• Full-­‐text
Search
across
data
set
• Scales
horizontally
as
more
content
gets
added
to
the
system
• Fast
access
to
metadata
and
content
via
object-­‐managed
cache
• JSON
provides
schema
flexibility
to
store
all
types
of
content
and
metadata
• Indexing
and
querying
provides
real-­‐7me
analy7cs
capabili7es
across
dataset
• Integra7on
with
Elas7cSearch
for
full-­‐text
search
• Ease
of
scalability
ensures
that
the
data
cluster
can
be
grown
seamlessly
as
the
amount
of
user
and
ad
data
grows
Data
Stored
in
Couchbase
Applica7on
Requirements
Why
NoSQL
and
Couchbase
Key
Ver7cals
-­‐ Media
&
Publishing
-­‐
High-­‐tech
-­‐
Social
-­‐
Fin
serv.

11. • Social
media
feeds:
Twiger,
Facebook,
LinkedIn
• Blogs,
news,
press
ar7cles
• Data
service
feeds:
Hoovers,
Reuters
• Data
form
other
systems
Data
Aggrega7on
Use
Case:
Data
Aggrega7on
• Flexibility
to
store
any
kind
of
content
• Flexibility
to
handle
schema
changes
• Full-­‐text
Search
across
data
set
• High
speed
data
inges7on
• Scales
horizontally
as
more
content
gets
added
to
the
system
• JSON
provides
schema
flexibility
to
store
all
types
of
content
and
metadata
• Fast
access
to
individual
documents
via
built-­‐in
cache,
high
write
throughput
• Indexing
and
querying
provides
real-­‐7me
analy7cs
capabili7es
across
dataset
• Integra7on
with
Elas7cSearch
for
full-­‐text
search
• Ease
of
scalability
ensures
that
the
data
cluster
can
be
grown
seamlessly
as
the
amount
of
user
and
ad
data
grows
Data
Stored
in
Couchbase
Applica7on
Requirements
Why
NoSQL
and
Couchbase
Key
Ver7cals
-­‐ Ad
targe7ng
-­‐
High-­‐tech
-­‐ Media
&
Publishing

12. Common
Use
Cases
Social
Gaming
• Couchbase
stores
player
and
game
data
• Examples
customers
include:
Zynga
• Tapjoy,
Ubiso?,
Tencent
Mobile
Apps
• Couchbase
stores
user
info
and
app
content
• Examples
customers
include:
Kobo,
PlayAka
Ad
Targe7ng
• Couchbase
stores
user
informaAon
for
fast
access
• Examples
customers
include:
AOL,
Mediamind,
Convertro
Session
store
• Couchbase
Server
as
a
key-­‐
value
store
• Examples
customers
include:
Concur,
Sabre
User
Profile
Store
• Couchbase
Server
as
a
key-­‐value
store
• Examples
customers
include:
Tunewiki
High
availability
cache
• Couchbase
Server
used
as
a
cache
Aer
replacement
• Examples
customers
include:
Orbitz
Content
&
Metadata
Store
• Couchbase
document
store
with
ElasAc
Search
• Examples
customers
include:
McGraw
Hill
3rd
party
data
aggrega7on
• Couchbase
stores
social
media
and
data
feeds
• Examples
customers
include:
LivePerson

13. McGraw
Hill
Educa7on
Labs
Learning
portal

14. Use
Case:
Content
and
metadata
store
Building
a
self-­‐adapAng,
interacAve
learning
portal
with
Couchbase

15. As learning move online in great numbers
Growing need to build interactive learning environments that
Scale!!
Scale
to
millions
of
learners
Serve
MHE
as
well
as
third-­‐party
content
Including
open
content
Support
learning
apps
0101001001
1101010101
0101001010
101010
Self-­‐adapt
via
usage
data
The Problem

16. • Allow
for
elasAc
scaling
under
spike
periods
• Ability
to
catalog
&
deliver
content
from
many
sources
• Consistent
low-­‐latency
for
metadata
and
stats
access
• Require
full-­‐text
search
support
for
content
discovery
• Offer
tunable
content
ranking
&
recommendaAon
funcAons
Backend is an Interactive Content Delivery Cloud that must:
XML
Databases
SQL/MR
Engines
In-­‐memory
Data
Grids
Enterprise
Search
Servers
Experimented with a combination of:
The Challenge

18. The Learning Portal
• Designed and built as a
collaboration between MHE Labs
and Couchbase
• Serves as proof-of-concept and
testing harness for Couchbase +
ElasticSearch integration
• Available for download and further
development as open source
code

19. • Document
Modeling
• Metadata
&
Content
Storage
• View
Querying
to
support
Content
Browsing
• ElasAc
Search
IntegraAon
(Full
Text
Search)
-­‐ Content
Updated
in
near
Real-­‐Time
-­‐ Search
Content
Summaries
-­‐ Relevancy
boosted
based
on
User
Preferences
• Real-­‐Time
Content
Updates
• Event
Logging
for
offline
analysis
Techniques
Used

20. Couchbase
2.0
+
Elas7csearch
Store
full-text articles
as
well
as
document metadata
for
image,
video
and
text
content
in
Couchbase
Combine
user
preferences
staAsAcs
with
custom
relevancy scoring
to
provide
personalized search results
Logs
user behavior
to
calculate
user
preference
staAsAcs
(e.g.
video
>
text)
1
2
4
ConAnuously
accept updates
from
Couchbase
with
new
content
&
stats
3

21. Data
Model
Content Metadata
Bucket
User Profiles
Bucket
Content Stats
Bucket
• Stores content metadata for
media objects and content for
articles
• Includes tags, contributors, type
information
• Includes pointer to the media
• Stores user view details per type
• Updated every time a user views
a doc with running count
• To be used for customizing ES
search results per user
preference
• Stores content view details
• Updated for every time a
document is viewed
• To be used for boosting ES
search results based on
popularity

22. Architecture

23. • Social
media
feeds:
Twiger,
Facebook,
LinkedIn
• Blogs,
news,
press
ar7cles
• Data
service
feeds:
Hoovers,
Reuters
3rd
Party
Data
Aggrega7on
Use
Case:
3rd
party
data
aggrega7on
• Flexibility
to
store
any
kind
of
content
• Flexibility
to
handle
schema
changes
• Full-­‐text
Search
across
data
set
• High
speed
data
inges7on
• Scales
horizontally
as
more
content
gets
added
to
the
system
• JSON
provides
schema
flexibility
to
store
all
types
of
content
and
metadata
• Fast
access
to
individual
documents
via
built-­‐in
cache,
high
write
throughput
• Indexing
and
querying
provides
real-­‐7me
analy7cs
capabili7es
across
dataset
• Integra7on
with
Elas7cSearch
for
full-­‐text
search
• Ease
of
scalability
ensures
that
the
data
cluster
can
be
grown
seamlessly
as
the
amount
of
user
and
ad
data
grows
Types
of
Data
Applica7on
Requirements
Why
NoSQL
and
Couchbase

24. LivePerson
–
Real
7me
visitor
engagement

25. Use
Case:
3rd
party
data
aggrega7on
with
analy7cs
Real
Ame
AnalyAcs
for
LivePerson's
customers
LiveEngage
DASHBOARD

26. LivePerson:
Leading
customer
engagement
plaeorm

27. Requirements Requirements Requirements
• High
throughput,
really
fast
• Linear
scale
• Searchable
(Views
and
M/R)
• Supports
both
K/V
&
Document
store
• Cross
data
center
replicaAon
• “Always
on”,
Resilience
soluAon
The Problem
13
TB
per
month
~1
PB
In
total
1.8
B
Visits
per
month
VOLUME

28. Couchbase
Java
SDK
ApplicaAon
server
Tomcat
M/R
views
cluster
M/R
views
cluster
XDCR
REST
API
Couchbase
Java
SDK
Storm
Topology
Couchbase
Java
SDK
Storm
Topology
Architecture

29. Visitor
Stream
Event
Processing
Visitor
Feed
-­‐
Storm
Topology
Customer
RepresentaAve
Kala
Couchbase
Visitor
Monitoring
Service
(1)
Visitor
browsing
(2)
Visitor
events
(4)
Write
event
to
user
document
(6)
Return
relevant
visitors
(7)
Return
relevant
visitors
(5)
Get
visitors
List
Every
3
sec
Visitor
Feed
API
(3)
Analyze
relevant
events
and
persist
Data
flow

30. Document
Structurestructure
{
"accountId": "64302875",
"id": 121640710013,
"rtSessionId": "643028754295878498",
"eventSequence": 5104,
"ipAddress": {
"fieldValue": "194.39.63.10",
"seq": 1
},
"browser": {
"fieldValue": "Chrome 27.0.1453.116",
"seq": 1
},
"state": {
"fieldValue": "LEFT_SITE",
"seq": 5104
}
......................................
}
MulA
tenant
DB
Basic
visitor
informaAon
Sequence
use
due
to
Kala

31. Couchbase
Server

32. Easy
Scalability
Consistent
High
Performance
Always
On
24x365
Grow
cluster
without
applicaAon
changes,
without
downAme
with
a
single
click
Consistent
sub-­‐millisecond
read
and
write
response
Ames
with
consistent
high
throughput
No
downAme
for
so?ware
upgrades,
hardware
maintenance,
etc.
JSON
JSON
JSON
JSONJSON
PERFORMANCE
Flexible
Data
Model
JSON
document
model
with
no
fixed
schema.
Couchbase
Server

33. Features
in
Couchbase
Server
2.0
JSON
support
Indexing
and
Querying
Cross
data
center
replica7on
Incremental
Map
Reduce
JSON
JSON
JSON
JSONJSON

34. Addi7onal
Features
Built-­‐in
clustering
–
All
nodes
equal
Data
replicaAon
with
auto-­‐failover
Zero-­‐downAme
maintenance
Built-­‐in
managed
cached
Append-­‐only
storage
layer
Online
compacAon
Monitoring
and
admin
API
&
UI
SDK
for
a
variety
of
languages

35. Ques7ons?

36. Thank
you!
dip7@couchbase.com