Couchbase Day

Wi-Fi: Network – Matrix | Password - 11223344

©2015 Couchbase Inc. 2
Today’s Agenda
Registration and Continental Breakfast
§ Labs and Environment Setup
Introduction to NoSQL
§ Why NoSQL?
§ Introduction to Couchbase
Couchbase Architecture
§ Single Node Operations
§ Cluster Operations
Break
HA and DR
§ Availability and XDCR
§ Backup/Restore
§ Security
12:00 – Lunch Break
Developing with Couchbase
§ Couchbase SDKs
Couchbase Indexing
§ Couchbase GSI andViews
§ Views Overview
Break
Indexing and Query
§ Couchbase Indexes (GSI/MOI)
§ Couchbase Query (N1QL)
Couchbase Mobile
§ Overview
Adjourn
§ Close

Why companies need a new database solution
§ What are the biggest problems with Relational Databases that are
driving adoption of NoSQL?
LACKOFFLEXIBILITY/
HASRIGIDSCHEMAS
INABILITY
TOSCALEOUT
PERFORMANCE
CHALLENGES
49%
69%
50%
47%
44%
COST
ALLOFTHEABOVE
35%
29%
16%
12%

Dynamo
October 2007
Cassandra
August 2008
Bigtable
November 2006
Voldemort
February 2009
Very few organizations want to (fewer can) build and maintain database software technology.
But every organization participating in the digital economy needs this technology.
Why NoSQL?
New data solutions to address emerging needs:
§ Performance and Scalability
§ High Availability with Data Synchronization (multi-datacenter)
§ Flexible Schema to accommodate frequent changes
§ Transparent/Auto-Sharding without application participation
§ Distributed System for Querying
§ Managed Cache

©2015 Couchbase Inc. 6Scaling out flattens the cost and performance curves
NoSQL Database Scales Out
Cost and performance mirrors app tier
Users
NoSQL Distributed Data Store
Web/App Server Tier
Application Scales Out
Just add more commodity web servers
Users
System Cost
Application Performance
Application Performance
System Cost
Why NoSQL

How has Couchbase Server evolved beyond NoSQL?
§ Discreet, scalable services (data, index, query)
§ Filtering for cross data center replication
§ SQL-based query language and secondary indexing
§ Geospatial search and indexing
§ CRUD, prepared statements, and covering indexes
§ Memory-optimized Indexes
§ Array Indexing within Nested JSON
§ Graphical QueryW0rkbench
§ Sub-Document API
§ LDAP integration and auditing; RBAC, X-509 Certificates

Next Generation Data Management Architecture
Data
Sources
Enterprise
Applications
Social Media
Web &
Ecommerce
Mobile AppsSensor Data Mainframe
External
Data
System Logs
Data
Integration
Speed Layer Batch Layer
Data
Management
Data Lake Data Warehouse
Business Intelligence Dashboards Query ToolsData
Access
In Memory Cache
Data Serving Layer

Next Generation Data Management Architecture
Enterprise
Applications
Social Media
Web &
Ecommerce
Mobile AppsSensor Data MainframeExternal DataSystem Logs
Speed Layer Batch Layer
Data Lake
Business Intelligence Dashboards Query Tools
In Memory Cache
Data Warehouse
N1QL Query Workbench
Data
Sources
Data
Integration
Data
Management
Data
Access
Data Serving Layer
Real Time
Data Ingestion

Couchbase Ecosystem

How Does Couchbase Fit In Big Data?
Couchbase and Hadoop are complementary.

Couchbase and Hadoop
New Data
Stream
MergedView
All Data
Precompute
Views
(Map Reduce)
Process
Stream
Incremental
Views
Partial
Aggregate
Partial
Aggregate
Partial
Aggregate
Real-Time Data
Batch
Recompute
BatchViews
Real-TimeViews
Real-Time
Increment
Merge
Batch Layer
Serving Layer
Speed Layer
Couchbase Hadoop
Connector (Sqoop)

Couchbase Hadoop
Connector (Sqoop)
Couchbase and Hadoop
New Data
Stream
MergedView
All Data
Precompute
Views
(Map Reduce)
Process
Stream
Incremental
Views
Partial
Aggregate
Partial
Aggregate
Partial
Aggregate
Real-Time Data
Batch
Recompute
BatchViews
Real-TimeViews
Real-Time
Increment
Merge
Batch Layer
Serving Layer
Speed Layer
Stream / Data
Ingestion
Store
Incremental
Data / Stream
processing
Serving merged
results /
responses

Couchbase Server on Docker and OrchestrationTechnologies
Official Docker Support with MacOS, Ubuntu is coming
Couchbase & Containers

Platform as a Service
§ Couchbase runs on PaaS environments
Current Focus Future Plan

Infrastructure as a Service
Couchbase supports IaaS environments
§ Available in AWS
marketplace
§ BYOL or hourly pricing
§ Available in Azure
marketplace
§ BYOL or hourly pricing
§ Marketplace image in
the works
§ BYOL at the moment
§ Hourly pricing soon
Other providers supported

Couchbase Overview
What is Couchbase?

Couchbase Server Defined
§ Couchbase Server 4.X is the first NoSQL Database to enable
you to develop with agility and operate at any scale.
Managed Cache Key-Value Store Document Database Embedded Database Sync Management

The Power of the Flexible JSON Schema
Ability to store data in multiple ways
• Denormalized single document, as opposed to normalizing data across multiple table
• Dynamic Schema to add new values when needed

Document Fundamentals - Composition
§ Similar to primary keys in relational databases
§ Documents are partitioned based on the
document ID
§ ID based document lookup is extremely fast
§ Must be unique
§ JSON
§ Binary - integers, strings, booleans
§ Common binary values include serialized objects,
compressed XML, compressed text, encrypted
values
Document ID or Key
Value
§ CASValue (unique identifier for concurrency)
§ TTL
§ Flags (optional client library metadata)
§ Revision #
Metadata
Document
Document ID /
Key
MetadataValue

Storing And Retrieving Documents

Couchbase Data Access
• Everything is layered on top of KeyValue
• A Document store is a special case of Key-
Value
• Views provide aggregation and real-time
analytics through incremental map-reduce
• Global Secondary Indexes provide low
latency/high throughput indexes
• N1QL is a language that provides a powerful
and expressive way of accessing documents

Single Node
23

• Data Service – builds and
maintains Distributed secondary
indexes (MapReduceViews)
• Indexing Engine – builds and
maintains Global Secondary
Indexes
• Query Engine – plans,
coordinates, and executes queries
against either Global or
Distributed indexes
• Cluster Manager – configuration,
heartbeat, statistics, RESTful
Management interface

Couchbase Read Operation
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
DOC 1
GET
DOC 1
DOC 1

APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
Write Operation
DOC 1
DOC 1DOC 1

Cache Ejection
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
DOC 1
DOC 2DOC 3DOC 4DOC 5
DOC 1
DOC 2 DOC 3 DOC 4 DOC 5

APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
DOC 1
Cache Miss
GET
DOC 1
DOC 1
DOC 1

Cluster

Auto sharding – Bucket and vBuckets
vB
Data buckets
vB
1 ….. 1024
Active Virtual buckets
vB vB
1 ….. 1024
Replica Virtual buckets

Cluster Map
Couchbase Cluster
CLUSTER MAP
Hashing Algorithm
CRC32
vBucket2
vBucket3
vBucket4
vBucket5
vBucket6
vBucket7
vBucket1024
vBucket1
...
Couchbase SDK
Couchbase Cluster
CLUSTER MAP
Hashing Algorithm
CRC32
vBucket2
vBucket3
vBucket4
vBucket5
vBucket6
vBucket7
vBucket1024
vBucket1
...
Couchbase SDK

ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
Basic Operation
SHARD
5
SHARD
2
SHARD
9
SHARD SHARD SHARD
SHARD
4
SHARD
7
SHARD
8
SHARD SHARD SHARD
SHARD
1
SHARD
3
SHARD
6
SHARD SHARD SHARD
SHARD
4
SHARD
1
SHARD
8
SHARD SHARD SHARD
SHARD
6
SHARD
3
SHARD
2
SHARD SHARD SHARD
SHARD
7
SHARD
9
SHARD
5
SHARD SHARD SHARD

Rebalance
ACTIVE ACTIVE
REPLICA REPLICA
Couchbase Server 4 Couchbase Server 5
SHARD
5
SHARD
2
SHARD SHARD
SHARD
4
SHARD SHARD
SHARD
1
SHARD
3
SHARD SHARD
SHARD
4
SHARD
1
SHARD
8
SHARD SHARD SHARD
SHARD
6
SHARD
3
SHARD
2
SHARD SHARD SHARD
SHARD
7
SHARD
9
SHARD
5
SHARD SHARD SHARD
SHARD
7
SHARD
SHARD
6
SHARD
SHARD
8
SHARD
9
SHARD
READ/WRITE/UPDATE

Fail Over
ACTIVE ACTIVE
REPLICA REPLICA
SHARD
5
SHARD
2
SHARD SHARD
SHARD
4
SHARD SHARD
SHARD
1
SHARD
3
SHARD SHARD
SHARD
4
SHARD
1
SHARD
8
SHARD SHARD
SHARDSHARD
6
SHARD
2
SHARD SHARD SHARD
SHARD
7
SHARD
9
SHARD
5
SHARD SHARD
SHARD
SHARD
7
SHARD
SHARD
6
SHARDSHARD
8
SHARD
9
SHARD
SHARD
3
SHARD
1
SHARD
3
SHARD

High Availability with Couchbase
35

Recovering from a Fail Over
ACTIVE ACTIVE
REPLICA REPLICA
SHARD
5
SHARD
2
SHARD SHARD
SHARD
4
SHARD SHARD
SHARD
1
SHARD
3
SHARD SHARD
SHARD
4
SHARD
1
SHARD
8
SHARD SHARD
SHARDSHARD
6
SHARD
2
SHARD SHARD SHARD
SHARD
7
SHARD
9
SHARD
5
SHARD SHARD
SHARD
SHARD
7
SHARD
SHARD
6
SHARDSHARD
8
SHARD
9
SHARD
SHARD
3
SHARD
1
SHARD
3
SHARD

Recovering from Failover
Three options:
1. Remove the node
2. Full recovery
3. Delta node recovery

Delta Node Recovery
Recover Failed Nodes with incremental Catch-up
§ Improve recovery time for nodes with large datasets
§ Add a failed node into cluster and recover data from where it left off

Rack Zone Awareness – Server Groups
Informs Couchbase as to the
physical distribution of nodes in
racks or availability groups.
The Active vBuckets in Server 1
will have its replica’s in a different
group. Never in the same group.
Couchbase Server 3
Couchbase Server 2
Couchbase Server
6
Couchbase Server 5
Couchbase Server 4
Couchbase Server
9
Couchbase Server
8
Couchbase Server 7
Ensures that replica vBuckets are
distributed across groups
REPLICA
REPLICA
Group: A Nodes 1,2,3
Group: B Nodes 4,5,6
Group: C Nodes 7,8,9
Couchbase Server 1
ACTIVE

Cross Datacenter Replication
(XDCR)
41

Cross Datacenter Replication (XDCR)
TO OTHER NODE
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
XDCR QUEUE
TO OTHER CLUSTER
DOC 1
DOC 1DOC 1DOC 1

TO OTHER NODE
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
QUEUE
REPLICATION
QUEUE
XDCR QUEUE
TO OTHER CLUSTER
DOC 1
DOC 1DOC 1DOC 1
Single-node type means easier
administration and scaling
CLUSTER
REPLICATION
vBucket
REPLICAS

XDCR: Cluster Topology Aware
• Applications can access both Clusters
(active-active)
• Can provide filtered replication
• One-one, one-many, many-one
• Differently sized clusters
• Different HW platforms
• Mixed versions supported (Upgrades)

Unidirectional or Bidirectional Replication
Unidirectional
§ Hot spare / Disaster Recovery
§ Development/Testing copies
§ Connectors (Solr, Elasticsearch)
§ Integrate to custom consumer
Bidirectional
§ Multiple Active Masters
§ Disaster Recovery
§ Datacenter Locality

Key Features
§ Zero downtime backup and restore
§ Single utility to manage multiple clusters
§ Restore from any point, to any bucket or topology
§ Fully differential – merge backups to maintain desired restore points
§ Built-in concurrency control
§ Resume interrupted backups
§ Compaction reduces storage requirements

Enterprise Backup and Restore
§ >10x Faster Backups
§ Better Data Protection with Enterprise Backup Restore
– Scale to larger datasets
– Fast backups and restores
– Better Consistency in backup data
– Resilient to failures/Topology Changes
288
25
0
50
100
150
200
250
300
350
BackUpMB/Sec
4.5 Backup Speed 4.1 Backup Speed

Data Import
§ Create backup repo
/opt/couchbase/bin/cbbackupmgr config --archive /tmp/backup --repo demo
§ Import of sample data
/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password
--bucket travel-sample --dataset /opt/couchbase/samples/travel-sample.zip --format sample
§ Import of list data
/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password
--bucket chicago --dataset file:///root/data/companies.json --format list --generate-key %permalink%::#MONO_INCR#

Security with Couchbase
§ Administrative Security:
§ Admin user (full access) vs. Read-only user (monitoring, developer access)
§ SSL encryption to REST API and Web UI
§ HTTP access log
§ Data Security:
§ Applications connect via SASL with single user/pass
§ Data-at-Rest encryption via partnership withVormetric
§ SSL encryption for over-the-wire
§ Couchbase 4.0:
§ LDAP integration
§ More extensive administrative action auditing
More at:
http://docs.couchbase.com/admin/admin/security/security-best-practices.html

Couchbase authentication overview
§ Application authentication
§ Buckets are protected with challenge-response SASL protocol
§ AuthN happens over CRAM-MD5
§ Admin authentication
§ Authentication through admin username and password
§ Authentication through LDAP (New in 4.0)
AUTHENTICATION

Couchbase authorization overview
§ Application data access
§ Full access to the bucket application is connected to
§ Admin access
§ Full administrator has full privileges on the cluster
§ Read-only administrator cannot change cluster settings
AUTHORIZATION

Couchbase encryption overview
§ Data-in-motion encryption
§ Client-server communication can be encrypted using SSL
§ Secure admin access using SSL over port 18091
§ Secure view access using SSL over port 18092
§ Secure XDCR for encryption across datacenters
Track all Access
SERVER 3SERVER 1 SERVER 2
Couchbase Server – New York
SERVER 3SERVER 1 SERVER 2
Couchbase Server – London
SSL
Client applications
Secure
XDCR over
SSL
Admin
access
over port
18091
SSL
View
access
over port
18092
SSL
https://couchbase_server:18091/…
https://couchbase_server:18092/…
ENCRYPTION

Couchbase encryption overview
§ Transparent data-at-rest encryption solution ENCRYPTION
Storage
Database
Application
User
File
Systems
Volume
Managers
DSM
Vormetric
Data Security Manager
on Enterprise premise or in cloud
virtual or physical appliance
• Centrally manage keys and policy
• Virtual and physical appliance
• High-availability with cluster
• Multi-tenant and strong separation of duties
• Proven 10,000+ device and key management scale
• Web, CLI, API Interfaces
• FIPS 140-2 certified
Secure Personally Identifiable Information
• User profile information
• Login Credentials
• IP Addresses

External identity management using LDAP
Centralized identity management
§ Define multiple read-only admins and full-admins
§ Centralized security policy management for admin accounts for
stronger passwords, password rotation, and auto lockouts
Individual accountability. Simplified compliance.
§ Define UIDs in LDAP, and map UIDs to read-only / full admin role in
Couchbase
§ Comprehensive audit trails with LDAP UIDs in audit records

New UI for authorizing LDAP administrators
Turn on/off
LDAP
Add UIDs to
read-only admins
Add UIDs to full
admins
Set default behavior
if UID is not mapped
Testing credentials
to verify what level
of access
Plus REST, and CLI integration for programmatic setup

Admin Auditing in Couchbase
Rich audit events
§ Over 25 different, detailed admin audit events
§ Auditing for tools including backup
Configurable auditing
§ Configurable file target
§ Support for time based log rotation and audit filtering
Easy integration
§ JSON format allows for easy integration with downstream
systems using flume, logstash, and syslogd

Auditing a successful login
{
"timestamp":"2015-02-20T08:48:49.408-08:00",
"id":8192,
"name":"login success",
"description":"Successful login to couchbase cluster",
"role":"admin",
"real_userid": {
"source":"ns_server",
"user":"bjones”
},
"sessionid":"0fd0b5305d1561ca2b10f9d795819b2e",
"remote":{"ip":"172.23.107.165", "port":59383}
}
WHEN
WHO
WHAT
HOW

Languages and Interfaces for Couchbase
§ Official SDKs
– Java
– .NET
– Node.js
– Python
§ For each of these we have
– Full Document support
– Interoperability
– Common Programming Model
Others: Erlang, Perl,TCL, Clojure, Scala
- PHP
- C
- Go
- Ruby
JDBC and ODBC

Interacting with Data
Cluster
Bucket
CRUD
View
Query
N1QL
Query

Couchbase SDKs
§ What does it mean to be a Couchbase SDK?
Cluster
Bucket
CRUD
Functional
Give the application developer a concurrent
API for basic (k-v) or document management
API
get()
insert()
replace()
upsert()
remove()

The Document
§ Documents are integral to the SDKs.
§ All SDKs support JSON format
§ In addition: Serialized objects, Unquoted Strings, Binary pass-through
§ A Document contains:
67
Property Description
ID The bucket-unique identifier
Content The value that is stored
Expiry An expiration time
CAS Check-and-Set identifier

Bucket API – Modifying Documents
§ insert() the document if it does not exist
§ replace() the document if it does exist
§ upsert() the document (insert or replace)
§ remove() the document
§ append() data to the document
§ prepend() data to the document
§ counter() for increment/decrement type operations

Bucket API – Retrieving Documents
§ get() the document
§ getFromReplica() if the master is not available
§ getAndTouch() to load the document and reset the expiry
§ getAndLock() to load the document and write-lock it

Unified API – DML, CAS Example
1. Two Clients retrieve the same document "XYZ"
2. Client A retrieves it first.
3. Client B then retrieves XYZ. Both clients will have the same CAS value for document XYZ
4. Client B tries to perform an update to document XYZ. The update succeeds as the CAS value
was unchanged from when Client B initially retrieved the document. Once the update
succeeds, the CAS value for XYZ changes.
5. Client A then tries to perform an update on XYZ immediately after Client B. The update will
fail as Client A's CAS value is out of date. When Client B updated XYZ, the CAS value
changed.
©2014 Couchbase, Inc.

Durability Options for writes
APPLICATION
SERVER
CACHE
DISK
XDCR QUEUE
TO OTHER CLUSTER
CACHE
DISK DISK
QUEUE
REPLICATION
QUEUE
APPLICATION
SERVER
DISK DISK
QUEUE
CACHE
REPLICATION
QUEUE
DISK
QUEUE
DOC
DOCDOC DOCDOCDOC DOC
DOC
CLUSTER 1
NODE 2
CLUSTER 2
NODE …
CLUSTER 1
NODE 1
DOC
( PUT ) By default, a write operation is successful when the data has been written to the
memory of the node with the active vBucket. Eventually (but pretty quickly), the data will
also be written to persistent storage and to replica vBuckets.
( replicateTo ) write operation is
successful when data has been
written to both the active
vBucket and replica vBuckets.
( persistTo ) write operation is successful when data has been
written to persistent storage
• Can be set to require writes to persistent storage for replicas.
• Using persistTo will slow performance.

Bucket API – Querying
§ query() is possible for
§ Views
§ N1QL
§ Streams N response rows as they arrive from the server
§ Loads more than one Document based on Criteria
§ Typically used to satisfy secondary and advanced querying use cases

Which to choose – GSI vsViews
Workloads New GSI in v4.X Map/ReduceViews
Complex
Reporting
Just InTime Aggregation Pre-aggregated
Ad-hoc
Querying
Ad-hoc View per Query
Flexible
Indexing Logic
N1QL Functions Javascript
Secondary
Lookups
Faster with
Single Node Lookup
Slower due to
Scatter-Gather
Range
Scans
Faster with
Single Node Lookup
Slower due to
Scatter-Gather

Comparing GSI andViews
Capabilities Global Secondary Indexes Map/ReduceViews
Partitioning Model Independent – Indexing Service Aligned to Data – Data Service
Scale Model Independently Scale Index Scale with Data Service
Secondary Lookup Single Node Scatter-Gather
Range Scan Single Node Scatter-Gather
Grouping, Aggregates With N1QL Built-in withViews API
Caching Managed Not Managed
Storage ForestDB Couchstore
Availability Identical Indexes load balanced Replica Based

Indexing Differences
§ Local secondary indexes (Views)
– Co-located with data
– Higher write performance / Lower read performance: scatter-gather
– Map/ReduceViews: programmable indexer for complex reporting and indexing logic.
– SpatialViews: Incremental R-tree indexing for powerful bounding-box queries
– Full partition alignment and paired scalability with Data Service
§ Global secondary indexes (GSI / MOI)
– Independently partitioned and independently scalable indexes in Indexing Service
– Isolated from Key-Value operations
– Low latency queries without compromising on mutation performance (insert/update/delete)
– Higher query performance
– Async writes to a large number of global indexes
– Memory Optimized Indexes (MOI) alternate storage option to standard GSI - small memory
footprint, optimized for lowest latency queries

Indexing with N1QL
Indexing in N1QL
Global and Local Indexes
IndexingTypes for N1QL
View Based Indexes
• Local Indexes:
• PartitionedAligned to Data
• Auto Partitioning
• Built in Replicas
• Simple Scaling Model
Global Secondary Indexes (GSI)
• Global Indexes:
• Independently Partitioned Indexes
• Predictable High Performance at Scale
• Sophisticated Scaling Model
Controlled with USING VIEW|GSI in CREATE INDEX. Set per Index. Drop and Recreate the index
using DROP INDEX & CREATE INDEX to change Index Type.

A little aboutViews…
§ Using views you can define
– Primary indexes
– Simple secondary indexes (most common use case)
– Complex secondary, tertiary and composite indexes
– Aggregations (reduction)
§ Built using MapReduce technology
– Map and Reduce functions are written in Javascript

A little aboutViews…
§ Captures user-defined key/value pairs about a document into an index
§ Can execute logic on data during indexing
§ Able to express relationships too cumbersome to maintain in Key/Value
§ Incremental indexing essentially pre-computes query results
§ Higher latency compared to key/value lookups
§ Common Use Cases
– Materialized view: project an alternate view of your document
– Complex aggregates
– Lightweight reporting

Queries run against stale indexes by default
§ stale=update_after (default if nothing is specified)
• always get fastest response
• can take two queries to read your own writes
§ stale=ok
• auto update will trigger eventually
• might not see your own writes for a few minutes
• least frequent updates -> least resource impact
§ stale=false
• Use with “set with persistence” if data needs to be included in view results
• BUT be aware of delay it adds, only use when really required

Lightweight Reporting Example
Document:
{
“date”: “2015-08-03T16:38:05.835Z”,
“source”: “192.168.56.101”,
“logtype”: “error”,
“message”: “An error occurred during startup”
}
Map:
function(doc, meta) {
emit(dateToArray(doc.date), doc.logtype);
}

function(key, values, rereduce) {
var response = {"warning" : 0, "error": 0, "fatal" : 0 };
for(i=0; i<values.length; i++) {
if (rereduce) {
response.warning = response.warning + values[i].warning;
response.error = response.error + values[i].error;
response.fatal = response.fatal + values[i].fatal;
} else {
if (values[i] == "warning")
response.warning++;
if (values[i] == "error" )
response.error++;
if (values[i] == "fatal" )
response.fatal++;
}
}
return response;
}

Query: ?group_level=2&startkey=[2010,7]&endkey=[2011,4]
{"rows":[
{"key":[2010,7], "value":{"warning":4,"error":2,"fatal":0}},
{"key":[2010,10],"value":{"warning":7,"error":6,"fatal":0}},
{"key":[2011,4], "value":{"warning":3,"error":6,"fatal":0}}
]
}

Data Service
Projector & Router
Indexing Service
Query ServiceIndex Service
Supervisor
Index maintenance &
Scan coordinator
Index#2Index#1
Query Processor
cbq-engine
Bucket#1 Bucket#2
DCP Stream
Index#4Index#3
...
B
u
c
k
e
t
#
2
B
u
c
k
e
t
#
1
Projector and Router:
1 Projector and Router per node
1 stream of changes per bucket per
supervisor
ForestDB
Storage Engine
Supervisor
1 Supervisor per node
Many indexes per Supervisor
MemDB Storage Engine for
MOI

Types of Indexes
§ Composite Indexes
§ Covering Indexes
§ Filtered Indexes
§ Function Based Indexes
§ Sub-Document Indexes
§ Incremental MapReduceViews
§ SpatialViews
§ FullText Indexes
§ Prepared Statements
§ N1QL Complete SQL CRUD support: Insert, Delete, Merge, Etc.

create index idx_odspilot on ods(DISTINCT ARRAY p.FILEN in p in PILOT END);
Array Indexing
§ Index only required elements or attributes in the array
§ Benefits are lot more significant for nested arrays/objects
§ Efficient on Index storage & search time
Transportation Company POC
• 422,137 documents.
• Query2: BEFORE array indexing
• Primary index scan
• 38.91 seconds.
• Query2: AFTER array indexing
• Array index scan [DistinctScan]
• 8.51 millisecond
• Improvement of 4572Times

Step by Step – Building indexes
§ Building and Index
1. Issue “CREATE INDEX …”
2. Place on the “next node” if WITH {“nodes”:[]}not specified
– “next node” picked by round robin
3. Create metadata for the index on the node
– “next node” picked by round robin
4. If deferred build = true
– Mark Index Status = “Pending”
– Wait for “BUILD INDEX ...”
5. Else
– Mark Index Status = “Pending”
– Start DCP Backfill with Projector
– When Done – Mark Index Status = “Online”

Step by Step – Ingest a Mutation
§ Ingest a Mutation on Data
1. DCP Feed communicates the mutation to Projector on the node.
2. Projector picks up the mutation for a bucket
– Projector strips the mutation to check relevance to existing indexes on each indexer
– Projector Sends stripped mutation to indexers on all index service nodes
3. Indexer receive the mutation on each index service node
– Queue the mutation
– ExtractionWorker Prepares and queues the update to each index.
– Persistence Worker persist the mutation to the index

Global Secondary Indexes – Storage Options
Global Secondary Index
Predictable low latency with Query and Key Based Operations,
even in presence of large number of indexes
Storage Option for GSI
Standard Global Secondary Indexes
• Optimized for Storage + Memory: efficient spill-
to-disk with ForestDB as the storage engine
• Advanced IO Performance: New circular-writes
that minimize compaction overhead
Memory-Optimized Global Secondary Indexes (MOI)
• Optimized for Memory: small memory footprint,
optimized for lowest latency queries
• Faster Indexing: fresh indexes under heavy mutations with
lock free index maintenance
Controlled with Cluster Settings (Settings Tab). Cluster-wide Setting for all Indexes and
Require removing all GSI indexes and index service nodes to change storage options.
(New in 4.5)

Memory Optimized (MOI) Global Secondary Indexes
• Optimized for Memory: small memory footprint, optimized for lowest
latency queries
• Faster Indexing: fresh indexes under heavy mutations with lock free
index maintenance

Which to Choose – Standard vs Memory Optimized GSI
Standard GSI Memory Optimized GSI
Initial Index BuildTimes Slower Faster
Ongoing Index Build
Times
Slower Faster
Ability to Keep up with
Mutation Rate
Lower Higher
Memory Demand for
Larger Indexes
Lower Higher
Behavior on OOM
(out of memory)
Spill to disk Stop processing mutations

Indexing & HA
§ Mirrored Indexes are load balanced and used for HA
– Size at least 2 indexes for production
– Add more indexes if all requests cannot be handled by a single index to protect against
failures

Indexes & Partitioning
§ Why Partition Indexes?
– Index may not fit on the node
– Fit index to the nodes you have
– Distribute Index load to many nodes
– Minimize the tree scanned
– Large/Tall indexes will take longer to scan
§ How to Partition Indexes?
– By list of values – great for scan latency
– ... WHERE type = “customer”
– ... WHERE type = “product”
– By range –
– ... WHERE id BETWEEN 10 AND 20
– ... WHERE id BETWEEN 20 AND 30
– By function – no range scan ability

Efficient IO with Circular Writes in Standard GSI
Standard Global Secondary Indexes
§ Reduced Disk IO Requirements
– Append-OnlyWrites with frequent full compaction (Version 4.1 & Earlier)
– Circular-Reuse Writes with reduced full compactions (New in 4.5: )
– Reused orphaned blocks in the index file
– Reduce the need for frequent full-compactions of the index file
Index
Mutation
Append-OnlyWrite
CircularWrite
File for Global Secondary Indexes
Index
Mutation
Index
Mutation

What is N1QL?
§ N1QL is a new query language that systematically extends SQL to
document-oriented data
§ N1QL extends SQL to handle data that is:
– Nested: Contains nested objects, arrays
– Heterogeneous: Schema-optional, non-uniform
– Distributed: Partitioned across a cluster

N1QL (EXPRESSIVE)
§ Access to every part of JSON document
§ Scalar & Aggregate functions
§ Issue subquery in any expressions
§ Subqueries
§ Subqueries in the FROM clause
Give developers and enterprises an expressive, powerful, and complete
language for querying, transforming, and manipulating JSON data.

N1QL (POWERFUL)
§ Access to every part of JSON document
§ JOINS, Aggregations, standard scalar functions
§ Aggregation on arrays
§ NEST & UNNEST operations
§ Covering Index

N1QL (QUERYING)
§ INSERT
§ UPDATE
§ DELETE
§ MERGE
§ SELECT
§ EXPLAIN

N1QL (TRANSFORMING & MANIPULATING)
§ FullTransformation of the data via Query.
§ INSERT
§ INSERT single & multiple documents
§ INSERT result a SELECT statement
§ DELETE documents based on complex filter
§ UPDATE any part of JSON document & use complex filter.
§ MERGE two sets of documents using traditional MERGE statement
§ SUBQUERIES

Optimizing Query Speed
§ There are four options for N1QL queries:
Primary Index
Secondary Index
(GSI or MOI)
Covered Index
“ON KEYS”

Query Execution Flow
1. Application submits
N1QL query
2. Query is parsed,
analyzed and plan is
created
1
2

3. Query Service makes
request to Index Service
4. Index Service returns
document keys and data
3
4

5. If Covering Index, skip
step 6
6. If filtering is required,
fetch documents from
Data Service56

7. Apply final logic (e.g.
SORT, ORDER BY)
8. Return formatted results
to application
7
8

Inside a Query Service (without covering index)
Client
FetchParse Plan Join Filter
Pre-Aggregate
Offset Limit ProjectSortAggregateScan
Query Service
Index
Servic
e
Data
Servic
e
Data-parallel — Query is N data streams over N cores*
Memory-based
Pluggable architecture — datastore, index…
Request Response

Query Consistency
Strict Request-Time
Consistency
(request_plus)
Query execution is delayed until all
indexes process mutations up toT3
RYOW Consistency
(at_plus)
Query execution is delayed until all
indexes process mutations up toT1
T1: insert (k1, v1)
T3: issue query on (k1,v1)
T2: do other business logic computation
…
…

Joins (Referential)

Document-oriented Extensions
§ Nested
– Paths—user.profile.email, children[0], children[0:2]
– NEST, UNNEST
– Ranging—EVERY child.age > 5 FOR child IN children END
– Transformations—SELECT {“name”: first_name || last_name}
§ Non-uniform
– IS MISSING
– Type checking and conversion
§ Distributed
– Direct lookup—USE KEYS
– Efficient joins—ON KEYS

SELECT statement
SELECT customers.id,
customers.NAME.lastname,
customers.NAME.firstname
Sum(orderline.amount)
FROM orders UNNEST orders.lineitems AS orderline
JOIN customers ON KEYS orders.custid
WHERE customers.state = 'NY'
GROUP BY customers.id,
customers.NAME.lastname
HAVING sum(orderline.amount) > 10000
ORDER BY sum(orderline.amount) DESC
Dotted sub-document reference
Names are CASE-SENSITIVE
UNNEST to flatten the arrays
JOINS with Document KEY of
customers

UNNEST example: Customer Document
{
"C_ZIP" : "828011111",
"C_STATE" : "vt",
"C_FIRST" : "ykfdbqku",
"C_CREDIT" : "GC",
"C_DELIVERY_CNT" : 0,
"C_W_ID" : 1,
"C_CITY" : "quhpismkzumehqhr",
"C_STREET_1" : "rmtxadlsxqefdcwf",
"C_D_ID" : 1,
"ORDERS" : [
{
"ORDER_LINE" : [
{
"OL_AMOUNT" : 0,
"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z",
"OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye",
"OL_I_ID" : 35828,
"OL_SUPPLY_W_ID" : 1,
"OL_QUANTITY" : 5
},
{
"OL_AMOUNT" : 0,
"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z",
"OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb",
"OL_I_ID" : 26024,
"OL_QUANTITY" : 5
},
}
….

UNNEST example: N1QL Code
{
"C_ZIP" : "828011111",
"C_STATE" : "vt",
"C_FIRST" : "ykfdbqku",
"C_CREDIT" : "GC",
"C_DELIVERY_CNT" : 0,
"C_W_ID" : 1,
"C_CITY" : "quhpismkzumehqhr",
"C_STREET_1" : "rmtxadlsxqefdcwf",
"C_D_ID" : 1,
"ORDERS" : [
{
"ORDER_LINE" : [
{
"OL_AMOUNT" : 0,
"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z",
"OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye",
"OL_I_ID" : 35828,
"OL_QUANTITY" : 5
},
{
"OL_AMOUNT" : 0,
"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z",
"OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb",
"OL_I_ID" : 26024,
"OL_QUANTITY" : 5
},
}
….
SELECT COUNT(my_order_line)AS total_orders,
MAX(my_order_line.ol_delivery_d)AS max_delivery_date,
MAX(my_order_line.ol_quantity)AS max_order_quantity,
MAX(my_orders.o_entry_d) AS max_customer_entry,
MAX(my_orders.o_ol_cnt) AS max_orderline_entry,
COUNT(customer)AS total_customers
FROM CUSTOMER MY_CUSTOMER
UNNEST ORDERS AS my_orders
UNNEST my_orders.order_line AS my_order_line
;

Named Prepared Statement
url="http://localhost:8093/query"
s = requests.Session()
s.keep_alive = True
s.auth = ('Administrator','password')
query = {'statement':’prepare select * from `beer-sample` where type=’beer’
and name = 'Guinness’}
r = s.post(url, data=query, stream=False)
prepared = str(r.json()['results'][0]['name'])
query = {'prepared': '"' + prepared + '"' }
for i in range (0, 5000):
print i, r.json()['metrics']['executionTime']

Named Prepared Statement
url="http://localhost:8093/query"
s = requests.Session()
s.keep_alive = True
s.auth = ('Administrator','password')
query = {'statement':'prepare select * from `beer-sample` where name = [$1]’}
prepared = str(r.json()['results'][0]['name'])
for i in range (0, 5):
query={'prepared': '"' + prepared + '"', 'args': '["old_hat_brewery"]' }
print i, r.json()['metrics']['executionTime']
Prepare Once
Bind Values Many Times

Query Workbench – Query Editing
Query Editor
Autocomplete
Syntax highlighting
Multi-line formatting
History
Create, edit, save, run
Results
JSON /Table /Tree

Query Workbench – Schema Browsing
Schema
Inferred
Document types
Element types
Frequency
Sample data
Bucket
Schema
Indexes

Query Workbench – Query Monitoring
Catalogs
Vitals
Active Requests
Prepared Statements
Completed Requests
Interfaces
QueryWorkbench
CLI (Query Shell)
REST API

Monitoring: Active requests
§ List / Delete requests currently being run by the query service
§ Through N1QL
– SELECT * FROM system:active_requests
– DELETE FROM system:active_requests WHERE…
§ Through REST
– GET http://localhost:8093/admin/active_requests
– GET http://localhost:8093/admin/active_requests/<request_id>
– DELETE http://localhost:8093/admin/active_requests/<request_id>

Monitoring: Prepared statements
§ List / Delete requests prepared on the query node
§ Through N1QL
– SELECT * FROM system:prepareds
– DELETE FROM system:prepareds WHERE…
§ Through REST
– GET http://localhost:8093/admin/prepareds
– GET http://localhost:8093/admin/prepareds/<request_id>
– DELETE http://localhost:8093/admin/prepareds/<request_id>

Monitoring: Completed requests
§ List / Delete completed requests deemed to be of high cost
§ Through N1QL
– SELECT * FROM system:completed_requests
– DELETE FROM system:completed_requests where…
§ Through REST
– GET http://localhost:8093/admin/completed_requests
– GET http://localhost:8093/admin/completed_requests/<request_id>
– DELETE http://localhost:8093/admin/completed_requests/<request_id>
§ Provide an overall health picture of the query service
– Using REST: GET http://localhost:8093/admin/vitals

Couchbase Mobile
Introduction
128

Key Challenges When Building Mobile Apps
According to study done by Bock & Company Enterprise Developers Rank Critical Requirements for Success in the Coming Mobile App Explosion

What is Couchbase Mobile
• Removed continual network dependency
• Faster development cycles
• Less long term maintenance than
traditional solutions
• Enterprise class mobile/embedded NoSQL
database + sync platform
• Fast and consistent access to data
Built-in enterprise level security throughout the entire stack includes user authentication, user and
role based data access control (RBAC), secure transport (TLS), and 256-bit AES full database
encryption.
SECURITY

Couchbase Lite Overview
§ Cross-platform support for all major
operating systems and platforms
§ Built native from the ground up
§ 500kb for most platforms
§ 256-bit AES full database encryption
131
Couchbase Lite
Embedded Database

Sync Gateway Overview
§ Synchronize data between Couchbase Lite
and Couchbase Server
§ REST, Stream, Batch, and Event APIs
§ Pluggable authentication
§ Fine grained user and role based access
control
§ Elastically scalable in real-time
132
Sync Gateway
Synchronization

Couchbase Server Overview
§ Scale easily to thousands of nodes
§ Consistent high performance that supports
millions of concurrent users
§ Flexible JSON data model
§ 24x365 always-on availability
133
Couchbase Server
Database Server

End-to-End Security
ü Pluggable authentication
ü Fine grained user and role based access control
ü Transport layer security (TLS)
ü 256-bit AES full database encryption on device
ü File system encryption in the cloud
Security

Who s using Couchbase Mobile today

Common Mobile Use CasesCommon Mobile Use Cases

Offline Data Access
The Problem
• When an app doesn’t work offline, users are cut off from important transactions
• Mobile apps need to work regardless of network connection
How Couchbase Mobile helps
• Couchbase Lite stores data locally so your app isn’t dependent on the network
• If a network connection is lost, the database continues to operate and is automatically
synced with the cloud when the network is restored

Local Caching
The Problem
• Today’s mobile applications require fast access to data
• Slow access to data results in slow mobile apps, unhappy users and loss of
productivity
• By storing and caching application data locally in the embedded Couchbase Lite
database, you eliminate the need to constantly request data from the server
• You can synchronize data automatically with Sync Gateway

Digitization of Paper Assets
The Problem
• Reducing operational time and costs is critical for any business, and mobile
application deployments are a key component of making it happen
• By digitizing paper assets, you save time and resources, reduce storage, and cut
logistics costs
• Store paper assets with Couchbase Mobile, both locally and in the cloud
• Your data is securely synced between devices and the cloud

Content Delivery & Storage
The Problem
• Guaranteed delivery and storage of assets is crucial for a great user experience
• Users need real-time data updates
• Couchbase Mobile gives you the ability to store and access data locally so content is
always available
• Data can be synced between the cloud and device, allowing business-critical
updates to be reflected in real-time

Real-time Asset Tracking
The Problem
• Real-time asset tracking is a key requirement in mobile applications across many
industries
• Tracking, storing, and delivering real-time updates can be tricky to manage
• Asset tracking data can be shared with supervisors and/or across field employees
• Deliver updates on shipments, routing, service orders, ETAs, or anything else in real-
time with Couchbase Mobile

Peer-to-Peer Usage
The Problem
• Without a network connection, devices can’t talk to each other
• In isolated environments like retail stores, airplanes, the field, or any other places
where the network isn’t reliable, this can cause lost revenue and productivity
• With Couchbase Mobile, connect devices directly to each other with peer-to-peer
replication
• Apps can connect to exchange data and will continue to work regardless of network
availability

User Data/Preferences Storage
The Problem
• Storing user data and preferences across devices and the cloud is complex for
games, CRM systems, e-commerce apps, or any other app
• Making sure user information is secure, both on-device and in transport, is difficult
• Couchbase Mobile gives your users a consistent experience across all of their
devices
• Securely store and sync user data like preferences, profiles, sign-in, game state and
more across devices

Couchbase Server and Intel’s high-performance processors and storage
subsystems combine to power millions of operations/sec for the world’s
largest customer 360 systems and profile stores, popular online games, and
more. Additional information about Couchbase and Intel can be found here:
http://www.couchbase.com/partners/intel
Power with Intel

Consulting and Expert Services
Packaged Services
– Quick Start
– Architecture Reviews
– Health Checks
– Upgrade Services
– Managed Services
Custom Consulting
– Migration
– Application Integration
– Big-Data Integration, Analytics, Reporting
– Others
Email inquiries: Services@couchbase.com OR Contact your Account Representative

Couchbase Hands-OnTraining
For System Admins:
§ CS300 Couchbase NoSQL Server Administration, 4-day
For Developers:
§ CD220 Developing Couchbase NoSQL Applications, 3-day
For Anyone:
§ Free Online On-Demand Courses: http://training.couchbase.com/online
Email inquiries: training@couchbase.com

Additional Resources
148
§ Getting Started: http://www.couchbase.com/download
§ Couchbase Labs: http://github.com/couchbaselabs
§ Developer Portal: http://developer.couchbase.com
§ General Docs: http://docs.couchbase.com
§ Query Portal: http://query.couchbase.com
§ Blog: http://blog.couchbase.com
§ Forum: http://forums.couchbase.com
§ Sample Applications:
§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=try
§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=beer

Couchbase Day

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (15)

Semelhante a Couchbase Day

Semelhante a Couchbase Day (20)

Mais de Idan Tohami

Mais de Idan Tohami (20)

Último

Último (20)

Couchbase Day