O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
What’s New in Couchbase
Top 10 Capabilities in 3.0
Dipti Borkar
Sr. Director |WW Solution Engineering
Couchbase Server Overview
©2014 Couchbase Inc.
Couchbase provides a complete Data Management solution
3
High availability
cache
Key-value
store
Docu...
©2014 Couchbase Inc.
What makes Couchbase unique?
4
Performance &
scalability leader
Sub millisecond latency
with high thr...
©2014 Couchbase Inc.
Journey so far…
©2014 Couchbase Inc.
DEVELOPER
EMPOWERMENT
MISSION CRITICAL
SCALE
ULTRA HIGH
AVAILABILITY
SECURE
COMMUNICATION
SIMPLIFIED
...
#0
The Foundation
©2014 Couchbase Inc.
 New DCP (Database Change Protocol) protocol for cluster-wide
communication
 Eliminate Disk-IO from...
©2014 Couchbase Inc.
33 2
XDCR after Write
9
Managed Cache
DiskQueue
Disk
Replication
Queue
App Server
Couchbase Server No...
©2014 Couchbase Inc.
33 2
View processing after write
10
Managed Cache
DiskQueue
Disk
Replication
Queue
App Server
Couchba...
#1
Tunable memory
©2014 Couchbase Inc.
 Optimized Memory Usage with Metadata Ejection Policy
 Better optimization of memory for massive da...
©2014 Couchbase Inc.
 Each hash table item consists of {key, metadata, value}
 Metadata memory overhead is 40 bytes at l...
©2014 Couchbase Inc.
 Application’s entire key space is maintained in the hash table
 Highly cache-oriented architecture...
©2014 Couchbase Inc.
 Application’s entire key space doesn’t need to be loaded in cache
 Reduce the memory overhead sign...
©2014 Couchbase Inc.
Tunable Memory - Optimization for Massive Databases
100sx Reduction in Metadata Memory Consumption fr...
#2
Recovery
©2014 Couchbase Inc.
 Delta Node Recovery
 Shorter time to add nodes back into the cluster!
 Bring back a server increm...
#3
Persistence and I/O
©2014 Couchbase Inc.
 Efficient Auto-Tuning Engine
 Detect and allocate threads based on HW resources
 Pool threads for...
©2014 Couchbase Inc.
Inside data manager
©2014 Couchbase, Inc. 21
Append-only B-Tree Storage Engine
Engine APIs
(get, set,...
©2014 Couchbase Inc.
 Efficient warm-up
 Parallelization on cache warm-up
 Further warm-up time improvements under meta...
©2014 Couchbase Inc.
 Priority Setting on Buckets
 Ability to assign high and low priority to buckets
 High priority bu...
#4
Replication
©2014 Couchbase Inc.
 Faster Durability with Replication
 Improved latency on ReplicateTo
 DCP Improves replication spe...
©2014 Couchbase Inc.
 Improved Protections forYour Data
 Drastically Lower XDCR Latency
 In memory change detection wit...
#5
Indexing data
©2014 Couchbase Inc.
 More up-to-date views
 Mutation show up at the indexes
faster with the new DCP protocol
 No longe...
#6
Backup
©2014 Couchbase Inc.
 Minimize time and resources during backups
Efficient Recovery with Incremental Backup & Restore
• B...
#7
XDCR
©2014 Couchbase Inc.
 XDCR Pause and Resume
 Want to suspend replication? No longer need to drop replications
 Ability ...
#8
Connection
management
©2014 Couchbase Inc.
Connection Management
Optimized Connection Management
 Cluster map / topology information transmitte...
©2014 Couchbase Inc.
Connection Management
Increased connection limits
Data port : 11210
 max_conns_on_port_11210: 30000
...
#9
Maintenance
©2014 Couchbase Inc.
 New option to gracefully failover a node.
 Failover with a graceful handoff
 May take longer to f...
#10
SDKs
©2014 Couchbase Inc.
 Componentized libraries to simplify use cases
• Couchbase JVM Core - Reusable Foundation
• Handles ...
©2014 Couchbase Inc.
Integrated native JSON data type programmability
• Simplified Development
• Native JSON handling bui...
©2014 Couchbase Inc.
 Node.js, PHP
• Native JSON handling built-in
• Core IO enhancements through updated C libcouchbase
...
What’s ahead
Couchbase Roadmap
2015 – Couchbase Server 4.0
POWERFUL QUERY
PERFORMANCE &
SCALE
HIGH
AVAILABILITY
SECURITY
COMPLIANCE
DEV...
©2014 Couchbase Inc.
Introduction to N1QL – SQL for Documents
 Next generation, NoSQL query language
 SQL-like : SELECT ...
©2014 Couchbase Inc.
N1QL Architecture
 Single node installation, services
defined dynamically
 Query service access Ind...
Q & A
Thank you.
dipti@couchbase.com
@dborkar
Próximos SlideShares
Carregando em…5
×

Couchbase Live Europe 2015: What’s New in Couchbase: Top 10 Capabilities that Eliminate Barriers to Availability, Performance & Scale

541 visualizações

Publicada em

This presentation covers new capabilities in Couchbase Server 3.0. Eliminate barriers to performance, scale and availability.

Presented by: Dipti Borkar, Couchbase

Publicada em: Software
  • Seja o primeiro a comentar

Couchbase Live Europe 2015: What’s New in Couchbase: Top 10 Capabilities that Eliminate Barriers to Availability, Performance & Scale

  1. 1. What’s New in Couchbase Top 10 Capabilities in 3.0 Dipti Borkar Sr. Director |WW Solution Engineering
  2. 2. Couchbase Server Overview
  3. 3. ©2014 Couchbase Inc. Couchbase provides a complete Data Management solution 3 High availability cache Key-value store Document database Embedded database Sync management Multi-purpose capabilities support a broad range of apps and use cases Enterprises often start with cache, then broaden usage to other apps and use cases
  4. 4. ©2014 Couchbase Inc. What makes Couchbase unique? 4 Performance & scalability leader Sub millisecond latency with high throughput; memory-centric architecture Multi- purpose Simplified administration Easy to deploy & manage; integrated Admin Console, single- click cluster expansion & rebalance Cache, key value store, document database, and local/mobile database in single platform Always-on availability Data replication across nodes, clusters, and data centers Enterprises choose Couchbase for several key advantages 24x365
  5. 5. ©2014 Couchbase Inc. Journey so far…
  6. 6. ©2014 Couchbase Inc. DEVELOPER EMPOWERMENT MISSION CRITICAL SCALE ULTRA HIGH AVAILABILITY SECURE COMMUNICATION SIMPLIFIED ADMINISTRATION • Fully Integrated JSON Programmability • Native Async and Reactive Programmability • New Integration with Popular Frameworks • N1QL Preview Support in SDKs • Optimized Massive Databases with Tunable Memory • Auto-Tuning Thread Pool for Extreme IO Throughput • Faster Replication with DCP • Improved Durability • Faster view building • Low Latency, In- Memory XDCR • Delta Node Recovery • Faster Rebalance • Faster Node Restart with Efficient Warm- up • New Incremental Backup option • Encryption for Data Access • Encryption for Administrative Access • XDCR Pause and Resume • Improved Resource Governance • Graceful Failover • Cluster-wide Diagnostics What’s new in Couchbase Server v3.0
  7. 7. #0 The Foundation
  8. 8. ©2014 Couchbase Inc.  New DCP (Database Change Protocol) protocol for cluster-wide communication  Eliminate Disk-IO from the replication access path  High-Performance, Stream-based, In-memory  Ordered based on global Sequence Numbers per partition  Snapshot-able  Restart-able • Powers Node to Node Replication • PowersView • Powers XDCR Database Change Protocol This is non-trivial stuff!!!
  9. 9. ©2014 Couchbase Inc. 33 2 XDCR after Write 9 Managed Cache DiskQueue Disk Replication Queue App Server Couchbase Server Node Doc 1 Doc 1 XDCR Queue Doc 1Doc 1 (New in 3.0) Memory- to-Memory Replication to remote cluster Memory-to-Memory Replication to other node
  10. 10. ©2014 Couchbase Inc. 33 2 View processing after write 10 Managed Cache DiskQueue Disk Replication Queue App Server Couchbase Server Node Doc 1 Doc 1 To other node View engine Doc 1Doc 1
  11. 11. #1 Tunable memory
  12. 12. ©2014 Couchbase Inc.  Optimized Memory Usage with Metadata Ejection Policy  Better optimization of memory for massive databases  Enable efficient management of rarely accessed data set  Cache only keys and data for the working set & eject all historic data Tunable Memory - Optimization for Massive Databases
  13. 13. ©2014 Couchbase Inc.  Each hash table item consists of {key, metadata, value}  Metadata memory overhead is 40 bytes at least A Key in the HashTable ©2014 Couchbase, Inc. 13 Key Metadata Blob pointer Blob value Expiration time CAS identifier Sequence number (DCP) Revision number (XDCR) Lock expiry (GetLocked API) Flag, NRU, … Hash Table Item Pointer to next item
  14. 14. ©2014 Couchbase Inc.  Application’s entire key space is maintained in the hash table  Highly cache-oriented architecture  Item pager ejects only an item’s value from the hash table Value-Only Ejection ©2014 Couchbase, Inc. 14 Key: “foo” Metadata Blob pointer Blob value Pointer to next item Hash Table Item Storage Engine Batch Reader Get(“foo”) read_value(“foo”)
  15. 15. ©2014 Couchbase Inc.  Application’s entire key space doesn’t need to be loaded in cache  Reduce the memory overhead significantly in heavy DGM (Disk Greater than Memory) cases  Item pager ejects an item’s key and metadata along with its value Full Metadata Ejection ©2014 Couchbase, Inc. 15 Key: “foo” Metadata Blob pointer Blob value Pointer to next item Hash Table Item Storage Engine Batch Reader Get(“foo”) read_meta_value(“foo”)
  16. 16. ©2014 Couchbase Inc. Tunable Memory - Optimization for Massive Databases 100sx Reduction in Metadata Memory Consumption from 2.5 to 3.0 Note: The graph represents characteristics under data mutations. ~50M docs with value size ~0.5KB v.3.0 – Large DB with HotWorking-set 3 GB Consumed for Metadata in RAM 2.5.1 or earlier – Real-time Latency 80 MB Consumed for Metadata in RAM
  17. 17. #2 Recovery
  18. 18. ©2014 Couchbase Inc.  Delta Node Recovery  Shorter time to add nodes back into the cluster!  Bring back a server incrementally into the cluster without having to rebuild the full node. Incremental Node Catch-up with Delta Node Recovery 100sx Reduction inTime to Re-Add Node from 2.5 to 3.0 Note: The absolute performance improvement depend on data size and mutation count that needs to be caught up
  19. 19. #3 Persistence and I/O
  20. 20. ©2014 Couchbase Inc.  Efficient Auto-Tuning Engine  Detect and allocate threads based on HW resources  Pool threads for best resource utilization across buckets  Improved latency across the board: Faster Reads Faster Rebalance Faster Node reactivation Faster Durability withWrites & PersistTo (more details on this on later) …and many more AutoTuningThread Pool
  21. 21. ©2014 Couchbase Inc. Inside data manager ©2014 Couchbase, Inc. 21 Append-only B-Tree Storage Engine Engine APIs (get, set, del, add, append, DCP, …) Partition Hash Table (active) Partition Hash Table (replica) Partition Hash Table (active) … Checkpoints Checkpoints Checkpoints Reader Threads Non-IO Threads Data Replicator I/O Completion Notifier Aux-IO Threads Flushers Data Backfill User Configured Replica Count = 1 Batch Readers Writer Threads … Item Pager Expiry Pager Checkpoint Manager Shared Thread Pool
  22. 22. ©2014 Couchbase Inc.  Efficient warm-up  Parallelization on cache warm-up  Further warm-up time improvements under metadata ejection  Up to 60% Faster Warm-up between 3.0 and 2.5.1  Up to 8x Faster Warm-up with Full Ejection in 3.0  *Note: Absolute latency depend on disk IO latency and bandwidth Faster Node Re-Activation
  23. 23. ©2014 Couchbase Inc.  Priority Setting on Buckets  Ability to assign high and low priority to buckets  High priority buckets get higher IO priority and more throughput  Low priority buckets can still be processed without starvation Improved Resource Governance with Bucket Priority
  24. 24. #4 Replication
  25. 25. ©2014 Couchbase Inc.  Faster Durability with Replication  Improved latency on ReplicateTo  DCP Improves replication speed and durability  Better protects your data! Up to 150x Improvement on ReplicateTo latency from 2.5 to 3.0 *Note: Absolute latency depend on network bandwidth and latency  *95th percentile ReplicateTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node Faster Replication within the cluster & across clusters 0 200 400 600 Time(ms) 95 Percentile ReplicateTo=1 Latency* 2.5.1 3.0
  26. 26. ©2014 Couchbase Inc.  Improved Protections forYour Data  Drastically Lower XDCR Latency  In memory change detection with DCP No longer require persisting to disk Up to 4x better on XDCR latency between clusters between 3.0 & 2.5.1 Lower Latency across clusters 0 100 200 300 400 500 600 Time(ms) 90th Percentile. XDCR Latency 2.5.1 3.0 90th percentile replication lag (ms), 5 -> 5 UniDir, 2 buckets x 500M x 1KB, 10K SETs/sec, LAN
  27. 27. #5 Indexing data
  28. 28. ©2014 Couchbase Inc.  More up-to-date views  Mutation show up at the indexes faster with the new DCP protocol  No longer require persisting to disk for change detection  Improved consistency with views (stale=false behavior)  Ports to C in view engine Very fast index building 0 10000 20000 30000 40000 Time(ms) Indexing Under Load 2.5.1 3.0 Up to 50x Faster Indexing from 2.5 to 3.0 *Note: Absolute latency depend on disk IO speed *95th percentile Indexing latency (ms), 1 bucket x 20M x 2KB, non-DGM, 1 view, 250 mutations/sec/node, 400 queries/sec
  29. 29. #6 Backup
  30. 30. ©2014 Couchbase Inc.  Minimize time and resources during backups Efficient Recovery with Incremental Backup & Restore • Back up only the data updated since the last backup • Differential Backups • Cumulative Backups
  31. 31. #7 XDCR
  32. 32. ©2014 Couchbase Inc.  XDCR Pause and Resume  Want to suspend replication? No longer need to drop replications  Ability to pause XDCR traffic anytime  Resume and incrementally catch-up from where you left off Efficient Maintenance with XDCR
  33. 33. #8 Connection management
  34. 34. ©2014 Couchbase Inc. Connection Management Optimized Connection Management  Cluster map / topology information transmitted via 11210 port rather than 8091  Much more scalable and better performance during topology changes 34
  35. 35. ©2014 Couchbase Inc. Connection Management Increased connection limits Data port : 11210  max_conns_on_port_11210: 30000 Internal Cluster communication : 11209  max_conns_on_port_11209: 5000 35
  36. 36. #9 Maintenance
  37. 37. ©2014 Couchbase Inc.  New option to gracefully failover a node.  Failover with a graceful handoff  May take longer to failover but prevents potential for any data loss Node Maintenance
  38. 38. #10 SDKs
  39. 39. ©2014 Couchbase Inc.  Componentized libraries to simplify use cases • Couchbase JVM Core - Reusable Foundation • Handles core IO, topology changes. Reactive and asynchronous. • Java Client Library • Native JSON handling built-in • Provides synchronous and asynchronous interface • Preview of N1QL language integrated query What’s New in SDK 2.0
  40. 40. ©2014 Couchbase Inc. Integrated native JSON data type programmability • Simplified Development • Native JSON handling built-in • Preview of N1QL language integrated query • Performance & Scale • Improved connection management • Native .Net: • Simplified Configuration Management with Web.config & App.config What’s New in SDK 2.0
  41. 41. ©2014 Couchbase Inc.  Node.js, PHP • Native JSON handling built-in • Core IO enhancements through updated C libcouchbase • Better connection management  Python, Ruby • Core IO enhancements through updated C libcouchbase • Better connection management What’s New in SDK 2.0
  42. 42. What’s ahead
  43. 43. Couchbase Roadmap 2015 – Couchbase Server 4.0 POWERFUL QUERY PERFORMANCE & SCALE HIGH AVAILABILITY SECURITY COMPLIANCE DEVELOPER EMPOWERMENT • N1QL: SQL for Documents • Secondary indexing for interactive queries • Improved Spatial Queries • Multi Dimensional Scaling • Scalability Enhancements for Massive Database Deployments • Improve Performance for Full Eviction – Bloom Filters • Filtering Replication for Efficient Geo Data Distribution with XDCR • Auditing Framework and Admin Auditing • External Account Management System Integration for the Enterprise (LDAP) • Enhanced Consistency Options • Legacy Data interfaces ODBC/JDBC
  44. 44. ©2014 Couchbase Inc. Introduction to N1QL – SQL for Documents  Next generation, NoSQL query language  SQL-like : SELECT * FROM WHERE/LIKE/JOIN/GROUP/etc, CREATE INDEX  Extended for JSON to support nested and hierarchical data structures  Support for views and newly-developed secondary indexes  Query (DQL), Manipulation (DML), Description (DDL)  ODBC/JDBC drivers in development  Built into Couchbase Server:  Single installation package  Multi-threaded, stateless query and indexing components  Leverages high-performance, high-scale Couchbase buckets  Coming in 2015, preview at query.couchbase.com
  45. 45. ©2014 Couchbase Inc. N1QL Architecture  Single node installation, services defined dynamically  Query service access Index and Data to formulate response  All queries and direct access is topology aware and dynamically scalable
  46. 46. Q & A Thank you. dipti@couchbase.com @dborkar

×