n this talk we will address how we developed our Cassandra environments utilizing Cisco UCS Open Stack Platform with the DataStax Enterprise Edition software. In addition we are utilizing OpenSource CEPH storage in our Infrastructure to optimize the Performance and reduce the costs.

1. 1
TOMORROW starts here

2. Cassandra Adoption
on
Cisco UCS & OpenStack
Nayeem Khaja, Program Manager, Cisco IT
Bidhu Das, Database Domain Architect, Cisco IT

3. 3
• About Cisco
• Why Cassandra at Cisco
• Use Cases
• Architecture & Implementation
• Key Callout
• Q & A
Agenda

4. 4
About Cisco
locations in
countries
offices
employees
2000+ Applications
1500+ Databases (Prod & Non-Prod)
HANA, Legacy EDW, Hadoop
Supporting Mission Critical Environments
32 data centers and server rooms
of data center space
of UPS power to raised floors
servers virtualized in new DCs,
overall
Virtualization goal =

5. 5
Cisco Products
http://www.cisco.com/c/en/us/products/index.html

6. 6
Cisco UCS Models

7. 7
Why Cassandra at Cisco
• Active ~ Active i.e. distributed Architecture
• Linear scalability
• High availability with zero downtime
• Better throughput with Multi-DC Architecture
• Align with Industry Cloud Native Application strategy

8. 8
Use Cases
Cloud and Operations!
v Cisco Commerce"
v Cisco Collaboration"
v Quote & Validate system"
v AutoTest scanning for Cisco Services Group"
Mobility!
v Cisco Network
Management"
Security & Fraud
Detection!
v Cisco Security"
v Cisco Finance
Fraud Analytics"
Personalization and
Recommendation!
v Cisco Video"
IoT !
v Cisco Supply Chain"
v Cisco Project Polaris"
v Cisco Webex"
v Cisco Voicemail Player"
Cassandra

9. 9
Cisco Commerce Renewals Cloud
Clients
IaaS StorageCompute Networking
Platform
Cassandra (Database)
Applications
Notifications Pricing Search Quoting
Data Loader
Validation
(drools)
Cisco UCE Browser
App
AndroidPartner app IOS
Upload
Conversion
Nginx
(Web Server)
Platform,build,TestAutomation
(Puppet,Nagios,Jenkins
Ordering Web
Tomcat
(Java appServer)
Elastic Search
(Search Engine, Log Mining)
Rabbit MQ
(Messaging)
HAProxy
(Load Balancer)
Memcached
(In Memory
Cache)
Logstash
(Log Forwarder)
Kibana
(Log Visualizer)
Quartz
(Scheduler)

10. 10
Commerce Analytics & Reporting
Transactional
(Oracle)

11. 11
eStore : Database Provisioning Tool
DB Category
RDBMS
BigData - Hadoop
NoSQL
RDBMS Capability
Open Source
High Transactional
Immediate Consistency
Cost & Support
Vendor Supported – P1 Apps
Community Support < P1 Apps
Maria DB
Multi-Master replication
Large Database Support
In-Memory Capability
Better Security
NoSQL Capability
Open Source
Distributed across Geo-location
Active ~ Active
Multi-Master replication
Columnar Search
Key-Value pair – In-Memory, small read/ small write,
large objects
Document Oriented
Very high Data volume
Graph Database – for complex, highly connected
hierarchical data
Schema less Architecture
BigData Capability
Huge Dataset with structures as well
as unstructured
Highly Scalable
Master – Slave Architecture
Batch Oriented

12. 12
Architecture & Implementation

13. 13
What is OpenStack ?
OpenStack is an open source platform consisting of set of software tools ,
used for building and managing the cloud computing platform for both
public & private clouds.

14. 14
OpenStack Components
libvirt ovs
KVM
Haproxy
(Load balancer)
Swift
(Object Storage)
Glance
(Imaging)
Cinder
(block storage)
Nova
(Compute)
Neutron
(Networking)
Heat
(Orchrastration)
Cellometer
(Telemetry)
Keystone
(Identity)

15. 15
OpenStack on Cisco UCS
Red Hat OpenStack Infrastructure
RHEL RHEL RHEL
……
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
…
…..
RHEL RHEL
Red Hat OpenStack Infrastructure
Ceph Cluster
….
FI -A FI - B
FEX -A FEX -B
Compute Nodes Storage Nodes
APP
OS

16. 16
Cassandra on OpenStack Architecture
Automati
on Packs
Process
Orchestrator
Prime
Service
Catalog
Users c
Openstack
Mangement
Nodes
Network
Fabric
UCS B
Series
UCS C
Series
Storage Cluster
Compute Cluster
Nexus
5000
Nova
Compute
Nodes
RADOS
GW (Swift)
OSD /
MON
OpenStack Infrastructure
Cassandra Cluster Config :
• 8 CPU & 64 GB Memory per Node
• Ceph storage
• OS : RHEL 6.4
• Apache Cassandra 2.1
Replication Factor = 3
Consistency Level = Local Quorum
Replication Strategy = NetworkTopologyStrategy
Node 2
Node
3
Node 4
Node
5Node
6
Node
9
Node 8
Node 7
R
A
C
K
3
R
A
C
K
2
Co-ordinator Node
Node
1
Client Driver

17. 17
Cassandra & Spark on Physical host & SSD
4
1
6
5
2
3
(Transactional
Physical/SSD) 3
1
2
(ETL /Spark
Physical/SSD)
• C220 M4 Servers
• 256 GB Memory each
• 8 SSD Drives 960GB each
• RHEL 6.5 OS 64bit
• JBOD Configuration
• Datastax 4.7

18. 18
Ephemeral storage is a storage solution where the storage is directly
attached to compute (nova) layer i.e. local to the hypervisor. The data is not persistent
on the disk which means the data is lost in case of terminating the VM, but not in case of
VM shutdown.
Hypervisor 1 Hypervisor 2
Hypervisor 3 Hypervisor 4
Ceph
Hypervisor 1 Hypervisor 2
Hypervisor 3 Hypervisor 4
Hypervisor with local
Storage (dedicated
per hypervisor)
Hypervisor with Ceph
Storage (shared)

19. 19
Redhat
Openstack
Redhat
Openstack
Ephemeral solution for Cassandra
C-Series
Host
C-Series
Host
C-Series
Host
Redhat
Openstack
Cass
andra
OS
4
35
6
1
2
Dedicated Cluster at DB level for each applications, but Multi-
Tenant at OpenStack level with common storage pool per
Hypervisor.
Cluster 1
4
35
6
1
2
Cluster 2
Nova Nova Nova
This configuration provides consistent/ better I/O
throughput as compared to OpenStack w/Ceph
(shared) storage.
…
…
…
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS
Cass
andra
OS….
Storage (Locally
attached or External
(connected thru FC))
Storage (Locally
attached or External
(connected thru FC))
Storage (Locally
attached or External
(connected thru FC))

20. 20
Platform Migration & Upgrade with zero downtime
VTL/Data
Domain
Snapshot
Snapshot
8 days
retention 2 weeks
retention
.
.
.
.
.
.
Snapshot
Snapshot
Snapshot
30 days
retention
2 weeks
retention
.
.
.
.
.
.
.
.
.
Backup
Type
Frequency Retention Comments
Cluster VTL
Metadata Daily 2 weeks 30 days
Full Daily 2 weeks 30 days
Incremental 6 hrs 1 week 30 days Only in case of high
critical applications
as an exception.
5
7
1
6
2
3
9
4
8
5
1
9
2
3
4DC2- PROD
Physical/SSD
7
8
6
DC1- PROD
Openstack
Shared Ceph Storage

21. 21
Monitoring
Cassandra :
• Compaction status
• Nodetool tpstats for pending request or dropped mutations
• Nodetool cfstats/ cfhistograms for latency distribution
• Recent restarts, Dead Node
• Node Health check wrt Gossip, Thrift & Native transport
• Event driven / metrics driven Alerts
System :
• CPU , Memory, IO status
• Load average

22. 22
Centralized Operations Dashboard
123.45.678.001
123.45.678.002
123.45.678.003
123.45.678.004
123.45.678.005
123.45.678.006
123.45.678.007
123.45.678.001
123.45.678.002
123.45.678.003
123.45.678.004
123.45.678.005
123.45.678.006
123.45.678.007
123.45.678.008
123.45.678.009
CCW Cassandra
cass-prd-08:8900
cass-prd-09:8900
cass-prd-08.cisco.com
cass-prd-09.cisco.com
8
9
123.45.678.008
123.45.678.009
P3
P3
P3
P3
P3
P3
P3
P3
P3
P2
P2
P2
P2
P2
P2
P2
P2
P2
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
CCW Cassandra
1
2
3
4
5
6
7
8
9

23. 23
Integrated Real time Database Dashboard
xyz-prd-03.cisco.com
xyz-prd-03.cisco.com
xyz-prd-03
xyz-prd-03
abc-prd-01
cass-prd-07
cass-prd-07
123.45.678.007
cass-prd-07.cisco.com
abc-prd-01.cisco.com
XYZPRD.CISCO.COM_XYZPRD1
LISTENER_XYZPRD1_xyz-prd-03
ABCPRD

24. 24
Monitoring using DataStax Opscenter

25. 25
• Do not use Nodetool command with more concurrency.
• Enable Incremental backup, only if required.
• Snapshot retention policy.
• Use Nodetool repair thru OpsCenter
• Compaction Strategy (Size tiered ~ Level Tiered)
• Disable replication at storage layer
Lesson Learn & Key Call out

26. 26

27. 27
Thank you