Managing the Basho Data Platform with the Cloudsoft UX, including Riak blueprints in Apache Brooklyn and building up to tiered dynamic IoT analytics management

1. Managing the Basho Data Platform with the Cloudsoft UX
RICON Nov 2015
Alex Heneveld @ahtweetin
Mike Zaccardo @ItsMeMikeZ
Cuyler Jones (Basho)
@cuylerjones

2. @ahtweetin
Recap from 2014
Riak Blueprints in Apache Brooklyn
The easiest way to deploy and manage Riak anywhere!
Blueprints for Riak nodes and clusters
available off-the-shelf in Apache Brooklyn
Node and cluster metrics & auto-scaling policies
Supporting AWS, Azure, Openstack, GCE, Softlayer,
Docker, VMware, Virtustream, Bluebox, and more

3. @ahtweetin
What is a Brooklyn Blueprint?
location: aws-ec2
services:
- type: riak-cluster
initialSize: 5

4. @ahtweetin
What is a Brooklyn Blueprint?
location:
byon:
hosts: [ 10.11.12.{101-105} ]
services:
- type: riak-cluster
initialSize: 5

5. @ahtweetin
What is a Brooklyn Blueprint?
location: openstack-nova:http://10.9.8.7:5357/
services:
- type: riak-cluster
initialSize: 5
provisioning.properties:
minRam: 16gb
osFamily: centos
brooklyn.policies:
- type: autoscaler
brooklyn.config:
metric: $brooklyn:sensor("riak.node.ops.1m.perNode")
metricLowerBound: 60000
metricUpperBound: 90000

6. @ahtweetin
What is a Brooklyn Blueprint?
location: softlayer
services:
- type: riak-cluster
initialSize: 5
- type: spark
- type: redis

7. @ahtweetin
What is a Brooklyn Blueprint?
location: softlayer
services:
- type: riak-cluster
initialSize: 5
- type: spark
- type: redis
- type: bash-server
launch.command: |
curl http://... | bash

8. @ahtweetin
What is a Brooklyn Blueprint?
location: softlayer
services:
- type: riak-cluster
initialSize: 5
brooklyn.config:
network.list: [ riak ]
- type: load-balanced-web-cluster
memberSpec:
type: tomcat
brooklyn.config:
network.list: [ webapp, riak ]
controllerSpec:
type: haproxy
brooklyn.config:
network.list: [ public, webapp ]

9. @ahtweetin

10. @ahtweetin

11. @ahtweetin
Run$me'
Loca$onsLoca%on'
OSS'Blueprint'Catalog'
Bridge'
PAYG%Subscrip.on%
REST'API'
Debugger'
Designer'
GUI'
OpenShi>'
Brooklyn'
Bridge'
Mesos'
Brooklyn'
Bridge'
Custom'
Custom'
Tools'
COTS'Blueprint'Catalog'
Extensions'
Custom'
Tools'
Custom'
Extensions'
OASIS'CAMP' OASIS'TOSCA'
24x7%Produc.on%
Support%
VPC'

12. @ahtweetin
New for 2015 (and 2016)
Bring your own data tools blueprints
Blueprints for the Basho Data Platform
Consistent across many infrastructures,
now including Docker and Mesos
Riak Enterprise support in Cloudsoft AMP

13. @ahtweetin
The Basho Data Platform
BDP: Combines Riak with other analytics tools to give a simpler solution to a
much larger class of analysis challenges, including Riak, Cache-Proxy, and Spark
Coming soon: Riak TS and range scan queries

14. @ahtweetin
The Basho Data Platform
Redis and Cache-Proxy: Enables natural use of Redis as a read cache for Riak KV
Coming soon: Write-through support for SET, DEL, and PEXPIRE to invalidate cache
BDP: Combines Riak with other analytics tools to give a simpler solution to a
much larger class of analysis challenges, including Riak, Cache-Proxy, and Spark
Coming soon: Riak TS and range scan queries

15. @ahtweetin
The Basho Data Platform
Redis and Cache-Proxy: Enables natural use of Redis as a read cache for Riak KV
Coming soon: Write-through support for SET, DEL, and PEXPIRE to invalidate cache
Spark: Streaming optimization for Riak KV, with 1000 key-object pairs per request.
BDP Leader Election Service allowing HA without ZK, simplifying cluster operation.
BDP: Combines Riak with other analytics tools to give a simpler solution to a
much larger class of analysis challenges, including Riak, Cache-Proxy, and Spark
Coming soon: Riak TS and range scan queries

16. @ahtweetin
The Basho Data Platform
Riak
RiakRiakRiak
RedisRedis
Basho Redis Proxy
Spark Connector
Riak
SparkSpark
Client
Application
Read
Write
Read
Riak Ensemble
Leader Election
Basho Data Platform
Service
Manager
Read
Write
Read
Query
Spark Services
Redis Services
Riak Client
Write
Read / Solr
Spark Client

17. @ahtweetin
The Basho Data Platform
BDP typically runs with a secondary Riak cluster, alongside a primary Riak cluster.
After installing BDP on nodes, start services by running:
data-platform-admin start-service »node« »group« »service«

18. @ahtweetin
The Basho Data Platform
BDP typically runs with a secondary Riak cluster, alongside a primary Riak cluster.
After installing BDP on nodes, start services by running:
data-platform-admin start-service »node« »group« »service«
Considerations for Spark
Spark workers should be run on different nodes from Riak to avoid interference.
Start with a one-to-one correspondence between Riak nodes and Spark nodes.
Follow general Spark provisioning guidelines for Disk, Memory, CPU, and Network.
Considerations for Redis and Cache-Proxy
Redis should be run on different nodes from Riak and Spark to avoid interference.
Cache Proxy and Redis should be run together to create a mesh of cache servers.
Set CACHE_TTL to meet the Eventual Consistency SLA of the application.
Monitor usage/allocation via Redis INFO (used_memory and maxmemory).

19. @ahtweetin
The BDP and Cloudsoft AMP
Riak
RiakRiakRiak
RedisRedis
Basho Redis Proxy
Spark Connector
Riak
SparkSpark
Client
Application
Read
Write
Read
Riak Ensemble
Leader Election
Basho Data Platform
Service
Manager
Read
Write
Read
Query
Spark Services
Redis Services
Riak Client
Write
Read / Solr
Spark Client

20. @ahtweetin
The BDP and Cloudsoft AMP
Riak
RiakRiakRiak
RedisRedis
Basho Redis Proxy
Spark Connector
Riak
SparkSpark
Client
Application
Read
Write
Read
Riak Ensemble
Leader Election
Basho Data Platform
Service
Manager
Read
Write
Read
Query
Spark Services
Redis Services
Riak Client
Write
Read / Solr
Spark Client
Cloudsoft AMP
Deploy
Scale

21. @ahtweetin
The BDP and Cloudsoft AMP
Riak
RiakRiakRiak
RedisRedis
Basho Redis Proxy
Spark Connector
Riak
SparkSpark
Client
Application
Read
Write
Read
Riak Ensemble
Leader Election
Basho Data Platform
Service
Manager
Read
Write
Read
Query
Spark Services
Redis Services
Riak Client
Write
Read / Solr
Spark Client
Cloudsoft AMP
Deploy
Scale
Riak
MoreEven

22. @ahtweetin

23. @ahtweetin
The BDP and Cloudsoft AMP
Riak
RiakRiakRiak
RedisRedis
Basho Redis Proxy
Spark Connector
Riak
SparkSpark
Client
Application
Read
Write
Read
Riak Ensemble
Leader Election
Basho Data Platform
Service
Manager
Read
Write
Read
Query
Spark Services
Redis Services
Riak Client
Write
Read / Solr
Spark Client
Cloudsoft AMP
Deploy
Scale
Riak
MoreEven

24. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis

25. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis

26. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute

27. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App

28. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App

29. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute

30. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute

31. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute

32. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute

33. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute

34. @ahtweetin
Riak
Sensors
Sensors
Sensors
Analysis
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute Edge Compute
Datacenter 1
BDP
Riak
Consumer App
Consumer App
DC3
BDP
Riak Web App
Spark Consumer
DC2
BDP
Riak Riak Consumer
Web App
Consumer App
Consumer App
Riak
Sensors
Sensors
Sensors
Analysis
Regional Hub
Analysis
BDP
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute
Riak
Sensors
Sensors
Sensors
Analysis
Edge Compute

35. @ahtweetin
The BDP and Cloudsoft AMP
One-click deployment of BDP, following best practices
Scale each sub-system, manually or automatically
Consistency across bare metal, many clouds, containers, and more
Handle replication, migration and DR
Blueprints extensible for the systems you need today
and the flexibility and agility you will need in the future