Rakuten Technology Conference 2014 "At Rakuten, The Rakuten OpenStack Platform and Beyond" Kentaro Sasaki, Koichi Yoshigoe (Rakuten)

1. At Rakuten, the Rakuten OpenStack
Platform and Beyond
Oct/25/2014
Kentaro Sasaki, Koichi Yoshigoe
Global Operations Department, Rakuten, Inc.

2. 2
Agenda
• OpenStack
• Design
• Customize
• Impressions
• Roadmap
• OSS Activity
http://goo.gl/3Qitfy

3. 3
Presenter
Kentaro Sasaki
Storage Architect since 2010
Research fields:
Condensed Matter Physics
Skills and Experience:
Python, C, Golang, OpenStack,
NetApp, ZFS, EMC, Object Storage

4. 4
Presenter
Koichi Yoshigoe
Cloud Architect since 2012
Research fields:
High Energy Astrophysics
Skills and Experience:
Python, C, Fortran,
OpenStack, VMware, Xen,
Open Compute

5. 5
OpenStack
100% OSS cloud management software:
OpenStack project began in 2010.
NASA Nebula -> Nova
Rackspace Cloud Files -> Swift
10 core components in Icehouse(2014.1)

6. 6
OpenStack Project
19303"
people
144"
countries
Community

7. 7
Why we choose OpenStack?
REST API
Abstract Infrastructure
Generalize / OSS
API
OSS
Internal
Agility
Extensibility
Tech Driven

8. 8
Design
Complicated, not sure yet, no best practice
Introduce current system architecture

9. 9
Middleware for OpenStack
OpenStack Components:
Keystone (Icehouse)
Nova (Icehouse)
Cinder (Icehouse)
Glance (Icehouse)
Neutron (Icehouse)
Horizon (Icehouse)
http://goo.gl/FxLCYi

10. 10
System Design Overview
All nodes are on VMware.
1DB / 1server
Auth = LDAP
Glance = ZFS
LB = A10
Min = 20servers
A10 LB
keystone
API nodes
nova/cinder/neutron
horizon
API nodes
glance
ZFS
A10 LB
nova misc. nodes
scheduler, etc
nova compute node
cinder volume
A10 LB
A10 LB
LDAP
region1
RabbitMQVMware
neutron-vcenter
Mamcached
each components
MariaDB
keystone
MariaDB
cinder misc. nodes
scheduler
Complicated !!

11. 11
Design Key Factor
Pros:
High SLA
A few compute
Cons:
Complex HW
Far from defect
High Cost!!
A10 Load Balancer
OpenStack
RabbitMQ
MariaDB 10
Ubuntu 14.04 LTS
vSphere 5.5
3PAR, Pure Storage
Splunk

12. 12
Region and Availability Zone
AZ: development
IOPS: 500
AZ: staging
IOPS: 500
region1
region2
region3
common
keystone
AZ: production
IOPS: 500, 4000
AZ: sandbox
Disposable environment
IOPS: 100
horizon
Flavor Spec Catalog: standard or high (defined by IOPS)
Region limitation: 5000VMs / 1Region

13. 13
IaaS in numbers
Pros:
Much less power
Cons:
High overcommit rate
AZ
VMs
HVs
Sandbox
438
2
Development
2039
44
Staging
3000
93
Production
6909
371
CentOS: 80%, Ubuntu(used in paas): 20%
Tenant: 600, User = 2000

14. 14
Redundancy Concept Overview
Concept Overview:
1Process / 1server except for API nodes
Use Load Balancer for API nodes
Use VMware HA for active / standby

15. 15
Redundancy Concept
Scale out with Load Balancer:
nova-api
cinder-api
neutron-server
glance-api/registry
keystone
horizon
API node
Load Balancer
API node
API node

16. 16
Redundancy Concept
Scale out with servers:
nova-conductor
nova-scheduler
cinder-scheduler
nova-conductor
nova-conductor
nova-conductor
nova-scheduler
nova-scheduler
nova-scheduler
cinder-conductor
cinder-conductor
cinder-conductor
1process / 1server, No Load Balancer necessary

17. 17
Redundancy stack of API nodes
A10 Load Balancer
nova
cinder
glance
VMware HA
keystone
neutron
horizon
Active / Active: A10 Load Balancer
Active / Standby: VMware HA
Active / Active
Active / Standby

18. 18
Deployment Automation flow
Local PC
Chef cookbook
CI environment
Chef server
Chef cookbook
git push
hook
deploy
OpenStack nodes
MariaDB
RabbitMQ
http://goo.gl/MGIUwA
http://goo.gl/hRogWM
SERVERSPEC
http://goo.gl/PPEj46
http://goo.gl/WIFg8A

19. 19
Customize
Basically community based, but… still have
difficulty in default, develop by ourselves

20. 20
Customizing
Customizing is scared…, but required!!:
Difficult to upgrade / apply patch / maintain
Rakuten specific configuration / General Bugs
nova vmware driver
keystone ldap driver
neutron-vcenter-agent
cinder driver
glance_store driver
neutron ml2 vds driver

21. 21
neutron-vcenter-agent
Official neutron vDS:
Single port group
Rakuten neutron vDS:
Multiple port group
ESXi
port group
ESXi
ESXi
vSphere Distributed Switch
port group
ESXi
port group
port group
ESXi
ESXi
vSphere Distributed Switch

22. 22
Config Drive and cloud-init
start creation
boot instance
mount config drive
extend root LVM
Use per-instance
module in cloud-init
set static IP
Register DNS record
Mount /dev/disk/by-label/config-2
Call in-house DNS API
finish creation

23. 23
Impressions
Basically community based, but… still have
difficulty in default, develop by ourselves

24. 24
What a difficult…!!!
Complicated:
• So many components
No best practice:
• Different from one by one

25. 25
What a difficult…!!!
Bugs & Bugs:
• Read source code!!
Big changing on culture:
• Most difficult thing is mind-shift

26. 26
What a fun…!!!
Full Stack Skills:
• Read / Write source codes
• Broader technology range
Be a pioneer:
• Eternal Beta
• Cutting Edge
Let’s Begin!!

27. 27
What a fun…!!!
Contribution to World Wide Project:
• Everyone will be happy!!
Community activities:
• Knowledge sharing
• Regular workshop
Let’s Join!!

28. 28
Roadmap

29. 29
Roadmap #1
LBaaS / DNSaaS: Introduce “Official” XaaS project
Current
In-house LBaaS
Neutron LBaaS
Future
In-house DNSaaSDesignate

30. 30
Roadmap #2
Expand to the other Regions: [soon] 4th Region
Introduce new Network /
Storage / Compute technology
Current
region1: sdbx
region2: dev/stg
Future
region4: prod2
region3: prod1

31. 31
Roadmap #3
Online Migration of existing VMs:
Migration method had been established!!
Run only
DB INSERT
at nova boot
UPDATE
nova’s DB
to be active
SELECT VM’s
instance_id
from vCenter

32. 32
Roadmap #4
Hybrid Cloud:
Abstract Private / Public with paas and Terraform
Private Cloud
Cloud Foundry v2 / Terraform
Public Cloud
OpenStack
Amazon Web ServiceMicrosoft Azure

33. 33
Roadmap #5
Auto Scaling:
• Validate Heat component
• Hook by monitoring
• Scheduled-based
• Event-based
VM
VM
VM
VM
VM
VM
VM
VM
VM

34. 34
Challenges at OpenStack activity
KVM / Multi-Hypervisor
Other Components
OpenStack x LeoFS
Region / AZ
OpenStack
VMwareKVM
http://goo.gl/Ot8Ef4http://goo.gl/VUqPnb

35. 35
OSS Activity

36. 36
Open Source Strategy at Rakuten OPS
“Take advantage of global developers and open source”
http://goo.gl/a3S7Sp
• Follow open source mainstream
• Don’t fork unless required
• Build plugin when needed
• Contribute back

37. 37
Upstream training
Training for contributing newbie:
• Faster integration
• Contribute real world patch
• Master the technical tools
• Understand workflow
Educating Contributors
Makigaki @Atlanta, 2014 Summer

38. 38
Contribution at Juno (2014.2) Release
24061
20503
16851
12563
8749
3900 3474 2718
5
Contribution by companies
Mirantis
HP
Red Hat
Rackspace
IBM
Cisco
Independent
VMware
Rakuten
http://goo.gl/N3c5pT (at Sep 4 2014)
Rakuten started commitment from last quarter.

39. 39
GitHub
Our source code in GitHub!!:
• Our Chef recipe
• Bootstrap script
• GPLv3
• By the winter of 2014
http://goo.gl/DRxKxT

40. 40
Wrap Up
We started OpenStack!:
• Contributed back
• OSS activity is a lot of fun!
• Our knowledge will be shared in public

41. 41
Thank you