Waldemar Quevedo of Rakuten's RPaaS DevOps team presented lessons learned from using CloudFoundry in production. Key points include: 1) RPaaS is Rakuten's internal PaaS built on CloudFoundry that started serving production apps in mid-2012. It provides benefits like improved agility and simplified operations. 2) To effectively support users, the RPaaS team took an approach of understanding everything - the CloudFoundry stack, infrastructure environment, user needs, and capacity trends. 3) Managing user expectations was important as not all "cloud concepts" could currently be fulfilled and limitations needed clarification. The field of PaaS continues to

1. RPaaS DevOps:
Lessons from using
Cloudfoundry in Production
Oct/26/2013
Waldemar Quevedo
PaaS DevOps team. Rakuten, Inc.

2. About me
• Waldemar Quevedo
• From Mexico
• Joined Rakuten in 2010 as an App developer
• Joined the RPaaS team in 2012 in its earlier stages
• Github: https://github.com/wallyqs
2

3. WARNING
• No funny pictures in these slides! Sorry!
3

4. What is RPaaS?
• Internal PaaS at use in Rakuten
• Built on Cloudfoundry
• Started to service production applications since
mid 2012
• Derek Collison, its creator is a technical fellow of
our team
4

5. Example site: tech.rakuten.co.jp
5

6. 6

7. Why have an application Platform as a Service?
• Natural evolution of how to develop, release and
maintain applications.
• Improve productivity by automating the
provisioning of resources on demand.
• Scale horizontally (more instances)
• Scale vertically (more cpus, memory, etc..)
• Guarantees high availability
7

8. Example use case: Scaling
How to go from this,
aaa.bbb.ccc.ddd:80
tech.rakuten.co.jp
to this,
aaa.bbb.ccc.ddd:9393
aaa.bbb.ccc.ddd:9395
aaa.bbb.ccc.ddd:9291
aaa.bbb.ccc.ddd:8888
tech.rakuten.co.jp
tech.rakuten.co.jp
tech.rakuten.co.jp
tech.rakuten.co.jp
• …in a matter of seconds?
8

9. Increasing instances
9

10. Increasing instances
10

11. Benefits from an application PaaS
• Improves agility of the application team
• Application team can focus on the application
not maintaining servers.
• Simplified operations
11

12. Simplified operations
12

13. Benefits from an application PaaS
• Enforces automation and better practices for
developers.
• Reduces the amount of workflows required to get
things done.
• Integration with internal environment.
• And many more…
13

14. Topic of the day: How to adopt an application PaaS
• An application PaaS is all goodness for
developers, since they can stop caring about
servers.
• Instead, they can rely on the RPaaS team!
14

15. Usual DevOps metaphor: Vampires vs. Werewolves
• Vampires (吸血鬼)
app developers
provide value
• Werewolves (オオカミ人間)
sys admins
protect value
15

16. Vampires vs. Werewolves
• A PaaS admin is somewhere in the middle.
• We take care of the servers on behalf of users but
in the end we also care about:
• How the user is actually using the runtimes.
• Demand from application runtimes available.
• Support the users when they run into problems
with one of the runtimes.
16

17. Different metaphor with PaaS:
Villagers and Samurai
• Villagers (村人)
Application team can survive without PaaS but
has lots of issues (-agility, -flexibility, -availability)
• Samurai (侍)
Solves the issues of the application team by
providing a better environment.
17

18. In order to deliver to users…
• RPaaS team approach:
• “you should know everything”
1. Know your stack
(Cloudfoundry, Nginx, Services)
2. Know your environment (infrastructure, the network…)
3. Know your users
(runtimes: Ruby, Java, support)
4. Know your capacity
(metrics & trends)
18

19. Know your stack
• We needed to learn the internals of Cloudfoundry
(OpenSource software FTW)
•
•
•
•
•
CloudController
DEA
HealthManager
Router
NATS
19

20. Cloudfoundry
20

21. Know your stack
• Specially need to be prepared for failure scenarios:
• How long does it take for the HM to execute the
application failover when a DEA goes down?
• What happens when there is a NATS failure?
• How long does it take for the Router to stop
balancing an application when the app goes
down?
21

22. Know your environment
• Remember the „Fallacies of distributed
computing‟
http://en.wikipedia.org/wiki/Fallacies_of_Distribut
ed_Computing
• Need to be familiar with your infrastructure, the
network etc…
• Don‟t be afraid of tcpdump
• Share your findings about the environment with
the users
• Each datacenter is a different beast
22

23. Know your environment
• Again, need to be prepared for trouble scenarios:
• Storage will fail
• Physical hosts will fail
• That VM that you expected to never to restart,
will restart (hint: NATS)
23

24. Know your users
• Insight: When introducing a PaaS into your
company, you need an extra effort during the
adoption phase.
• „Do things that Don‟t Scale‟:
http://paulgraham.com/ds.html
24

25. Know your users
• At the time we started Cloudfoundry was
relatively new technology
• Roadmap mismatch: Private PaaS vs Public
PaaS
• Must features for our users were different:
• Logging, monitoring, billing, team based
deployment
• Need highly available services
• These were blockers we need to comply with
so that teams would start adoption of the PaaS
25

26. Know your capacity
• re: „Auto-scale‟:
• You usually don‟t want to do it
• You do need to automate provisioning of
resources though
• Monitor the trends and provision according to
them:
• DEAs (app resources)
• Disks
• Services (cache, dbs)
• Traffic (increase max qps limit)
26

27. Managing user expectations
• “Under promise and over deliver”
• Need to make it very clear for users what is
possible and what is just not feasible according
to the current limitations of the system.
• The concept of the cloud is cause of confusion
sometimes
• “it should be elastic”
• “it should auto scale”
• “it should self heal”
27

28. Managing user expectations
• “The RPaaS team will fix it for us”
• It helps becoming familiar with popular libraries
for different runtimes to give good advice to the
users on how to solve their issues in the
environment you provide.
• Learn to say no sometimes
• Ideally: Make PaaS users collaborators
• Provide a straightforward way for users to
have a single node setup of the PaaS.
28

29. Wrapping up
• The PaaS field continues to evolve
• The application PaaS we run is still
first,second generation technology
• We already sort of see traces of the next
evolution
29

30. Wrapping up
• Keep up with the trend
• When we started it seemed Ruby was the
language from the cloud
https://twitter.com/mart_brooks/status/27311502
8008341506
30

31. Wrapping up
• Nowadays, it seems that Go is looking out to
displace it little by little.
• Biggest gains seem to be performance, and it
being so practical to deploy.
• Also, docker.
https://twitter.com/derekcollison/status/245522124666716160
http://www.slideshare.net/derekcollison/go-conference-japan
31

32. Thanks!
• Questions? Comments?
32