Two years ago at Devoxx UK we talked about DevOps, what it was, why it was important and how to get started. Boy, was it scary. Now we’re wiser. More battle-scarred. The large scale of the challenge for application writers exploiting cloud and DevOps is clearer, but so is the path forward. Understanding the DevOps approach is important, but equally you must understand specific deployment technologies, security issues, operational reliability, and how to drive organisational transformation. Whether creating simple applications or sophisticated microservice architectures many of the challenges are the same. Join us to learn how you can apply this within your team and company.

1. DevOps: Microservices, containers,
platforms, tooling... Oh yeah, and people
Daniel Bryant
@danielbryantuk
Steve Poole
@spoole167

2. On the Previous Episode of Devoxx UK…
• 2014 “Moving to DevOps: Easy, Hard or Just Plain Terrifying”
• DevOps is about extending agility across your IT org
• Breaking down the silos is vital
• The business needs to react to the industry change
– DevOps, cloud and containers

3. Today
• Microservices are (operationally/conceptually) distributed systems
• The application/infrastructure ‘platform’ is still not fully baked
• Think “Safety first”
– Security, networking cyber criminals
• DevOps is (still) all about the organisation, people and processes

4. I (we) am the one who knocks…
Steve Poole
IBM Developer
@spoole167
Daniel Bryant
Chief (Mad) Scientist,
OpenCredo
@danielbryantuk
Making Java Real Since Version 0.9
Open Source Advocate
DevOps Practitioner (whatever that means!)
Driving Change
“Biz-dev-QA-ops”
Leading change in organisations
Experience of Docker, k8s, Go, Java
InfoQ, DZone, Voxxed contributor

5. Part 1- Painful Lessons…
https://www.flickr.com/photos/sarahmstewart/

6. All I hear is microservices…
“In computing, microservicesis a software
architecture style in which complex
applications are composed of small,
independent processes communicating with
each other using language-agnostic APIs. These
services are small, highly decoupled and
focus on doing a small task, facilitating a
modularapproach to system-building.”
https://en.wikipedia.org/wiki/Microservices

7. Microservices
Turn applications into small,
independent, highly decoupled,
modular services
https://www.flickr.com/photos/daikrieg/
You want to make
my life more
complicated?

8. https://www.flickr.com/photos/tahini/
Where’s the problem?

9. browser
App A
V1.0
Database
browser
You
Your
Customers
server
Data Centre

10. browser
Load
balancer
App A
V2.0
App A
V2.0
App B
V1.0
Database
browser
Database
Ops

11. browser
Load
balancer
App A
V2.0
App A
V2.0
App B
V1.0
Database
browser
Database
browser
App A
V2.0
App A
V2.0
App B
V1.0
browser

12. browser
Load
balancer
App A
V2.0
App A
V2.0
App B
V1.0
Database
Database
browser App A
V2.0
App A
V2.0
App B
V1.0
browser
browser
browser
browser
browser
Load
balancer
Database

13. browser
Load
balancer
App A
V2.1
App A
V2.1
App B
V1.0
Database
Database
browser App A
V2.1
App A
V2.1
App B
V1.0
browser
browser
browser
browser
browser
Load
balancer
Database

14. A simple upgrade or a major impact?
lost revenue or going out of business?

15. https://www.flickr.com/photos/24151087@N00/
What lessons have we learnt?
Sharing data stores sounds like it saves effort but
introduces cohesion between applications
Big-bang versioning of applications means putting
existing unchanged use cases at risk
Scaling is challenging when you try to duplicate whole
systems
Infrastructure – it’s much more important than we
initially realized

16. Part 2 - Build Your Own Platform?

17. Adrian Cockcroft’s (@adrianco) Thoughts

18. http://wikibon.com/wp-content/uploads/container_implementations.png
Technology Choices

19. 09/06/2016 @danielbryantuk

20. What’s Wrong with PaaS?

21. Core Features
• Continuous deployment
• Health checks
• Logging
• Monitoring
www.opencredo.com/2015/10/31/javaone-building-a-
microservice-development-ecosystem-video

22. Logging
• “The Log: What every software engineer
should know about real-time data's
unifying abstraction”
• “10 Tips for Proper Application Logging”
• ElasticSearch-Logstash-Kibana (ELK)
– Buffer/proxy log sending or…
– Mount directory into container
09/06/2016 @danielbryantuk

23. Monitoring
• Push
– Spring Boot actuator e.g.
InfluxDbExporter
• Pull
– E.g. Telegraf (TICK), Prometheus
• InfluxDB vs prometheus vs graphite
vs opentsdb
• Information radiators - Grafana
– Aggregate vs individual
09/06/2016 @danielbryantuk

24. Aggregation: Sick Cattle, Not Sick Pets

25. Distributed Tracing
• Correlation: github.com/daniel-bryant-uk/correlation-id-async
• MDC logging: logback.qos.ch/manual/mdc.html
• OpenZipkin: github.com/openzipkin
09/06/2016 @danielbryantuk

26. Looking Inside the Container

27. Common Java / Docker Issues
• No disk space for docker logging
– Increase disk space (move logs to mount)
• Restricting resources to only Xmx memory limit
– Set memory limit = Heap (Xmx) + Metaspace + JVM
• Security or crypto issues as /dev/random limited in containers
– -Djava.security.egd=file:/dev/urandom
• See Chris Batey’s “The JVM and Docker” talk here at 15:00 today

28. Debugging Tools
• Java
– jstat, jstack, jmap
– “5 things you didn’t know”
• OS
– Top, htop, ps, free, df –h,
vmstat,iostat
– /proc filesystem meminfo and
vmstat not cgroup aware!
– Use sysdig
www.joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x

29. Problems?
• Rob Ewaschuk’s “Philosophy on Alerting”
• Brendan Gregg’s USE method
– “check utilization, saturation, and errors.”
• “DevOps Troubleshooting”
– Kyle Rankin
09/06/2016 @danielbryantuk

30. Part 3 – Safety first

31. https://www.flickr.com/photos/miriamdelirium/
Tooling – what’s left to do?

32. Dynamic Development
Capacity
Predefined static
VM’s LPARs etc
OpenStack
Cloud(s)
Docker
Cloud
Infrastructure as Code
Chef, Puppet, UCD …
OS
Infra On Prem Data
Centres
Cloud Providers
SoftLayer / Amazon etc
Config Containerized
Applications
Continuous Availability
Mesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev SaaSPrimary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
✔
✔
✔
✔
✔
Application DIY

33. Dynamic Development
Capacity
Predefined static
VM’s LPARs etc
OpenStack
Cloud(s)
Docker
Cloud
Infrastructure as Code
Chef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud Providers
SoftLayer / Amazon etc
Config Containerized
Applications
Continuous Availability
Mesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev SaaSPrimary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static
hosts (‘BYOD’)
DIY
Application DIY

34. Dynamic Development
Capacity
Predefined static
VM’s LPARs etc
OpenStack
Cloud(s)
Docker
Cloud
Infrastructure as Code
Chef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud Providers
SoftLayer / Amazon etc
Config Containerized
Applications
Continuous Availability
Mesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev SaaSPrimary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static
hosts (‘BYOD’)
DIY
Compliance
/ Security ContainersVM Images
Application DIY

35. Dynamic Development
Capacity
Predefined static
VM’s LPARs etc
OpenStack
Cloud(s)
Docker
Cloud
Infrastructure as Code
Chef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud Providers
SoftLayer / Amazon etc
Config Containerized
Applications
Continuous Availability
Mesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev SaaSPrimary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static
hosts (‘BYOD’)
DIY
Compliance
/ Security ContainersVM Images
Application DIY

36. You do understand about security and compliance
right?
https://www.flickr.com/photos/adulau/

37. Where’s your data?
Legal restrictions on data location
Vary by country even within the EU.
Different rules depending on types of data
You already know this?
But now you’re putting the data in the
cloud.
DO YOU understand where its going
Can you control / manage / audit the
situation?
Now it’s your problem.
Having fun finding tools to help

38. How’s your security knowledge?
• Again – now it’s your problem
– Your code is running in the cloud
– You created the services & the containers
– Are they secure?
– How do you test?
– Are you sure?
– Are those web services you’re buying secure?
• How much do you know about networking?
– Not enough…
• Cyber crime is big business – you will get targeted.
https://www.flickr.com/photos/61423903@N06/

39. “Organized Cybercrime is the most profitable type of crime”
• Cybercrime is estimated to be worth 445 Billion Dollars a Year
• In 2013 the United Nations Office on Drugs and Crime (UNODC) estimated
globally the illicit drug trade was worth 435 Billion Dollars
• Guess which one has the least risk to the criminal?
• Guess which is growing the fastest?
• Guess which one is the hardest to prosecute?
• Guess which one is predicted to reach 2100 Billion Dollars by 2019?

40. Talk to your Ops team
• They are your best friends.
– They know about security and networking.
• You need to know too
– They know (some) of the answers
• It’s a whole new domain for you
– It’s not a new problem for them
• Time to learn

41. Part 4 - Sharing is Caring

42. The Results of the Survey Are In…
• Puppet Labs 2015 State of DevOps
– Available: puppetlabs.com/2015-devops-report
• Accelerates deployment
– High performers 30x more deploys
– Code committed to production 200x faster
• Prevents failures and streamlines recovery
– High performers 60x fewer failures
– Recovery 168x faster

43. DevOps Topologies (Bad)
• DevOps Anti-Types
– Dev and Ops
– DevOps Silo
– No Ops Needed
– Tools Team
– Sysadmin
– Embedded Ops
http://web.devopstopologies.com/
@matthewpskelton

44. DevOps Topologies (Good)
• DevOps Team Topologies
– Dev+Ops
– Shared Ops
– Ops as IaaS
– DevOps-as-a-Service
– Teamp DevOps Team
– DevOps Evangelists
– SRE Team
– Container-Driven
– DB Capability
http://web.devopstopologies.com/
@matthewpskelton

45. The ’Spine Model’ – The Right Conversations
• Effective conversations make for
effective collaboration
• People get stuck in a dilemma where
equally plausible options are available
• “Going up the Spine” breaks deadlock
• It’s a TOOL Problem
– As a species, we have always been Tool
users and makers.
– We use _____ to get our work done
http://spinemodel.info/explanation/introduction/

46. DevOps != Tooling
• PRACTICES before Tools
– Decide on the Practices that the tools are there to
support
– We do _____ to create value
• PRINCIPLES before Practices
– Decide on the Principles to measure those Practices
against.
– We leverage _____ to change the system
• VALUES before Principles
– Make as explicit as possible the Values at play in the
system.
– We optimise for _____
• NEEDS before Values
– It all starts at Needs. Why does this system exist in the
first place?
– We are here to satisfy _____
http://spinemodel.info/explanation/introduction/

47. When Things (Inevitably) Go Wrong
09/06/2016 @danielbryantuk

48. Optimise for Learning (and Feedback)
09/06/2016 @danielbryantuk
www.infoq.com/news/2015/06/too-big-to-fail

49. Part 5 – Conclusions

50. Key Messages
• Microservices force you to know about distributed systems
– You need to learn how to design and run applications in a new way
• The application/infrastructure ‘platform’ stack is still not fully baked
– Essentials are CI/CD, health checks, logging and monitoring
• Think “Safety first”
– It’s a wild world out there - security, networking cyber criminals
• Make your ops team your best friend
– DevOps is all about the organisation, people and processes

51. Thanks – Questions?
Daniel Bryant
@danielbryantuk
Steve Poole
@spoole167

52. Bonus: Containers Are Not Immutable (By Default)
• Containers can be as susceptible to
configuration drift as VMs/bare metal
– This surprises many people
• Suggestions
– docker run --read-only
– docker run --tmpfs /tmp
• Gareth Rushgrove’s CraftConf talk
– http://www.ustream.tv/recorded/86186490