DevSecCon London 2017: Permitting agility whilst enforcing security by Alina Radu

1. Join the conversation #DevSecCon
BY ALINA RADU
Permitting agility while enforcing
security - a story of making Docker a
win-win proposition

2. Overview
• PaaSTA: Yelp’s open source Platform as a service
• Microservices in Docker containers
• Jenkins
• build pipelines
• multiple steps
• security-check step
• runs a suite of security tests
• Notification system for failures & runbook
• Take aways

3. whoami
• Alina
• Software Engineer at Yelp London
• Security team: Infrastructure security
• Politehnica University of Bucharest Alumni
• Avid traveller

4. Yelp’s Mission
Connecting people with great
local businesses.

5. PaaSTA: Platform as a Service
Mesos, Marathon & Chronos
• Platform as a Service
• Mesos
• distributed job scheduler
• Marathon
• Mesos framework for long running tasks
• Chronos
• Mesos framework to schedule batch jobs

6. PaaSTA: Platform as a Service
Mesos, Marathon & Chronos
• Mesos
• distributed job scheduler
• master & agents
• offers compute resources to frameworks
• frameworks provide task - schedules it on an agent

7. PaaSTA: Platform as a Service
Mesos, Marathon & Chronos
• Marathon
• Mesos framework for long running tasks
• upstart at datacenter level

8. PaaSTA: Platform as a Service
Mesos, Marathon & Chronos
• Chronos
• Mesos framework to schedule batch jobs

9. PaaSTA: Platform as a Service
• microservices in Docker containers
• autoscaling cluster & resource specification
• move from:
• hundreds of specialized servers to
• heterogenous Mesos agents
• SOA architecture
• operational ownership of individual services
• from operations team to the service authors

10. PaaSTA contract
• service: 1 git repo and 1 Dockerfile
• Docker image
• runs the service
• same image multiple use cases: worker daemon vs web task
• contains all the code necessary for the service
• Service:
• stateless
• filesystem I/O but ephemeral disk
• log to external processors (Yelp: Scribe or Kafka)
• all checked, all good

11. Build Pipeline of a Service
• configuration repository
• Jenkins
• orchestrates build and deployment
• pipelines of sequential steps
• security-check step

12. PaaSTA security-check
• security status of the service
• run a set of tests at every build
• high level security health of the service
• something changed?
• actionable alerts for failures
• faster response time from team

13. Security tests
• Ubuntu packages up to date
• Docker container best practices
• Well known vulnerabilities
• No secrets in the service repo
• Python/Java dependency check

14. Ubuntu packages up to date
• Check if the latest packages are installed against our apt
repositories
• apt-get update && apt-get --simulate dist-upgrade

15. Docker container best practices
• container not running as user root
• Dockerfile
• Yelp maintained Docker images, no public images
• Latest images
• no packages pinned to certain versions
• .dockerignore contains .git

16. Whitelisting
• Certain version(s) of a package
• Docker images
• public images for open source projects
• non-standard Yelp images

17. Well known vulnerabilities
• bash shellshock
• extending the list in the future
• heartbleed - server side applications

18. No secrets into the service repo
• detect and prevent high entropy strings from entering our code
base
• assumes the existing code has no secrets
• checks only the new code
• solution loosely based off truffleHog

19. Package vulnerability checking
• look for known vulnerabilities in python/Java packages
• Docker container scanning
• vulnerabilities in Ubuntu packages
• classified by severity
• weekly report via email

20. Failures & alerts
• Solves: ticket creation to track failures that need to be fixed
• security-check failed?
• email
• Jira ticket
• sensu
• Runbook

21. How we got here
• run bash tests with goss
• get around the libc incompatibilities with rspec
• moving everything to python
• race condition between the Ubuntu packages that are updated
upstream and the Docker base images we build daily
• some services are not built regularly

22. You can do it too
• You don’t have to
• run our PaaS (PaaSTA)
• use our containerisation solution (Docker)
• write high-level security tests
• integrate them in your build pipeline
• open source our security tests suite next year

23. Take aways
• PaaSTA
• PaaS
• microservices in Docker containers
• build pipeline: multiple steps
• security-check
• runs a set of tests and send notifications
• most important - service owners:
• more aware of the security of their service
• involved in keeping it safe

24. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp

25. www.yelp.com/careers/
We're Hiring!

26. Join the conversation #DevSecCon
Thank you

27. Join the conversation #DevSecCon
Q & A