Hashicorp usage at Holaluz: From no automation at all to infrastructure-as-code and a DevOps culture in 8 months

1. BARCELONA HUG - 14 SEP 2017
at

2. 7YEARS AGO…
• Vagrant first commit was in 17 Jan 2010
• Holaluz was founded in 12 Nov 2010
• DevOps starts becoming a concept
• A backend developer in a big e-commerce

4. WHO AM I?
• Currently working as CTO at Holaluz
• Ex Wonga, Hailo, SocialPoint, Ulabox, Privalia…
• Developer for many years, been automating
things for a while, DevOps before it was trendy!
• https://devops.barcelona conference co-organiser

6. HOLALUZ WEBAPPS INFRA
Web servers
under ELBS
running Apache
+ PHP7.1
Dedicated
cron job boxes
using PHP7.1
RDS PostgreSQL
instances
EC2 Elasticsearch
Some S3
buckets to
store files
Elasticache
Redis
for message
deduping
SQS queues
background
jobs
RDS MySQL
instances
EFS to store
shared files
(moving to
S3 & ELK)

7. HOLALUZ BIGDATA INFRA
Python apps
& cron tasks
using Django
and Celery
R running
instances
RDS
PostgreSQL
instances
EC2 Elasticsearch
EC2 MongoDB
Some S3
buckets to
store files
Elasticache
Redis
for Celery

8. HOLALUZ ERP INFRA
RDS
SQLServer
Per env S3 buckets and
SQS communicating
with Web Infra
License
Servers
Cluster
Windows 2012R2
Servers Cluster
running Java8
PROD
INTEGRATION
DEVELOPMENT

9. OTHERTHINGS AROUND
Shared ELK Stack for logs
aggregation and inspection
Middleman static sites
generator
Grafana forTime Series
inspection and alerting
InfluxDB forTime Series
metrics storage
E-Invoice Document
Management + approval
flows
CloudwatchTime Series
metrics storage and alerting

10. HASHICORPTOOLS
• Confession: I am a massive fan
• Work everywhere, automate anything
• Excellent open-source community
• Well documented, specially BC changes

11. Development Environments
Made Easy

12. VAGRANT AT HOLALUZ
• Static website generator middleman (Ruby based)
• Local environments for PHP projects
• Local environments for Python Big Data projects
• Coming soon, Windows boxes for people who
need to connect to sites needing old IE

14. Build Automated Machine
Images

15. CONCEPTS
• Builders: Platforms you build images in. It is all
about what you start from!
• Provisioners: Installs and configures
• Post-processors: Optional final steps

16. WHAT I LIKE
• Builds for multiple platforms from a single
source configuration
• VERY Easy to understand
• Works (and can provision) in Win, Mac, Linux
• Easy to share provisioning scripts or use Puppet /
Ansible recipes

17. CAVEATS
• Need to be very prescriptive or you end up
with multiple very similar templates
• A bit hard to go with a DRY approach
• Some things are hard to destroy / replace with
new images

18. PACKER AT HOLALUZ
• All PHP infrastructure with ASGs using
AMIs we recycle periodically
• Kibana, Logstash, Grafana, InfluxDB
• Python BigData infrastructure as well (R soon)
• ERP first steps on top of official Windows AMIs

21. Write, Plan and Create
Infrastructure as Code

22. CONCEPTS
• Provider: Platform we are automating
• Resources:Automatable things in the Provider
• Modules: Reusable set of resources
• State: Used to diff desired state to existing. Can be
stored remotely and supports distributed locking

23. WHAT I LIKE
• Can integrate with anything that has an API
• Easy to extend, contribute and really quick to add
new features. Excellent Github community
• Existing resources can be imported (PAIN)
• Have used it for 18 months, multiple providers, rarely
hit a bug and was always quickly fixed

24. CAVEATS
• Once you goTerraform, STOP using Console
• Some providers don´t have nice update support
• Terraform modules feel a bit hacky
• Sometimes state needs manual edition (getting
much better but beware new providers)

25. TERRAFORM AT HOLALUZ
• Most infrastructure rebuilt usingTerraform
• VPC / Network for test environments
• Some S3 buckets and SQS configuration
• Most IAM policies

26. INTERESTINGTRICKS (I)
Automatic code
deployment via EC2
userdata on startup

27. INTERESTINGTRICKS (II)
Always apply latest
built AMI using a filter
on tags created by
Packer

28. INTERESTINGTRICKS (III)
Environments Modules
State in S3 per env
Lock via DynamoDB
Use env output as
variables in another env

29. BACK IN JANUARY
• Deployments were painful
• Unreliable test and local dev environments
• Hard to integrate new things
• Poor logging and monitoring metrics
• Painful error recovery, no real DR strategy

30. BAD HABITS AND PROBLEMS
• Development teams did not talk much
• Little understanding of network and infra
• Lack of technical guidelines
• No time to rethink processes
• Lots of legacy code with no tests at all

31. HOLALUZ IN SEPTEMBER
• One-click deployments almost everywhere
• Development teams talk and collaborate
• Releases speed MASSIVELY improved
• Decent logging and monitoring
• Better error recovery, decent DR strategy

32. LONG WAYTO GO
• Lots of technical debt still to pay
• Fragility in crucial parts of our business
• Still too many bugs, still too little testing
• Incoming changes in Energy (electric cars,
smart batteries, prosumers, blockchain…)

33. NEAR FUTURE
• Complete missing Vagrant environments
• Packer generating Docker containers for Dev envs
• Import all existing resources into Terraform and
rebuild the production network (PAIN)
• Increase AWS usage (Redshift, DynamoDB…)
• Full automated EVERYTHING

34. DEVOPS IS…
• Devs and Ops working together to deliver value
• Empower teams, reduce hard dependencies
• Communicaton, Integration, Collaboration
• Boosting productivity, make life easier!
• Automation, CI/CD, Infrastructure as code…

35. ENABLETHE BUSINESS!
That´s what they pay us for!

36. BOOKS AND REPOS
• The Phoenix Project - Gene Kim, Kevin Behr, George Spafford
• The DevOps Handbook - Gene Kim, Patrick Debois
• Terraform: Up and Running - JamesTurnbull
• https://github.com/ricardclau/geekshubsbcn (Packer +Terraform)
• https://github.com/kaorimatz/packer-templates (Linux) & https://github.com/
joefitzgerald/packer-windows (Windows)
• https://github.com/terraform-community-modules (MANY examples)

37. QUESTIONS? CONTACT?
• Email: ricard.clau@gmail.com
• Twitter: @ricardclau
• Github: https://github.com/ricardclau
• If you think these techniques help your company,
let´s talk!