Physical, virtual, containers. Public cloud, private cloud, hybrid cloud. IaaS, PaaS, SaaS. These are the choices that we're faced with when architecting a datacenter of today. And the choice is not one or the other; it is often a combination of many of these. How do we remain in control of our datacenters? How do we deploy and configure software, manage change across disparate systems, and enforce policy/security? How do we do this in a way that operations engineers and developers alike can rejoice in the processes and workflow? In this talk, I will discuss the problems faced by the modern datacenter, and how a set of open source tools including Vagrant, Packer, Consul, and Terraform can be used to tame the rising complexity curve and provide solutions for these problems.

1. Automa'ng the
Modern Datacenter

2. @mitchellh
Mitchell Hashimoto

3. Powering the so9ware-­‐managed datacenter.
HashiCorp
hashicorp.com

6. The Modern Datacenter

7. Single Server
Datacenter
Server

8. Mul'-­‐Server
Datacenter
Server Server
ServerServer

9. Virtualiza'on
Datacenter
Server Server
VM VM
VM VM
VM VM
VM VM

10. Containeriza'on
Datacenter
Server Server
VM
VMVM
VM
VM
VMVM

11. Service Prolifera'on
Datacenter
Server Server Server
Server Server Server
Server Server Server
DNS
Database
CDN

12. Etc…
• Hybrid cloud: Physical datacenter vs. Cloud provider
• Mul'-­‐paradigm: Physical, virtual, container
• IaaS, PaaS, SaaS depending on the app
• Opera'ng systems: Windows, Linux, Mac, Other
• Realis'cally a mixture of everything just shown

13. But… why?

14. Common Goal: Efficiently deliver and
maintain applica9ons.

15. Applica'on Delivery
• Consistent
• Shareable
• Readily Available
• High producCon
parity.
• Start and configure
servers / services
• Deploy and run
applicaCon
• Update servers or
applicaCons
• Reconfigure, feature
flag
• Monitor health
• Orchestrate
complex changes
Development Deployment Maintenance

16. HashiCorp’s Open Source Tools
Development Deployment Maintenance

17. Taming the Datacenter
Deployment + Maintenance

18. Deployment + Maintenance
1. Acquisi'on
2. Provision
3. Update
4. Destroy

19. Historically
• Servers: Days, weeks
• Provisioning: Hours, days
• SaaS: <didn’t exist>

20. Today
• Servers: Minutes
• Provisioning: Minutes
• SaaS: Minutes

21. Historically
• Rela'vely fixed set of servers
• Fewer applica'ons to deploy
• Fewer SaaS
• Less demanding web traffic

22. Today
• Poten'ally elas'c set of servers of varying sizes
• Push towards SoA
• SaaS for everything
• More internet connected devices than ever before => higher traffic

23. What do we need?
• Zero to deployed in one command
• Resiliency through distributed systems
• Autoscaling, autohealing
• Beder teamwork through codified knowledge

24. But how? Automa6on.

25. HashiCorp’s Open Source Tools
Development Deployment Maintenance

26. Automa9ng the Datacenter
Deployment + Maintenance

27. terraform.io

28. Build, combine, and launch
infrastructure safely and efficiently.
terraform.io

29. What If I asked you to…
• create a completely isolated second environment to run an applica'on
(staging, QA, dev, etc.)?
• deploy a complex new applica'on?
• update an exis'ng complex applica'on?
• document how our infrastructure is architected?
• delegate some ops to smaller teams? (Core IT vs. App IT)

30. What If I asked you to…
• create a completely isolated second environment to run an applica'on
(staging, QA, dev, etc.)? One command.
• deploy a complex new applica'on? Code it, diff it, pull request.
• update an exis'ng complex applica'on? Code it, diff it, pull request.
• document how our infrastructure is architected? Read the code.
• delegate some ops to smaller teams? (Core IT vs. App IT) Modules,
code reviews.

31. But how?

32. Terraform
• Create infrastructure with code: servers, load balancers, databases, email
providers, etc.
• One command to create, update infrastructure.
• Preview changes to infrastructure, save diffs.
• Use code + diffs to treat infrastructure change just like code change:
make a pull request, show the differences, review it, and accept.
• Break infrastructure into modules to encourage/allow teamwork without
risking stability.

33. Infrastructure as Code
DigitalOcean Droplet with DNS in DNSimple
resource "digitalocean_droplet" "web" {
name = "tf-web"
size = "512mb"
image = "centos-5-8-x32"
region = "sfo1"
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = "${digitalocean_droplet.web.ipv4_address}"
type = "A"
}

34. Infrastructure as Code
DigitalOcean Droplet with DNS in DNSimple
resource "digitalocean_droplet" "web" {
name = "tf-web"
size = "512mb"
image = "centos-5-8-x32"
region = "sfo1"
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = "${digitalocean_droplet.web.ipv4_address}"
type = "A"
}

35. Infrastructure as Code
DigitalOcean Droplet with DNS in DNSimple
resource "digitalocean_droplet" "web" {
name = "tf-web"
size = "512mb"
image = "centos-5-8-x32"
region = "sfo1"
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = "${digitalocean_droplet.web.ipv4_address}"
type = "A"
}

36. Infrastructure as Code
DigitalOcean Droplet with DNS in DNSimple
resource "digitalocean_droplet" "web" {
name = "tf-web"
size = "512mb"
image = "centos-5-8-x32"
region = "sfo1"
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = "${digitalocean_droplet.web.ipv4_address}"
type = "A"
}

37. Infrastructure as Code
• Human friendly config, JSON compa'ble
• Text format makes it version-­‐able, VCS-­‐friendly
• Declara've
• Infrastructure as code on a level not before possible

38. Zero to Done in One Command
Terraform Apply
$ terraform apply
digitalocean_droplet.web: Creating…
dnsimple_record.hello: Creating…
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

39. Zero to Done in One Command
• Idempotent
• Highly parallelized
• Will only do what the plan says

40. Safely Change/Iterate
Terraform Plan
+ digitalocean_droplet.web
backups: "" => "<computed>"
image: "" => "centos-5-8-x32"
ipv4_address: "" => "<computed>"
ipv4_address_private: "" => "<computed>"
name: "" => "tf-web"
private_networking: "" => "<computed>"
region: "" => "sfo1"
size: "" => "512mb"
status: "" => "<computed>"
+ dnsimple_record.hello
domain: "" => "example.com"
domain_id: "" => "<computed>"
hostname: "" => "<computed>"
name: "" => "test"
priority: "" => "<computed>"
ttl: "" => "<computed>"
type: "" => "A"
value: "" => "${digitalocean_droplet.web.ipv4_address}"

41. Safely Change/Iterate
Terraform Plan
+ digitalocean_droplet.web
backups: "" => "<computed>"
image: "" => "centos-5-8-x32"
ipv4_address: "" => "<computed>"
ipv4_address_private: "" => "<computed>"
name: "" => "tf-web"
private_networking: "" => "<computed>"
region: "" => "sfo1"
size: "" => "512mb"
status: "" => "<computed>"
+ dnsimple_record.hello
domain: "" => "example.com"
domain_id: "" => "<computed>"
hostname: "" => "<computed>"
name: "" => "test"
priority: "" => "<computed>"
ttl: "" => "<computed>"
type: "" => "A"
value: "" => "${digitalocean_droplet.web.ipv4_address}"

42. Safely Change/Iterate
Terraform Plan
+ digitalocean_droplet.web
backups: "" => "<computed>"
image: "" => "centos-5-8-x32"
ipv4_address: "" => "<computed>"
ipv4_address_private: "" => "<computed>"
name: "" => "tf-web"
private_networking: "" => "<computed>"
region: "" => "sfo1"
size: "" => "512mb"
status: "" => "<computed>"
+ dnsimple_record.hello
domain: "" => "example.com"
domain_id: "" => "<computed>"
hostname: "" => "<computed>"
name: "" => "test"
priority: "" => "<computed>"
ttl: "" => "<computed>"
type: "" => "A"
value: "" => "${digitalocean_droplet.web.ipv4_address}"

43. Safely Change/Iterate
• Plan shows you what will happen
• Save plans to guarantee what will happen
• Plans show reasons for certain ac'ons (such as re-­‐create)
• Prior to Terraform: Operators had to “divine” change ordering,
paralleliza'on, rollout effect.

44. Workflow
• Make code changes
• `terraform plan`
• Pull request with code changes + plan to make changes
• Review and merge
• `terraform apply pr1234.pplan`

45. Knowledge Sharing: Modules
Terraform Plan
module “consul” {
source = “github.com/hashicorp/consul/terraform/aws”
servers = 3
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = “${module.consul.server_address}”
type = "A"
}

46. Knowledge Sharing: Modules
Terraform Plan
module “consul” {
source = “github.com/hashicorp/consul/terraform/aws”
servers = 3
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = “${module.consul.server_address}”
type = "A"
}

47. Knowledge Sharing: Remote Modules
Terraform Plan
resource “terraform_remote_state” “consul” {
type = “atlas”
name = “hashicorp/consul”
}
resource "dnsimple_record" "hello" {
domain = "example.com"
name = "test"
value = “${terraform_remote_state.consul.outputs.server_address}”
type = "A"
}

48. Knowledge Sharing: Modules
• Self-­‐contained infrastructure components
• Allows delega'on of responsibility to mul'ple teams
• Some teams create modules, other teams consume modules
• Remote modules let teams share outputs, but not affect infrastructure

49. Terraform
• Zero to fully deployed in one command
• Change/maintain infrastructure predictably
• Teamwork-­‐oriented workflow to infrastructure
• Goal: Sta'c deploy/provisioning of infrastructure. Real'me monitoring,
discovery, configura'on provided by Consul (discussed next).

50. consul.io

51. Service discovery, configura9on, and
orchestra9on made easy. Distributed,
highly available, and datacenter-­‐aware.

52. Ques'ons that Consul Answers
• Where is the service foo? (ex. Where is the database?)
• What is the health status of service foo?
• What is the health status of the machine/node foo?
• What is the list of all currently running machines?
• What is the configura'on of service foo?
• Is anyone else currently performing opera'on foo?

53. Service Discovery
Where is service foo?

54. Service Discovery
Service Discovery via DNS or HTTP
$ dig web-frontend.service.consul. +short
10.0.3.89
10.0.1.46
$ curl http://localhost:8500/v1/catalog/service/web-frontend
[{
“Node”: “node-e818f1”,
“Address”: “10.0.3.89”,
“ServiceID”: “web-frontend”,
…
}]

55. Service Discovery
• DNS is legacy-­‐friendly. No applica'on changes required.
• HTTP returns rich metadata.
• Discover both internal and external services
(such as service providers)

56. Failure Detection
Is service foo healthy/available?

57. Failure Detec'on

58. Failure Detec'on
• DNS won’t return non-­‐healthy services or nodes.
• HTTP has endpoints to list health state of catalog.

59. Key/Value Storage
What is the config of service foo?

60. Key/Value Storage
Serng and Gerng a Key
$ curl –X PUT –d ‘bar’ http://localhost:8500/v1/kv/foo
true
$ curl http://localhost:8500/v1/kv/foo?raw
bar

61. Key/Value Storage
• Highly available storage of configura'on.
• Turn knobs without big configura'on management process.
• Watch keys (long poll) for changes
• ACLs on key/value to protect sensi've informa'on

62. Multi-­‐Datacenter

63. Mul'-­‐Datacenter
Service Discovery
$ dig web-frontend.singapore.service.consul. +short
10.3.3.33
10.3.1.18
$ dig web-frontend.germany.service.consul. +short
10.7.3.41
10.7.1.76

64. Mul'-­‐Datacenter
Serng and Gerng a Key
$ curl http://localhost:8500/v1/kv/foo?raw&dc=asia
true
$ curl http://localhost:8500/v1/kv/foo?raw&dc=eu
false

65. Mul'-­‐Datacenter
• Local by default
• Can query other datacenters however you may need to
• Can view all datacenters within one UI

66. Orchestration
Events, Exec, Watches

67. Events, Exec, Watches
Dispatching Custom Events
$ consul event deploy 6DF7FE
…
$ consul watch -type event -name deploy /usr/bin/deploy.sh
…
$ consul exec -service web /usr/bin/deploy.sh
…

68. Events, Exec, Watches
• Powerful orchestra'on tools
• Pros/cons to each approach, use the right tool for the job
• All approaches proven to scale to thousands of agents

69. Easiest Distributed System Deploy
Deploy Consul to AWS
$ terraform apply github.com/hashicorp/consul/terraform/aws
var.servers
The number of Consul servers to launch.
Default: 3
Enter a value: 3
…

70. Easiest Distributed System Deploy
Deploy Consul to AWS (manually)
$ consul agent -atlas-join
-atlas=USERNAME/NAME
-atlas-token=API_TOKEN

71. Workflow
• Server is started (via Terraform, etc.)
• Consul agent is started, joins cluster
• Star'ng services (ex. web app) query Consul for configura'on
• Once healthy, services are discovered via DNS!

72. Opera'onal Bullet Points
• Leader elec'on via Ra9
• Gossip protocol for aliveness
• Three consistency models: default, consistent, and stale
• Encryp'on, ACLs available
• Real world usage to thousands of agents per datacenter

73. Thanks!
hashicorp.com