In this talk, I cover the typical evolution of Terraform in the organizations I have seen over the years, how most teams end up getting eventually buried in Pull Request reviews for infrastructure changes, and the easiest way to unbury a team by starting small with Conftest and Terraform policy automation.

1. 1
Jay Wallace, Production Engineering
Feb 20th, 2020
Twitter: @mootpt
Terraform Code Reviews:
Supercharged with Conftest

2. HashiTalks
2
Morning
Routine

3. HashiTalks
3
What we will cover
Typical Evolution of Terraform
Conftest
Easy Recipes For Success
Demo In Practice
OUTLINE

4. 4
Typical Evolution of
Terraform

5. HashiTalks
Works on my machine
5
Looks good to me 🤖🤖🤖🤖
Evolution broken into 3 eras

6. HashiTalks
6
Agility vs
Stability
Agility
Stability

7. HashiTalks
7
Agility vs
Stability
Stability
Agility
Help Wanted

8. HashiTalks
8
Write some code
Works on
my
machine

9. HashiTalks
Apply locally
Works on
my
machine
9

10. HashiTalks
Uh Oh. Someone already
created the resource
Works on
my
machine
10

11. HashiTalks
11
This is a simple fix. We will just add remote
state and it won’t happen again.

12. HashiTalks
Resource was created, but
it’s not in our config
Works on
my
machine
12

13. HashiTalks
13
Works on my machine
Make Changes Quick
Engineers can just ship.
Terraform apply is easy to just
run.
Nothing to implement
Outside configuring state
buckets, there isn’t much work
on the frontend.
State & Config inconsistent
There is a chance engineers
running Terraform
independently will run into some
config/state drift.
High Risk
Engineers may inadvertently
introduce security risks. Also,
possibly destroy production
resources.
No Accountability
No audit trail of who did what
and when. Little to no
collaboration

14. HashiTalks
Agility vs
Stability
Stability
Agility
Help Wanted
Startup Mode
14

15. HashiTalks
15
All of this should be happening in version
control. That’ll fix it!

16. HashiTalks
16
Atlantis
Terraform Pull Request Automation

17. HashiTalks
17
Engineer writes some code
LGTM

18. HashiTalks
Pushes to GitHub.
LGTM
18

19. HashiTalks
Explains what they are
doing
LGTM
19

20. HashiTalks
20
Atlantis runs the plan
instead of the engineer
LGTM

21. HashiTalks
21
Requests an approval
before an apply can occur
LGTM

22. HashiTalks
22
Comments in GitHub to
trigger Atlantis to run apply
LGTM

23. HashiTalks
23
Looks good to me
Collaborative
Engineers able to collaborate
effectively. Allows for code
review and approval process
Auditable and Accountable
Clear change logs and the
ability to know who did what
and when.
Bottlenecks
More code reviews equates to
more engineering hours. Adds
up the more infrastructure you
manage.
Config reflects state
No more developers operating
on their own machine. Code is
accurate reflection of state of
world

24. HashiTalks
24
Agility vs
Stability
Stability
Agility
Help Wanted
Startup Mode
Slow and Steady

25. HashiTalks
25
Now I’m a full-time Pull Request reviewer.
How can I automate this?

26. 26
Conftest

27. HashiTalks
27
These (policies) are the things were actually looking for when we
do code reviews. We should be checking for violations before
there is ever a human involved.
Policy Enforcement

28. HashiTalks
28
OPA is an engine that
enables a single approach
for policy enforcement
across different services
Conftest

29. HashiTalks
29
Built on top of OPA. With a
focus on testing data
Conftest

30. HashiTalks
30
Conftest
Purposeful Design
Designed specifically to be used
with CI or local testing.
Functionally same as OPA.
Works well with Atlantis
Not just HCL/Terraform
OPA supports a wide variety of
structured data, today we are
just talking about testing
Terraform
Everything is Rego
DSL for making policy
assertions on the given
structured data. Rego
playground awesome place to
experiment

31. HashiTalks
Conftest functionality
geared towards user
Conftest
31

32. HashiTalks
For us it started small.
Conftest
32

33. HashiTalks
Example policy execution
Conftest
33

34. HashiTalks
Example policy checking if
our security groups are too
open.
Conftest
34

35. HashiTalks
35
Great. How do I run this in my pipelines?

36. HashiTalks
GitHub Actions
36
There is a GitHub action for contest CircleCI has a conftest orb
Traditional CI
Atlantis is a great fit
GitOps

37. 37
Recipes for Success

38. HashiTalks
38
Recipe 1: Resource whitelisting

39. HashiTalks
Do we care about this
resource?
Recipe 1
39

40. HashiTalks
40
Recipe 2: Approved Modules

41. HashiTalks
Is the module approved?
Recipe 2
41

42. HashiTalks
42
Recipe 3: CRUD

43. HashiTalks
Is this engineer about to
delete a bunch of
resources?
Recipe 3
43

44. 44
Example setup

45. HashiTalks
Atlantis + Custom Workflow
45
Execute Terraform and Conftest Pul; Request mergeable
GitHub Status Checks
Determining approvers
PullApprove
Our setup

46. HashiTalks
Note the custom conftest
workflow
atlantis
46

47. HashiTalks
We both run conftest in this
scripts and update GitHub
with custom statuses
opa.sh
47

48. HashiTalks
We use this service over
CODEOWNERS as it
supports conditional
approvals on custom
statuses
pullapprove
48

49. 49
Demo - Approved
Modules

50. HashiTalks
50
Conftest
Agility
With automated testing
engineers can move faster
without approvals
Auditable and Accountable
Clear change logs and the
ability to know who did what
and when.
More work on frontend
Implementing some initial
policies and a workflows around
conftest takes some work
Reliability
By codifying our policies, we
can feel confident on what can
and can’t slip into our codebase

51. HashiTalks
51
Agility vs
Stability
Stability
Agility
Help Wanted
Startup Mode
Slow and Steady
Supercharged

52. 52
Photo by Laura Johnston on Unsplash
Thank you