Configuration management is a great tool for helping with hardening and securing servers. But with any addition of new technology comes a new attack vector: Who watches the watchers? Security is painful. Luckily the invention of configuration management tools has made this process easier, by allowing repeatable configuration for common hardening. However there comes a catch-22: How do we harden the configuration management itself? When you have a tool that enables you to change systems at a fundamental level, it's a fairly tempting target for malicious agents, and one that would cause a lot of problems if compromised. We'll be discussing some general patterns we can use to mitigate these problems: - Whitelisting "master" API's - Encrypting sensitive data - Adding a security element to code review And we'll talk about some application specific options for some of most popular tools out there, such as Puppet, Chef, Ansible, cfengine and Salt.

1. HARDENING YOUR CONFIG
MANAGEMENT
SECURITY AND ATTACK VECTORS IN
CONFIG MANAGEMENT

2. WHO AM I?
> Peter Souter
> @petersouter
> @petems - IRC/GitHub
> Professional Services Engineer at
Puppet Labs
> Work with customers when they buy
services and teach Puppet classes

3. THIS IS MY 3RD FOSDEM!

4. WHAT IS THIS
ALL ABOUT?HTTPS://FLIC.KR/P/BHYT8B

5. SECURITY IS
HARD
AND UNDER
APPRECIATED!
HTTPS://TWITTER.COM/PETECHESLOCK/STATUS/595617204273618944

6. SPECIFIC REQUIREMENTS
MULTIPLE SYSTEMS
EVERY OS HAS IT'S OWN QUIRKS AND
NUANCES

7. CONFIG MANAGEMENT IS
HERE TO SAVE THE DAY!

8. HOWEVER...

9. QUIS CUSTODIET
IPSOS
CUSTODES?

10. A SYSTEM CAPABLE OF PERFORMING
CHANGES FOR CONFIGURATION ACROSS
THOUSANDS OF SERVERS...

11. COULD CAUSE A LOT OF
DAMAGE!

12. CONFIG MANAGEMENT: A
PRETTY BIG ATTACK
VECTOR....

13. HOW DO WE HARDEN
CONFIG MANAGEMENT
ITSELF?

14. DON'T WANT TO FOCUS
TOO MUCH ON THE TOOLS
THEMSELVES

15. I HAVE BIASES
BOTH CONSCIOUS AND SUBCONSCIOUS

16. THERE IS NO ONE-SIZE-
FITS-ALL TO HARDEN
CONFIG MANAGEMENT!

17. IT’S A LOT OF CHANGES
TO PROCESSES

18. PEOPLE ARE HARDER TO
CHANGE THAN
COMPUTERS!

19. ACCEPT THAT YOU WILL
FAIL, PLAN ACCORDINGLY

20. THE BADDIES HAVE MORE
TIME/MONEY/ENERGY
THAN YOU DO!

21. YOU WILL FAIL AT SOME
POINT. YOU NEED TO FAIL
SECURELY

22. A QUICK SURVEY

23. WHO HERE USES...
ANSIBLE
CFENGINE
CHEF
PUPPET
SALTSTACK

24. WHERE TO START?

26. FIRST 3 RESULTS ARE
FROM A COMPANY THAT
RHYMES WITH RIPTIRE...

27. 4TH RESULT: OWASP
PRINCIPLES

29. 5TH RESULT...

31. 8TH RESULT: PRETTY
GOOD BLOG POST

33. STILL, NOT SUPER IN-
DEPTH...

34. GUESS I'LL HAVE TO
ACTUALLY DO SOME
RESEARCH...

36. DATA

37. IT'S EASY TO LEAK DATA...

38. ESPECIALLY SOMETHING
YOU CAN LOOK FOR
AUTOMATICALLY

40. BEST PRACTICE
SEPARATION OF CONCERNS

41. REMOVE DATA FROM CODE
ESPECIALLY COMPANY SPECIFIC DATA!

42. DATA ABSTRACTION:
PUPPET - HIERA
CHEF - DATA BAGS/ATTRIBUTES
ANSIBLE - ROLES
SALT - GRAINS/PILLAR

43. BAD

45. GOOD

47. ADVANTAGE:
NOT ONLY MORE SECURE,
CLEANER CODE THAT'S
MORE REUSABLE!

48. THEORETICAL SCENARIO:

49. YOU SHOULD BE ABLE TO RELEASE
MOST CODE YOU WRITE PUBLICALLY
WITHOUT ANY SORT OF SECURITY
ISSUES

50. ANYTHING SENSITIVE SHOULD BE KEPT
IN THE DATA ABSTRACTION LAYER

51. EXAMPLE: GDS

53. HTTPS://GITHUB.COM/ALPHAGOV/
GOVUK-PUPPET
HTTPS://
GDSTECHNOLOGY.BLOG.GOV.UK/
2016/01/19/OPENING-GOV-UKS-
PUPPET-REPOSITORY/

54. YOUR DATA SHOULD IS
NOW SEPARATED. HOORAY!

55. BUT IT'S PLAINTEXT. BOO!

56. ENCRYPTION

57. ENCRYPTING DATA WITH
YOUR APPLICATION
SPECIFIC TOOLS:

58. PUPPET - HIERA-EYAML
CHEF - CHEF-VAULT
ANSIBLE - ANSIBLE VAULT
SALT - SALT.MODULES.GPG
CFENGINE - CF-KEYCRYPT

59. TOOL-SPECIFIC VAULTS ARE GREAT,
BUT ARE OFTEN LIMITED IN
FUNCTIONALITY OUTSIDE THAT TOOL.

60. YOU DON'T WANT TO STORE THE SAME
PASSWORD IN 10 DIFFERENT SYSTEMS
IF YOU CAN HELP IT
THAT'S 10X MORE THAT NEEDS TO BE
SECURED

61. EXTERNAL
SECRET
SERVERS?

62. OPEN SOURCE POTENTIALS CHOICES:
OPENSTACK'S BARBICAN
CLOUDFLARE'S REDOCTOBER
HASHICOP'S VAULT

63. GOING DEEPER:

64. SECURING DATA WITH
SOURCE CONTROL

65. "I wanted to make a configuration
management repository open for
others to look at and contribute
to (à la Wikimedia's Puppet
repository)...

66. However, the repository contained
secret material, like SSL keys and
passwords...

67. git-crypt was developed so the
secret material could be
protected without having to
remove it from the repository
(which is what Wikimedia had to
do).
- ANDREW AYER

68. GIT-CRYPT?
HTTPS://WWW.AGWA.NAME/
PROJECTS/GIT-CRYPT/

69. GIT-SUBMODULES OR
SEPARATE REPOS

70. STAY IN (VERSION)
CONTROL

71. GATE CONFIG
MANAGEMENT CHANGES
BEHIND VERSION CONTROL

72. REMEMBER TO KEEP
COMMITS CLEAN AS WELL!

73. commit 88a055c4c3dcec34d5r9054011963649be89d49c
Merge: 783d425 1743488
Author: Peter Souter <petems@users.noreply.github.com>
Date: Mon April 1 23:47:43 2030 +0000
Turned off SSL, we don't need that right?
also password is now password123

74. RBAC FOR GIT REPOS
CONTAINING THE DATA

75. RBAC

76. SPLIT ACCESS TO CONFIG
MANAGEMENT TOOLS
BASED ON NEED

77. MOST APPLICATIONS HAVE SOME FORM
OF RBAC HOOKS TO ANOTHER
AUTHORIZED SYSTEM (LDAP, AD, ETC.)

78. REVIEW PROCESSES
(AUTOMATED AND MANUAL)

79. AUTOMATED
SPEC TESTING
AUTOMATED TESTING SUITES
LINTING/SYNTAX CHECKING

80. MANUAL
CODE REVIEWS
GET SECURITY TEAM INVOLVED IN THE
PROCESSES!

81. WORK WITH AUDITORS

82. PEOPLE LOVE TO HATE
AUDITORS

83. ADVERSARIAL
ENVIRONMENTS ARE NOT
FUN

84. IF YOU HAVE A GOOD WORKING
RELATIONSHIP WITH THEM, THEY'RE LIKE
AN ADDITION TO YOUR TEAM.

85. LET'S FACE IT, YOU'LL HAVE TO DEAL
WITH THEM ANYWAY, SO YOU MIGHT AS
WELL MAKE IT ENJOYABLE!

86. ASK AROUND

87. SOFT SKILL/CULTURAL
SOLUTION

88. COMPARE YOUR SECURITY
WITH OTHERS WHEN
POSSIBLE

89. A SECURITY MODEL MADE
IN A VACUUM IS A SMELL

90. IF YOU'RE A CUSTOMER,
ASK YOUR VENDOR
IF YOU'RE A FOSS USER,
ASK ON MAILING LISTS

91. GAME DAYS AND DRILLS

92. IF SOMEONE HAD ACCESS
TO THE VARIOUS PARTS OF
YOUR CONFIG
MANAGEMENT INFRA...

93. HOW MUCH DAMAGE COULD THEY DO?
HOW FAST COULD YOU REVOKE
ACCESS?
HOW LONG WOULD IT TAKE YOU TO
NOTICE?

94. MONITOR, DON'T
JUST LOG

95. GET A BASELINE OF WHAT
YOUR CONFIG
MANAGEMENT DEPLOYS
LOOK LIKE

96. ELK, STATSD, RIEMANN,
COLLECTD, ETC.

97. GET DATA ON WHAT LOOKS
SUSPICIOUS

98. ACTIVITY WHEN YOU DON'T EXPECT IT
4XX, 5XX ERRORS FROM YOUR CONFIG
MANAGEMENT INFRA
UNEXPLAINED INCREASES IN THE
TEMPERATURE OF YOUR MACHINES IN
THE DATA CENTRE
GENERAL ERRORS IN VARIOUS LOGS

99. COULD BE MALICIOUS,
COULD BE ACCIDENTAL,
COULD BE A BUG...

100. ALL OF WHICH YOU
SHOULD KNOW ABOUT!

101. REDUCE
SURFACE LEVEL
OF ATTACK

102. NOT SECURITY THROUGH
OBSCURITY!

103. A BASIC EXAMPLE AT THE
APPLICATION LEVEL

104. > Chef: sensitive: true
> Puppet: show_diff=false
> Ansible: no_log: True
> Salt: --state-verbose=false

105. SECURITY
BASELINE

106. USE THE SAME SECURITY BASELINE FOR
ANY SORT OF SYSTEM:
NO DIRECT INTERNET ACCESS UNLESS ABSOLUTELY NECESSARY
USE BASTION HOSTS FOR DIRECT INTERNET ACCESS
MIRROR REPOS AND ARTIFACTS
KEEP PACKAGES UP TO DATE AND PATCHED
SENSIBLE FIREWALL RULES

107. HARDEN CONFIG
MANAGEMENT
INFRASTRUCTURE WITH
CONFIG MANAGEMENT!

108. CENTER FOR INTERNET SECURITY
BENCHMARKS

113. HARDENING.IO

115. SOME 3 LETTER AGENCIES
HAVE EVEN RELEASED
THEIR CONFIG
MANAGEMENT CODE...

117. IN LIGHT OF RECENT EVENTS, THAT
MIGHT BE NOT SUCH A GREAT THING
BUT HEY, IT'S CONFIG MANAGEMENT, SO
YOU CAN INSPECT AND ADAPT WHERE
NECESSARY!

118. SSH

119. PRIMARILY FOR ANSIBLE

120. BUT SSH CAN BE USED
FOR OTHER TOOLS AS
WELL...

121. PUPPET - SUPPLY DROP/CAPISTRANO
CHEF - KNIFE SOLO
SALT - SALT SSH
CUSTOM MADE SSH-LOOPS WRAPPING
LOCAL MODES FOR TOOLS

122. SSH HARDENING STANDARDS
> Whitelisted access
> Bastion hosts
> Restrict users
> Increase key strength
> Rotate keys
> Pre-populated knownhosts

123. HARDEN YOUR SSH WITH
CONFIG MANAGEMENT! :)

124. IF YOU'RE USING ~/.ssh/id_rsa
FOR EVERYTHING...
YOU'RE DOING IT WRONG :(

125. DEEPER SSH HARDENING...

126. SSH KEYS ON HARDWARE
YUBIKEY
SMARTCARD

127. THOUGHT EXPERIMENT:
DISABLE SSH COMPLETELY?

128. CONCLUSION
> Get your data out of your code
> Encrypt it and control access
> Most normal security conventions apply
> Follow best practices from communities and
organizations
> Auditing and gating help
> Work together! :)

129. GOING TO CONFIG
MANAGEMENT CAMP?

131. QUESTIONS? IDEAS?
HOW ARE YOU HARDENING YOUR CONFIG
MANAGEMENT?