A presentation I gave on September 26 at the Melbourne Symfony developers group on using Environment Variables (envvars) in Symfony and managing secrets in your PHP applications.
For more information on these subjects, check out the supporting piece I wrote: https://samjarrett.com.au/swipe-right
AWS Community Day CPH - Three problems of Terraform
Symfony finally swiped right on envvars
1. Or,“why do we keep baking secrets into our artifacts?”
Symfony finally
swiped right on
envvars
2. Or,“why do we keep baking secrets into our artifacts?”
Symfony finally
swiped right on
envvars
just a collection of
cool gifs
3.
4. Hi, I’m Sam
● Long time listener, first time caller.
● Co-organiser of this group.
● Using Symfony since 2.0 beta.
● Day job - ops @ REA.
● Docker & build automation fanboy.
● Have committed terrible atrocities with VCS and
build systems in the past.
12. ● Database credentials
● SMTP creds
● Payment gateway keys
● AWS credentials
● SMS tokens
● EDM service API keys
Beyond 11 herbs and spices
13. The modern app has so many..
● S3 bucket names
● SSH or deploy keys
● JWT keys or certificates
● Service n’s API key
● Other internal system hostnames
But really, an application secret is anything you wouldn’t
put directly in the HTML markup itself.
14. Here’s why we get a bad
rep as php developers[1][2]
…
1: I’ve seen all of these examples used
2: you probably have, too
15.
16. <?php
class Config {
const LIVE_DATABASE_STRING = 'mysql://root:root@localhost:3306/pro
const TEST_DATABASE_STRING = 'mysql://root:root@localhost:3306/tes
public static function getDatabaseString() {
// @TODO add magic to determine environment
return self::LIVE_DATABASE_STRING;
}
// ...etc
The know it all “pattern”
17. The build it & they’ll come “pattern”
1. put your prod & test values in a config file
2. run “make config” or some other magic build script
3. generate PHP from template:
<?php
class Config {
const DATABASE_STRING = '{{ database_string }}';
public static function getDatabaseString() {
return self::DATABASE_STRING;
18.
19. dropping a prod
database on a
misconfigured
test site. - my biggest professional fuck up.
probably?
23. Note: This is the best of the bad, but Symfony makes it insecure & hard to
update.
1. define your config keys in a file called parameters.yml.dist, set sane defaults
2. store the actual configuration on S3 / build servers / CI / somewhere else
3. at build-time, copy the file in & build the container (cache-warm)
The Fabien told me to “pattern”
24.
25. OK. What’s
wrong with this?
How secure is your CI?
Who has access to wherever the
parameter values are stored?
27. OK. What’s
wrong with this?
Secrets are written into the
compiled Container class file.
Try it yourself:
$ grep “database.host”
var/cache/prod/*.php
32. Introducing environment variables
✅ Easy to set up in the app
✅ Functions just like another parameter
✅ Not compiled into the container
✅ Like other parameters, allows default values to be set
✅ Updating values is as simple as updating wherever it’s
stored & reload PHP
33. Introducing environment variables
❌ Can’t access outside services - i.e. controllers etc -
may require app refactoring
❌ Can be difficult to implement if you run your app
outside of Docker
❌ Some libraries don’t quite support envvars yet
(there’s a workaround though!)
38. While you’re porting
● Consider what values actually change between
deployment environments
● Anything that doesn’t change, should go into
config.yml, config_dev.yml or
config_prod.yml (a la current standard edition
practice).
● Anything that changes is a good candidate to live in
the environment.
39. Any gotchas?
● Need to defer making decisions on anything that
changes per-deployment until later.
i.e. any decisions made in a container extension.
● Dynamic parameters don’t solve this
%param% => %env(PARAM)%
● If it’s your bundle/extension, you can use factory
services to work around this.
40. The Dotenv component in 3.3
● Symfony flex uses it out of the box!
● A small class that is added in your app/app_dev.php
and console scripts
● Provides env-var like support in places you can’t
easily set variables (dev!)
● In Flex, it replaces the parameters.yml paradigm
● Relatively easy to back-port into the full-stack
framework
47. Environment vars are not perfect
● Does not solve security for anyone with access to the machine.
● Can still be accessed from outside your app - i.e. anything in
shell_exec() calls.
● The PHP user or someone on CLI can expose to another program.
● Any insecure libraries in your app can access it, too.
● Can unintentionally turn up in logs, the Symfony profiler, etc.
● The plain-text values are probably still stored somewhere.
48. How do I make it better?
● Current PR: github/symfony/symfony#23901 (due 3.4) adds features to
support docker/kubernetes secrets & make it extendable.
● Once it’s available to us, it might be like this…
○ on AWS? KMS-encrypted envvars or SSM Param. Store.
○ Not AWS? G-Cloud KMS, or other open-source, e.g. Hashicorp
Vault, Pinterest Knox, Lyft Confidant.
● In the meantime: do you host on AWS? Use KMS-encrypted environment
variables or fetch from SSM Param. Store at container boot.
49. ● Any of these principles a surprise? Read about 12-factor apps
● Segment have a great write-up on managing secrets on AWS on their
blog.
● KMS-ed envvars (enc. strings): REA’s Shush is a tiny go library to help.
● SSM param. store (central management): Segment’s Chamber can help.
● Taking inspo from k8s, Docker swarm mode now has secrets built-in.
● Not using Docker? Can be quite complicated to set env across multiple
users + daemons. Apache, and CLI users share a syntax, PHP-FPM is
different.
● Example of porting a Symfony app to Docker - for the adventurous.
Further reading & tools
50. thank you.
stalky stalk: @sammyjarrett or linkedin.com/in/samjarrett
more detail and these slides
published at samjarrett.com.au/swipe-right
Editor's Notes
Since 2.0 beta, before all the current patterns we now use existed, before the documentation was any good.
Terrible atrocities: here to atone, really.
For PHP apps, booting refers to the app loading on each page load
Not just something that occurs in PHP. Ruby, Python, Node, etc all do this.
In comparison, Symfony demands that you compile the application before you boot it. We call it building the container. Or clearing the cache.
Symfony steals a lot from Java, which also also suffers from the same issue, but you notice it differently...
Here’s something I’m sure we’re all familiar with...
This is familiar with any time you’ve ever booted a java app. Building the configuration tree and configuring the services is mostly why they take so long to start accepting traffic!
Which gives us a good speed improvement because the app doesn’t have to load all of it’s configuration and build the container at runtime!
This creates a big problem when we have complex build and deploy processes that need to operate at scale (how do you deploy to 20+ instances quickly? prebuild!) and still protect secret values
Modern deployment practices emphasise keeping the artifact static between environments, to be able to more easily replicate issues back in development or testing environments.
Container compilation means that we can’t guarantee that our artifact compiles correctly or that it is exactly the same in production. So, how do we solve this in symfony?
But this naturally creates issues when we have secrets in our apps, and yep, the modern app has a lot of them
What is a secret?
Secrets can be any form of data that you wouldn’t want to be widely spread around.
Internal hostnames (i.e. unprotected internal systems you’re hiding): redis, memcached, finance systems, etc
Lead developer’s email address (for emailing emerging issues, I guess)
Pagerduty/OpsGenie alert topics/email address (don’t want these abused overnight waking up on call staff!)
Slack tokens
Summary: secret is anytthing you wouldn’t want to put in the markup
Create a single class or factory class, and delegate responsibility for it to provide you with answers,
Ultimately embedding the secret for all environments in it.
Compromise here on either test or production means all of your secrets are gone.
Use ini, json, yaml, java.properties style or some other configuration format to specify all of your values
Create a php-class as a template full of placeholders that a template parser can quickly generate the final value for you
Generate the PHP from the template before deploying (as part of a build or deploy process)
Include the final configuration (without the config file) as part of your deployment artifact.
Profit?
Since we’re having an amnesty session, here’s my big professional mistake.
Dropping a production database because a test-site was misconfigured to use it by one of the above secret management methods (you can guess)
Let’s have some drinks after and I’ll expand on the unprofessional ones.
Same management principal as the config class (“know it all”), finally we add some Symfony flair!
Use config_dev.yml and config_prod.yml to store secrets for each environment
Largely seen in remnants of old Symfony 2.0 projects, from before the framework didn’t suggest a way of swapping values out yet (i.e. pre-parameters.ini)
Some apps that have existed since then and just been upgraded still contain bits and pieces of this
No disrespect to Fabien. He’s a super smart guy and this works pretty good. But we evolve.
Symfony provides such a simple way forward and we all go running to do it! Let’s make everything a parameter!
What’s wrong with this?
CI systems can be hacked.
If it’s in S3 or on a server somewhere that the CI can access, can others?
Do you audit access to this?
In the midst of a security incident, this may allow an unacceptable delay to roll a pivotal secret, or mean that your application “goes down” or has degraded functionality until the build/deploy completes.
If the key is something critical (e.g. database), you can’t do a rolling deploy.
Lost your DB creds? You can recover them in var/cache/prod/appProdContainer.php!
Great idea until the framework actually gained traction and then become subject to every hacker and their dog knowing how to do this.
Want to add a pre-prod? Or test a new feature on prod-like infra?
New deployments now involve: updating your CI & build process, building a new artifact, updating your deployment process & tools
How sure are you that the issue doesn’t exist solely on production?
Can developers drag the exact artifact down to their machine to replicate?
Wow this project was inconsistent with config values and quotes
In symfony we think of app environments, and try to keep agnostic to deployment environments (test server is still prod app environment), but for this, consider deployment envs.
Biggest challenge is dev environments where setting envvars is traditionally hard
I’ve talked a bit about security, but it’s not really. And her’es the big gotcha
Has the same flaws as any other parameter, plus a few new ones