I have had countless conversations with developers, projects managers, and even executives that end up being about nginx and what it can do. Usually, the phrase "nginx can do that?" comes up. More often than not, the answer is YES. What happens though, is the nginx config file can get unwieldy. How can we assert that it will behave how it needs to over time? How can we avoid introducing inadvertent regressions?

1. NGINX CAN DO THAT?
TEST DRIVE YOUR CONFIG FILE!
Presented By Jeff Anderson / @programm3rq / programmerq

2. MY BACKGROUND
Linux/Apache since middle school. webdev since I was 10.
Worked with Apache professionally since my first job in the
industry.
Started using nginx for my first project when I started at HP
back in 2010.
I have been using nginx every chance since.

3. NGINX CAN DO THAT?
More often than not... YES!

4. STORY TIME
http://www.example.com/ server layout diagram.

5. http://www.example.com/ nginx config file.
upstream backend {
server app1:8000;
server app2:8000;
server app3:8000;
} server {
server_name www.example.com;
location / {
proxy_pass http://backend;
}
}

6. But a simple nginx config file can be added to.
You can only keep your config file simple for so long while taking
on additional features.

7. REQUIREMENTS
Multi-environment (staging, production, dev, test, qa, etc...)
Caching - some content will be cached by nginx
IP-based Access Control - some URLs will be protected
Maintenance Page - for site maintenance (scheduled or
otherwise)
Multiple upstreams - feature B is done by team X in language Y

8. REQUIREMENTS (CONTINUED)
NON-PRODUCTION-SPECIFIC REQUIREMENTS
Caching - developers want to toggle caching features for
development.
Maintenance Page - we won't ever use this feature outside
production.
Keep out the Internet! - Our QA team does not have VPN
access, but needs to access our site still. Basic Auth.

9. REQUIREMENTS (CONTINUED)
PRODUCTION-SPECIFIC REQUIREMENTS (STAGING TOO)
Additional Nodes - More traffic and real customers means
more resources.
Geographic Redundancy - Two or more datacenters.
CDN - Our nginx servers will be behind Akamai or another
CDN to speed up asset delivery.
SSL - https needs to be on.

10. REQUIREMENTS (CONTINUED)
THESE TRICKLE IN AS PEOPLE FIND THINGS BROKEN.
Maintenance Page (cont'd) - Make sure the maintenance page
does not block /admin/.
Basic Auth - Should not block /api/callbacks/.
Block some URLs - Appserver URLs for healthchecks should
not be served via nginx.

11. TESTING SOFTWARE
"We will write tests before we code, minute
by minute. We will preserve these tests
forever, and run them all together frequently.
We will also derive tests from the customer's
perspective."
- Kent Beck "eXtreme Programming
explained" Chapter 18 heading

12. TEST ALL THE THINGS!

13. WRITING TESTS IS AWESOME!

14. TEST NGINX?

15. CONTINUOUS INTEGRATION

16. CONTINUOUS INTEGRATION
I have the most experience with Jenkins.

17. INFRASTRUCTURE TESTS

18. INFRASTRUCTURE TESTS
Test Kitchen (aka KitchenCI) was written to enable automated
testing of chef cookbooks and recipes.
Runs tests inside virtualbox, vmware, aws, digitalocean,
docker, and more
Can run chef, puppet, salt, and more
ability to use Bats, shUnit2, rspec, serverspec, and more

19. BACK TO OUR ORIGINAL EXAMPLE:
http://www.example.com/ server layout diagram.

20. BACK TO OUR ORIGINAL EXAMPLE:
http://www.example.com/ nginx config file.
upstream backend {
server app1:8000;
server app2:8000;
server app3:8000;
} server {
server_name www.example.com;
location / {
proxy_pass http://backend;
}

21. How do we make this support multiple environments?

22. CONFIG GENERATORS
What if we want to dynamically build the config file?
upstream backend {
<% upstreams.each do |host| %>
server <%= host %>;
<% end %>
} server {
server_name <%= server_name %>;
location / {
proxy_pass http://backend;
}
}

23. TESTING STRATEGY 1
Mock data to be used by the config generator
run nginx -t against the result.
make assertions with regex
compare all generated output to a "golden image" (approvals)

24. Mock data
[{
'environment': 'production',
'server_name': 'www.example.com',
'upstreams': ['app1:8000',
'app2:8000',
'app3:8000']},
{
'environment': 'staging',
'server_name': 'staging.example.com',
'upstreams': ['stage_app1:8000',
'stage_app2:8000',
'stage_app3:8000']}
]

25. That should produce the following configuration files:
upstream backend {
server app1:8000;
server app2:8000;
server app3:8000;
} server {
server_name www.example.com;
location / {
proxy_pass http://backend;
}
}
upstream backend {
server stage_app1:8000;
server stage_app2:8000;
server stage_app3:8000;
}
server {
server_name staging.example.com;
location / {
proxy_pass http://backend;
}
}

26. WORKFLOW FOR STRATEGY #1
Check in golden copies.
Make a change to the template.
If you introduce a syntax error, you are notified.
Test will fail if output is different than golden.
Examine differences between the new and the golden.
"Approve" the changes by checking in the changes as the new
golden.
This is known as "approvals" testing.

27. A MORE COMPLEX EXAMPLE:
Mock data
[{
'environment': 'production',
'server_name': 'www.example.com',
'upstreams': ['app1:8000',
'app2:8000',
'app3:8000'],
'no_access_locations': ['/api/healthcheck'],
'basic_auth': false,
'basic_auth_excluded_paths': []},
{
'environment': 'staging',
'server_name': 'staging.example.com',
'upstreams': ['stage_app1:8000',
'stage_app2:8000',
'stage_app3:8000'],
'no_access_locations': ['/api/healthcheck'],
'basic_auth': {'file': '.htpasswd'},
'basic_auth_excluded_paths': ['/api/callback', '/api/rss']
}]

28. A MORE COMPLEX EXAMPLE:
upstream backend {
<% upstreams.each do |host| %>
server <%= host %>;
<% end %>
} server {
server_name <%= server_name %>;
error_page 503 /maintenance.html;
error_page 502 /maintenance.html;
error_page 500 /500.html;
error_page 504 /504.html;
error_page 404 /404.html;
root /www/<%= server_name %>;
proxy_intercept_errors on;
proxy_connect_timeout 2;
<% no_access_locations.each do |blocked| %>
location ^~ <%= blocked %> {
return 404;
}
<% end %>
location / {
if ( -f $request-filename ) {
break;
}
if ( -e $document_root/maintenance-on) {
return 503;
}
try_files $uri @backend;
<% if basic_auth %>

29. TESTING STRATEGY 2
Generate the config.
Actually turn on nginx.
send requests to it and assert it does the right thing.
These are integration tests.

30. INTEGRATION TESTING
Assert that ip-address-based access works.
{"server_name": "test-app.example.com",
"ip_access": [{"127.0.0.1": ["/admin"]}]}
assert `curl -kIs -H 'Host: test-app.example.com' http://192.168.0.2/admin
| head -n 1 | awk '{print $2}'` == 404
assert `curl -kIs -H 'Host: test-app.example.com' http://127.0.0.1/admin
| head -n 1 | awk '{print $2}'` == 502

31. INTEGRATION TESTING
Assert that the existence of
$document_root/maintenance-on turns on the
maintenance page.
rm -f /www/maintenance-on
assert `curl -kIs -H 'Host: example.com' http://127.0.0.1/
| head -n 1 | awk '{print $2}'` == 502
touch /www/maintenance-on
assert `curl -kIs -H 'Host: example.com' http://127.0.0.1/
| head -n 1 | awk '{print $2}' == 503
rm -f /www/maintenance-on

32. ADDITIONAL METHODS OF TESTING
more integration options
run nginx using strace and examine strace output.
run nginx inside a proxifier and intercept nginx upstream
requests.
configure nginx to use a dummy webserver and examine its
upstream requests.

33. LOOKING FORWARD
Using a web server testing framework instead of curl requests.
"native" testing framework - maybe an extension that runs
inside nginx?
Something else? maybe there's an even better method that I
have not even thought of.

34. DEMO TIME
REQUIREMENTS
There are three separate applications that share an
authentication service.
The NOC must ssh in to 10 different nginx servers and touch
/www/maintenance-on.
Management wants maintenance page up faster during an
outage.

35. BUTTONS!

36. POSSIBLE IMPLEMENTATIONS
admin.example.com to ssh in to nginx servers?...Nope.
admin.example.com to trigger chef runs?...Nope.
admin.example.com to hit nginx directly?...Yes!

37. HITTING NGINX
I want to touch /www/maintenance-on
I want only admin.example.com's server to be able to do it.
Webdav PUT/DELETE seems like a good fit.
Security team wants webdav to appear disabled to scanners.

38. THE END
Presented By Jeff Anderson / @programm3rq / programmerq
Slides created using reveal.js -
Jenkins -
Chef -
Berkshelf -
Test Kitchen / KitchenCI -
BATS -
Docker -
cURL -
nginx -
http://lab.hakim.se/reveal-js/#/
http://jenkins-ci.org/
https://www.getchef.com/
http://berkshelf.com/
http://kitchen.ci/
https://github.com/sstephenson/bats
https://www.docker.com/
http://curl.haxx.se/
http://nginx.org/