NGINX Can Do That? Test Drive Your Config File!

NGINX CAN DO THAT? TEST DRIVE YOUR CONFIG FILE! Presented By Jeff Anderson / @programm3rq / programmerq

MY BACKGROUND Linux/Apache since middle school. webdev since I was 10. Worked with Apache professionally since my first job in the industry. Started using nginx for my first project when I started at HP back in 2010. I have been using nginx every chance since.

NGINX CAN DO THAT? More often than not... YES!

STORY TIME http://www.example.com/ server layout diagram.

http://www.example.com/ nginx config file. upstream backend { server app1:8000; server app2:8000; server app3:8000; } server { server_name www.example.com; location / { proxy_pass http://backend; } }

But a simple nginx config file can be added to. You can only keep your config file simple for so long while taking on additional features.

REQUIREMENTS Multi-environment (staging, production, dev, test, qa, etc...) Caching - some content will be cached by nginx IP-based Access Control - some URLs will be protected Maintenance Page - for site maintenance (scheduled or otherwise) Multiple upstreams - feature B is done by team X in language Y

REQUIREMENTS (CONTINUED) NON-PRODUCTION-SPECIFIC REQUIREMENTS Caching - developers want to toggle caching features for development. Maintenance Page - we won't ever use this feature outside production. Keep out the Internet! - Our QA team does not have VPN access, but needs to access our site still. Basic Auth.

REQUIREMENTS (CONTINUED) PRODUCTION-SPECIFIC REQUIREMENTS (STAGING TOO) Additional Nodes - More traffic and real customers means more resources. Geographic Redundancy - Two or more datacenters. CDN - Our nginx servers will be behind Akamai or another CDN to speed up asset delivery. SSL - https needs to be on.

REQUIREMENTS (CONTINUED) THESE TRICKLE IN AS PEOPLE FIND THINGS BROKEN. Maintenance Page (cont'd) - Make sure the maintenance page does not block /admin/. Basic Auth - Should not block /api/callbacks/. Block some URLs - Appserver URLs for healthchecks should not be served via nginx.

TESTING SOFTWARE "We will write tests before we code, minute by minute. We will preserve these tests forever, and run them all together frequently. We will also derive tests from the customer's perspective." - Kent Beck "eXtreme Programming explained" Chapter 18 heading

CONTINUOUS INTEGRATION I have the most experience with Jenkins.

INFRASTRUCTURE TESTS Test Kitchen (aka KitchenCI) was written to enable automated testing of chef cookbooks and recipes. Runs tests inside virtualbox, vmware, aws, digitalocean, docker, and more Can run chef, puppet, salt, and more ability to use Bats, shUnit2, rspec, serverspec, and more

BACK TO OUR ORIGINAL EXAMPLE: http://www.example.com/ server layout diagram.

BACK TO OUR ORIGINAL EXAMPLE: http://www.example.com/ nginx config file. upstream backend { server app1:8000; server app2:8000; server app3:8000; } server { server_name www.example.com; location / { proxy_pass http://backend; }

How do we make this support multiple environments?

CONFIG GENERATORS What if we want to dynamically build the config file? upstream backend { <% upstreams.each do |host| %> server <%= host %>; <% end %> } server { server_name <%= server_name %>; location / { proxy_pass http://backend; } }

TESTING STRATEGY 1 Mock data to be used by the config generator run nginx -t against the result. make assertions with regex compare all generated output to a "golden image" (approvals)

Mock data [{ 'environment': 'production', 'server_name': 'www.example.com', 'upstreams': ['app1:8000', 'app2:8000', 'app3:8000']}, { 'environment': 'staging', 'server_name': 'staging.example.com', 'upstreams': ['stage_app1:8000', 'stage_app2:8000', 'stage_app3:8000']} ]

That should produce the following configuration files: upstream backend { server app1:8000; server app2:8000; server app3:8000; } server { server_name www.example.com; location / { proxy_pass http://backend; } } upstream backend { server stage_app1:8000; server stage_app2:8000; server stage_app3:8000; } server { server_name staging.example.com; location / { proxy_pass http://backend; } }

WORKFLOW FOR STRATEGY #1 Check in golden copies. Make a change to the template. If you introduce a syntax error, you are notified. Test will fail if output is different than golden. Examine differences between the new and the golden. "Approve" the changes by checking in the changes as the new golden. This is known as "approvals" testing.

A MORE COMPLEX EXAMPLE: Mock data [{ 'environment': 'production', 'server_name': 'www.example.com', 'upstreams': ['app1:8000', 'app2:8000', 'app3:8000'], 'no_access_locations': ['/api/healthcheck'], 'basic_auth': false, 'basic_auth_excluded_paths': []}, { 'environment': 'staging', 'server_name': 'staging.example.com', 'upstreams': ['stage_app1:8000', 'stage_app2:8000', 'stage_app3:8000'], 'no_access_locations': ['/api/healthcheck'], 'basic_auth': {'file': '.htpasswd'}, 'basic_auth_excluded_paths': ['/api/callback', '/api/rss'] }]

A MORE COMPLEX EXAMPLE: upstream backend { <% upstreams.each do |host| %> server <%= host %>; <% end %> } server { server_name <%= server_name %>; error_page 503 /maintenance.html; error_page 502 /maintenance.html; error_page 500 /500.html; error_page 504 /504.html; error_page 404 /404.html; root /www/<%= server_name %>; proxy_intercept_errors on; proxy_connect_timeout 2; <% no_access_locations.each do |blocked| %> location ^~ <%= blocked %> { return 404; } <% end %> location / { if ( -f $request-filename ) { break; } if ( -e $document_root/maintenance-on) { return 503; } try_files $uri @backend; <% if basic_auth %>

TESTING STRATEGY 2 Generate the config. Actually turn on nginx. send requests to it and assert it does the right thing. These are integration tests.

INTEGRATION TESTING Assert that ip-address-based access works. {"server_name": "test-app.example.com", "ip_access": [{"127.0.0.1": ["/admin"]}]} assert `curl -kIs -H 'Host: test-app.example.com' http://192.168.0.2/admin | head -n 1 | awk '{print $2}'` == 404 assert `curl -kIs -H 'Host: test-app.example.com' http://127.0.0.1/admin | head -n 1 | awk '{print $2}'` == 502

INTEGRATION TESTING Assert that the existence of $document_root/maintenance-on turns on the maintenance page. rm -f /www/maintenance-on assert `curl -kIs -H 'Host: example.com' http://127.0.0.1/ | head -n 1 | awk '{print $2}'` == 502 touch /www/maintenance-on assert `curl -kIs -H 'Host: example.com' http://127.0.0.1/ | head -n 1 | awk '{print $2}' == 503 rm -f /www/maintenance-on

ADDITIONAL METHODS OF TESTING more integration options run nginx using strace and examine strace output. run nginx inside a proxifier and intercept nginx upstream requests. configure nginx to use a dummy webserver and examine its upstream requests.

LOOKING FORWARD Using a web server testing framework instead of curl requests. "native" testing framework - maybe an extension that runs inside nginx? Something else? maybe there's an even better method that I have not even thought of.

DEMO TIME REQUIREMENTS There are three separate applications that share an authentication service. The NOC must ssh in to 10 different nginx servers and touch /www/maintenance-on. Management wants maintenance page up faster during an outage.

POSSIBLE IMPLEMENTATIONS admin.example.com to ssh in to nginx servers?...Nope. admin.example.com to trigger chef runs?...Nope. admin.example.com to hit nginx directly?...Yes!

HITTING NGINX I want to touch /www/maintenance-on I want only admin.example.com's server to be able to do it. Webdav PUT/DELETE seems like a good fit. Security team wants webdav to appear disabled to scanners.

THE END Presented By Jeff Anderson / @programm3rq / programmerq Slides created using reveal.js - Jenkins - Chef - Berkshelf - Test Kitchen / KitchenCI - BATS - Docker - cURL - nginx - http://lab.hakim.se/reveal-js/#/ http://jenkins-ci.org/ https://www.getchef.com/ http://berkshelf.com/ http://kitchen.ci/ https://github.com/sstephenson/bats https://www.docker.com/ http://curl.haxx.se/ http://nginx.org/

NGINX Can Do That? Test Drive Your Config File!

More Related Content

What's hot

Similar to NGINX Can Do That? Test Drive Your Config File!

Recently uploaded

NGINX Can Do That? Test Drive Your Config File!