Background
An HTTP request sent by an end-user to www.stuff.com/things is processed as follows:
- Received by an application Load Balancer (the public load balancer) that load balances to an NGINX reverse proxy (Let's call this the security proxy).
- The security proxy routes the request to a private application load balancer.
- The private load balancer routes traffic based on the URL. if the url is
/things/...then the request is routed to another NGINX proxy. - This second NGINX proxy (lets call this the
thingsproxy) routes all requests to a classic load balancer. - The classic load balancer listens on ports 80 and 50000 and balances requests to instances hosting the
thingsapplication.
So to recap: I have the public load balancer, the security proxy, the private load balancer, the things proxy, and the classic load balancer.
The problem
When a user tries to visit www.stuff.com/things, they wait, and they wait, and they eventually receive an HTTP 504: gateway timeout. Or, the page loads immediately. Or, some elements load quickly while other hang.
Debugging
I've tried issuing requests from several places in the chain:
curlthe classic load balancer, response is instantaneouscurlthethingsproxy, response is instantaneouscurlthe private load balancer, response is instantaneouscurlthe security proxy, response is instantaneouscurlthe public load balancer, response is slooooow... except when it's fast.
Perhaps I need to reconfigure my proxies?
Proxy Configuration
The things proxy is configured as follows:
events { worker_connections 1024; } http { server { location ^~ /proxycheck { return 200 'available'; add_header Content-Type text/plain; } location / { proxy_pass ${PROXY_ADDRESS}; } } } The security proxy is configured as follows:
events { worker_connections 1024; } http { server { location ^~ /proxycheck { return 200 'available'; add_header Content-Type text/plain; } location / { proxy_pass ${PROXY_ADDRESS}; proxy_read_timeout 90; proxy_redirect default; proxy_set_header Host $host:$server_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; } } } In the future, the security proxy will check certificates. Until then, I've got this other problem.
Any suggestions?
thingsproxy is required because I can't connect a load balancer to a load balancer (AFAIK, AWS doesn't provide a means of doing this). The public load balancer is required to balance load across security proxies, the private load balancer balances for applications (all butthings, where it balances those proxies), and thethingsload balancer is required because an AWS application load balancer doesn't support non-http health checks or more than a single port.