How to configure nginx to start if upstream host is down when using upstream blocks

Question

We use nginx as a reverse proxy to forward requests to our various servers. Currently, if any of those upstream servers are down/unavailable, nginx will not start. This is frustrating, since it means if any of our upstream servers are temporarily down, our entire system becomes unavailable.

We'd like to configure nginx such that it will start successfully even when one or more of the upstream servers are down.

I stumbled upon https://sandro-keil.de/blog/let-nginx-start-if-upstream-host-is-unavailable-or-down/ , which, if I understand correctly, says that using setting the location to be proxy_passed as a variable moves the hostname resolution to request time instead of startup time, which resolves the problem. However, the examples in that post and that I've been able to find elsewhere don't describe how to implement this "workaround" when using upstream configuration blocks to define multiple upstream servers, e.g. for load balancing. Unfortunately, since my understanding of the nginx startup and request processing phases and details are minimal, I'm not sure how to adapt the example "workaround" for when upstream blocks are used.

Will this approach still work to allow nginx to start when upstream server(s) are down? If so, how and where do I set and reference the variable? If not, how can I accomplish this?

Our (abbreviated) nginx configuration:

upstream webserver { server webserver:8080; } upstream websocketserver { server websocketserver:8080; } server { ... # Reverse-proxy web requests to the web server(s) location / { proxy_pass http://webserver; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } # Reverse-proxy Server-Push requests to the WebSocket server(s) location /websocket/ { proxy_pass http://websocketserver; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Prefix /websocket; } } 

(Failed) attempt 1, setting the variable in the upstream block:

upstream webserver { set $upstream_webserver webserver:8080 server $upstream_webserver; } server { ... # Reverse-proxy web requests to the web server(s) location / { proxy_pass http://webserver; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } } 

results in

root@039266c002d9:/# nginx -t 2024/10/11 17:29:06 [emerg] 40#40: "set" directive is not allowed here in /etc/nginx/conf.d/psif.conf:9 nginx: [emerg] "set" directive is not allowed here in /etc/nginx/conf.d/app.conf:9 nginx: configuration file /etc/nginx/nginx.conf test failed 

(Failed) attempt 2, setting the variable in the server block (to be able to reuse it in the multiple location blocks):

upstream webserver { server webserver:8080; } server { set $upstream_webserver webserver ... # Reverse-proxy web requests to the web server(s) location / { proxy_pass http://$upstream_webserver; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } } 

results in

root@039266c002d9:/# nginx -t 2024/10/11 17:31:44 [emerg] 41#41: host not found in upstream "webserver:8080" in /etc/nginx/conf.d/app.conf:7 nginx: [emerg] host not found in upstream "webserver:8080" in /etc/nginx/conf.d/app.conf:7 nginx: configuration file /etc/nginx/nginx.conf test failed 

How can I configure nginx to start when the upstream server(s) are down while using upstream configuration blocks?

Answer 1

upstream { ... server webserver:8080 max_fails=0 resolve ; ... } 

Nginx uses Passive Health Checks whether or not you configure Active Health Checks. Using the max_fails=0 (zero means Unlimited failures) in this case will stop the 502's... however it could introduce a new problem as you would be telling Nginx "hey no matter what just send the request..." which means it will continue routing even if there aren't any Live Upstreams!!!

After reading your post a little closer, I notice you might also benefit from slow_start=30s or something if your servers restart frequently or randomly.. you can change that to proper number of seconds for your env.

Note usage instructions on all of the above from: https://nginx.org/en/docs/http/ngx_http_upstream_module.html