Having just setup a project that is essentially identical to what you describe, I'll share my approach - no guarantees that it is 'the best', but it does work.
My server stack is
- Varnish (v3.0.2) - all interfaces, port 80
- Nginx (v1.0.14) - local interface, port 81
- Node.js (v0.6.13) - local interface, port 1337
- Operating system is CentOS 6.2 (or similar)
My Node.js app uses Websockets (sockets.io - v0.9.0) and Express (v2.5.8) - and is launched using forever. (The same server also has other sites on it - primarily PHP which use the same instances of Nginx and Varnish).
The basic intention of my approach is as follows:
- Single public port/address for both websocket and 'regular' data
- Cache some assets using Varnish
- Serve (uncached) static assets directly from nginx
- Pass requests for 'web pages' to nginx, and from their proxy to Node.js
- Pass web socket requests directly (from Varnish) to Node.js (bypass nginx).
Varnish config - /etc/varnish/default.vcl:
#Nginx - on port 81 backend default { .host = "127.0.0.1"; .port = "81"; .connect_timeout = 5s; .first_byte_timeout = 30s; .between_bytes_timeout = 60s; .max_connections = 800; } #Node.js - on port 1337 backend nodejs{ .host = "127.0.0.1"; .port = "1337"; .connect_timeout = 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 60s; .max_connections = 800; } sub vcl_recv { set req.backend = default; #Keeping the IP addresses correct for my logs if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } #remove port, if included, to normalize host set req.http.Host = regsub(req.http.Host, ":[0-9]+", ""); #Part of the standard Varnish config if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } #Taken from the Varnish help on dealing with Websockets - pipe directly to Node.js if (req.http.Upgrade ~ "(?i)websocket") { set req.backend = nodejs; return (pipe); } ###Removed some cookie manipulation and compression settings## if(req.http.Host ~"^(www\.)?example.com"){ #Removed some redirects and host normalization #Requests made to this path, even if XHR polling still benefit from piping - pass does not seem to work if (req.url ~ "^/socket.io/") { set req.backend = nodejs; return (pipe); } #I have a bunch of other sites which get included here, each in its own block }elseif (req.http.Host ~ "^(www\.)?othersite.tld"){ #... } #Part of the standard Varnish config if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } #Everything else, lookup return (lookup); } sub vcl_pipe { #Need to copy the upgrade for websockets to work if (req.http.upgrade) { set bereq.http.upgrade = req.http.upgrade; } set bereq.http.Connection = "close"; return (pipe); } #All other functions should be fine unmodified (for basic functionality - most of mine are altered to my purposes; I find that adding a grace period, in particular, helps.
Nginx config - /etc/nginx/*/example.com.conf:
server { listen *:81; server_name example.com www.example.com static.example.com; root /var/www/example.com/web; error_log /var/log/nginx/example.com/error.log info; access_log /var/log/nginx/example.com/access.log timed; #removed error page setup #home page location = / { proxy_pass http://node_js; } #everything else location / { try_files $uri $uri/ @proxy; } location @proxy{ proxy_pass http://node_js; } #removed some standard settings I use } upstream node_js { server 127.0.0.1:1337; server 127.0.0.1:1337; }
I am not particularly crazy about the repetition of the proxy_pass statement, but haven't gotten around to finding a cleaner alternative yet, unfortunately. One approach may be to have a location block specifying the static file extensions explicitly and leave the proxy_pass statement outside of any location block.
A few settings from /etc/nginx/nginx.conf:
set_real_ip_from 127.0.0.1; real_ip_header X-Forwarded-For; log_format timed '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' '$request_time $upstream_response_time $pipe'; port_in_redirect off;
Among my other server blocks and settings, I also have gzip and keepalive enabled in my nginx config. (As an aside, I believe there is a TCP module for Nginx which would enable the use of websockets - however, I like using 'vanilla' versions of software (and their associated repositories), so that wasn't really an option for me).
A previous version of this setup resulted in an unusual 'blocking' behaviour with the piping in Varnish. Essentially, once a piped socket connection was established, the next request would be delayed until the pipe timed out (up to 60s). I haven't yet seen the same recur with this setup - but would be interested to know if you see a similar behaviour.
