Tell Nginx not to cache based on a response header from upstream

Question

I'm using Nginx to serve MediaWiki, and I use FastCGI caching to avoid unnecessary PHP calls.

(In the MediaWiki LocalSettings.php file, I have $wgUseCdn = true; and in the Nginx config, I use fastcgi_cache and related config options.)

Problem is, many HTTP responses from MediaWiki/PHP (aka upstream) contain a Vary header with User-Agent in it. And since user agent strings are so detailed and variable, this probably makes Nginx needlessly cache a ton of identical copies of some pages and rarely ever give a new visitor a cached version.

I can give the Nginx FCGI cache a whole 12 GiB of space, and it fills it all up, but I suspect it's mostly junk due to this problem, and that only a fraction of the cache contents are actually useful.

I'd like to tell the Nginx FastCGI cache to not cache a response if the Vary response header sent back from the MediaWiki PHP code contains User-Agent.

(There are also lots of resources sent back from PHP which don't vary based on user-agent, and Nginx can still cache those.)

I've tried the following, but it doesn't seem to work:

# In the http context: map $sent_http_vary $fcgi_nocache { default 0; "~*user-agent" 1; } # In a location *.php context: add-header X-Fcgi-Nocache $fcgi_nocache; # Or, if it worked fine, to actually make the nocache take effect: fastcgi_no_cache $fcgi_nocache; 

This allows me to check the X-Fcgi-Nocache header received by my HTTP client (e.g. web browser) to see if the map directive does what I want, but the value is always 0 so I guess the $sent_http_vary variable doesn't work the way I hoped it would.

Any way to make this work?

Answer 1

I don't know what I was doing wrong before, but suddenly it's working. Maybe I had typed "*~user-agent" instead of "~*user-agent" or maybe it was a similarly stupid typo, but currently it seems to be working as intended.

That being said, by suggestion of someone in the #nginx IRC channel on Libera, I've now also changed it to use the variable $upstream_http_vary instead of $sent_http_vary. This may be more reliable, although it seemed to be working with the $sent_xyz variant as well.

Also, I've noticed that MediaWiki either specifies no Vary header at all, or one that includes User-Agent, so there's actually no need for the map directive, although it's good to know that it's a possibility.

In short, simple solution:

# In location context, where fastcgi_pass and fastcgi_cache are specified: fastcgi_no_cache $upstream_http_vary; 

More refined solution, if needed, e.g. checking the Vary header for User-Agent:

# In http context, where fastcgi_cache_path is specified: map $upstream_http_vary $fcgi_nocache { default 0; "~*user-agent" 1; } 

# In location context, where fastcgi_pass and fastcgi_cache specified: fastcgi_no_cache $fcgi_nocache;