Ubuntu upgrade broke django mod_wsgi

Question

I'm banging my head here and can't seem to find the answers anywhere.

My server is running Ubuntu 14.04. I run several django sites using apache and mod_wsgi Today after an update, the sites will not work.

I can see that libapache2-mod-wsgi:amd64 (3.4-4ubuntu2.1.14.04.1, 3.4-4ubuntu2.1.14.04.2) is one of the updates

This error is appearing in the apache error log

mod_wsgi (pid=6624): Failure to configure the daemon process correctly and process left in unspecified state. Restarting daemon process after delay. 

Whilst this one appears in a specific django app log

Fatal Python error: PyEval_AcquireThread: NULL new thread state <...> Script timed out before returning headers: wsgi.py 

I can't remember if I install mod_wsgi using pip or not.

Any pointers would be great.

Answer 1

Just in case you haven't solved this yet:

In mod_wsgi 2.0+, you need to specify the home directory if using the WSGIDaemonProcess directive. In my case I was using a user without a home directory, and that was causing the same error you saw.

home=directory

Defines an absolute path of a directory which should be used as the initial current working directory of the daemon processes within the process group. If this option is not defined, in mod_wsgi 1.X the current working directory of the Apache parent process will be inherited by the daemon processes within the process group. Normally the current working directory of the Apache parent process would be the root directory. In mod_wsgi 2.0+ the initial current working directory will be set to be the home directory of the user that the daemon process runs as.

https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess

Answer 2

It appears that mod_wsgi will fail to start if the daemon does not have permissions to its working directories. From the source code:

if (wsgi_setup_access(daemon) == -1) { /* * If we get any failure from setting up the appropriate * permissions or working directory for the daemon process * then we exit the process. Don't die immediately to avoid * a fork bomb. */ ap_log_error(APLOG_MARK, APLOG_ALERT, 0, wsgi_server, "mod_wsgi (pid=%d): Failure to configure the " "daemon process correctly and process left in " "unspecified state. Restarting daemon process " "after delay.", getpid()); sleep(20); wsgi_exit_daemon_process(-1); } 

Answer 3

The behaviour around the current working directory for daemon mode has changed over time.

Way back in time, whatever Apache set current working directory to would be used. This would usually be '/'.

It then got changed to try and use the working directory of the user that the daemon process ran as. If that user didn't have a valid home directory as is sometimes the case for default Apache user, it would silently fail and you would still be left with it as being at '/'.

When some changes were made as part of a cleanup related to security issues, the silent failure was removed and it was inadvertently made a hard error if the user didn't have a home directory. This caused problem as it wasn't realised the Apache user sometimes didn't have a valid home directory.

This triggered a further change such that the home directory would only be used if the user the daemon process runs as is not the default Apache user. If you were explicitly setting the user for the daemon process to a non Apache user and it didn't have a home directory, you would have to take action to set the 'home' option.

It therefore sounds like you are not using the latest mod_wsgi version, but a version which has the second to last behaviour.

Because some Linux distributions back ported some of the related changes to much older mod_wsgi versions when it wasn't even a security issue that required the changes be back ported, they have created somewhat of a mess as they never picked up the last change to address that Apache user doesn't have a home directory.

So ultimately your problem is caused by your Linux distribution using an out of date mod_wsgi and possibly back ported incomplete sets of changes.

The workaround for you is going to be to set home='/' as option to WSGIDaemonProcess.

Better still, somehow upgrade to the latest mod_wsgi version instead of the ancient version your Linux distribution uses.

The latest version of mod_wsgi 4.4.8. Your Linux distribution ships a version which is over 20 versions behind so you are missing out on many bug fixes and improvements.