Way to configure apache for ~125 django sites to optimize memory usage (mod_python v mod_wsgi; worker vs prefork; static files)

Question

I'm currently running 125 Django sites on a single dedicated box at GoDaddy. The sites are relativelly low traffic and I'm wondering what i can do to optimize the memory usage in my apache config. Before I tuned the prefork directives to a lower MaxServer and MaxrequestsPerChild, the box would hang after about 5 hours of being live.

I've been scowering google for answers, but I can't come up with anything definite for the following:

• Are there any arguments for mod_python vs mod_wsgi in multi site django installs
• Is there a way to optimize static file serving to conserve memory in apache virtual hosts? (/site_media/ vs media.domain.com)
• What is the argument for prefork vs worker and memory usage in a multi VirtualNameHost environment?

Current config file included below:

<VirtualHost *:80> ServerAdmin [email protected] DocumentRoot /var/www/projectroot/django/chat/static ServerName domain.com ServerAlias www.domain.com Alias /media /usr/lib/python2.5/site-packages/django/contrib/admin/media ErrorLog logs/www.domain.com-error_log CustomLog logs/www.domain.com-access_log common <Location "/"> SetHandler python-program PythonHandler django.core.handlers.modpython PythonPath "['/var/www/domain/django'] + sys.path" SetEnv DJANGO_SETTINGS_MODULE chat.settings_domain1 #chat is the name of the project all the sites reside in PythonDebug Off </Location> <Location "/media"> SetHandler None </Location> <Location "/site_media"> SetHandler None </Location> </VirtualHost> 

Thanks! -Tom

Answer 1

I'd definitely go for mod_wsgi. It allows you to define users, number of threads/processes on a per application basis.

I'm not quite sure about the memory requirements but mod_python is considered inferior to mod_wsgi on just about every FAQ or hint you see. WSGIDaemonProcess allows you to configure a lot of options, stacksize, number of processes and the different timeouts may be of interest to you.

I have no experience with GoDaddy so I can't tell you about how far you can go configuring everything.

For the apache part I'd definitely use prefork with the right numbers (depends on your expected user numbers how many childs you want to allow)

For static hosting you could disable all the handlers and even force a certain MIME-Type so that the configuration will just work.

If memory is your bottleneck you might want to check on ngninx from my experience (not that much) memory usage can be predicted a lot better with nginx than with apache, I have no idea about mod-wsgi + ngninx however.

Answer 2

It would definitely be wise to avoid mod_python. That said, don't use mod_wsgi embedded mode either. The reasons why embedded mode is bad are detailed in:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

This is possibly the problem you were encountering.

You can use mod_wsgi daemon mode. It would be preferable though that your Django sites be thread safe as that would allow you to minimise number of daemon processes overall.

The claim that mod_wsgi daemon mode is not compatible with Apache prefork is wrong. You can use mod_wsgi daemon mode with any of the Apache MPMs. The issue is whether the underlying Apache Runtime Library has threading capabilities built in to it, not whether prefork or worker MPM is used. This would normally be the case and only system where it isn't is older BSD systems which had borked threading implementations. Either way, you are still better of using worker MPM as reduces number of Apache server child processes needed.

Anyway, using mod_wsgi daemon mode should be relatively straight forward. The issue will be determining if your Django sites are thread safe or not.

Answer 3

Edit: Disregard the first paragraph. See Graham's comment and answer.

Apache prefork is not compatible with wsgi daemon mode. Per the documentation, daemon mode requires threading. So if you want the flexibility of daemon mode you need to use Apache mpm worker.

Based on this blog post, I wouldn't use nginx's mod_wsgi.

So my preferred setup would be nginx as a lightweight, front-end/static file server with Apache mpm worker and mod_wsgi in daemon mode serving the Django content.