I am running an apache2 server with several virtual hosts. I run it on a rather resource constrained raspberry pi 4. In particular RAM is very tight (4GB). There isn't a whole lot of traffic, I mainly use it for myself to host nextcloud, a custom streaming website, and some static files.
The problem is the static files. Whenever I, or anyone else, downloads a file, it seems like the whole file is loaded into RAM. This isn't a problem for small files, but anything > 500MB causes serious issues, often filling up SWAP and eventually causing an OOM kill of apache.
I can not figure out for the life of me how to make it simply stream the files from disk, instead of caching them in memory before sending them. That might be a viable thing to do if you have 128GB of RAM on a proper server, but it's killing me on my poor pi.
So how do I make apache just stream the file from disk? And why does it load the file to ram anyway? disk i/o is way faster than network, so there is no point caching the file from disk to memory.
I have tried:
- switching from mpm_prefetch to mpm_event.
EnableSendfile On
, which doesn't seem to do anything.- mod_xsendfile with
XSendFile On
- Looking for other solutions, including
- this one, not applicable because a single worker is using all the RAM for loading a single huge file into RAM
- this one, not applicable because even though I do run PHP, the problem is with get requests to static files on disk, not PHP. Also can't install more RAM.
- this one, where I don't even know what the heck they are talking about with that config file. Also again, not a PHP issue.
- Many more talking about large uploads to PHP and PUT requests, and other PHP stuff.
I would like to avoid doing a reverse proxy to nginx hosted files if possible, as that would just increase the complexity of everything.
One of my virtual hosts, for which the issue exists:
<IfModule mod_ssl.c> <VirtualHost *:443> ServerName host.tld ServerAdmin [email protected] DocumentRoot /var/www/web CustomLog ${APACHE_LOG_DIR}/access.log combined # Generate more pleasing and modern looking listing IndexOptions FancyIndexing IconsAreLinks FoldersFirst EnableSendfile On EnableMMAP Off <IfModule mod_xsendfile.c> <Directory "/var/www/web"> XSendFile On XSendFilePath /var/www/web </Directory> </IfModule> <IfModule mod_rewrite.c> RewriteEngine on RewriteRule ^/\.well-known/carddav /nextcloud/remote.php/dav [R=301,L] RewriteRule ^/\.well-known/caldav /nextcloud/remote.php/dav [R=301,L] RewriteRule ^/\.well-known/webfinger /nextcloud/index.php/.well-known/webfinger [R=301,L] RewriteRule ^/\.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo [R=301,L] RewriteRule ^/ocm-provider/?$ index.php [QSA,L] </IfModule> <IfModule mod_headers.c> # disallow embedding of non https Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains" # Prevents browsers from interpreting text files as javascript Header onsuccess unset X-Content-Type-Options Header always set X-Content-Type-Options "nosniff" # enable browser side cross site script filter Header onsuccess unset X-XSS-Protection Header always set X-XSS-Protection "1; mode=block" # prevent search engines from indexing the site Header onsuccess unset X-Robots-Tag Header always set X-Robots-Tag "noindex, nofollow" # prevent embedding in other sites Header onsuccess unset X-Frame-Options Header always set X-Frame-Options "SAMEORIGIN" Header onsuccess unset X-Permitted-Cross-Domain-Policies Header always set X-Permitted-Cross-Domain-Policies "none" # prevent browser from sending referrer data when clicking links Header onsuccess unset Referrer-Policy # Header always set Referrer-Policy "no-referrer" Header always set Referrer-Policy "same-origin" </IfModule> # Private, password protected directory <Location "/private"> # authentication AuthType basic AuthName "My Server" # Cache credentials with socache AuthBasicProvider socache dbd # also needed for caching: tell cache to cache dbd lookups AuthnCacheProvideFor dbd AuthDBDUserPWQuery "SELECT password FROM authn WHERE user = %s" # block all non users. Require valid-user </Location> <Location /nextcloud/> Require all granted AllowOverride All Options FollowSymLinks MultiViews <IfModule mod_dav.c> Dav off </IfModule> # Disable Basic auth for this Satisfy Any </Location> # SSL settings SSLEngine on SSLCertificateFile /etc/letsencrypt/live/host.tld/cert.pem SSLCertificateChainFile /etc/letsencrypt/live/host.tld/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/host.tld/privkey.pem #Include /etc/letsencrypt/options-ssl-apache.conf # Add vhost name to log entries: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common </VirtualHost> </IfModule> # vim: syntax=apache ts=4 sw=4 sts=4 sr et
My enabled modules:
access_compat.load authn_core.load authz_core.load autoindex.load dir.conf headers.load mpm_event.load proxy.load reqtimeout.conf security3.load socache_shmcb.load status.load alias.conf authn_dbd.load authz_host.load dbd.load dir.load mime.conf negotiation.conf proxy_fcgi.load reqtimeout.load setenvif.conf ssl.conf unique_id.load alias.load authn_file.load authz_user.load deflate.conf env.load mime.load negotiation.load proxy_http.load rewrite.load setenvif.load ssl.load xsendfile.load auth_basic.load authn_socache.load autoindex.conf deflate.load filter.load mpm_event.conf proxy.conf proxy_wstunnel.load security3.conf socache_memcache.load status.conf
mpm_event.conf:
<IfModule mpm_event_module> StartServers 2 MinSpareThreads 25 MaxSpareThreads 75 ThreadLimit 64 ThreadsPerChild 25 MaxRequestWorkers 150 MaxConnectionsPerChild 0 </IfModule>
apachectl -V:
Server built: 2024-07-17T18:57:26 Server's Module Magic Number: 20120211:126 Server loaded: APR 1.7.0, APR-UTIL 1.6.1 Compiled using: APR 1.7.0, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_PROC_PTHREAD_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/etc/apache2" -D SUEXEC_BIN="/usr/lib/apache2/suexec" -D DEFAULT_PIDLOG="/var/run/apache2.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="mime.types" -D SERVER_CONFIG_FILE="apache2.conf"
/proc/meminfo when downloading a 2GB static file with curl. Text is when the ssh connection died approximately 10 seconds after starting the download. By that time not a single byte was received by curl yet.
MemTotal: 3880848 kB MemFree: 190000 kB MemAvailable: 479864 kB Buffers: 52840 kB Cached: 458780 kB SwapCached: 63000 kB Active: 502232 kB Inactive: 2665432 kB Active(anon): 318696 kB Inactive(anon): 2479580 kB Active(file): 183536 kB Inactive(file): 185852 kB Unevictable: 26756 kB Mlocked: 26756 kB SwapTotal: 4095996 kB SwapFree: 2418916 kB Dirty: 52 kB Writeback: 192 kB AnonPages: 2626292 kB Mapped: 172496 kB Shmem: 134752 kB KReclaimable: 292096 kB Slab: 411324 kB SReclaimable: 292096 kB SUnreclaim: 119228 kB KernelStack: 8768 kB PageTables: 22876 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6036420 kB Committed_AS: 8174592 kB VmallocTotal: 133143592960 kB VmallocUsed: 27632 kB VmallocChunk: 0 kB Percpu: 3216 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB CmaTotal: 65536 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB
cat /proc/meminfo
while doing a large file transfer. This look into kernel memory accounting will answer if it is private allocations or something else.