WordPress + NGINX + PHP-FPM fastcgi + Memcached is extremely slow

Question

I'm at a loss here and would love to get some pointers. I'm running AlmaLinux 9.5 with NGINX with fastcgi to PHP-FPM 8.3. WordPress is extremely slow when using more heavy plugins such as WooCommerce or Divi (looks like Elementor isn't too bad). Measuring time curl https://DOMAIN/wp-admin/admin-ajax.php gives a good approximation of the time it takes wp-load to load, which is needed on all pages. For example, three configurations of plugins:

• WooCommerce + Events Calendar: 0.5s
• Divi: 0.5s
• Bare WordPress: 0.05s

While 0.5s for every page load is already very high (even though considering PHP is slow, I was hoping for at least below 0.1s). However, some of the pages with WooCommerce/EventsCalendar or Divi take 5s to 10s to load in the admin panel, which is absurd. Below each aspect that may influence the speed:

Hardware and configuration

• CPU: 8 cores AMD EPYC 7443 (not CPU bound, never goes above 20%, usually below 5%)
• Memory: 32 GB DDR4 (not memory bound, only 6 GB used, rest is buff/cache)
• Disk: SSD NVMe (not disk bound, usually only a few ioops per sec on average)
• OS: AlmaLinux 9.5
• Nginx: with gzip on-the-fly and sendfile on (not bound since HTML websites are very fast)
• fastcgi: with page caching enabled (not nearly full, and not used for admin panel)
• PHP-FPM: 512MB memory limit (no error in logs neither)
• opcache: not nearly full

zend_extension=opcache opcache.enable=1 opcache.enable_cli=0 opcache.memory_consumption=1536 opcache.interned_strings_buffer=100 opcache.max_accelerated_files=75000 opcache.validate_timestamps=1 opcache.revalidate_freq=1 opcache.blacklist_filename=/etc/php.d/opcache*.blacklist opcache.log_verbosity_level=2 opcache.huge_code_pages=0 opcache.validate_permission=1 

• database: MariaDB on the same another server, usually only a fraction of the load time, eg. less than 0.1s, so not database bound

Observations

All requests to /wp-admin/admin-ajax.php take 0.5-0.6s, such as the heartbeats. I can disable heartbeats but that's a symptom, the cause is something slow in wp-load.

Using wp-cli profile hook init tells me it took 0.62 seconds, of which 0.02s database queries. Cache ratio is 98%, with nearly no cache misses. et_setup_builder() takes 0.1s, Automattic\WooCommerce\Blocks\BlockTypesController->register_blocks() and WooCommerce->init() and google-site-kit/includes/Plugin.php all take 0.06s.

I understand there are some heavy plugins, but surely this server should be able to serve such a website quickly! Server resources are hardly being used (CPU and memory), so upgrading the server doesn't do anything. How can WordPress be this slow? Is there anything that can be done? Any tips on figuring our what's the culprit?

Edit: strace

I ran a strace and the first lines show:

 26.39 0.031918 6 4651 65 openat 21.71 0.026260 3 7520 450 newfstatat 10.92 0.013204 2 4802 read 9.62 0.011633 1 5985 3748 access 8.68 0.010497 2 4591 1 close 6.86 0.008299 1 4657 fstat 4.97 0.006018 9 666 mmap 2.37 0.002866 11 254 munmap 

So it seems it is disk bound and/or it reads A LOT of files. The OPCache is in memory, I was hoping to avoid the disk mostly but seems like it doesn't...not sure how to fix this.

Answer 1

Part of the answer is accepting the inherent slowness of PHP and WordPress. The big problem is that for somewhat larger WordPress setups, the amount of plugins is causing a direct slowdown in TTFB and in server response.

Server configuration

A non-exhaustive list is below. I run AlmaLinux RHEL 9 with NGINX + FastCGI + PHP-FPM with Memcached and MariaDB.

NGINX

Make sure you have

http { sendfile on; tcp_nopush on; tcp_nodelay on; } 

Use the ssl_session_cache. Disable Brotli since it uses significantly more CPU with compression rates only slightly superior to Gzip. Set up FastCGI (this is only an example!):

# NGINX wide configuration http { map $http_cache_control $cache_bypass { no-cache 1; } fastcgi_index index.php; fastcgi_buffer_size 8k; fastcgi_buffers 8 32k; fastcgi_cache_key "$scheme$request_method$host$request_uri"; fastcgi_cache_use_stale error timeout invalid_header updating http_429 http_500 http_503; fastcgi_cache_valid 60m; fastcgi_ignore_headers Cache-Control Expires Set-Cookie; } # Per WordPress/site configuration fastcgi_cache_path /srv/www/example/cache/example.com levels=1:2 keys_zone=example.com:100m inactive=60m; server { server_name .example.com; # ... root /srv/www/example/public_html/$host; index index.html index.php; include software.d/wordpress.conf; # ... location ~ [^/]\.php(/|$) { fastcgi_split_path_info ^(.+\.php)(.*)$; try_files $fastcgi_script_name /index.php =404; # Caching fastcgi_cache example.com; fastcgi_no_cache $skip_cache; fastcgi_cache_bypass $skip_cache $cache_bypass; # PHP-FPM include fastcgi_params; fastcgi_param PHP_ADMIN_VALUE "session.cookie_domain = example.com\nsendmail_path = /usr/sbin/sendmail -t -i -f [email protected] --"; fastcgi_pass unix:/var/run/php-fpm/example.sock; } # Fallback to /index.php location / { try_files $uri $uri.html $uri/index.html $uri/index.php$is_args$args; } } 

Content of /etc/nginx/software.d/wordpress.conf:

# Page caching set $skip_cache 0; if ($request_method != GET) { set $skip_cache 1; # Don't cache non-GET requests } if ($query_string != "") { set $skip_cache 1; # Don't cache GET requests with query } if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*\.php|/feed/|index.php|(page-)?sitemap(_index)?\.xml") { set $skip_cache 1; # Don’t cache certain URIs } if ($request_uri ~* "/store.*|/cart.*|/my-account.*|/checkout.*|/addons.*") { set $skip_cache 1; # Don't cache WooCommerce pages } if ($request_uri ~* "/tickets-cart.*|/tickets-payment.*|/tc-api.*") { set $skip_cache 1; # Don't cache Tickera pages } if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wordpress_logged_in|wp-cron") { set $skip_cache 1; # Don’t cache for logged in users or recent commenters } # Browser caching expires $expires; # Remove headers location ~* \.(?:css(\.map)?|js(\.map)?|jpe?g|png|gif|ico|xml|json|webmanifest|ogg|svgz?|mp4|webm|webp|avif|pdf|ttf|ttc|otf|otc|woff|woff2|eot)$ { add_header X ''; # Requires: load_module modules/ngx_http_headers_more_filter_module.so; try_files $uri /index.php?$is_args$args; } # Hardening location ~* ^/(?:license\.txt|readme\.html|wp-config\.php|wp-config-sample\.php|xmlrpc\.php)$ { deny all; } location ~* ^/wp-content/[^/]*\.php$ { deny all; } location ~* ^/wp-content/(?!(plugins/)).*\.php$ { deny all; } location ~* ^/wp-includes/.*\.php$ { deny all; } location ~* ^/wp-admin/install\.php$ { deny all; } # Convenience location = /wp-admin { return 301 https://$host$request_uri/; } 

You can check the cache by adding add_header X-FastCGI-Cache $upstream_cache_status;.

PHP-FPM

Enable OPCache in /etc/php.d/10-opcache.ini:

zend_extension=opcache opcache.enable=1 ; enable_cli is useless and slows down all CLI invocations opcache.enable_cli=0 ; Check and measure each! opcache.memory_consumption=1536 opcache.interned_strings_buffer=100 opcache.max_accelerated_files=75000 ; If you can, set validate_timestamps to 0 and refresh cache by restarting phpfpm.service opcache.validate_timestamps=1 opcache.revalidate_freq=2 ; Security must if you have more than one PHP-FPM pool opcache.validate_permission=1 ; Enable if you run each PHP-FPM pool in a chroot ;opcache.validate_root=0 

Also enable the OPCache file cache, this helps when restarting PHP-FPM the cache is pulled from disk instead of recompiling all scripts. Does not affect that the main cache is still in memory.

In /etc/php-fpm.d/example.conf (and for all other users), make sure to set

; Set memory limit as necessary, 256M is quite high for a small WordPress php_admin_value[memory_limit] = 256M php_admin_value[opcache.file_cache] = /srv/www/example/opcache ; Use memcached for sessions php_admin_value[session.save_handler] = memcached php_admin_flag[session.auto_start] = off ;php_admin_flag[session.use_trans_sid] = off php_admin_flag[session.use_strict_mode] = on php_admin_flag[session.use_cookies] = on php_admin_flag[session.use_only_cookies] = on ;php_admin_flag[session.cookie_lifetime] = 14400 php_admin_flag[session.cookie_secure] = on php_admin_flag[session.cookie_httponly] = on php_admin_value[session.cookie_samesite] = Strict php_admin_value[session.cache_expire] = 30 php_admin_value[session.sid_length] = 128 php_admin_value[session.sid_bits_per_character] = 6 php_admin_value[session.gc_maxlifetime] = 600 php_admin_value[session.gc_divisor] = 1000 php_admin_flag[session.lazy_write] = off php_admin_flag[memcached.sess_locking] = off php_admin_value[session.name] = example php_admin_value[session.save_path] = /var/run/memcached/example.sock 

Memcached

Set up memcached per user, edit /etc/memcached.d/example.conf:

USER="example" MAXCONN="256" CACHESIZE="32" # adjust as necessary OPTIONS="-s '/var/run/memcached/example.sock' -a 0760" 

In each WordPress, set in wp-config.php:

$memcached_servers = array('default' => array('/var/run/memcached/example.sock')); 

and add wp-content/object-cache.php (eg from https://github.com/Automattic/wp-memcached/blob/master/object-cache.php). If you installed php-memcached (as recommended per PHP manual) instead of php-memcache, you need to make some changes to that file.

Use system cron

In each WordPress, set in wp-config.php:

define( 'DISABLE_WP_CRON', true ); 

and run a cron every 10 minutes that calls

for site in /srv/www/*/public_html/*/wp-config.php; do curl --resolve "$domain:443:127.0.0.1" "https://$domain/wp-cron.php" done 

MariaDB

Make sure it's not on the same server, since a database needs A LOT of memory, and you need all the memory you can get for requests, memcached, and the Linux page/disk/file cache.

Open files limit

Check current system limit with cat /proc/sys/fs/file-max and process limit with ulimit -Sn; ulimit -Hn (soft and hard limit). For an active process check ps -e | grep nginx to get the PID, and then cat cat /proc/{PID}/limits.

Create or edit /etc/sysctl.d/limits.conf to contain

fs.file-max=100000 

In /etc/nginx/nginx.conf check

worker_processes auto; worker_rlimit_nofile 65535; events { worker_connections 65535; multi_accept on; use epoll; } 

In /etc/php-fpm.conf check

; Default is 1024, measure and increase. But keep in check to avoid abuse. rlimit_files = 4096 

Measuring wp-load

A good measure is to check and compare the following endpoints:

time curl https://example.com/wp-load.php

and

time php /srv/www/example/public_html/example.com/wp-load.php

The first will include our OPCache (run twice to make sure the cache is filled), while the latter uses the CLI. My results:

# Simple WordPress curl: ≈50ms cli: ≈250ms # WordPress + WooCommerce + Tickera curl: ≈500ms cli: ≈2s 

The latter is way too much. Checking with strace -c php /srv/www/example/public_html/example.com/wp-load.php:

 26.39 0.031918 6 4651 65 openat 21.71 0.026260 3 7520 450 newfstatat 10.92 0.013204 2 4802 read 9.62 0.011633 1 5985 3748 access 8.68 0.010497 2 4591 1 close 6.86 0.008299 1 4657 fstat 4.97 0.006018 9 666 mmap 2.37 0.002866 11 254 munmap 

System calls are not the main cause, but we have many failing access calls. Check output of strace -e trace=access /srv/www/example/public_html/example.com/wp-load.php, and most of the failing calls are from plugins that try to open non-existing files (buggy!), especially googleanalytics and the-events-calendar. Removing both plugins improved the curl time to 250ms!

Also check the Query Monitor WordPress plugin, it will show you if there is a database bottleneck and also shows the time spent on HTTP API calls (they can significantly delay a page).

You can run the CLI invocation in a loop with bash, and in another terminal check watch -n0.1 iostat (disk). Check if anything unusual happens, such as a high iowait. Do the same with vmstat (memory), netstat (network), etc. There are so many moving parts that it is hard to pinpoint, but it is key to discover what the bottleneck is. Is it disk, memory (or lack thereof), CPU, database?

Autoload

It is worth checking the wp_options autoload rows in the database by running SELECT SUM(LENGTH(option_value)) FROM wp_options WHERE autoload='yes' OR autoload='on' OR autoload='auto' OR autoload='auto-on';. WordPress advices a limit of 1MB, but be aware that for every page you load you need to transfer 1MB from the database server to the web server! I still need to investigate the impact.

Slow plugins at the server

Freemius API calls Since Tickera (an event ticketing plugin) is using Freemius (and so are other plugins), it requires checking the plugin licence, especially at the admin plugin page. These responses take 600ms roundtrip (including EU<->US roundtrip), and if you have several of these plugins it may take north of 5s to load the admin plugins page. Those responses ought to be asynchronous and cached...! Not easy to fix, unless Freemius fixes this or you override the Freemius class as a "muplugin" and cut the API calls.

Slow plugins at the client

Divi with huge images I've had a client that uploaded >20.000x20.000 PNG images so as to preserve quality. While the byte size may be several MBs, the problem is that displaying the page in the Divi editor was causing it to load it several times, each time took a few seconds to decompress the PNG image.

Yoast SEO premium Especially the premium version was much slower since it was running a slow regular expression, probably over the whole HTML page or all page resources. This was causing the browser to get stuck for a few seconds.

Conclusion

PHP is slow, WordPress is slow, add plugins and anything can happen. The enormous amount of files that need to be accessed by some plugins (over 5000! see strace above) is absolutely ridiculous. If the plugin doesn't use proper caching (using wp_cache for example) or tries to access a wild number of non-existing files, I'm not even sure you can fix the performance issues. You may try to put the whole WordPress directory in tmpfs (except wp-content/uploads) if you have only one installation, but that sounds over-the-top to me.

Database and network calls are the usual suspects. Other than that, we must contend with pages that load within around 1 second.