I'm on a server with the following config:
32 GB RAM 12 core processor
and it's running single wordpress on nginx, varnish, php5-fpm, fastcgi. Everything was running fine, but suddenly after yesterday night the whole site is extremely slow and unresponsive. Upon inspection, I found that mysql was using a lot of CPU. After a lot of research online, I disabled XML-RPC and mysql CPU load came down drastically.
But the problem still persists, my CPU was almost 100%. I see that php5-fpm processes are taking up all the CPU.
Here's the result of top
top - 10:30:33 up 7:10, 1 user, load average: 31.35, 31.87, 31.85 Tasks: 238 total, 27 running, 179 sleeping, 32 stopped, 0 zombie %Cpu(s): 84.9 us, 11.7 sy, 0.0 ni, 3.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 32948544 total, 11695168 used, 21253376 free, 408404 buffers KiB Swap: 0 total, 0 used, 0 free. 3151748 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12179 mysql 20 0 6233684 998836 7632 S 109.2 3.0 371:56.58 mysqld 18533 nginx 20 0 1007708 769740 26008 R 57.5 2.3 3:33.24 php5-fpm 18487 nginx 20 0 600672 361312 26212 R 51.7 1.1 4:13.51 php5-fpm 18657 nginx 20 0 280024 42640 25436 R 51.7 0.1 0:46.94 php5-fpm 18459 nginx 20 0 996592 753804 25764 R 40.2 2.3 4:53.79 php5-fpm 18469 nginx 20 0 1010296 768676 25704 R 40.2 2.3 4:22.98 php5-fpm 18545 nginx 20 0 277216 40060 25740 R 40.2 0.1 2:53.74 php5-fpm 18547 nginx 20 0 684952 441796 25616 R 40.2 1.3 2:58.57 php5-fpm 18629 nginx 20 0 280452 42752 25444 R 40.2 0.1 1:22.99 php5-fpm 18460 nginx 20 0 282796 45920 25828 R 34.5 0.1 5:22.28 php5-fpm 18461 nginx 20 0 276952 40200 25636 R 34.5 0.1 4:13.64 php5-fpm 18462 nginx 20 0 278752 41504 25672 R 34.5 0.1 4:16.16 php5-fpm 18540 nginx 20 0 277744 41096 25740 R 34.5 0.1 3:02.92 php5-fpm 18624 nginx 20 0 278648 41060 25588 R 34.5 0.1 1:27.54 php5-fpm 18452 nginx 20 0 278752 41696 25732 S 28.7 0.1 4:25.82 php5-fpm 18458 nginx 20 0 282488 45512 25684 S 28.7 0.1 4:44.26 php5-fpm 18472 nginx 20 0 282652 45976 25948 R 28.7 0.1 4:34.61 php5-fpm 18486 nginx 20 0 1007732 769472 25800 R 28.7 2.3 4:03.61 php5-fpm 18491 nginx 20 0 282488 45584 25764 R 28.7 0.1 4:06.75 php5-fpm 18495 nginx 20 0 280744 43216 25744 R 28.7 0.1 3:37.82 php5-fpm 18496 nginx 20 0 282488 44924 25908 R 28.7 0.1 3:38.28 php5-fpm 18562 nginx 20 0 282488 45412 25584 R 28.7 0.1 2:53.19 php5-fpm 18575 nginx 20 0 282760 46068 26192 R 28.7 0.1 2:34.81 php5-fpm 18455 nginx 20 0 278904 41532 25716 S 23.0 0.1 4:22.13 php5-fpm 18468 nginx 20 0 278776 42120 25732 R 23.0 0.1 4:09.41 php5-fpm 18490 nginx 20 0 280840 44148 26172 R 23.0 0.1 3:39.08 php5-fpm 18492 nginx 20 0 282232 45260 25652 R 23.0 0.1 4:07.21 php5-fpm 18514 nginx 20 0 282608 45716 25732 R 23.0 0.1 3:44.15 php5-fpm 18576 nginx 20 0 279416 41964 25496 R 23.0 0.1 2:09.75 php5-fpm 18656 nginx 20 0 279560 42428 25508 S 23.0 0.1 0:49.84 php5-fpm 18465 nginx 20 0 284492 47860 26252 R 17.2 0.1 5:09.60 php5-fpm 10165 nobody 20 0 11.233g 2.437g 83232 S 11.5 7.8 23:56.78 varnishd 18686 root 20 0 24948 1656 1080 R 11.5 0.0 0:00.04 top 8 root 20 0 0 0 0 S 5.7 0.0 2:48.56 rcuos/0 18122 nginx 20 0 36700 7428 1040 S 5.7 0.0 1:04.16 nginx 18129 nginx 20 0 33836 4568 1040 S 5.7 0.0 0:25.52 nginx As you can see Load Average is a whopping 31.35, 31.87, 31.85 What can cause such high CPU usage. I've tried tweaking everything from mysql to nginx to varnish, but nothing seems to be working. I've tried playing around with pm value in my php5 conf file too, but here I'm, tearing my hair apart.
Please help.
The website gets a lot of traffic, say 2000 to 3000 concurrent users on an average. :(
UPDATE:
Including the nginx conf files: /etc/nginx/nginx.conf
user nginx; worker_processes 12; worker_rlimit_nofile 20000; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 3000; use epoll; multi_accept on; } http { server_names_hash_bucket_size 64; include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; limit_req_zone $binary_remote_addr zone=one:100m rate=3r/m; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; client_header_timeout 180; client_body_timeout 180; fastcgi_read_timeout 180; client_body_buffer_size 10M; client_max_body_size 32m; gzip on; gzip_disable "msie6"; gzip_vary on; gzip_proxied any; gzip_comp_level 6; gzip_buffers 16 8k; gzip_http_version 1.1; gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; include /etc/nginx/conf.d/*.conf; port_in_redirect off; } /etc/nginx/conf.d/default.conf
server { listen 8080; server_name mysite.com; return 301 $scheme://www.mysite.com$request_uri; } server { ## Your website name goes here. server_name www.mysite.com; ## Your only path reference. root /var/www/; listen 8080; ## This should be in your http block and if it is, it's not needed here. index index.html index.htm index.php; include conf.d/drop; location / { # This is cool because no php is touched for static content try_files $uri $uri/ /index.php?q=$uri&$args; } location ~ \.php$ { try_files $uri =404; # proxy buffers - no 502 errors! proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; fastcgi_buffers 256 16k; fastcgi_buffer_size 128k; fastcgi_max_temp_file_size 0; fastcgi_intercept_errors on; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_pass unix:/dev/shm/php-fpm-www.sock; #fastcgi_pass 127.0.0.1:9000; location ~* wp\-login\.php { limit_req zone=one burst=1 nodelay; include fastcgi_params; fastcgi_pass unix:/dev/shm/php-fpm-www.sock; } } location /xmlrpc.php { deny all; } } 
pm.max_requestscould be lowered so they recycle more.