I had a server which was based on apache; it was used to serve multiple image tiles of a map. I thought of improving it by shifting it to HTTP3, for this I had to shift to nginx and quick protocol. But since shifting a weird problem have started after some time the network response code for images become 500 and nginx have to be restarted to further load any other image. On further investigating the error logs of nginx I get "Too many open files". To resolve I made following changes:
- Increase "worker_rlimit_nofile"
- Removed "open_file_cache"
While 2nd modification brought some improvement in the issue, the issue is not resolved. I am unable to debug or resolve this issue. Some observation which I had:
- While apache had 13 workers nginx had 92 workers (obtained using
ps -auxbash command) - System wide max open files limit is 9223372036854775807 (obtained using
cat /proc/sys/fs/file-max)
Configuration for apache was:
DefaultRuntimeDir ${APACHE_RUN_DIR} PidFile ${APACHE_PID_FILE} Timeout 300 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5 User ${APACHE_RUN_USER} Group ${APACHE_RUN_GROUP} HostnameLookups Off ErrorLog ${APACHE_LOG_DIR}/error.log LogLevel warn IncludeOptional mods-enabled/*.load IncludeOptional mods-enabled/*.conf Include ports.conf <Directory /> Options FollowSymLinks AllowOverride None Require all denied </Directory> <Directory /usr/share> AllowOverride None Require all granted </Directory> <Directory /var/www/> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> AccessFileName .htaccess <FilesMatch "^\.ht"> Require all denied </FilesMatch> LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %O" common LogFormat "%{Referer}i -> %U" referer LogFormat "%{User-agent}i" agent IncludeOptional conf-enabled/*.conf IncludeOptional sites-enabled/*.conf And the configuration for nginx is:
worker_processes auto; worker_rlimit_nofile 100000; error_log logs/error.log; events { worker_connections 32768; multi_accept on; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log logs/access.log main; sendfile on; keepalive_timeout 65; http3_stream_buffer_size 2m; quic_gso on; http3_max_concurrent_streams 512; tcp_nodelay on; tcp_nopush on; server { listen 80; server_name catmap.ptmil.cd.in; location / { root html; index index.php index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } server { server_name map.ptmil.cd.in; ssl_certificate /etc/nginx/ssl/ptmil.cd.in/PTMIL.crt; ssl_certificate_key /etc/nginx/ssl/ptmil.cd.in/PTMIL.key; quic_retry on; ssl_early_data on; ssl_prefer_server_ciphers off; ssl_session_timeout 1d; ssl_session_cache shared:MozSSL:64m; root /var/www/html/map; index index.php; client_max_body_size 2048m; gzip_static on; location / { try_files $uri $uri/ =404; } location ~ /\.ht { deny all; } location ~ \.php$ { include snippets/fastcgi-php.conf; fastcgi_pass unix:/var/run/php/php8.3-fpm.sock; fastcgi_param PHP_VALUE "upload_max_filesize=2048M \n post_max_size=2048M"; } location = /favicon.ico { access_log off; log_not_found off; } location /newsat { expires 2d; add_header Cache-Control "public"; } location ~* \.(js|css)$ { expires 10h; add_header Cache-Control "public"; } location ~* \.(jpg|jpeg|png|gif)$ { expires 7d; add_header Cache-Control "public"; } location ~ \.gz$ { deny all; } add_header Alt-Svc 'h3=":443"; ma=86400, h3-29=":443"; ma=86400' always; add_header Alt-Svc 'h2=":443"; ma=2592000; persist=1'; add_header Alt-Svc 'h2=":443"; ma=2592000;'; add_header X-Frame-Options "SAMEORIGIN"; add_header X-Content-Type-Options "nosniff"; } } systemd file for apache:
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=https://httpd.apache.org/docs/2.4/ [Service] Type=forking Environment=APACHE_STARTED_BY_SYSTEMD=true ExecStart=/usr/sbin/apachectl start ExecStop=/usr/sbin/apachectl graceful-stop ExecReload=/usr/sbin/apachectl graceful KillMode=mixed PrivateTmp=true Restart=on-abort [Install] WantedBy=multi-user.target systemd file for nginx:
[Unit] Description=The NGINX HTTP and reverse proxy server After=network.target [Service] LimitNOFILE=100000 Type=forking PIDFile=/usr/local/nginx/logs/nginx.pid ExecStartPre=/usr/local/nginx/sbin/nginx -t ExecStart=/usr/local/nginx/sbin/nginx ExecReload=/bin/kill -s HUP $MAINPID ExecStop=/bin/kill -s QUIT $MAINPID PrivateTmp=true [Install] WantedBy=multi-user.target output of ulimit -a command is:
real-time non-blocking time (microseconds, -R) unlimited core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 880215 max locked memory (kbytes, -l) 28181404 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 880215 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited The host is Ubuntu 20 based. I have bummed my head in it but cannot do much. What next step should I take? Thanks in advance.
