3

The nginx configuration server block:

localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name' server { listen 443 ssl http2 default_server; ssl_certificate /etc/letsencrypt/live/$ssl_server_name/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/$ssl_server_name/privkey.pem; location / { include /etc/nginx/snippets/set-headers.conf; proxy_pass http://localhost:8080; } }

This is using the variable $ssl_server_name in the certificate directives which is supported since nginx 1.15.9. Relevant part of the nginx docs.

The configuration passes nginx -t and loads without issues, but page does not load in browser, and there is a permissions denied error opening the cert in error.log even though nginx is running as root:

localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443 localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep www-data 5676 24653 0 18:49 ? 00:00:00 nginx: worker process root 24653 1 0 15:08 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem -rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443 localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep www-data 5676 24653 0 18:49 ? 00:00:00 nginx: worker process root 24653 1 0 15:08 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem -rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem localhost:/etc/nginx$ openssl OpenSSL> version OpenSSL 1.0.2g 1 Mar 2016 OpenSSL> ^C localhost:/etc/nginx$ nginx -v nginx version: nginx/1.17.0

When I replace $ssl_server_name with the domain name in the nginx configuration then there is no permissions error reading the very same cert file, and the page loads in the browser.

Why does using the variable in the cert path not work?

UPDATE:

I updated the archive folder group to www-data, still seing the permissions error:

localhost:/etc/nginx$ sudo chgrp -R www-data /etc/letsencrypt/archive localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem drwxr-xr-x root root / drwxr-xr-x root root etc drwxr-xr-x root root letsencrypt drwx------ root www-data archive drwxr-xr-x root www-data [DOMAIN NAME REDACTED] -rw-r--r-- root www-data fullchain1.pem localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 07:18:58 [error] 4897#4897: *6 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 2:

Added group read and execute permissions to archive folder, still seing the permissions error:

localhost:/etc/nginx$ sudo chmod g+r /etc/letsencrypt/archive localhost:/etc/nginx$ sudo chmod g+x /etc/letsencrypt/archive localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/ [DOMAIN NAME REDACTED]/fullchain1.pem f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem drwxr-xr-x root root / drwxr-xr-x root root etc drwxr-xr-x root root letsencrypt drwxr-x--- root www-data archive drwxr-xr-x root www-data [DOMAIN NAME REDACTED] -rw-r--r-- root www-data fullchain1.pem localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 07:39:58 [error] 4897#4897: *22 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/ [DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 3:

Tried becoming www-data using sudo but got an error:

localhost:/etc/nginx$ sudo su - www-data No directory, logging in with HOME=/ This account is currently not available.

Update 4:

I also updated the permissions on the symlinked path live folder, still seing the permissions error:

localhost:/etc/nginx$ ll /etc/letsencrypt | grep live drwx------ 5 root root 4096 Apr 17 18:53 live/ localhost:/etc/nginx$ sudo chgrp www-data /etc/letsencrypt/live localhost:/etc/nginx$ sudo chmod g+rx /etc/letsencrypt/live localhost:/etc/nginx$ ll /etc/letsencrypt | grep live drwxr-x--- 5 root www-data 4096 Apr 17 18:53 live/ localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live f: /etc/letsencrypt/live drwxr-xr-x root root / drwxr-xr-x root root etc drwxr-xr-x root root letsencrypt drwxr-x--- root www-data live localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 07:57:48 [error] 5104#5104: *17 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 5:

Listing the permissions of all dirs in path including symlinks:

localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem f: /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem drwxr-xr-x root root / drwxr-xr-x root root etc drwxr-xr-x root root letsencrypt drwxr-x--- root www-data live drwxr-xr-x root root [DOMAIN NAME REDACTED] lrwxrwxrwx root root fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem drwxr-x--- root www-data .. drwxr-xr-x root root .. drwxr-x--- root www-data archive drwxr-xr-x root www-data [DOMAIN NAME REDACTED] -rw-r--r-- root www-data fullchain1.pem

Update 6:

Tried temporarily changing the shell for www-data user, became www-data using sudo and tested reading the cert was possible, but the permission error is still happening:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin localhost:/$ cat /etc/passwd | grep www-data www-data:x:33:33:www-data:/var/www:/bin/bash localhost:/etc/nginx$ sudo vim /etc/passwd localhost:/etc/nginx$ sudo su - www-data No directory, logging in with HOME=/ localhost:01:/$ whoami www-data localhost:/$ cat /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -----BEGIN CERTIFICATE----- [REDACTED CERT] -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- [REDACTED CERT] -----END CERTIFICATE----- localhost:/$ exit logout localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 08:40:23 [error] 5259#5259: *14 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 7:

Tried exporting the certs to another folder:

localhost:/etc/nginx$ mkdir /tmp/exported-certs localhost:/etc/nginx$ sudo rsync -razL /etc/letsencrypt/live/ /tmp/exported-certs localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem -rw-r--r-- 1 root www-data 3591 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem -rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem localhost:/etc/nginx$ sudo namei -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem f: /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem drwxr-xr-x root root / drwxrwxrwt root root tmp drwxr-x--- root www-data exported-certs drwxr-xr-x root root [DOMAIN NAME REDACTED] -rw-r--r-- root www-data fullchain.pem localhost:/etc/nginx$ sudo vim nginx.conf localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name' server { listen 443 ssl http2 default_server; ssl_certificate /tmp/exported-certs/$ssl_server_name/fullchain.pem; ssl_certificate_key /tmp/exported-certs/$ssl_server_name/privkey.pem; location / { include /etc/nginx/snippets/set-headers.conf; proxy_pass http://localhost:8080; } } localhost:/etc/nginx$ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful localhost:/etc/nginx$ sudo nginx -s reload localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 10:52:48 [notice] 6250#6250: signal process started localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log 2019/06/20 10:53:08 [error] 6251#6251: *67 cannot load certificate key "/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Then decided to check again as the www-data user because last time I checked it was when the certs were in the letsencrypt folder, also this time I remembered to check both cert and key:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin localhost:/etc/nginx$ sudo vim /etc/passwd localhost:/etc/nginx$ cat /etc/passwd | grep www-data www-data:x:33:33:www-data:/var/www:/bin/bash localhost:/etc/nginx$ sudo su - www-data No directory, logging in with HOME=/ localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem -----BEGIN CERTIFICATE----- [CERT REDACTED] -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- [CERT REDACTED] -----END CERTIFICATE----- localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem cat: /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem: Permission denied <---- THERE IT IS! localhost:/$ ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem -rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem localhost:/$ exit logout localhost:/etc/nginx$ sudo chmod g+r /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem localhost:/etc/nginx$ sudo su - www-data No directory, logging in with HOME=/ localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem -----BEGIN PRIVATE KEY----- [CERT REDACTED] -----END PRIVATE KEY----- localhost:/$ exit logout localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/access.log 139.162.202.226 - [DOMAIN NAME REDACTED]:443 - [20/Jun/2019:11:04:08 +0100] "GET / HTTP/2.0" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

Once I addd the group read permission for www-data to the privkey.pem, the browser was able to load the page. :)

Thanks to all that commented on this question.

Improve this question
15
  • 2
    When you provide full absolute name this can be resolved when nginx starts, and it has root credentials then, so can load everything. If you provide a path with a variable, the loading can happen only later, when the variable is resolved that is when the connection is made and then nginx has only www-data credentials, so the certificate needs to be readable by that user, including all elements of the path leading to it. It is written in the documentation you link to: "Give read-access to the certificates" Commented Jun 19, 2019 at 19:31
  • @PatrickMevzek Yes I read that but though that nginx had root permissions or that it read all the certs at startup time, thanks for the details I will try to add read permissions to www-data Commented Jun 20, 2019 at 4:59
  • Certificates can appear on disk after the process started... As you can see from your ps output, there is a master process running as root but another process deals with HTTP exchanges and it runs as www-data... Commented Jun 20, 2019 at 5:23
  • @PatrickMevzek I updated the group of all files in the archive folder, but I am still seing the same permissions error, see the updated question with the console output. Shouldn't the worker process now be able to read the files? Commented Jun 20, 2019 at 6:32
  • 1
    You need to check all directories in the path, starting from /etc, www-data must have rx on all of them. Use sudo to become www-data momuntarily and try accessing the files Commented Jun 20, 2019 at 6:37

1 Answer 1

Reset to default
0

OK, thanks for feedback in comment. Let me share two ideas - not directly solution on what you have mentioned but possible workarounds....

prepare instead of linking

I am using in some specific case certbot (I guess the same as you) and haproxy. I have cron job running certbot with certonly and in case the cert is issued it is concatenating cert and key (one file is preferred) and copy out to location where haproxy expect it in configuration. The haproxy is restarted afterwards. In case you will not solve it and need to have it working may be try to "prepare" the cert for nginx somewhere instead of linking to the "original" location...

run certbot as www-data user

you can run certbot as user who is running nginx (www-data) so the output would fit for the purpose by the permission point of view. In that case you need to change the ownership first for the structure around - e.g. /etc/letsencrypt/account....

Improve this answer
2
  • Thanks for the suggestions - Not to keen to run certbot as a different use in case it breaks something in how certbot works. The prepare route sounds like it could be a good way to go, I wanted to use the default certbot setup because otherwise I have to figure out how to manage another folder full of certs, but it might not be as bad as I thought since certbot has a post-create/update/delete command that you can add to run automatically when it does it's stuff. I'll do a few tests today at least then, once all the symlinks are out of the way, it will be clear if it's an nginx issue. Commented Jun 20, 2019 at 8:48
  • As default certbot is installed and run under root so running it as non root user was first thing what I did :-). Until the location /etc/letsencrypt is writable for the user running certbot with first run it is ok. Once it is run as root for the first time the created structure should be re-owned as a easiest way. For the schedule I have utilized cron so certbot is run as user certbot (created for that purpose) and in case of new cert the rest of stuff (incl. copying the cert to needed location) is done by cron under root... But as I wrote it is just one of the option ;-). Commented Jun 20, 2019 at 9:33

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.