We are running rabbit v3.8.3-1.el7, erlang v23.3.3.el7, kernel 3.10.0-1062.12.1.el7.x86_64, release Centos 7.7
I have three nodes that I would like in disc mode, cdvlhbqr23[1-3]
However I'm running into an issue after attempting to enable TLS on erlang.
[ cdvlhbqr231:rabbitmq ] 10.128.3.231 :: root -> rabbitmqctl status Error: unable to perform an operation on node 'rabbit@cdvlhbqr231'. Please see diagnostics information and suggestions below. Most common reasons for this are: * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues) * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server) * Target node is not running In addition to the diagnostics info below: * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more * Consult server logs on node rabbit@cdvlhbqr231 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools DIAGNOSTICS =========== attempted to contact: [rabbit@cdvlhbqr231] rabbit@cdvlhbqr231: * connected to epmd (port 4369) on cdvlhbqr231 * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic * TCP connection succeeded but Erlang distribution failed * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more Current node details: * node name: 'rabbitmqcli-23412-rabbit@cdvlhbqr231' * effective user's home directory: /var/lib/rabbitmq/ * Erlang cookie hash: MudCW7tn3FA5sTmC1FlR0g== I've double checked the cookie file and it's identical across all nodes. All of the hostnames are correct and consistent across the nodes. So I assume this has to be directly the result of ssl / tls
Here's what the node config looks like:
[ cdvlhbqr231:rabbitmq ] 10.128.3.231 :: root -> cat /etc/rabbitmq/rabbitmq.config [ {rabbit, [ {vm_memory_high_watermark, 0.4}, {vm_memory_high_watermark_paging_ratio, 0.5}, {memory_alarms, true}, {disk_free_limit, 41686528}, {cluster_partition_handling, autoheal}, {tcp_listen_options, [binary, {packet, raw}, {reuseaddr, true}, {backlog, 128}, {nodelay, true}, {exit_on_close, false}, {keepalive, true} ] }, {cluster_nodes, {['rabbit@cdvlhbqr231', 'rabbit@cdvlhbqr232', 'rabbit@cdvlhbqr233'], disc}}, {loopback_users, []}, {tcp_listeners, [{"0.0.0.0",5672}]}, {ssl_listeners, [{"0.0.0.0",5671}]}, {ssl_options, [ {cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"}, {certfile,"/etc/rabbitmq/ssl/cert.pem"}, {keyfile,"/etc/rabbitmq/ssl/key.pem"}, {verify,verify_peer}, {versions, ['tlsv1.2']}, {fail_if_no_peer_cert,false} ]} ] }, {rabbitmq_management, [{ listener, [ {port, 15672}, {ip, "0.0.0.0"}, {ssl, true}, {ssl_opts, [ {cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"}, {certfile,"/etc/rabbitmq/ssl/cert.pem"}, {keyfile,"/etc/rabbitmq/ssl/key.pem"}, {verify,verify_peer}, {versions, ['tlsv1.2']} ]} ]} ] } ]. The private key is generated on the host and signed by an intermediate CA whose pub key is in the systems extracted cert bundle. We generate an "/etc/rabbitmq/ssl/allfile.pem" which is a bundle of the servers private key and signed cert.
The ssl environment for erlang is defined as the following:
[ cdvlhbqr231:rabbitmq ] 10.128.3.231 :: root -> cat rabbitmq-env.conf # Obtaining of an Erlang ssl library path export HOME=/var/lib/rabbitmq/ ERL_SSL_PATH=/usr/lib64/erlang/lib/ssl-9.6.2/ebin # Add SSL-related environment vars for rabbitmq-server and rabbitmqctl SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile '/etc/rabbitmq/ssl/allfile.pem' \ -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true" # CLI CTL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile /etc/rabbitmq/ssl/allfile.pem \ -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true" I'm not entirely clear what's causing the issue. I thought I had followed the documentation to a T. Can anyone help me review this and see if there is anything obvious I'm missing, or any suggestions on how to trace down this problem?
