DEV Community

balajivedagiri
balajivedagiri

Posted on

Installing and Configuring Elasticsearch/Kibana 8.x with Security

We will be installing,configuring elasticsearch and kibana 8.4, but steps should be same for most versions.

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

1) pre-requisites 

1a) create /var/lib/elasticsearch mount point on all the nodes. 1b) turn off swap on OS(to ensure JVM heap is not swapped out). 1c) since we are using packages to install elasticsearch, ulimits are enforced in systemd unit file /usr/lib/systemd/system/elasticsearch.service. 1d) settings like file descriptors, max processes, max virtual memory size , max file size, etc are controlled from the systemd unit file. 1e) change default value of TCP retransmission timeout value, update the net.ipv4.tcp_retries2 setting in /etc/sysctl.conf to 5, and sysctl -w net.ipv4.tcp_retries2=5.
Enter fullscreen mode Exit fullscreen mode

2) Installing elasticsearch

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

2a) Import elasticsearch PGP key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
Enter fullscreen mode Exit fullscreen mode

2b) Install apt-transport-https package

sudo apt install apt-transport-https
Enter fullscreen mode Exit fullscreen mode

2c) save the repo,

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Enter fullscreen mode Exit fullscreen mode

2d) update the repo and install the package,

apt update && apt install elasticsearch
Enter fullscreen mode Exit fullscreen mode 
apt-get install elasticsearch Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: elasticsearch 0 upgraded, 1 newly installed, 0 to remove and 119 not upgraded. Need to get 0 B/566 MB of archives. After this operation, 1,170 MB of additional disk space will be used. Selecting previously unselected package elasticsearch. (Reading database ... 111616 files and directories currently installed.) Preparing to unpack .../elasticsearch_8.4.3_amd64.deb ... Creating elasticsearch group... OK Creating elasticsearch user... OK Unpacking elasticsearch (8.4.3) ... Setting up elasticsearch (8.4.3) ... --------------------------- Security autoconfiguration information ------------------------------ Authentication and authorization are enabled. TLS for the transport and HTTP layers is enabled and configured. The generated password for the elastic built-in superuser is : B25meUI2L6WcfTWBNvNp If this node should join an existing cluster, you can reconfigure this with '/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>' after creating an enrollment token on your existing cluster. You can complete the following actions at any time: Reset the password of the elastic built-in superuser with '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'. Generate an enrollment token for Kibana instances with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'. Generate an enrollment token for Elasticsearch nodes with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.
Enter fullscreen mode Exit fullscreen mode

2e) Ansible playbook to install the package.

--- - hosts: elasticsearch become: true gather_facts: true tasks: - name: Import the Elasticsearch PGP key apt_key: url: https://artifacts.elastic.co/GPG-KEY-elasticsearch keyring: /usr/share/keyrings/elasticsearch-keyring.gpg state: present - name: Install apt-transport-https apt: name: apt-transport-https state: present # Add elasticsearch repo into sources list file /etc/apt/sources.list.d/elastic-8.x.list, after adding it will also run apt update or apt-get update by default - apt_repository: repo: 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main' state: present filename: elastic-8.x.list - name: Install a specific version of elasticsearch apt: name: elasticsearch=8.4.3 state: present update_cache: yes
Enter fullscreen mode Exit fullscreen mode

2f) enable the service to start automatically on boot

sudo systemctl daemon-reload sudo systemctl enable elasticsearch
Enter fullscreen mode Exit fullscreen mode

3) Generating certificates to enable TLS for transport and http.

3a) Generate CA certificate.

Login to one of the node where you installed elasticsearch , and issue below command to generate CA certificate. For higher protection, ensure you are setting password the certificate when it prompts below at the end and ensure you save that password in a secure location to use it later.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode 
/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12 This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'ca' mode generates a new 'certificate authority' This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode. Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority By default the 'ca' mode produces a single PKCS#12 output file which holds: * The CA certificate * The CA's private key If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key Enter password for elasticsearch-test-ca.p12 : root@jumperserver:~/elasticsearch_certs# ls elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode

3b) Generate node certificates

We use node certificates to join nodes to cluster and for transport layer encrytion. add all of your node details with dns name and ip into an yaml file like below,

root@jumperserver:~# cat /root/elasticsearch_certs/instances.yaml instances: - name: "test-elastic-master01" ip: "10.10.4.6" dns: "test-elastic-master01" - name: "test-elastic-master02" ip: "10.10.4.7" dns: "test-elastic-master02" - name: "test-elastic-master03" ip: "10.10.4.8" dns: "test-elastic-master03" - name: "test-elastic-hotdata01" ip: "10.10.4.2" dns: "test-elastic-hotdata01" - name: "test-elastic-hotdata02" ip: "10.10.4.3" dns: "test-elastic-hotdata02" - name: "test-elastic-hotdata03" ip: "10.10.4.4" dns: "test-elastic-hotdata03" - name: "test-elastic-warmdata01" ip: "10.10.4.11" dns: "test-elastic-warmdata01" - name: "test-elastic-warmdata02" ip: "10.10.4.12" dns: "test-elastic-warmdata02" - name: "test-elastic-warmdata03" ip: "10.10.4.13" dns: "test-elastic-warmdata03" - name: "test-elastic-ml01" ip: "10.10.4.10" dns: "test-elastic-ml01"
Enter fullscreen mode Exit fullscreen mode

below you need to enter CA certificate password that you entered in step 3a, and ensure you set password for each and every node certificate ( you can set same password for all the nodes or different password as per security compliance)

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12 This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'cert' mode generates X.509 certificate and private keys. * By default, this generates a single certificate and key for use on a single instance. * The '-multiple' option will prompt you to enter details for multiple instances and will generate a certificate and key for each one * The '-in' option allows for the certificate generation to be automated by describing the details of each instance in a YAML file * An instance is any piece of the Elastic Stack that requires an SSL certificate. Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats may all require a certificate and private key. * The minimum required value for each instance is a name. This can simply be the hostname, which will be used as the Common Name of the certificate. A full distinguished name may also be used. * A filename value may be required for each instance. This is necessary when the name would result in an invalid file or directory name. The name provided here is used as the directory name (within the zip) and the prefix for the key and certificate files. The filename is required if you are prompted and the name is not displayed in the prompt. * IP addresses and DNS names are optional. Multiple values can be specified as a comma separated string. If no IP addresses or DNS names are provided, you may disable hostname verification in your SSL configuration. * All certificates generated by this tool will be signed by a certificate authority (CA) unless the --self-signed command line option is specified. The tool can automatically generate a new CA for you, or you can provide your own with the --ca or --ca-cert command line options. By default the 'cert' mode produces a single PKCS#12 output file which holds: * The instance certificate * The private key for the instance certificate * The CA certificate If you specify any of the following options: * -pem (PEM formatted output) * -multiple (generate multiple certificates) * -in (generate certificates from an input file) then the output will be be a zip file containing individual certificate/key files Enter password for CA (/root/elasticsearch_certs/elasticsearch-test-ca.p12) : Enter password for test-elastic-master01/test-elastic-master01.p12 : Enter password for test-elastic-master02/test-elastic-master02.p12 : Enter password for test-elastic-master03/test-elastic-master03.p12 : Enter password for test-elastic-hotdata01/test-elastic-hotdata01.p12 : Enter password for test-elastic-hotdata02/test-elastic-hotdata02.p12 : Enter password for test-elastic-hotdata03/test-elastic-hotdata03.p12 : Enter password for test-elastic-warmdata01/test-elastic-warmdata01.p12 : Enter password for test-elastic-warmdata02/test-elastic-warmdata02.p12 : Enter password for test-elastic-warmdata03/test-elastic-warmdata03.p12 : Enter password for test-elastic-ml01/test-elastic-ml01.p12 : Certificates written to /root/elasticsearch_certs/server-cert-bundle.zip This file should be properly secured as it contains the private keys for all instances After unzipping the file, there will be a directory for each instance. Each instance has a single PKCS#12 (.p12) file containing the instance certificate, instance private key and the CA certificate For each Elastic product that you wish to configure, you should copy this '.p12' file to the relevant configuration directory and then follow the SSL configuration instructions in the product guide. For client applications, you may only need to copy the CA certificate and configure the client to trust this certificate. root@elasticsearch-jumperserver:~/elasticsearch_certs#
Enter fullscreen mode Exit fullscreen mode

below we are checking the generated certificates,

root@elasticsearch-jumperserver:~/elasticsearch_certs# ls elasticsearch-test-ca.p12 instances.yaml server-cert-bundle.zip root@elasticsearch-jumperserver:~/elasticsearch_certs# root@elasticsearch-jumperserver:~/elasticsearch_certs# unzip server-cert-bundle.zip Archive: server-cert-bundle.zip creating: test-elastic-master01/ inflating: test-elastic-master01/test-elastic-master01.p12 creating: test-elastic-master02/ inflating: test-elastic-master02/test-elastic-master02.p12 creating: test-elastic-master03/ inflating: test-elastic-master03/test-elastic-master03.p12 creating: test-elastic-hotdata01/ inflating: test-elastic-hotdata01/test-elastic-hotdata01.p12 creating: test-elastic-hotdata02/ inflating: test-elastic-hotdata02/test-elastic-hotdata02.p12 creating: test-elastic-hotdata03/ inflating: test-elastic-hotdata03/test-elastic-hotdata03.p12 creating: test-elastic-warmdata01/ inflating: test-elastic-warmdata01/test-elastic-warmdata01.p12 creating: test-elastic-warmdata02/ inflating: test-elastic-warmdata02/test-elastic-warmdata02.p12 creating: test-elastic-warmdata03/ inflating: test-elastic-warmdata03/test-elastic-warmdata03.p12 creating: test-elastic-ml01/ inflating: test-elastic-ml01/test-elastic-ml01.p12 root@elasticsearch-jumperserver:~/elasticsearch_certs# root@elasticsearch-jumperserver:~/elasticsearch_certs# ls elasticsearch-test-ca.p12 test-elastic-hotdata02 test-elastic-master01 test-elastic-master03 test-elastic-warmdata01 test-elastic-warmdata03 server-cert-bundle.zip test-elastic-hotdata01 test-elastic-hotdata03 test-elastic-master02 test-elastic-ml01 test-elastic-warmdata02 instances.yaml root@elasticsearch-jumperserver:~/elasticsearch_certs# root@elasticsearch-jumperserver:~/elasticsearch_certs# ls -ltr * -rw-r--r-- 1 root root 876 Oct 26 18:49 instances.yaml -rw------- 1 root root 2672 Oct 26 18:55 elasticsearch-test-ca.p12 -rw------- 1 root root 39406 Oct 26 18:56 server-cert-bundle.zip test-elastic-master01: total 4 -rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master01.p12 test-elastic-master02: total 4 -rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master02.p12 test-elastic-master03: total 4 -rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master03.p12 test-elastic-hotdata01: total 4 -rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata01.p12 test-elastic-hotdata03: total 4 -rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata03.p12 test-elastic-hotdata02: total 4 -rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata02.p12 test-elastic-warmdata01: total 4 -rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata01.p12 test-elastic-warmdata03: total 4 -rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata03.p12 test-elastic-warmdata02: total 4 -rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata02.p12 test-elastic-ml01: total 4 -rw-r--r-- 1 root root 3676 Oct 26 18:56 test-elastic-ml01.p12 root@elasticsearch-jumperserver:~/elasticsearch_certs#
Enter fullscreen mode Exit fullscreen mode

3c) Generate http certificate.

generate http certificates for http encryption, ensure you enter hostnames and ip's of the machines from which you would like you to communicate with elaticsearch over http, e.g jumpservers, kibana, elasticsearch nodes , so on.

/usr/share/elasticsearch/bin/elasticsearch-certutil http
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# /usr/share/elasticsearch/bin/elasticsearch-certutil http ## Elasticsearch HTTP Certificate Utility The 'http' command guides you through the process of generating certificates for use on the HTTP (Rest) interface for Elasticsearch. This tool will ask you a number of questions in order to generate the right set of files for your needs. ## Do you wish to generate a Certificate Signing Request (CSR)? A CSR is used when you want your certificate to be created by an existing Certificate Authority (CA) that you do not control (that is, you don't have access to the keys for that CA). If you are in a corporate environment with a central security team, then you may have an existing Corporate CA that can generate your certificate for you. Infrastructure within your organisation may already be configured to trust this CA, so it may be easier for clients to connect to Elasticsearch if you use a CSR and send that request to the team that controls your CA. If you choose not to generate a CSR, this tool will generate a new certificate for you. That certificate will be signed by a CA under your control. This is a quick and easy way to secure your cluster with TLS, but you will need to configure all your clients to trust that custom CA. Generate a CSR? [y/N]n ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate? If you have an existing CA certificate and key, then you can use that CA to sign your new http certificate. This allows you to use the same CA across multiple Elasticsearch clusters which can make it easier to configure clients, and may be easier for you to manage. If you do not have an existing CA, one will be generated for you. Use an existing CA? [y/N]y ## What is the path to your CA? Please enter the full pathname to the Certificate Authority that you wish to use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS (.jks) or PEM (.crt, .key, .pem) format. CA Path: /root/elasticsearch_certs/elasticsearch-test-ca.p12 Reading a PKCS12 keystore requires a password. It is possible for the keystore's password to be blank, in which case you can simply press <ENTER> at the prompt Password for elasticsearch-test-ca.p12: ## How long should your certificates be valid? Every certificate has an expiry date. When the expiry date is reached clients will stop trusting your certificate and TLS connections will fail. Best practice suggests that you should either: (a) set this to a short duration (90 - 120 days) and have automatic processes to generate a new certificate before the old one expires, or (b) set it to a longer duration (3 - 5 years) and then perform a manual update a few months before it expires. You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D) For how long should your certificate be valid? [5y] 10y ## Do you wish to generate one certificate per node? If you have multiple nodes in your cluster, then you may choose to generate a separate certificate for each of these nodes. Each certificate will have its own private key, and will be issued for a specific hostname or IP address. Alternatively, you may wish to generate a single certificate that is valid across all the hostnames or addresses in your cluster. If all of your nodes will be accessed through a single domain (e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it simpler to generate one certificate with a wildcard hostname (*.es.example.com) and use that across all of your nodes. However, if you do not have a common domain name, and you expect to add additional nodes to your cluster in the future, then you should generate a certificate per node so that you can more easily generate new certificates when you provision new nodes. Generate a certificate per node? [y/N]N ## Which hostnames will be used to connect to your nodes? These hostnames will be added as "DNS" names in the "Subject Alternative Name" (SAN) field in your certificate. You should list every hostname and variant that people will use to connect to your cluster over http. Do not list IP addresses here, you will be asked to enter them later. If you wish to use a wildcard certificate (for example *.es.example.com) you can enter that here. Enter all the hostnames that you need, one per line. When you are done, press <ENTER> once more to move on to the next step. test-elastic-master01 test-elastic-master02 test-elastic-master03 test-elastic-kibana01 test-elastic-clustmon01 elasticsearch-jumpserver You entered the following hostnames. - test-elastic-master01 - test-elastic-master02 - test-elastic-master03 - test-elastic-kibana01 - test-elastic-clustmon01 - elasticsearch-jumpserver Is this correct [Y/n]Y ## Which IP addresses will be used to connect to your nodes? If your clients will ever connect to your nodes by numeric IP address, then you can list these as valid IP "Subject Alternative Name" (SAN) fields in your certificate. If you do not have fixed IP addresses, or not wish to support direct IP access to your cluster then you can just press <ENTER> to skip this step. Enter all the IP addresses that you need, one per line. When you are done, press <ENTER> once more to move on to the next step. 10.10.4.6 10.10.4.7 10.10.4.8 10.10.4.5 10.10.4.16 10.10.4.17 10.10.4.18 10.10.4.1 10.10.4.31 You entered the following IP addresses. - 10.10.4.6 - 10.10.4.7 - 10.10.4.8 - 10.10.4.5 - 10.10.4.16 - 10.10.4.17 - 10.10.4.18 - 10.10.4.1 - 10.10.4.31 Is this correct [Y/n]Y ## Other certificate options The generated certificate will have the following additional configuration values. These values have been selected based on a combination of the information you have provided above and secure defaults. You should not need to change these values unless you have specific requirements. Key Name: test-elastic-master01 Subject DN: CN=test-elastic-master01 Key Size: 2048 Do you wish to change any of these options? [y/N]N ## What password do you want for your private key(s)? Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12". This type of keystore is always password protected, but it is possible to use a blank password. If you wish to use a blank password, simply press <enter> at the prompt below. Provide a password for the "http.p12" file: [<ENTER> for none] Repeat password to confirm: ## Where should we save the generated files? A number of files will be generated including your private key(s), public certificate(s), and sample configuration options for Elastic Stack products. These files will be included in a single zip archive. What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip] Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls elasticsearch elasticsearch-ssl-http.zip kibana root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch-ssl-http.zip elasticsearch-ssl-http.zip_old root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch elasticsearch_old root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv kibana kibana_old root@elasticsearch-jumperserver:~/elasticsearch_certs/http# pwd /root/elasticsearch_certs/http root@elasticsearch-jumperserver:~/elasticsearch_certs/http# cp /usr/share/elasticsearch/elasticsearch-ssl-http.zip . root@elasticsearch-jumperserver:~/elasticsearch_certs/http# unzip elasticsearch-ssl-http.zip Archive: elasticsearch-ssl-http.zip creating: elasticsearch/ inflating: elasticsearch/README.txt inflating: elasticsearch/http.p12 inflating: elasticsearch/sample-elasticsearch.yml creating: kibana/ inflating: kibana/README.txt inflating: kibana/elasticsearch-ca.pem inflating: kibana/sample-kibana.yml root@elasticsearch-jumperserver:~/elasticsearch_certs/http# root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls elasticsearch elasticsearch_old elasticsearch-ssl-http.zip elasticsearch-ssl-http.zip_old kibana kibana_old root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
Enter fullscreen mode Exit fullscreen mode

4) Copy the generated certificates

Copy the node certificate and http certificate to respective nodes to the path /etc/elasticsearch/certs/

Note: Node certificate is different for each and every elasticsearch node, http certificate is common for all the nodes.

*5) Setting keystore and trustore for transport and http *

Transport Truststore password is the password of CA certificate.
Transport Keystore password is the password of node certificates.

Transport http password is the password of http certificate.

set transport truststore/keystore and http keystore with below commands on all the nodes, you need run below commands on each and every elasticsearch nodes,

/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Enter fullscreen mode Exit fullscreen mode

6) Configuring elasticsearch parameters

Setting the configuration in /etc/elasticsearch/elasticsearch.yml, comment all the existing lines and append below after changing ip and hostnames to your node ip's and hostnames,

6a) Master nodes

path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.enrollment.enabled: true cluster.name: test-elasticsearch node.name: test-elastic-master01 network.host: 10.10.4.6 discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] node.roles: [ master ] xpack.watcher.enabled: true # transport SSL/TLS xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: full xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: certs/test-elastic-master01.p12 xpack.security.transport.ssl.truststore.path: certs/test-elastic-master01.p12 # http SSL/TLS http.host: 0.0.0.0 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6b) Hot nodes

path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.enrollment.enabled: true cluster.name: test-elasticsearch node.name: test-elastic-hotdata01 network.host: 10.10.4.2 discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] node.roles: [ data,ingest ] node.attr.box_type: hot xpack.watcher.enabled: true # transport SSL/TLS xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: full xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: certs/test-elastic-hotdata01.p12 xpack.security.transport.ssl.truststore.path: certs/test-elastic-hotdata01.p12 # http SSL/TLS http.host: 0.0.0.0 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6c) Warm nodes

path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.enrollment.enabled: true cluster.name: test-elasticsearch node.name: test-elastic-warmdata01 network.host: 10.10.4.11 discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] node.roles: [ data,ingest ] node.attr.box_type: warm xpack.watcher.enabled: true # transport SSL/TLS xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: full xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: certs/test-elastic-warmdata01.p12 xpack.security.transport.ssl.truststore.path: certs/test-elastic-warmdata01.p12 # http SSL/TLS http.host: 0.0.0.0 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6d) ML nodes

path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.enrollment.enabled: true cluster.name: test-elasticsearch node.name: test-elastic-ml01 network.host: 10.10.4.10 discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"] node.roles: [ ml ] xpack.watcher.enabled: true # transport SSL/TLS xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: full xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: certs/test-elastic-ml01.p12 xpack.security.transport.ssl.truststore.path: certs/test-elastic-ml01.p12 # http SSL/TLS http.host: 0.0.0.0 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

7) Starting elasticsearch

Start the nodes one by one using systemctl start elasticsearch, you can monitor the logs in /var/log/elasticsearch/test-elasticsearch.log

We need to ensure we remove the paramter once cluster is formed in /etc/elasticsearch/elasticsearch.yml

cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]

8) resetting elastic user password

you can also do this once you start the first node,

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic This tool will reset the password of the [elastic] user to an autogenerated value. The password will be printed in the console. Please confirm that you would like to continue [y/N]y Password for the [elastic] user successfully reset. New value: xxxxxxxxxxxxxxxxxxxxxxxx root@test-elastic-master01:/var/log/elasticsearch#
Enter fullscreen mode Exit fullscreen mode

9) Check the status of cluster and list nodes 

root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cluster/health?pretty" -u elastic -k Enter host password for user 'elastic': { "cluster_name" : "test-elasticsearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 6, "number_of_data_nodes" : 3, "active_primary_shards" : 2, "active_shards" : 4, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cat/nodes?pretty" -u elastic -k Enter host password for user 'elastic': 10.10.4.2 2 63 0 0.04 0.05 0.02 di - test-elastic-hotdata01 10.10.4.3 2 63 0 0.00 0.06 0.06 di - test-elastic-hotdata02 10.10.4.8 7 97 1 0.00 0.10 0.09 m - test-elastic-master03 10.10.4.7 11 96 2 0.00 0.03 0.01 m * test-elastic-master02 10.10.4.6 10 97 2 0.00 0.04 0.02 m - test-elastic-master01 10.10.4.4 2 62 0 0.00 0.06 0.05 di - test-elastic-hotdata03 root@test-elastic-master01:/var/log/elasticsearch#
Enter fullscreen mode Exit fullscreen mode

10) Install and Configure Kibana

10a) Installing kibana

root@test-elastic-kibana01:~# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg root@test-elastic-kibana01:~# apt-get install apt-transport-https Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: apt-transport-https 1 upgraded, 0 newly installed, 0 to remove and 118 not upgraded. Need to get 1,704 B of archives. After this operation, 0 B of additional disk space will be used. Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] Fetched 1,704 B in 1s (3,407 B/s) (Reading database ... 111616 files and directories currently installed.) Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... Unpacking apt-transport-https (2.0.9) over (2.0.8) ... Setting up apt-transport-https (2.0.9) ... root@test-elastic-kibana01:~# root@test-elastic-kibana01:~# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main root@test-elastic-kibana01:~# root@test-elastic-kibana01:~# sudo apt-get update && sudo apt-get install kibana 0% [Working] Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease Get:2 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB] Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease Hit:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease Get:5 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [34.0 kB] Hit:6 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease Get:7 https://artifacts.elastic.co/packages/8.x/apt stable/main i386 Packages [3,556 B] Fetched 48.0 kB in 1s (33.1 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: kibana 0 upgraded, 1 newly installed, 0 to remove and 118 not upgraded. Need to get 285 MB of archives. After this operation, 680 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.4.3 [285 MB] Fetched 285 MB in 3s (83.2 MB/s) Selecting previously unselected package kibana. (Reading database ... 111616 files and directories currently installed.) Preparing to unpack .../kibana_8.4.3_amd64.deb ... Unpacking kibana (8.4.3) ... Setting up kibana (8.4.3) ... Creating kibana group... OK Creating kibana user... OK Created Kibana keystore in /etc/kibana/kibana.keystore root@test-elastic-kibana01:~#
Enter fullscreen mode Exit fullscreen mode

10b) Copy the ca certificate to kibana server

Copy the ca certificate that was generated from the step 3c kibana/elasticsearch-ca.pem to /etc/kibana/elasticsearch-ca.pem

10c) Reset kibana_system password

To do below, login into one of the elasticsearch node which is added to http certificate.

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

Password for the [kibana_system] user successfully reset.
New value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
root@test-elastic-master01:/var/log/elasticsearch#

10d) Configuring kibana

set below parameters in /etc/kibana/kibana.yml, we are pointing to hot data nodes below

elasticsearch.hosts: ["https://10.10.4.2:9200","https://10.10.4.3:9200","https://10.10.4.4:9200"] elasticsearch.username: "kibana_system" elasticsearch.password: "xxxxxxxxxxxxxxxxxxxxxxxx" elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]
Enter fullscreen mode Exit fullscreen mode

10f) Start kibana and enable the service

systemctl start kibana systemctl enable kibana
Enter fullscreen mode Exit fullscreen mode

Access kibana using elastic user using url http://kibana-hostname:5601

Image description

Top comments (0)