First steps

After going through the Installation section and having installed all the operators, you will now deploy a NiFi cluster and the required dependencies. Afterwards you can verify that it works by querying the REST API.

Setup

Two things need to be installed to create a NiFi cluster:

  • A ZooKeeper cluster for internal use by NiFi

  • The NiFi cluster itself

We will create them in this order, each one is created by applying a manifest file. The operators you just installed will then create the resources according to the manifest.

Apache ZooKeeper

To create a ZooKeeper instance run the following command:

kubectl apply -f - <<EOF --- apiVersion: zookeeper.stackable.tech/v1alpha1 kind: ZookeeperCluster metadata: name: simple-zk spec: image: productVersion: 3.8.3 servers: roleGroups: default: replicas: 3 EOF

Create a Znode object:

kubectl apply -f - <<EOF --- apiVersion: zookeeper.stackable.tech/v1alpha1 kind: ZookeeperZnode metadata: name: simple-nifi-znode spec: clusterRef: name: simple-zk EOF

The ZNode makes sure that the NiFi cluster will operate in its own separated directory in ZooKeeper.

Apache NiFi

The NiFi cluster requires authentication. Create a set of credentials for this purpose:

kubectl apply -f - <<EOF --- apiVersion: v1 kind: Secret metadata: name: simple-admin-credentials stringData: admin: admin --- apiVersion: authentication.stackable.tech/v1alpha1 kind: AuthenticationClass metadata: name: simple-nifi-users spec: provider: static: userCredentialsSecret: name: simple-admin-credentials EOF

Finally create a NiFi instance:

kubectl apply -f - <<EOF --- apiVersion: nifi.stackable.tech/v1alpha1 kind: NifiCluster metadata: name: simple-nifi spec: image: productVersion: 1.23.2 clusterConfig: authentication: - authenticationClass: simple-nifi-users listenerClass: external-unstable sensitiveProperties: keySecret: nifi-sensitive-property-key autoGenerate: true zookeeperConfigMapName: simple-nifi-znode nodes: roleGroups: default: replicas: 2 EOF

Verify that it works

First, make sure all pods are ready:

kubectl wait -l statefulset.kubernetes.io/pod-name=simple-nifi-node-default-0 \ --for=condition=ready pod --timeout=1200s && \ kubectl wait -l statefulset.kubernetes.io/pod-name=simple-nifi-node-default-1 \ --for=condition=ready pod --timeout=1200s

Then make sure the StatefulSets are ready:

kubectl get statefulset

The output should show all pods ready:

NAME READY AGE simple-nifi-node-default 2/2 5m simple-zk-server-default 3/3 7m

Congratulations! You successfully created your first NiFi cluster!

Access the NiFi web interface

You can retrieve the URL for the NiFi cluster web interface via stackablectl or kubectl.

stackablectl

Use the service command of stackablectl to get a list of all available endpoints:

stackablectl stacklet list

which should return something like this:

 PRODUCT NAME NAMESPACE ENDPOINTS EXTRA INFOS nifi simple-nifi default https https://172.18.0.3:32595 zookeeper simple-zk default zk 172.18.0.3:30173

You can also use the json output and parse the endpoint:

nifi_url=$(stackablectl stacklet ls -o json | jq --raw-output '.[] | select(.name == "simple-nifi") | .endpoints.https')

Then connect to https://172.18.0.3:32595/nifi and you should see the NiFi web login. After providing the username admin and password admin you are redirected to the NiFi web interface.

nifi web ui

Via kubectl

Extracting the IP and port via kubectl is cumbersome. We recommend using stackablectl instead. The following kubectl commands store their output for further use in a variable and write its content to stdout afterwards. Make sure to run these commands in the same terminal:

nifi_node_name=$(kubectl get endpoints simple-nifi --output=jsonpath='{.subsets[0].addresses[0].nodeName}') && \ echo "NodeName: $nifi_node_name"

which should output a single node name where a NiFi pod is scheduled:

NodeName: kind-worker

Retrieve the IP of that node:

nifi_node_ip=$(kubectl get nodes -o jsonpath="{.items[?(@.metadata.name==\"$nifi_node_name\")].status.addresses[?(@.type==\"InternalIP\")].address}") && \ echo "NodeIp: $nifi_node_ip"

which should output the internal IP of that node:

NodeIp: 172.18.0.3

You might need to replace InternalIP with ExternalIP depending on how you connect to your Kubernetes cluster.

Finally, retrieve the NodePort of the simple-nifi service:

nifi_service_port=$(kubectl get service -o jsonpath="{.items[?(@.metadata.name==\"simple-nifi\")].spec.ports[?(@.name==\"https\")].nodePort}") && \ echo "NodePort: $nifi_service_port"

which should output the NodePort:

NodePort: 32595

Now build the full URL:

nifi_url="https://$nifi_node_ip:$nifi_service_port" && \ echo "NiFi web interface: $nifi_url"

which should output a URL to connect to the NiFi web interface:

NiFi web interface: https://172.18.0.3:32595

Then connect to https://172.18.0.3:32595/nifi and you should see the NiFi web login. After providing the username admin and password admin you are redirected to the NiFi web interface.

What’s next

Have a look at the Usage guide page to find out more about the features of the NiFi Operator.