How to connect to Kafka over TLS to a vanilla (Strimzi) Kafka cluster

[landbaychrisburrell]

Hi

I'm trying to set up Kafbat with TLS configuration. I run an internal Kafka cluster using Strimzi, and am trying to use the Helm Chart to set up kafbat. When running over a plain listener, everything works well, and I have managed to get scram-sha-512 working over a plain connection.

However, when I try to connect over TLS, then this fails. My idea would be to run TLS connections with TLS authentication, but failing that TLS connection with scram-sha-512 over TLS would be fine also.

The error I'm currently getting is

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 

Kafbat is given the following chart:

apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: kafbat-ui namespace: kafka spec: releaseName: kafbat-ui chart: spec: chart: kafka-ui version: "1.5.*" sourceRef: kind: HelmRepository name: kafbat namespace: kafka interval: 5m values: securityContext: capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false seccompProfile: type: RuntimeDefault priorityClassName: "landbay-critical-priority" volumes: - name: kafka-credentials secret: secretName: kafbat-user - name: kafka-clients-ca secret: secretName: kafka-mycluster-clients-ca-cert volumeMounts: - name: kafka-credentials mountPath: /etc/kafka/secrets readOnly: true - name: kafka-clients-ca mountPath: /etc/kafka/clients-ca readOnly: true yamlApplicationConfig: kafka: clusters: - name: mycluster bootstrapServers: kafka-mycluster-kafka-bootstrap:9093 ssl: truststoreLocation: "/etc/kafka/clients-ca/ca.p12" truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}" verifySsl: false properties: security.protocol: SASL_SSL sasl.mechanism: SCRAM-SHA-512 sasl.jaas.config: ${SECRET_JAAS_CONFIG} ssl.truststore.location: "/etc/kafka/clients-ca/ca.p12" ssl.truststore.password: "${SECRET_TRUSTSTORE_PASSWORD}" ssl.truststore.type: "PKCS12" ssl.verifySsl: false env: - name: JAVA_TOOL_OPTIONS value: "-Xmx512m -Xms256m" - name: "SPRING_CONFIG_ADDITIONAL-LOCATION" value: "/etc/kafka/config" - name: SECRET_JAAS_CONFIG valueFrom: secretKeyRef: name: kafbat-user key: sasl.jaas.config - name: SECRET_TRUSTSTORE_PASSWORD valueFrom: secretKeyRef: name: kafka-mycluster-clients-ca-cert key: ca.password 

On start-up, I get the following details output before the cert-path error, which seem to show that some of the configuration above is not being picked up? e.g. the truststore type for example?

ssl.truststore.type = JKS	ssl.truststore.password = [hidden]	ssl.truststore.location = /etc/kafka/clients-ca/ca.p12	ssl.truststore.certificates = null	ssl.trustmanager.algorithm = PKIX	ssl.secure.random.implementation = null	ssl.provider = null	ssl.protocol = TLSv1.3	ssl.keystore.type = JKS	ssl.keystore.password = null	ssl.keystore.location = null	ssl.keystore.key = null	ssl.keystore.certificate.chain = null	ssl.keymanager.algorithm = SunX509	ssl.key.password = null	ssl.engine.factory.class = null	ssl.endpoint.identification.algorithm =	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]	ssl.cipher.suites = null	socket.connection.setup.timeout.ms = 10000	socket.connection.setup.timeout.max.ms = 30000	send.buffer.bytes = 131072	security.providers = null	security.protocol = SASL_SSL	sasl.oauthbearer.token.endpoint.url = null	sasl.oauthbearer.sub.claim.name = sub	sasl.oauthbearer.scope.claim.name = scope	sasl.oauthbearer.jwks.endpoint.url = null	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000	sasl.oauthbearer.header.urlencode = false	sasl.oauthbearer.expected.issuer = null	sasl.oauthbearer.expected.audience = null	sasl.oauthbearer.clock.skew.seconds = 30	sasl.mechanism = SCRAM-SHA-512	sasl.login.retry.backoff.ms = 100	sasl.login.retry.backoff.max.ms = 10000	sasl.login.refresh.window.jitter = 0.05	sasl.login.refresh.window.factor = 0.8	sasl.login.refresh.min.period.seconds = 60	sasl.login.refresh.buffer.seconds = 300	sasl.login.read.timeout.ms = null	sasl.login.connect.timeout.ms = null	sasl.login.class = null	sasl.login.callback.handler.class = null	sasl.kerberos.ticket.renew.window.factor = 0.8	sasl.kerberos.ticket.renew.jitter = 0.05	sasl.kerberos.service.name = null	sasl.kerberos.min.time.before.relogin = 60000	sasl.kerberos.kinit.cmd = /usr/bin/kinit	sasl.jaas.config = [hidden]	sasl.client.callback.handler.class = null	retry.backoff.ms = 100	retry.backoff.max.ms = 1000	retries = 2147483647	request.timeout.ms = 30000	reconnect.backoff.ms = 50	reconnect.backoff.max.ms = 1000	receive.buffer.bytes = 65536	metrics.sample.window.ms = 30000	metrics.recording.level = INFO	metrics.num.samples = 2	metric.reporters = []	metadata.recovery.strategy = none	metadata.max.age.ms = 300000	enable.metrics.push = true	default.api.timeout.ms = 60000	connections.max.idle.ms = 300000	client.id = kafbat-ui-admin-1754937263-182	client.dns.lookup = use_all_dns_ips	bootstrap.servers = [kafka-george-kafka-bootstrap:9093]	bootstrap.controllers = []	auto.include.jmx.reporter = true 

Any hints as to where I'm going wrong? I checked open_ssl s_client against my kafka, and it does seem like kafka-mycluster-kafka-bootstrap is listed, amongst others such as kafka-mycluster-kafka-bootstrap.kafka.svc, etc.

For mutual TLS, i would also appreciate a hint also - is it just setting the kafka.clusters.properties.keystore values in the same way?

Cheers

[fallen-up]

 yamlApplicationConfig: kafka: clusters: - name: mycluster bootstrapServers: kafka-mycluster-kafka-bootstrap:9093 ssl: truststoreLocation: "/etc/kafka/ca/ca.p12" truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}" verifySsl: false properties: security.protocol: SSL ssl.keystore.location: "/etc/kafka/user/user.p12" ssl.keystore.password: "${SECRET_TRUSTSTORE_PASSWORD}"

also you need take another CA from another strimzi-secret

[landbaychrisburrell]

Hi

Thanks for the prompt answer - i now still get the same issue, despite passing the user's p12 cert in the key store.

The config i tried:

 yamlApplicationConfig: kafka: clusters: - name: mycluster bootstrapServers: kafka-mycluster-kafka-bootstrap:9094 ssl: truststoreLocation: "/etc/kafka/clients-ca/ca.p12" truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}" verifySsl: false properties: security.protocol: SSL ssl.truststore.type: "PKCS12" ssl.keystore.type: "PKCS12" ssl.verifySsl: false ssl.keystore.location: "/etc/kafka/user/user.p12" ssl.keystore.password: "${SECRET_KEYSTORE_PASSWORD}" 

The error i still get

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383) 

Here is the output from kafbat
``
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null │
ssl.keystore.location = /etc/kafka/user/user.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = PKCS12
ssl.protocol = TLSv1.3 │
ssl.provider = null │
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null │
ssl.truststore.location = /etc/kafka/clients-ca/ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12

 I've double checked my cert itself and the DNS entry is there: 

 X509v3 Subject Alternative Name: DNS:kafka-mycluster-kafka-brokers.kafka.svc, DNS:kafka-mycluster-kafka-bootstrap, DNS:kafka-mycluster-kafka-bootstrap.kafka, DNS:kafka-mycluster-kafka-bootstrap.kafka.svc.cluster.local, DNS:kafka-mycluster-kafka-brokers.kafka.svc.cluster.local, DNS:kafka-mycluster-kafka-bootstrap.kafka.svc, DNS:kafka-mycluster-dual-role-0.kafka-mycluster-kafka-brokers.kafka.svc, DNS:kafka-mycluster-kafka-brokers, DNS:kafka-mycluster-kafka-brokers.kafka, DNS:kafka-mycluster-dual-role-0.kafka-mycluster-kafka-brokers.kafka.svc.cluster.local 

 In terms of the chain, i have 

depth=1 O=io.strimzi, CN=cluster-ca v0
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O=io.strimzi, CN=cluster-ca v0
verify return:1
depth=0 O=io.strimzi, CN=kafka-mycluster-kafka
verify return:1

[fallen-up]

not sure, but it looks like you're using the wrong CA. еhe error message indicates this - Java cannot build a trusted certificate chain.
it's strimzi-kafka magic. the CA should be taken from $clustername-cluster-ca-cert, not from the secret $clustername-clients-ca-cert.

[landbaychrisburrell]

Hi

Thanks for that - I actually thought of that too just as i posted and that worked - thanks for the hint. For posterity and for others, here is the final configuration I settled on:

listeners: - name: tls port: 9094 type: internal tls: true authentication: type: tls 

apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: kafbat-ui namespace: kafka spec: releaseName: kafbat-ui chart: spec: chart: kafka-ui version: "1.5.*" sourceRef: kind: HelmRepository name: kafbat namespace: kafka interval: 5m values: securityContext: capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false seccompProfile: type: RuntimeDefault priorityClassName: "my-high-priority" volumes: - name: kafka-mycluster-cluster-ca-cert secret: secretName: kafka-mycluster-cluster-ca-cert - name: kafka-user-keystore secret: secretName: mykafbatuser-tls-user volumeMounts: - name: kafka-mycluster-cluster-ca-cert mountPath: /etc/kafka/cluster-ca readOnly: true - name: kafka-user-keystore mountPath: /etc/kafka/user readOnly: true ingress: enabled: true ingressClassName: internal-portal-ingress-class path: "/" host: kafbat-uat.landbay.uk annotations: alb.ingress.kubernetes.io/healthcheck-path: / alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' yamlApplicationConfig: kafka: clusters: - name: mycluster bootstrapServers: kafka-mycluster-kafka-bootstrap:9094 ssl: truststoreLocation: "/etc/kafka/cluster-ca/ca.p12" truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}" properties: security.protocol: SSL ssl.keystore.location: "/etc/kafka/user/user.p12" ssl.keystore.password: "${SECRET_KEYSTORE_PASSWORD}" env: - name: "SPRING_CONFIG_ADDITIONAL-LOCATION" value: "/etc/kafka/config" - name: SECRET_TRUSTSTORE_PASSWORD valueFrom: secretKeyRef: name: kafka-mycluster-cluster-ca-cert key: ca.password - name: SECRET_KEYSTORE_PASSWORD valueFrom: secretKeyRef: name: mykafbatuser-tls-user key: user.password 

Thanks again for the help.

[Haarolean]

@fallen-up thanks!

[discostur]

Hi,

thanks for the good answer @landbaychrisburrell ; Just if someone wants to connect via SSL but authenticate via SCRAM here is the correct configuration:

Strimzi kafka listener:

spec: kafka: ... listeners: - name: scramsha port: 9093 type: internal tls: true authentication: type: scram-sha-512 authorization: type: simple

Strimzi kafka user:

apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaUser metadata: name: kafbat-ui labels: strimzi.io/cluster: dev1-kafka spec: authentication: type: scram-sha-512 password: valueFrom: secretKeyRef: name: kafbat-ui-kafka-user-password key: password authorization: type: simple acls: - resource: type: topic name: "*" patternType: literal operations: - All - resource: type: group name: "*" patternType: literal operations: - All

Kafbat-ui helm values:

... yamlApplicationConfig: kafka: clusters: - name: dev1-kafka bootstrapServers: dev1-kafka-kafka-brokers:9093 ssl: truststoreLocation: /etc/kafka/cluster-ca/ca.p12 truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}" properties: security.protocol: SASL_SSL sasl.mechanism: SCRAM-SHA-512 sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="kafbat-ui" password="XXX"; volumes: - name: dev1-kafka-cluster-ca-cert secret: secretName: dev1-kafka-cluster-ca-cert volumeMounts: - name: dev1-kafka-cluster-ca-cert mountPath: /etc/kafka/cluster-ca readOnly: true env: - name: SECRET_TRUSTSTORE_PASSWORD valueFrom: secretKeyRef: name: dev1-kafka-cluster-ca-cert key: ca.password