How to connect to Kafka over TLS to a vanilla (Strimzi) Kafka cluster #1260

landbaychrisburrell · 2025-08-11T20:52:37Z

landbaychrisburrell
Aug 11, 2025

Hi

I'm trying to set up Kafbat with TLS configuration. I run an internal Kafka cluster using Strimzi, and am trying to use the Helm Chart to set up kafbat. When running over a plain listener, everything works well, and I have managed to get scram-sha-512 working over a plain connection.

However, when I try to connect over TLS, then this fails. My idea would be to run TLS connections with TLS authentication, but failing that TLS connection with scram-sha-512 over TLS would be fine also.

The error I'm currently getting is

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Kafbat is given the following chart:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kafbat-ui
  namespace: kafka
spec:
  releaseName: kafbat-ui
  chart:
    spec:
      chart: kafka-ui
      version: "1.5.*"
      sourceRef:
        kind: HelmRepository
        name: kafbat
        namespace: kafka
  interval: 5m
  values:
    securityContext:
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
    priorityClassName: "landbay-critical-priority"
    volumes:
      - name: kafka-credentials
        secret:
          secretName: kafbat-user
      - name: kafka-clients-ca
        secret:
          secretName: kafka-mycluster-clients-ca-cert

    volumeMounts:
      - name: kafka-credentials
        mountPath: /etc/kafka/secrets
        readOnly: true

      - name: kafka-clients-ca
        mountPath: /etc/kafka/clients-ca
        readOnly: true


    yamlApplicationConfig:
      kafka:
        clusters:
          - name: mycluster
            bootstrapServers:  kafka-mycluster-kafka-bootstrap:9093
            ssl:
              truststoreLocation: "/etc/kafka/clients-ca/ca.p12"
              truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}"
              verifySsl: false
            properties:
              security.protocol: SASL_SSL
              sasl.mechanism: SCRAM-SHA-512
              sasl.jaas.config: ${SECRET_JAAS_CONFIG}
              ssl.truststore.location: "/etc/kafka/clients-ca/ca.p12"
              ssl.truststore.password: "${SECRET_TRUSTSTORE_PASSWORD}"
              ssl.truststore.type: "PKCS12"
              ssl.verifySsl: false


    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-Xmx512m -Xms256m" 

      - name: "SPRING_CONFIG_ADDITIONAL-LOCATION"
        value: "/etc/kafka/config"

      - name: SECRET_JAAS_CONFIG
        valueFrom:
          secretKeyRef:
            name: kafbat-user
            key: sasl.jaas.config

      - name: SECRET_TRUSTSTORE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: kafka-mycluster-clients-ca-cert
            key: ca.password

On start-up, I get the following details output before the cert-path error, which seem to show that some of the configuration above is not being picked up? e.g. the truststore type for example?

		ssl.truststore.type = JKS
		ssl.truststore.password = [hidden]
		ssl.truststore.location = /etc/kafka/clients-ca/ca.p12
		ssl.truststore.certificates = null
		ssl.trustmanager.algorithm = PKIX
		ssl.secure.random.implementation = null
		ssl.provider = null
		ssl.protocol = TLSv1.3
		ssl.keystore.type = JKS
		ssl.keystore.password = null
		ssl.keystore.location = null
		ssl.keystore.key = null
		ssl.keystore.certificate.chain = null
		ssl.keymanager.algorithm = SunX509
		ssl.key.password = null
		ssl.engine.factory.class = null
		ssl.endpoint.identification.algorithm = 
		ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
		ssl.cipher.suites = null
		socket.connection.setup.timeout.ms = 10000
		socket.connection.setup.timeout.max.ms = 30000
		send.buffer.bytes = 131072
		security.providers = null
		security.protocol = SASL_SSL
		sasl.oauthbearer.token.endpoint.url = null
		sasl.oauthbearer.sub.claim.name = sub
		sasl.oauthbearer.scope.claim.name = scope
		sasl.oauthbearer.jwks.endpoint.url = null
		sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
		sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
		sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
		sasl.oauthbearer.header.urlencode = false
		sasl.oauthbearer.expected.issuer = null
		sasl.oauthbearer.expected.audience = null
		sasl.oauthbearer.clock.skew.seconds = 30
		sasl.mechanism = SCRAM-SHA-512
		sasl.login.retry.backoff.ms = 100
		sasl.login.retry.backoff.max.ms = 10000
		sasl.login.refresh.window.jitter = 0.05
		sasl.login.refresh.window.factor = 0.8
		sasl.login.refresh.min.period.seconds = 60
		sasl.login.refresh.buffer.seconds = 300
		sasl.login.read.timeout.ms = null
		sasl.login.connect.timeout.ms = null
		sasl.login.class = null
		sasl.login.callback.handler.class = null
		sasl.kerberos.ticket.renew.window.factor = 0.8
		sasl.kerberos.ticket.renew.jitter = 0.05
		sasl.kerberos.service.name = null
		sasl.kerberos.min.time.before.relogin = 60000
		sasl.kerberos.kinit.cmd = /usr/bin/kinit
		sasl.jaas.config = [hidden]
		sasl.client.callback.handler.class = null
		retry.backoff.ms = 100
		retry.backoff.max.ms = 1000
		retries = 2147483647
		request.timeout.ms = 30000
		reconnect.backoff.ms = 50
		reconnect.backoff.max.ms = 1000
		receive.buffer.bytes = 65536
		metrics.sample.window.ms = 30000
		metrics.recording.level = INFO
		metrics.num.samples = 2
		metric.reporters = []
		metadata.recovery.strategy = none
		metadata.max.age.ms = 300000
		enable.metrics.push = true
		default.api.timeout.ms = 60000
		connections.max.idle.ms = 300000
		client.id = kafbat-ui-admin-1754937263-182
		client.dns.lookup = use_all_dns_ips
		bootstrap.servers = [kafka-george-kafka-bootstrap:9093]
		bootstrap.controllers = []
		auto.include.jmx.reporter = true

Any hints as to where I'm going wrong? I checked open_ssl s_client against my kafka, and it does seem like kafka-mycluster-kafka-bootstrap is listed, amongst others such as kafka-mycluster-kafka-bootstrap.kafka.svc, etc.

For mutual TLS, i would also appreciate a hint also - is it just setting the kafka.clusters.properties.keystore values in the same way?

Cheers

Answered by landbaychrisburrell

Aug 12, 2025

Hi

Thanks for that - I actually thought of that too just as i posted and that worked - thanks for the hint. For posterity and for others, here is the final configuration I settled on:

listeners:
      - name: tls
        port: 9094
        type: internal
        tls: true
        authentication:
          type: tls

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kafbat-ui
  namespace: kafka
spec:
  releaseName: kafbat-ui
  chart:
    spec:
      chart: kafka-ui
      version: "1.5.*"
      sourceRef:
        kind: HelmRepository
        name: kafbat
        namespace: kafka
  interval: 5m
  values:
    securityContext:
      capabilities:
        drop:
         …

View full answer

fallen-up · 2025-08-12T07:07:06Z

fallen-up
Aug 12, 2025

    yamlApplicationConfig:
      kafka:
        clusters:
          - name: mycluster
            bootstrapServers:  kafka-mycluster-kafka-bootstrap:9093
            ssl:
              truststoreLocation: "/etc/kafka/ca/ca.p12"
              truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}"
              verifySsl: false
            properties:
              security.protocol: SSL
              ssl.keystore.location: "/etc/kafka/user/user.p12"
              ssl.keystore.password: "${SECRET_TRUSTSTORE_PASSWORD}"

also you need take another CA from another strimzi-secret

0 replies

landbaychrisburrell · 2025-08-12T10:38:40Z

landbaychrisburrell
Aug 12, 2025
Author

Hi

Thanks for the prompt answer - i now still get the same issue, despite passing the user's p12 cert in the key store.

The config i tried:

    yamlApplicationConfig:
      kafka:
        clusters:
          - name: mycluster
            bootstrapServers:  kafka-mycluster-kafka-bootstrap:9094
            ssl:
              truststoreLocation: "/etc/kafka/clients-ca/ca.p12"
              truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}"
              verifySsl: false
            properties:
              security.protocol: SSL
              ssl.truststore.type: "PKCS12"
              ssl.keystore.type: "PKCS12"
              ssl.verifySsl: false
              ssl.keystore.location: "/etc/kafka/user/user.p12"
              ssl.keystore.password: "${SECRET_KEYSTORE_PASSWORD}"

The error i still get

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target                                                                            
     at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)                                                                                                         
     at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)                                                                                                   
     at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)                                                                                                                           
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)

Here is the output from kafbat
``
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null │
ssl.keystore.location = /etc/kafka/user/user.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = PKCS12
ssl.protocol = TLSv1.3 │
ssl.provider = null │
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null │
ssl.truststore.location = /etc/kafka/clients-ca/ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12


I've double checked my cert itself and the DNS entry is there:

        X509v3 Subject Alternative Name:
            DNS:kafka-mycluster-kafka-brokers.kafka.svc, DNS:kafka-mycluster-kafka-bootstrap, DNS:kafka-mycluster-kafka-bootstrap.kafka, DNS:kafka-mycluster-kafka-bootstrap.kafka.svc.cluster.local, DNS:kafka-mycluster-kafka-brokers.kafka.svc.cluster.local, DNS:kafka-mycluster-kafka-bootstrap.kafka.svc, DNS:kafka-mycluster-dual-role-0.kafka-mycluster-kafka-brokers.kafka.svc, DNS:kafka-mycluster-kafka-brokers, DNS:kafka-mycluster-kafka-brokers.kafka, DNS:kafka-mycluster-dual-role-0.kafka-mycluster-kafka-brokers.kafka.svc.cluster.local


In terms of the chain, i have

depth=1 O=io.strimzi, CN=cluster-ca v0
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O=io.strimzi, CN=cluster-ca v0
verify return:1
depth=0 O=io.strimzi, CN=kafka-mycluster-kafka
verify return:1

1 reply

fallen-up Aug 12, 2025

not sure, but it looks like you're using the wrong CA. еhe error message indicates this - Java cannot build a trusted certificate chain.
it's strimzi-kafka magic. the CA should be taken from $clustername-cluster-ca-cert, not from the secret $clustername-clients-ca-cert.

landbaychrisburrell · 2025-08-12T11:13:41Z

landbaychrisburrell
Aug 12, 2025
Author

Hi

Thanks for that - I actually thought of that too just as i posted and that worked - thanks for the hint. For posterity and for others, here is the final configuration I settled on:

listeners:
      - name: tls
        port: 9094
        type: internal
        tls: true
        authentication:
          type: tls

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kafbat-ui
  namespace: kafka
spec:
  releaseName: kafbat-ui
  chart:
    spec:
      chart: kafka-ui
      version: "1.5.*"
      sourceRef:
        kind: HelmRepository
        name: kafbat
        namespace: kafka
  interval: 5m
  values:
    securityContext:
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
    priorityClassName: "my-high-priority"
    volumes:
      - name: kafka-mycluster-cluster-ca-cert
        secret:
          secretName: kafka-mycluster-cluster-ca-cert

      - name: kafka-user-keystore
        secret:
          secretName: mykafbatuser-tls-user

    volumeMounts:

      - name: kafka-mycluster-cluster-ca-cert
        mountPath: /etc/kafka/cluster-ca
        readOnly: true

      - name: kafka-user-keystore
        mountPath: /etc/kafka/user
        readOnly: true

    ingress:
      enabled: true
      ingressClassName: internal-portal-ingress-class
      path: "/"
      host: kafbat-uat.landbay.uk
      annotations:
        alb.ingress.kubernetes.io/healthcheck-path: /
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'

    yamlApplicationConfig:
      kafka:
        clusters:
          - name: mycluster
            bootstrapServers:  kafka-mycluster-kafka-bootstrap:9094
            ssl:
              truststoreLocation: "/etc/kafka/cluster-ca/ca.p12"
              truststorePassword: "${SECRET_TRUSTSTORE_PASSWORD}"
            properties:
              security.protocol: SSL
              ssl.keystore.location: "/etc/kafka/user/user.p12"
              ssl.keystore.password: "${SECRET_KEYSTORE_PASSWORD}"

    env:
      - name: "SPRING_CONFIG_ADDITIONAL-LOCATION"
        value: "/etc/kafka/config"

      - name: SECRET_TRUSTSTORE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: kafka-mycluster-cluster-ca-cert
            key: ca.password

      - name: SECRET_KEYSTORE_PASSWORD
        valueFrom:
          secretKeyRef:
            name:  mykafbatuser-tls-user
            key: user.password

Thanks again for the help.

1 reply

Haarolean Aug 15, 2025
Maintainer

@fallen-up thanks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

How to connect to Kafka over TLS to a vanilla (Strimzi) Kafka cluster #1260

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

How to connect to Kafka over TLS to a vanilla (Strimzi) Kafka cluster #1260

Uh oh!

landbaychrisburrell Aug 11, 2025

Replies: 3 comments · 2 replies

Uh oh!

fallen-up Aug 12, 2025

Uh oh!

landbaychrisburrell Aug 12, 2025 Author

Uh oh!

fallen-up Aug 12, 2025

Uh oh!

landbaychrisburrell Aug 12, 2025 Author

Uh oh!

Haarolean Aug 15, 2025 Maintainer

landbaychrisburrell
Aug 11, 2025

Replies: 3 comments 2 replies

fallen-up
Aug 12, 2025

landbaychrisburrell
Aug 12, 2025
Author

landbaychrisburrell
Aug 12, 2025
Author

Haarolean Aug 15, 2025
Maintainer