Strimzi: Deploy Kafka in easy mode

The first time I installed Kafka, I did it in an Ubuntu Jammy VM with native packages and custom CAs for both server and client authentication. I did not enjoy it. The native packages update faster than app requirements, the custom CA setup is error-prone and distributing the certificates is rather clunky. I don’t particularly like the configuration files and updating the ACLs was an exercise in frustration.

Second time, I went with a simple compose file so I could keep several versions in parallel. That helped with the cluster maintenance, but did nothing to simplify the certificate distribution or ACL editing.

Meanwhile, we also had an AMQ Streams-based cluster (now Streams for Apache Kafka) running in our development OpenShift cluster, which I’d also helped set up. This, on the other hand, was a breeze to maintain. The CRDs the operator bundled made editing ACLs easy, and the certificates were there as secrets, directly in the cluster. However, neither we nor our customers always run OpenShift. Digging into how AMQ Streams did its magic, I ended up at Strimzi’s doorstep.

Strimzi is a Kubernetes operator that turns Kafka deployment into a declarative, repeatable process that just works. Once you’ve tried it, going back to manual setup feels like a step backward. So I set out to understand how I could have the AMQ Streams experience in a vanilla K8s cluster.

Strimzi keeps your concerns in CRDs

My first Kafka deployment meant provisioning a VM, installing Java, configuring server.properties by hand, setting up ZooKeeper, and then wiring up TLS, users, and ACLs. This was without any HA (fine, as this was a dev instance for a single team), and getting the cluster running was easy enough, but I still struggled with getting the configuration just right for what the applications needed.

Strimzi replaces all of that with Kubernetes custom resources. You declare Kafka clusters, node pools, topics, users, and the operator takes care of all the reconciliation. It handles TLS certificate generation, rolling restarts, scaling, and the KRaft migration (no more ZooKeeper).

Deploying Strimzi

To test this, I deployed an AKS single-node cluster via OpenTofu and installed cert-manager via Helm. Strimzi does not strictly require cert-manager, but enabling TLS encryption (Kafka.spec.kafka.listeners.tls) or rack awareness for KafkaNodePools does. We install it for TLS support.

With that out of the way, the Strimzi operator can be installed with a Helm chart invocation:

helm repo add strimzi https://strimzi.io/charts/
helm repo update

helm upgrade --install strimzi-operator strimzi/strimzi-kafka-operator \
  --namespace strimzi \
  --create-namespace \
  --set watchAnyNamespace=true \
  --wait

watchAnyNamespace=true lets the operator watch for Kafka resources in all namespaces. For production, scope it down with --set watchNamespaces={kafka}.

Verify:

kubectl get pods -n strimzi

You should see the Strimzi Cluster Operator pod running:

NAME                                   READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-6d7c8f5b9b   1/1     Running   0          45s

Creating a Kafka cluster

For MES Kafka use, we create users with specific ACLs for the MES topics. These topics follow a strict nomenclature, so to ease new project onboarding I created a Helm chart that deploys a Kafka cluster, then creates a set of topics and a set of users for each project. This makes it easy to add new projects in the future as well. Configuring the Helm chart is as easy as setting up a values.yaml such as:

kafkaCluster:
  name: my-kafka
  namespace: kafka
  version: "3.9.0"
  listeners:
    - name: plain
      port: 9092
      type: internal
      tls: false
    - name: tls
      port: 9093
      type: internal
      tls: true
      authentication:
        type: tls
  config:
    offsets.topic.replication.factor: 1
    default.replication.factor: 1
    min.insync.replicas: 1

kafkaNodePools:
  - name: dual-role
    replicas: 1
    roles:
      - controller
      - broker
    storage:
      type: ephemeral

topics:
  - name: app-events
    partitions: 3
    replicas: 1
    config:
      retention.ms: 604800000

projects:
  - name: MyApp
    systemName: my_app

One helm upgrade --install creates the Kafka cluster, node pool, topics, and KafkaUsers with the correct ACLs for MES use. Note that this is for a test or development setup. For production, you need to increase the node pool and the topic replicas.

Ditching ZooKeeper for KRaft

Strimzi v1 uses KRaft mode by default, which means no ZooKeeper dependency. The KafkaNodePool resource lets you assign roles (controller, broker) to nodes, and in KRaft a single dual-role node pool is fine for development. For production you’d split the roles over three controller nodes and three+ broker nodes, but this is just an edit in the resource, not setting up VMs, distributing the CAs, signing the server certs, etc.

Lessons learned: storage and throughput

When we first deployed AMQ Streams on an OpenShift cluster backed by OpenShift Data Foundation, the storage had plenty of capacity, but not enough throughput. Kafka is I/O-heavy: producers and consumers hammer the disks with sequential reads and writes, and if the underlying storage can’t keep up, you get latency spikes, and lag increases.

Make sure you have storage with high IOPS and throughput. On my Azure setup, that means choosing the managed-csi-premium storage class. On-prem, NVMe-backed volumes are a good bet, or a good storage appliance if you have it. Even in development clusters, if many projects will use them, you will notice the lag pretty quickly.

Once you have performant storage, Strimzi makes the switch easy by setting storage.type: persistent-claim and pointing the Persistent Volumes to the right storageClass in your KafkaNodePool spec.

---

Looking back at the configuration, there are surprisingly few steps needed for deploying Kafka via Strimzi. We aren’t messing about with custom CAs and their certificates, and upgrading a Kafka cluster is now a resource update. The operator has regular releases and is the bedrock of Red Hat’s AMQ Streams offering. If you have a Kubernetes cluster with good storage, it’s a good approach for a mostly hands-free maintenance story of your Kafka clusters.