Apache Kafka¶

Published: 11 Nov. 2026 Last updated: 11 Nov. 2026

Summary¶

Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications. Kafka stores streams of records in topics, which are partitioned and replicated across a cluster of brokers for fault tolerance and high throughput.

When deployed on Kubernetes, Kafka brokers persist topic data, consumer offsets, and cluster metadata to PersistentVolumes. Modern Kafka deployments use KRaft (Kafka Raft) mode, which eliminates the dependency on Apache ZooKeeper by using an internal Raft-based consensus protocol for metadata management.

Kafka’s persistent state includes:

Topic data — messages (records) stored as append-only log segments across partitions.
Consumer offsets — stored in the internal __consumer_offsets topic, tracking each consumer group’s position.
Transaction state — stored in the internal __transaction_state topic for exactly-once semantics.
Cluster metadata (KRaft) — stored in the internal __cluster_metadata topic, containing topic configurations, partition assignments, broker registrations, and ACLs.

This Application Note discusses how to use CloudCasa to back up and restore Apache Kafka running in containers under Kubernetes.

CloudCasa has been tested for this application note with a 3-node Kafka cluster deployed using the Strimzi Kafka operator as well as with a plain StatefulSet using the official Apache Kafka Docker image. The information herein is expected to apply to more recent versions and to different cluster sizes as well.

Prerequisites¶

The following requirements must be met for a successful backup and restore of Kafka with CloudCasa:

Kafka broker pods must use PersistentVolumes for data storage. Kafka deployments using emptyDir or in-memory storage will lose all data when pods are restarted and cannot be backed up.
Kafka brokers should be deployed as a StatefulSet with stable network identities. This is the default for both the Strimzi operator and the official Kafka Docker image.
Messages that need to be backed up must be persisted to disk. Kafka writes all messages to the log directory on the PV by default, so this is satisfied in a standard deployment.

Note

Kafka uses an append-only, log-structured storage engine. Each partition is a sequence of immutable log segment files that are only appended to or deleted (by retention policy). This makes PV-level backup generally safe, as there is no risk of capturing a partially written record — unlike databases that use write-ahead logs with in-place updates.

No application hooks are required for Kafka backup. CloudCasa backs up the Kafka PVs directly.

Backup¶

Run a full cluster backup that includes the Kafka namespace and all cluster-scoped resources. If using the Strimzi operator, this ensures that all Strimzi CRDs (Kafka, KafkaNodePool, KafkaTopic, etc.) are included.

No application hooks need to be configured. Kafka’s log-structured storage is append-only and does not require quiescing before a PV-level backup.

Restore¶

The restore procedure depends on how Kafka was deployed. The Strimzi operator requires additional post-restore steps because it generates a new internal cluster ID on reconciliation, which conflicts with the cluster ID stored on the restored PVs. A plain StatefulSet deployment does not have this issue.

Plain StatefulSet Deployment¶

When Kafka is deployed as a plain StatefulSet (without an operator), restore is straightforward:

Navigate to Clusters → Recovery Points, find the recovery point you want to restore from, and then click Actions → Restore.
Enable “Select namespaces” and select the Kafka namespace.
Enable “Include all cluster-scoped resources” to restore any associated resources.
If restoring to a namespace where Kafka PVCs already exist, enable the “Overwrite existing PVCs” option. Without this option, existing PVCs are silently skipped and no data is restored.
Run the restore job and verify that all Kafka broker pods start successfully.

Verify the restored data:

# List topics
kubectl -n <NAMESPACE> exec <BROKER_POD> -c kafka -- \
  /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Verify messages in a topic
kubectl -n <NAMESPACE> exec <BROKER_POD> -c kafka -- \
  /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic <TOPIC_NAME> --from-beginning --timeout-ms 10000

Note

It is recommended to restore Kafka to the same namespace as the original deployment. Kafka broker configurations (such as advertised.listeners) often include the namespace in their DNS names. Restoring to a different namespace may require updating these configurations.

Strimzi Operator Deployment¶

When Kafka is managed by the Strimzi operator, a post-restore fix is required to resolve a cluster ID mismatch.

Background: Strimzi generates a unique KRaft cluster ID and stores it in the KafkaNodePool status. When the Strimzi operator reconciles after a restore, it generates a new cluster ID. However, the restored PVs contain the original cluster ID in each broker’s meta.properties file. This mismatch causes broker pods to crash with the error:

Invalid cluster.id in: /var/lib/kafka/data/kafka-log0/meta.properties.
Expected <NEW_ID>, but read <ORIGINAL_ID>

To restore a Strimzi-managed Kafka cluster:

Restore the Kafka namespace from CloudCasa, including all cluster-scoped resources (Strimzi CRDs).

After the restore completes, one or more broker pods will be in CrashLoopBackOff due to the cluster ID mismatch. Before deleting the custom resources, ensure the KafkaNodePool has deleteClaim: false so that PVCs are preserved:

# Ensure PVCs are not deleted with the CR
kubectl -n <NAMESPACE> patch kafkanodepool <NODEPOOL_NAME> --type=merge \
  -p '{"spec":{"storage":{"volumes":[{"id":0,"type":"persistent-claim","deleteClaim":false}]}}}'

# Delete the Kafka and KafkaNodePool CRs
kubectl -n <NAMESPACE> delete kafka <CLUSTER_NAME>
kubectl -n <NAMESPACE> delete kafkanodepool <NODEPOOL_NAME>

Warning

If deleteClaim is true (the default), deleting the KafkaNodePool will also delete the PVCs and all restored data will be lost.

Create a temporary pod for each PVC to update the cluster ID in meta.properties:

# Create a temp pod for each broker PVC (adjust the loop for your broker count)
for i in 0 1 2; do
  kubectl -n <NAMESPACE> run fix-meta-$i --image=busybox --restart=Never \
    --overrides="{
      \"spec\": {
        \"containers\": [{
          \"name\": \"fix\",
          \"image\": \"busybox\",
          \"command\": [\"sleep\", \"3600\"],
          \"volumeMounts\": [{\"name\": \"data\", \"mountPath\": \"/data\"}]
        }],
        \"volumes\": [{
          \"name\": \"data\",
          \"persistentVolumeClaim\": {\"claimName\": \"data-<CLUSTER_NAME>-<NODEPOOL_NAME>-$i\"}
        }]
      }
    }"
done

# Wait for temp pods to be ready
kubectl -n <NAMESPACE> wait --for=condition=Ready pod/fix-meta-0 pod/fix-meta-1 pod/fix-meta-2 --timeout=60s

Recreate the KafkaNodePool and Kafka custom resources. The Strimzi operator will generate a new cluster ID:

# Apply your KafkaNodePool and Kafka CRs
kubectl apply -n <NAMESPACE> -f kafka-nodepool.yaml
kubectl apply -n <NAMESPACE> -f kafka-cluster.yaml

Wait for the new cluster ID to appear:

# Retrieve the new cluster ID
NEW_CLUSTER_ID=$(kubectl -n <NAMESPACE> get kafkanodepool <NODEPOOL_NAME> \
  -o jsonpath='{.status.clusterId}')
echo "New cluster ID: $NEW_CLUSTER_ID"

Update meta.properties on each PVC with the new cluster ID:
```
for i in 0 1 2; do
  kubectl -n <NAMESPACE> exec fix-meta-$i -- sh -c \
    "sed -i 's/cluster.id=.*/cluster.id=$NEW_CLUSTER_ID/' \
    /data/kafka-log$i/meta.properties"
done
```
Note

The log directory name (kafka-log0, kafka-log1, etc.) corresponds to the broker’s node ID. Adjust the path if your deployment uses a different naming convention.

Delete the temporary pods to release the PVCs:

kubectl -n <NAMESPACE> delete pod fix-meta-0 fix-meta-1 fix-meta-2 --grace-period=0

The Kafka broker pods will now start successfully with matching cluster IDs. Wait for the cluster to become ready:
```
kubectl wait kafka/<CLUSTER_NAME> --for=condition=Ready -n <NAMESPACE> --timeout=300s
```

Verify the restored data:

# List topics
kubectl -n <NAMESPACE> exec <BROKER_POD> -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Verify messages in a topic
kubectl -n <NAMESPACE> exec <BROKER_POD> -- \
  bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic <TOPIC_NAME> --from-beginning --timeout-ms 10000

What Gets Backed Up¶

CloudCasa performs a PV-level backup of all Kafka broker data directories. The following data is captured:

Data Type	Captured
Topic messages (log segments)	Yes — all persisted messages across all partitions
Consumer group offsets	Yes — stored in the internal `__consumer_offsets` topic on the PVs
Transaction state	Yes — stored in the internal `__transaction_state` topic
Cluster metadata (KRaft)	Yes — stored in the internal `__cluster_metadata` topic
Topic configurations	Yes — stored as part of the cluster metadata
ACLs	Yes — stored as part of the cluster metadata
In-flight messages (not yet flushed)	Depends — Kafka batches writes to disk. Messages that have not been flushed at the time of the PV snapshot may not be captured. In practice, Kafka flushes frequently and replication across brokers provides additional durability.