HashiCorp Consul

Published: 06 Nov. 2026 Last updated: 06 Nov. 2026

Summary

HashiCorp Consul is a service networking platform that provides service discovery, configuration management, and service mesh capabilities. When deployed on Kubernetes using the official Helm chart, Consul server pods maintain a replicated state store using the Raft consensus protocol, with all data persisted to PersistentVolumes.

Consul’s state store holds:

  • Key/Value (KV) data — application configurations, feature flags, and other structured data stored via the KV API.

  • Service catalog — registered services, health check definitions, and node information.

  • Intentions — service-to-service access control rules (connect/deny policies).

  • ACL tokens and policies — access control lists, roles, and authentication configuration.

  • Prepared queries and sessions — saved queries and distributed locking sessions.

This Application Note discusses how to use CloudCasa to properly back up and restore HashiCorp Consul running in containers under Kubernetes.

CloudCasa has been tested for this application note with a 3-node Consul server cluster deployed using the official HashiCorp Consul Helm chart. The information herein is expected to apply to more recent versions and to different cluster sizes as well.

Prerequisites

The following requirements must be met for a successful backup and restore of Consul with CloudCasa:

  • Consul server pods must use PersistentVolumes for data storage (server.storage in the Helm chart). Consul deployments using emptyDir or in-memory storage will lose all data when pods are restarted and cannot be backed up.

  • The storage class used for Consul server PVs must support CSI volume snapshots. Verify that a VolumeSnapshotClass exists for your storage provisioner.

  • Consul servers should be deployed as a StatefulSet (this is the default when using the official Helm chart).

  • If Consul ACLs are enabled, a token with operator:read permission is required for the snapshotter.

  • The consul snapshot save command requires connectivity to a Consul server’s HTTP API (default port 8500). Ensure network policies allow the snapshotter pod to reach the Consul server service.

Note

Consul’s consul snapshot save command produces a point-in-time, Raft-consistent snapshot of all server state. This is the recommended approach because a raw PV-level snapshot may capture the Raft write-ahead log and BoltDB database in an inconsistent state. CloudCasa uses a pre-backup hook to invoke this command before taking the PV snapshot.

Configure the CloudCasa Consul Snapshotter

To perform a consistent backup of Consul, the consul snapshot save command is used to create a snapshot during the backup. CloudCasa requires a running Pod with access to the Consul HTTP API to execute the snapshot command. The following instructions create the Pod that CloudCasa will use.

Note

If Consul ACLs are enabled, the snapshotter requires a token with at minimum the operator:read permission (for non-ACL clusters, no token is needed). The steps below cover both configurations.

  1. If Consul ACLs are enabled, create a snapshot policy and token. Skip this step if ACLs are not enabled:

    # Get the Consul bootstrap token
CONSUL_HTTP_TOKEN=$(kubectl -n consul get secret consul-bootstrap-acl-token \
  -o jsonpath='{.data.token}' | base64 -d)

# Create a policy for snapshots
kubectl -n consul exec -i consul-server-0 -- sh -c "
export CONSUL_HTTP_TOKEN=$CONSUL_HTTP_TOKEN
consul acl policy create -name snapshot-policy -rules '
acl = \"read\"
key_prefix \"\" { policy = \"read\" }
node_prefix \"\" { policy = \"read\" }
service_prefix \"\" { policy = \"read\" }
session_prefix \"\" { policy = \"read\" }
operator = \"read\"
'
"

# Create a token with this policy
kubectl -n consul exec -i consul-server-0 -- sh -c "
export CONSUL_HTTP_TOKEN=$CONSUL_HTTP_TOKEN
consul acl token create -description 'CloudCasa snapshot token' \
  -policy-name snapshot-policy -format=json
"

    Save the SecretID from the output. This will be used as the snapshot token.

  2. Create the namespace and a secret to hold the Consul snapshot token (if ACLs are enabled):

    kubectl create namespace cc-consul-backup

# Only if ACLs are enabled — replace <SNAPSHOT_TOKEN> with the SecretID from above
kubectl -n cc-consul-backup create secret generic consul-snapshot-token \
  --from-literal=token=<SNAPSHOT_TOKEN>

  3. Create a PVC for the Consul snapshotter. Replace STORAGE_CLASS_NAME with the name of your storage class:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: cc-consul-snapshots
    namespace: cc-consul-backup
    labels:
        app: cloudcasa-consul-snapshotter
spec:
    accessModes: [ReadWriteOnce]
    storageClassName: <STORAGE_CLASS_NAME>
    resources:
        requests:
            storage: 10Gi

  4. Deploy the Consul snapshotter Pod. Adjust the image and CONSUL_HTTP_ADDR environment variable as necessary for your environment:

    If ACLs are enabled:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudcasa-consul-snapshotter
  namespace: cc-consul-backup
  labels:
    app: cloudcasa-consul-snapshotter
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: cloudcasa-consul-snapshotter
  template:
    metadata:
      labels:
        app: cloudcasa-consul-snapshotter
    spec:
      containers:
        - name: snap
          image: hashicorp/consul:1.20
          command: ["sleep", "infinity"]
          env:
            - name: CONSUL_HTTP_ADDR
              value: "http://consul-server.consul.svc.cluster.local:8500"
            - name: CONSUL_HTTP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: consul-snapshot-token
                  key: token
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: snap
              mountPath: /snapshots
      volumes:
        - name: snap
          persistentVolumeClaim:
            claimName: cc-consul-snapshots

    If ACLs are not enabled:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudcasa-consul-snapshotter
  namespace: cc-consul-backup
  labels:
    app: cloudcasa-consul-snapshotter
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: cloudcasa-consul-snapshotter
  template:
    metadata:
      labels:
        app: cloudcasa-consul-snapshotter
    spec:
      containers:
        - name: snap
          image: hashicorp/consul:1.20
          command: ["sleep", "infinity"]
          env:
            - name: CONSUL_HTTP_ADDR
              value: "http://consul-server.consul.svc.cluster.local:8500"
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: snap
              mountPath: /snapshots
      volumes:
        - name: snap
          persistentVolumeClaim:
            claimName: cc-consul-snapshots

  5. Verify that the snapshot command works from within the Consul snapshotter Pod:

    • Open a shell in the Pod:

      kubectl -n cc-consul-backup exec -it deploy/cloudcasa-consul-snapshotter -- sh

    • Create a test snapshot:

      consul snapshot save /snapshots/consul-test.snap

    • Verify the snapshot:

      consul snapshot inspect /snapshots/consul-test.snap

      The output should show a valid snapshot with a non-zero index and the Raft term.

    • Delete the test snapshot:

      rm -f /snapshots/consul-test.snap

Backup

The following instructions can be used to configure backup of HashiCorp Consul using the cloudcasa-consul-snapshotter.

  1. From the CloudCasa UI, navigate to Configuration → App Hooks and click the “Add app hook” button.

  2. Specify a name for the App Hook and select “Pre-backup” as the type. Set the pod selector to app = cloudcasa-consul-snapshotter and the container to snap. Add the following into the command field and then click “Save”:

    /bin/sh -c "consul snapshot save /snapshots/consul.snap && consul snapshot inspect /snapshots/consul.snap"

    No post-backup hook is needed. The consul snapshot save command is an atomic Raft operation that completes instantly and does not require any cleanup.

  3. Navigate to Clusters → Overview, select the cluster containing the Consul deployment, and then click “Define backup”.

  4. In the App hooks section of the backup definition, add a pre-backup app hook. Select the cc-consul-backup namespace and select the App Hook created previously.

  5. Run a full cluster backup that includes both the cc-consul-backup namespace and the consul namespace. Enable the “Include all cluster-scoped resources” switch to ensure all CRDs are included.

  6. Run the backup and confirm the job completes.

Restore

Restoring Consul is a two-part operation. CloudCasa restores the Consul snapshot file from the snapshotter PVC. A Consul cluster is then deployed and the snapshot is loaded into it using the consul snapshot restore command.

Note

The consul snapshot restore command must be run against a running Consul cluster with an elected leader. The snapshot replaces all state in the target cluster.

  1. Navigate to Clusters → Recovery Points, find the recovery point you want to restore from, and then click Actions → Restore.

  2. Enable “Select namespaces” and select the cc-consul-backup namespace.

    Note

    The namespace containing the Consul servers (typically consul) does not need to be selected. The snapshot file produced by the App Hook is the recoverable copy of all Consul state. Restoring the consul namespace PVs directly is not recommended, as the Raft log on PVs may be in an inconsistent state.

  3. In the Destination section, select the target cluster for the restore.

  4. Run the restore job and verify that the cloudcasa-consul-snapshotter Pod and its PVC were restored.

  5. Deploy a fresh Consul cluster on the target if one is not already running. Use the official Helm chart:

    helm install consul hashicorp/consul \
  --namespace consul --create-namespace \
  --set global.name=consul \
  --set server.replicas=3 \
  --set server.storage=10Gi

  6. Wait for the Consul cluster to be ready:

    kubectl wait --for=condition=Ready pod -l app=consul,component=server \
  -n consul --timeout=300s

  7. Copy the snapshot from the restored snapshotter Pod and restore it into the new cluster:

    # Copy the snapshot out
POD=$(kubectl -n cc-consul-backup get pod -l app=cloudcasa-consul-snapshotter -o name)
kubectl -n cc-consul-backup cp ${POD#pod/}:/snapshots/consul.snap ./consul.snap

# Copy into a Consul server pod and restore
kubectl -n consul cp ./consul.snap consul-server-0:/tmp/consul.snap
kubectl -n consul exec consul-server-0 -- consul snapshot restore /tmp/consul.snap

    Note

    If ACLs are enabled on the target cluster, provide the bootstrap token when running the restore:

    kubectl -n consul exec consul-server-0 -- sh -c \
  "CONSUL_HTTP_TOKEN=<BOOTSTRAP_TOKEN> consul snapshot restore /tmp/consul.snap"

  8. Verify the restored data. Confirm that KV entries, service registrations, and other state are present:

    # Verify KV data
kubectl -n consul exec consul-server-0 -- consul kv get -recurse

# Verify service catalog
kubectl -n consul exec consul-server-0 -- consul catalog services

# Verify ACL tokens (if ACLs are enabled)
kubectl -n consul exec consul-server-0 -- sh -c \
  "CONSUL_HTTP_TOKEN=<BOOTSTRAP_TOKEN> consul acl token list"

What Gets Backed Up

The consul snapshot save command captures a point-in-time, Raft-consistent snapshot of all Consul server state, including:

Data Type

Captured

KV store entries

Yes — all key/value pairs across all namespaces

Service registrations (catalog)

Yes — services registered via the catalog API

Service intentions

Yes — all connect authorize rules

ACL tokens, policies, and roles

Yes

Prepared queries

Yes

Sessions

Yes

Raft configuration

Yes — peer set and cluster metadata

Agent-local service registrations

No — services registered only via the local agent API on client pods are not part of the server snapshot. These are typically re-registered by the application on startup.

Note

Consul Connect sidecar proxy configurations are typically managed by the connect-injector and are re-created when application pods restart. They do not need to be explicitly backed up.

See also