SUSE Storage/Longhorn¶

CloudCasa DR uses Longhorn Disaster Recovery Volumes (DR volumes). Longhorn backs up each volume on the source cluster to a backup target that both clusters share. On the DR cluster, a DR volume for each volume restores those backups incrementally. CloudCasa activates the DR volumes and restores the workloads on the DR cluster. This page describes what is specific to Longhorn.

Requirements¶

Backups and DR volumes must already be configured. CloudCasa doesn’t create backup targets, backups, recurring jobs, or DR volumes. Configure them in the Longhorn UI before you create a DR plan.
Both clusters must use the same backup target, with the same URL. All of a DR plan’s volumes must be backed up to the same backup target.
Each volume needs regular backups. Use a recurring backup job for the volumes that the DR plan protects. A volume counts as replicated only after at least one backup has completed.
Each volume needs exactly one DR volume on the DR cluster, created from that volume’s backups. The DR volume must be in standby mode and attached, which is how Longhorn keeps it up to date.
Add one storage system for each cluster’s Longhorn. The only setting is the Longhorn installation namespace, which is typically longhorn-system. When you validate the storage system, choose the cluster where that Longhorn is installed as the proxy cluster. CloudCasa connects to Longhorn through its longhorn-frontend service, which only that cluster can reach. The backup target must be available.
The inventory PVC must be replicated in the same way as the application PVCs. It must be a Longhorn volume, with a recurring backup job and its own DR volume on the DR cluster. The Kubernetes resources that are restored on the DR cluster are only as recent as the inventory PVC’s last backup, so back it up at least as often as the application volumes.

Storage inventory¶

During inventory, CloudCasa only reads from Longhorn. It collects the Longhorn volumes and backup targets. CloudCasa finds the Longhorn volume for each PV from the volume’s Kubernetes status.

For each PVC, the DR plan details page shows:

The replication target, which is the volume’s backup target.
The replication mode, which is always periodic.
The replication status. It is shown as synced when the volume has at least one backup and its backup target is available.
The last sync time, which is the time of the volume’s last backup.

Each Longhorn volume is replicated, and failed over, on its own, so Longhorn plans don’t show consistency groups.

PVCs that aren’t replicated are listed, but they are not failed over.

Creating a DR plan¶

Creating a DR plan doesn’t create or change anything in Longhorn on either cluster. On the source cluster, CloudCasa starts the DR agent, which records the plan’s Kubernetes resources on the inventory PVC. CloudCasa then runs an inventory job immediately, and again at the configured interval.

When the plan is saved, and each time inventory runs, CloudCasa checks that:

Both storage systems are connected, both use the same backup target URL, and the backup target is available.
The inventory PVC is bound, is replicated, and isn’t used by another DR plan.
All of the plan’s volumes are backed up to the same backup target.

Volumes¶

Each Longhorn volume is failed over on its own.
CloudCasa finds the DR volume for each volume using the backup volume label that Longhorn sets on it. DR volumes can have any name.
During recovery, CloudCasa has Longhorn create the PV for each activated DR volume. The PV has the same name as the source PV.

What happens during DR recovery¶

When a DR recovery job runs, CloudCasa does the following:

Checks the DR volume for every volume in the plan, and checks the DR cluster for a leftover inventory PVC that would block the recovery. If a DR volume is missing, or isn’t in standby mode and attached, the job stops before anything is changed.
Stops the workloads on the source cluster if Stop workloads before failover is enabled, which is the default. Otherwise, it stops only the DR agent. If the source cluster can’t be reached, the recovery continues.
Activates the DR volume for the inventory volume, restores the inventory PVC, and starts the DR agent on the DR cluster.
Activates the remaining DR volumes.
Restores the Kubernetes resources from the inventory, and creates a PV for each activated volume.

Note

After activation, a DR volume becomes a regular Longhorn volume, and the DR plan can’t be used again.
If some of the DR volumes fail to activate, the recovery continues, and the workloads that use them may not start. If all of them fail, the recovery stops.
If the job fails, nothing is rolled back. Activated volumes stay active. A retry doesn’t continue from there: an activated volume is no longer a DR volume, so the retry stops before execution. Contact CloudCasa support before retrying.

How DR volumes are activated¶

CloudCasa runs every Longhorn operation on the DR cluster. The source cluster doesn’t need to be reachable. For each volume, CloudCasa does the following:

Asks Longhorn on the DR cluster to check the backup target for new backups of the volume. CloudCasa doesn’t wait for this check to finish, and if it fails, CloudCasa continues.
Activates the DR volume, and waits up to 5 minutes for it to be ready.

CloudCasa doesn’t take a backup on the source cluster before failover. Anything written after the last backup that the DR volume has restored is lost. If the source cluster is still running, you can reduce this by stopping the workloads, taking a backup of each volume, and waiting until the DR volumes have restored it before you run the recovery.

Keeping DR plans separate¶

Because each volume is failed over on its own, recovering one DR plan doesn’t affect the volumes of another. Don’t include the same volume in more than one DR plan. Each DR plan also needs its own inventory PVC.

Failback¶

Longhorn doesn’t reverse replication on its own after a failover. To fail back, set up replication in the reverse direction first:

On the DR cluster, configure recurring backups of the activated volumes to the shared backup target.
On the primary cluster, create a DR volume for each of these volumes.

Then create a DR plan in the reverse direction, with the DR cluster as the source, and run a DR recovery job for it. CloudCasa fails back in the same way as it fails over.

Troubleshooting¶

The storage system can’t be validated

Failed to connect to Longhorn: check the Longhorn installation namespace, and check that the proxy cluster is the cluster where Longhorn is installed.
No active replication targets found: Longhorn’s backup target isn’t available. Check it in the Longhorn UI.

The inventory PVC isn’t replicated

The message Inventory PVC "<pvc>" in the "cloudcasa-io" namespace is not replicated according to the storage system means that the inventory volume has no completed backup yet, or that its backup target isn’t available.

A PVC can’t be matched to a Longhorn volume

The message Could not find Longhorn volume means that the PV isn’t a Longhorn volume on this cluster. The PVC isn’t failed over.

The recovery stops before execution

An error such as Aborting disaster recovery volume activation before execution because failed to verify activation eligibility for volume "<volume>" means that the DR volume isn’t ready to be activated. Check that the volume has exactly one DR volume on the DR cluster, and that it is in standby mode and attached. The agent log on the DR cluster names the problem, for example No DR volume found for volume <volume> or DR volume <volume> is not attached. See Collecting CloudCasa Kubernetes agent log files.

A DR volume fails to activate

The job shows Failed to activate disaster recovery volume for "<volume>". Check the volume in the Longhorn UI on the DR cluster. CloudCasa waits up to 5 minutes for an activated volume to be ready.

The DR agent never becomes ready

The DR agent reads every API resource type before it reports ready. A CRD with a broken conversion webhook, for example one left behind by an uninstalled operator, stops it from becoming ready. To find it, list each resource type as the cloudcasa-io service account:

for r in $(kubectl api-resources --verbs=list,watch -o name); do
  kubectl get "$r" -A --as=system:serviceaccount:cloudcasa-io:cloudcasa-io --request-timeout=30s -o name >/dev/null 2>/tmp/err \
    || echo "$r: $(head -1 /tmp/err)"
done