HPE Alletra¶

CloudCasa DR uses the HPE Remote Copy replication built into HPE Alletra Storage 9000 and Alletra Storage MP B10000 arrays. Remote Copy replicates the data. CloudCasa runs the failover and failback operations on the arrays and restores the workloads on the other cluster. This page describes what is specific to HPE Alletra.

Requirements¶

Replication must already be configured. CloudCasa doesn’t create Remote Copy targets or remote copy groups, and doesn’t add volumes to groups. Configure them on the arrays before you create a DR plan. Each remote copy group must have exactly one target, and all of a DR plan’s groups must replicate to the same target. A volume that isn’t in a remote copy group is not failed over.
The HPE CSI Driver for Kubernetes must be installed on both clusters. The driver on the DR cluster must be configured for the destination array.
Add one storage system for each array. The array must be an HPE Alletra 9060, Alletra 9080, or Alletra Storage MP. Other models fail validation. The Endpoint URL is the address of the array’s WSAPI, for example https://<array-address>. The user account must be able to view storage resources and run Remote Copy operations (sync, stop, failover, recover, and restore). The array must have at least one Remote Copy target in the ACTIVE state. Each cluster needs HTTPS access to its own array.
Enter a secret when you link each cluster to its storage system. For HPE storage systems, the link asks for a secret name and namespace. Use the HPE CSI driver’s backend secret for that cluster’s array, for example hpe-backend in the hpe-storage namespace. The secret must exist on the cluster. On the DR cluster, the PVs that CloudCasa creates during recovery use this secret.
The inventory PVC must be replicated in the same way as the application PVCs. It can share a remote copy group with the plan’s own workloads, but not with another DR plan. Its remote copy group, and everything in it, is failed over on every recovery of the plan.
Each DR plan needs its own remote copy groups. See Keeping DR plans separate.

Storage inventory¶

During inventory, CloudCasa only reads from the arrays. It collects volumes, remote copy groups, and Remote Copy targets. CloudCasa finds the array volume for each PV through the HPEVolumeInfo resource that the HPE CSI driver creates for it.

For each PVC, the DR plan details page shows:

The consistency group, which is the volume’s remote copy group.
The replication mode of the group’s target: synchronous, periodic, or asynchronous.
The replication status, role, and last sync time.

PVCs that aren’t replicated are listed, but they are not failed over.

If a remote copy group contains volumes that aren’t in the plan’s inventory, CloudCasa shows a warning. These volumes are failed over with the group.

Creating a DR plan¶

Creating a DR plan doesn’t create or change anything on either array. On the source cluster, CloudCasa starts the DR agent, which records the plan’s Kubernetes resources on the inventory PVC. CloudCasa then runs an inventory job immediately, and again at the configured interval.

When the plan is saved, and each time inventory runs, CloudCasa checks that:

Both storage systems are connected and the replication target is active.
The secret entered for each cluster’s storage system link exists.
The inventory PVC is bound, is replicated, and isn’t used by another DR plan.
All of the plan’s remote copy groups have the same role and replicate to the same target.

Block volumes and NFS volumes¶

Block volumes¶

The remote copy group is the unit of failover.
During recovery, CloudCasa creates each PV on the DR cluster itself, pointing at the replica volume. The PV has the same name as the replica volume on the DR array. CloudCasa also creates the HPEVolumeInfo resource for it.
The PV uses the same access protocol (iSCSI or Fibre Channel) as the source PV, taken from its accessProtocol attribute. If the source PV doesn’t specify one, iscsi is used.
The PV has no StorageClass. Restored PVCs get the StorageClass of the PV they bind to.

NFS volumes¶

CloudCasa supports RWX volumes that the HPE CSI driver serves through the HPE NFS Server Provisioner (StorageClasses with nfsResources: "true").

Each NFS volume is backed by a block volume and an NFS server Deployment in the cluster. The remote copy group that holds the backing volume is the unit of failover.
During recovery, CloudCasa creates the backing PV and PVC for the replica volume, and the NFS PV with the same name as on the source cluster.
The NFS server objects (Deployment, Service, ConfigMap, ServiceAccount, Role, and RoleBinding) are restored with the workloads. Their namespaces are added to the restore automatically.
On OpenShift, CloudCasa creates the hpe-csi-nfs-scc SecurityContextConstraints on the DR cluster if it is missing. If this fails, the job shows a warning, and NFS server pods may not start until it exists.
When workloads are stopped, the NFS server Deployments are stopped after the workloads that use them.

What happens during DR recovery¶

When a DR recovery job runs, CloudCasa does the following:

Checks every remote copy group in the plan on the DR array, and checks the DR cluster for a leftover inventory PVC that would block the recovery.
Stops the workloads on the source cluster if Stop workloads before failover is enabled, which is the default. Otherwise, it stops only the DR agent. If the source cluster can’t be reached, the recovery continues.
Fails over the remote copy group that holds the inventory volume, restores the inventory PVC, and starts the DR agent on the DR cluster.
Fails over the remaining remote copy groups.
Restores the Kubernetes resources from the inventory. Each restored PVC is bound to the PV that CloudCasa created for its replica.

Note

A whole remote copy group is always failed over, even if only some of its workloads are selected. The job shows a warning when this happens.
After failover, replication isn’t reversed until you run a failback, and the DR plan can’t be used again.
If some of the remote copy groups fail to fail over, the recovery continues, and the workloads that use them may not start. If all of them fail, the recovery stops.
If the job fails, nothing is rolled back. Remote copy groups that were failed over stay failed over. A retry doesn’t continue from there: it stops with Consistency group roles are reversed. The failover was already performed. Contact CloudCasa support before retrying.

How remote copy groups are failed over¶

CloudCasa runs every Remote Copy command on the DR array, from the DR cluster. The source cluster and the source array don’t need to be reachable. For each remote copy group, CloudCasa does the following:

Requests a sync and waits for the volumes to be in sync. If the sync fails, for example because the source array can’t be reached, CloudCasa continues.
If the group isn’t already stopped, waits up to 30 minutes for its volumes to be in sync. It then stops the group if the group uses asynchronous mode or doesn’t have auto synchronize enabled.
Fails over the group.

If the source array is still running, the final sync means that no data is lost, provided the workloads were stopped first. If the source array is down, Remote Copy has usually stopped the group already, so CloudCasa fails it over without a sync. With periodic or asynchronous replication, changes made since the last sync are lost.

Note

If a group isn’t stopped and its volumes don’t get in sync within 30 minutes, that group isn’t failed over. The job shows timed out after 30m0s waiting for remote copy group "<group>" volumes to be in sync.

Keeping DR plans separate¶

Remote Copy fails over, recovers, and restores a whole remote copy group at a time. If two DR plans share one, recovering or failing back either plan also moves the other plan’s volumes. Each DR plan must therefore have its own remote copy groups. CloudCasa doesn’t prevent groups from being shared. The inventory warning about volumes that aren’t in the plan’s inventory is a sign that a group is shared.

Inventory PVCs need particular care. They are all in the cloudcasa-io namespace, so make sure that each plan’s inventory volume is in that plan’s own remote copy group.

Failback¶

To fail back, create a DR plan in the reverse direction, with the DR cluster as the source, and run a DR recovery job for it. When the plan’s remote copy groups are in the Primary-Reverse role, CloudCasa does the following:

Stops the workloads on the DR cluster, starting with the DR agent. As with failover, application workloads are stopped only if Stop workloads before failover is enabled.
Runs Recover on the DR array, which reverses the replication direction and copies back the data changed since the failover, and waits for the sync to finish. If replication was already restarted, this step is skipped.
Runs Restore on the DR array, which returns the remote copy groups to their original roles. No separate failover is needed.
Restores the workloads on the primary cluster.

Both clusters and both arrays must be reachable during failback. All of the plan’s remote copy groups must be in the same role. The Recover and Restore steps must finish within 30 minutes. If any group fails, the job fails. A retry skips the groups that were already restored.

HPE CSI driver¶

CloudCasa doesn’t use HPE CSI driver replication (HPEReplicationDeviceInfo). It runs the Remote Copy operations on the arrays directly.
HPE CSI driver resources such as HPEVolumeInfo and HPENodeInfo aren’t restored from the inventory. CloudCasa creates the HPEVolumeInfo resources for the restored PVs itself.
CloudCasa reuses an existing PV on the DR cluster if it has the same name and volume handle, and the same access protocol.

Troubleshooting¶

A leftover PV or PVC blocks a retry

PVs or an inventory PVC left by a failed recovery can block the next attempt, with an error such as PersistentVolume "<pv>" is bound to a different volume or Inventory PVC "<pvc>" in namespace "cloudcasa-io" already exists in the cluster and is not bound to the failed over volume. Check that the PV’s reclaim policy is Retain, so that the volume on the array is kept, then delete the PV or PVC and run the recovery again.

A PV has the wrong access protocol

A PV on the DR cluster with the same name but a different accessProtocol can’t be reused while it is bound. Stop the pods that use it, delete the PV, and run the recovery again.

A PVC can’t be matched to an array volume

The message Could not find information about the volume using HPEVolumeInfo resource means that the PV wasn’t provisioned by the HPE CSI driver on this cluster, or its HPEVolumeInfo resource is missing. The PVC isn’t failed over.

The DR agent never becomes ready

The DR agent reads every API resource type before it reports ready. A CRD with a broken conversion webhook, for example one left behind by an uninstalled operator, stops it from becoming ready. To find it, list each resource type as the cloudcasa-io service account:

for r in $(kubectl api-resources --verbs=list,watch -o name); do
  kubectl get "$r" -A --as=system:serviceaccount:cloudcasa-io:cloudcasa-io --request-timeout=30s -o name >/dev/null 2>/tmp/err \
    || echo "$r: $(head -1 /tmp/err)"
done