Dell PowerStore¶

CloudCasa DR uses the synchronous or asynchronous replication built into Dell PowerStore arrays. PowerStore replicates the data. CloudCasa runs the failover and failback operations on the arrays and restores the workloads on the other cluster. This page describes what is specific to PowerStore.

Requirements¶

Replication must already be configured. CloudCasa doesn’t create volume groups, protection policies, or replication rules. Configure them in PowerStore Manager or with Dell CSM replication. Block PVCs must be in a replicated volume group, and NFS PVCs must be on a replicated NAS server. A volume that is replicated on its own, outside a volume group, is not failed over.
The PowerStore CSI driver must be installed on both clusters. The driver on the DR cluster must be configured for the destination array.
Add one storage system for each array. The Endpoint URL is the array’s management address, for example https://<mgmt-ip>. The user account must be able to view storage resources and run replication operations (sync, failover, and Reprotect). The array must have at least one remote system with an OK connection. Each cluster needs HTTPS access to its own array.
The inventory PVC must be replicated in the same way as the application PVCs. It can share a volume group or NAS server with the plan’s own workloads, but not with another DR plan. Its volume group or NAS server, and everything in it, is failed over on every recovery of the plan.
Each DR plan needs its own volume groups and NAS servers. See Keeping DR plans separate.

Storage inventory¶

During inventory, CloudCasa only reads from the arrays. It collects volumes, volume groups, file systems, NAS servers, replication sessions, protection policies, and remote systems.

For each PVC, the DR plan details page shows:

The consistency group, which is the volume group for block volumes and the NAS server for NFS file systems.
The replication mode: synchronous when the protection policy’s RPO is zero, and asynchronous otherwise.
The replication status, role, and last sync time.

PVCs that aren’t replicated are listed, but they are not failed over.

Creating a DR plan¶

Creating a DR plan doesn’t create or change anything on either array. On the source cluster, CloudCasa starts the DR agent, which records the plan’s Kubernetes resources on the inventory PVC. CloudCasa then runs an inventory job immediately, and again at the configured interval.

When the plan is saved, and each time inventory runs, CloudCasa checks that:

Both storage systems are connected and the replication target is active.
The inventory PVC is bound, is replicated, and isn’t used by another DR plan.

Block volumes and NFS file systems¶

Block volumes¶

The volume group is the unit of failover.
During recovery, CloudCasa creates each PV on the DR cluster itself, pointing at the replica volume. The PV has the same name as the volume on the array.
The PV gets the filesystem type of the source PV, or ext4 if the source PV doesn’t specify one.

NFS file systems¶

The NAS server is the unit of failover.
The PV refers to the file system and to the NAS server by name. The CSI driver looks up the NAS server’s address when the volume is mounted.
Network. The NAS server’s network settings (IP address, gateway, VLAN, and port) are replicated with it. After failover, the destination array brings the NAS server up with the same settings, so the DR cluster’s nodes must be able to reach that IP address. To check, run ip route get <nas-ip> on a DR node. If the route goes through a router (via) and the NAS server’s file interface has no gateway, NFS won’t work after failover. In that case, set a destination override on the file interface in PowerStore Manager on the destination array. The override can only be set while that array is still the replication destination, so set it before you need to fail over.
Export access. When a pod mounts the volume, the DR cluster’s driver adds the node’s address to the NFS export. If the nodes reach the NAS server from a different network, the mount fails with access denied by server. To avoid this, add that network to X_CSI_POWERSTORE_EXTERNAL_ACCESS (externalAccess in Helm) on the DR cluster’s PowerStore driver, for example 192.168.10.0/24.

What happens during DR recovery¶

When a DR recovery job runs, CloudCasa does the following:

Checks every volume group and NAS server in the plan, and checks the DR cluster for leftover PVs or PVCs that would block the recovery.
Stops the workloads on the source cluster if Stop workloads before failover is enabled, which is the default. Otherwise, it stops only the DR agent.
Fails over the volume group or NAS server that holds the inventory volume, restores the inventory PVC, and starts the DR agent on the DR cluster.
Fails over the remaining volume groups and NAS servers.
Restores the Kubernetes resources from the inventory. Each restored PVC is bound to the PV that CloudCasa created for its replica.

Note

A whole volume group or NAS server is always failed over, even if only some of its workloads are selected. The job shows a warning when this happens.
After failover, replication isn’t reversed until you run a failback, and the DR plan can’t be used again.
If the job fails, nothing is rolled back. Volume groups and NAS servers that were failed over stay failed over, and a retry continues from there.

Planned and unplanned failover¶

PowerStore only allows a planned failover to be started from the source array, so CloudCasa asks the source cluster to run it.

Planned failover: Used when the source cluster’s CloudCasa agent is connected. The source array syncs the latest changes before failing over, so no data is lost.
Unplanned failover: Used when the source cluster can’t be reached. CloudCasa fails over from the destination array without a final sync. With asynchronous replication, changes made since the last sync are lost. The volume may also have been in use when it was last synced, so the CSI driver can refuse to mount it with an FSCheckFailed error. See Troubleshooting.

Note

If the source cluster accepts the request but the failover fails on the source array, CloudCasa doesn’t fall back to an unplanned failover. That volume group or NAS server isn’t failed over.
PowerStore doesn’t allow a NAS server to be failed over from the destination array while it is still running on the source array.

Keeping DR plans separate¶

PowerStore fails over and fails back a whole volume group or NAS server at a time. If two DR plans share one, recovering or failing back either plan also moves the other plan’s volumes. Each DR plan must therefore have its own volume groups and NAS servers.

Block volumes with Dell CSM replication. CSM puts the PVCs that a StorageClass creates in the same namespace into one volume group, named from the StorageClass’s volumeGroupPrefix. Use a separate StorageClass, with its own volumeGroupPrefix, for each DR plan. This matters most for inventory PVCs. They are all in the cloudcasa-io namespace, so with a shared StorageClass they would all be in the same volume group. Application PVCs share a volume group too if two plans use the same namespace, or if the StorageClass sets ignoreNamespaces to true.
NFS file systems. Use a separate DR plan for each NAS server. A separate StorageClass is not enough if it uses the same NAS server.
Without CSM. Don’t add volumes from different DR plans to the same volume group.

Failback¶

To fail back, create a DR plan in the reverse direction, with the DR cluster as the source, and run a DR recovery job for it. CloudCasa then does the following:

Stops the workloads on the DR cluster, starting with the DR agent. As with failover, application workloads are stopped only if Stop workloads before failover is enabled.
Runs Reprotect on the DR array, which copies back only the data changed since the failover, and waits for the sync to finish.
Runs a planned failover back to the primary array.
Restores the workloads on the primary cluster.
Runs Reprotect on the primary array, so that replication runs in its original direction again.

Both clusters and both arrays must be reachable during failback. All of the plan’s volume groups and NAS servers must be in the same state.

Dell CSM replication¶

Dell Container Storage Modules (CSM) replication is not required, because CloudCasa runs the replication operations on the arrays directly. If CSM replication is installed:

CloudCasa reuses the target PVs that CSM creates on the DR cluster, without changing their StorageClass or filesystem type. Restored PVCs get the StorageClass of the PV they bind to.
PVCs that CSM created and that no pod is using are replaced by the restored PVCs.
Set fsGroupPolicy: File on the PowerStore CSIDriver of every cluster that can be a DR target, in the driver’s Helm values. CSM’s target PVs have no filesystem type. With the driver’s default policy (ReadWriteOnceWithFSType), Kubernetes then doesn’t apply the pod’s fsGroup. The CloudCasa DR agent runs as a non-root user and can’t write to its volume without it.

Troubleshooting¶

A leftover PV or PVC blocks a retry

PVs or an inventory PVC left by a failed recovery can block the next attempt, with an error such as PersistentVolume "<pv>" already exists and is bound to a different volume. Check that the PV’s reclaim policy is Retain, so that the volume on the array is kept, then delete the PV or PVC and run the recovery again. Otherwise, a leftover PV with the same name is reused as it is, including its filesystem type.

The DR agent can’t write to its volume

Set fsGroupPolicy: File on the PowerStore CSIDriver. See Dell CSM replication.

The DR agent never becomes ready

The DR agent reads every API resource type before it reports ready. A CRD with a broken conversion webhook, for example one left behind by an uninstalled operator, stops it from becoming ready. To find it, list each resource type as the cloudcasa-io service account:

for r in $(kubectl api-resources --verbs=list,watch -o name); do
  kubectl get "$r" -A --as=system:serviceaccount:cloudcasa-io:cloudcasa-io --request-timeout=30s -o name >/dev/null 2>/tmp/err \
    || echo "$r: $(head -1 /tmp/err)"
done

A volume fails to mount with FSCheckFailed

This can happen after an unplanned failover. Repair the filesystem (for example using e2fsck) from a DR node, then restart the pod.