Error “Mover pod status is not READY and TimeoutExpired” for Longhorn PVCs

Summary

When backing up Longhorn PVCs with the Copy to backup storage option, you may sometimes see this type of error in the job activity logs:

Failed to copy PVC <PVC-NAME> in namespace <NAMESPACE>. Error is Mover pod status is not READY and
TimeoutExpired. Status is Pending. Last event: AttachVolume.Attach failed for volume "<PV-NAME>" : rpc error:
code = Aborted desc = volume <PV-NAME> is not ready for workloads

Details

To copy the data of a PVC to backup storage, CloudCasa creates a CSI snapshot of the PVC, creates a temporary PVC from the snapshot and then mounts it in a data mover pod. The above error means that the temporary PVC was not ready to be mounted before the data mover pod timed out. This usually happens because Longhorn has not yet finished copying data from the snapshot to the new PVC.

Solution

There are two solutions and both can be applied together.

Increase the data mover pod timeout

Edit the backup definition and increase Data mover pod timeout in the advanced options to a higher value such as 60 minutes. Lower values can be tried as well depending on the size of the PVCs. For details, see Cluster Backup Wizard (look for “Data mover pod timeout”).

Use a storage class with fewer replicas for the temporary PVCs

If the Longhorn storage class used by the PVCs is configured with more than one replica (parameter “numberOfReplicas”), Longhorn creates the same number of replicas for the temporary PVC as well, which increases the time taken for it to become ready. Since the temporary PVC is deleted after the backup anyway, multiple replicas are not required. To avoid this, create a new storage class with the same contents as the original storage class and set “numberOfReplicas” to “1”. You can then configure this storage class in backup advanced options by selecting

Enable storage class mapping for PV snapshots mounted during backup

For details, see Cluster Backup Wizard (look for “Enable storage class mapping for PV snapshots mounted during backup”).