Support for PVCs whose actual size is different from configured size in the manifest

Summary

In some cases a PVC’s manifest shows one size while the data size of the actual volume underneath is much larger. This usually happens if the underlying storage volume is extended outside Kubernetes, causing PVC and the storage volume to get out of sync. In these cases, CloudCasa will backup all the data in the PVC so there is no problem there. But when such PVCs are restored, CloudCasa would provision a new PVC using the size set in the PVC manifest and when it tries to restore the data, the writes will fail with the error:

no space left on device

To give a specific example, let us assume that there is a PVC with 10G configured as “capacity” but the actual data on the volume is 30 GB. When this PVC is backed up, CloudCasa will backup 30 GB data without any issues. However, when the PVC is restored, CloudCasa will provision a PVC with 10 GB capacity. Since a new volume is created by the storage provider at this time, the volume will be of size 10 GB. CloudCasa then tries to restore 30 GB data which will obviously fail with the above error.

Solution

Ideally, PVC setting should always match the underlying volume size but there is a work around in cases they are out of sync. Add the annotation “cloudcasa.io/pvc-actual-size” to the PVC and set its value to the actual size (using appropriate units as understood by Kubernetes).

Example:

kubectl -n <NAMESPACE> annotate pvc <PVC-NAME> cloudcasa.io/pvc-actual-size=250Gi

Note that this annotation needs to be present at backup time.

If you need to restore from a recovery point that has this problem and if the recovery point was created before you could add the above annotation, you can still restore the data but you will need to first create an empty PVC in the target namespace in the target cluster. The PVC should be created with the same name as source PVC but with the right size value for “capacity” (30 GB using above example). After that, define CloudCasa restore by setting Overwrite existing resources option in Restore transforms. When you run the restore now, CloudCasa should restore all the data without any error.