Restore of Longhorn PVC Failed

Summary

Longhorn PVC restore may fail if multipathd is running on the host and is preventing the mover pod from mounting snapshots.

Identification

Check if the mover pod is not starting in the namespace that is being restored and if it encounters one of these error messages:

  • “/dev/longhorn/XXXXX is apparently in use by the system; will not make a filesystem here!”

  • “mount: /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXX/mount: /dev/longhorn/pvc-XXXXX already mounted or mount point busy”

You can view these error messages by running the kubectl describe pod command on the mover pod.

Solution

This issue may be caused by multipathd on the host creating a multipath device for any eligible device path, including every Longhorn volume device not explicitly blacklisted.

To resolve the issue follow the troubleshooting steps outlined in the Longhorn KB article: https://longhorn.io/kb/troubleshooting-volume-with-multipath