Filesystem freeze may fail for VM guests when SELinux is in enforcing mode

Environment

Harvester or other KubeVirt clusters with VMs running the following SUSE guest OSes when SELinux is in enforcing mode on the guest:

SUSE Linux Micro 5.3 and above
SUSE Linux Enterprise Server 16.0 and above
SUSE Linux Enterprise Server for SAP applications 16.0 and above

Summary

Users may encounter “Hook errors” when backups attempt to freeze VM guest OS filesystems using the QEMU guest agent on certain SUSE Linux distributions when SELinux is in enforcing mode.

Symptoms: The pre-backup FS freeze hook fails, and errors similar to the following are reported in the CloudCasa activity log.

Hook error: Phase=pre, Pod=virt-launcher-smlm-tv9pq, Namespace=labs, Container=compute,
Command=[/usr/bin/virt-freezer --freeze --name smlm --namespace labs], Error=command
terminated with exit code 1 (Stderr: Freezing VMI failed. Reason: server error. command
Freeze failed: "LibvirtError(Code=1, Domain=10, Message='internal error: unable to execute
QEMU agent command 'guest-fsfreeze-freeze': failed to open
/var/lib/containers/storage/volumes: Permission denied')")

Hook error: Phase=pre, Pod=virt-launcher-gpu-node-xmpl8, Namespace=labs, Container=compute,
Command=[/usr/bin/virt-freezer --freeze --name gpu-node --namespace labs], Error=command
terminated with exit code 1 (Stderr: Freezing VMI failed. Reason: server error. command
Freeze failed: "LibvirtError(Code=1, Domain=10, Message='internal error: unable to execute
QEMU agent command 'guest-fsfreeze-freeze': failed to open /var/lib/kubelet/pods/
b5cd6c86-ddac-4c9e-b9a3-8f3dd2be5934/volumes/kubernetes.io~csi/
pvc-02c08196-b6b0-46fd-bcd4-394d973afff0/mount: Permission denied')")

Note the path varies with what is mounted inside each guest (a podman storage volume in the first example, a CSI PVC in the second).

Cause: on SLES 16.0 guests with SELinux in enforcing mode but using otherwise default configurations, the QEMU gust agent’s FS freeze operation fails whenever the guest has any additional mounted filesystem labeled container_var_lib_t (or similarly non-container_file_t). The freeze operation succeeds on the root filesystem but aborts on the first such mount it can’t open.

SLES ships with SELinux dontaudit rules enabled by default, so it may not be immediately obvious what is causing the problem. You can confirm that SELinux enforcement is the cause by temporarily enabling auditing on the guest with semodule -DB and then checking the audit log, or by temporarily disabling SELinix enforcement with setenforce 0.

Solution

To resolve this issue, enable the SELinux boolean “virt_qemu_ga_read_nonsecurity_files” in the guest OS (not the host/worker OS) as follows.

setsebool -P virt_qemu_ga_read_nonsecurity_files on

No reboot or service restart is required for this setting to take effect. This setting will persist across reboots.