Changed Block Tracking¶

KubeVirt is the technology that allows virtual machines to be run on Kubernetes clusters and is used by platforms such as OpenShift Virtualization and SUSE Harvester. Starting from version 1.8, KubeVirt supports enabling changed block tracking of VMs so that backup applications such as CloudCasa can track the changes since last backup, and read only that data. This is much more efficient than reading entire PVC to find out what changed. Note that this is different from the CBT API that is part of the CSI spec. The latter requires each storage driver to implement the API while KubeVirt’s CBT is storage-agnostic. This page discusses CloudCasa’s support for KubeVirt’s CBT API.

Note that KubeVirt 1.8 is available in OpenShift 4.22 and Harvester 1.9.0, so OpenShift versions before 4.22 and Harvester versions before 1.9.0 cannot take advantage of this feature.

Warning

The KubeVirt CBT API is currently in Alpha state so it is not recommended for production workloads. For this reason, CBT support in CloudCasa is in Preview mode.

Enabling CBT¶

Since the CBT API is still in Alpha, it is not enabled by default. The following instructions show how to enable CBT on OpenShift and Harvester clusters. To enable CBT on other platforms, please contact CloudCasa support.

After enabling CBT on a cluster, you will need to enable the backup job option Back up VMs using changed block tracking under “Advanced options” while defining a backup.

OpenShift¶

Enable IncrementalBackup feature gate via the HCO followed by setting the changedBlockTrackingLabelSelectors.

Here is a sample HCO spec:

apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  featureGates:
    - name: incrementalBackup
      state: Enabled
  virtualization:
    changedBlockTrackingLabelSelectors:
      virtualMachineLabelSelector:
        matchLabels:
          cbt: "true"

Label each VM that should use CBT and restart it:

oc label vm <VM-NAME> -n <NAMESPACE> cbt=true
virtctl restart <VM-NAME> -n <NAMESPACE>

Verify:

oc get vm <VM-NAME> -n <NAMESPACE> -o jsonpath='{.status.changedBlockTracking}{"\n"}'

Harvester¶

Enable the IncrementalBackup, UtilityVolumes, Snapshot, and VMExport feature gates in the KubeVirt resource. First list the feature gates that are already enabled:

kubectl get kubevirt kubevirt -n harvester-system \
  -o jsonpath='{.spec.configuration.developerConfiguration.featureGates}{"\n"}'

Then add these gates, leaving out any that are already enabled:

kubectl patch kubevirt kubevirt -n harvester-system --type=json -p '[
  {"op":"add","path":"/spec/configuration/developerConfiguration/featureGates/-","value":"IncrementalBackup"},
  {"op":"add","path":"/spec/configuration/developerConfiguration/featureGates/-","value":"UtilityVolumes"},
  {"op":"add","path":"/spec/configuration/developerConfiguration/featureGates/-","value":"Snapshot"},
  {"op":"add","path":"/spec/configuration/developerConfiguration/featureGates/-","value":"VMExport"}
]'

Set the changedBlockTrackingLabelSelectors:

kubectl patch kubevirt kubevirt -n harvester-system --type=merge -p '{"spec":{"configuration":{
  "changedBlockTrackingLabelSelectors":{"virtualMachineLabelSelector":{"matchLabels":{"cbt":"true"}}}}}}'

Label each VM that should use CBT and restart it, either with “Restart” in the Harvester UI or with virtctl. A reboot from inside the guest operating system does not enable CBT:
```
kubectl label vm <VM-NAME> -n <NAMESPACE> cbt=true
virtctl restart <VM-NAME> -n <NAMESPACE>
```

Verify that the state is Enabled:

kubectl get vm <VM-NAME> -n <NAMESPACE> -o jsonpath='{.status.changedBlockTracking}{"\n"}'

When CBT is enabled, Harvester stores the state of each VM in a PVC of the storage class vmstate-persistence. The default snapshot based backup method does not work for this storage class as Harvester doesn’t allow PVCs to be created from snapshots. So this storage class needs to be excluded from the backups or a live backup method can be selected: in the backup’s advanced options, enable “Configure backup method by storage class (advanced)” and choose “Skip” or “Read from PVC” for the storage class vmstate-persistence.

Backup of Powered-off VMs¶

CBT method works only for running VMs. If the CBT job option is enabled, powered-off VMs are skipped by default. But if you still want to backup such VMs, enable the job option Backup powered-off VMs with non-CBT method (under the option Back up VMs using changed block tracking). When enabled, powered-off VMs are backed up by reading entire VM disk PVCs and detecting what changed. Only changed data is transferred to the backup target.

Scratch PVCs¶

When CBT is enabled, a scratch PVC is needed for each VM to store the disk data that is changing while the VM is being backed up. By default, CloudCasa creates a 20 GB size PVC using the storage class of the first VM disk. This PVC is automatically deleted after the backup is done.

The size of the scratch PVC and the storage class to be used to create such PVCs can be configured under cluster advanced option “Configure KubeVirt scratch PVC”.

Limitations¶

A VM must be running to be backed up with CBT. When CBT is enabled, a VM that is not running is skipped by default. To backup such VMs using the non-CBT method (which reads entire PVCs to find the changed data), enable the option “Back up VMs using changed block tracking => Backup powered-off VMs with non-CBT method” under backup advanced options.
Compression is not supported for CBT backups at this point.