Understanding Kubernetes Volume Snapshots

Photo by benjamin lehman on Unsplash

Backup and restore are often missing from the conversation when talking about Kubernetes architecture. Some mistakenly believe that such functionality is unnecessary in the world of stateless workloads. However, many users leverage Kubernetes to run stateful workloads, such as databases, in addition to their stateless applications. Kubernetes Volume Snapshots provide snapshot features that are familiar to many traditional storage administrators. While they do not solve all backup and recovery problems by themselves, Volume Snapshots should be part of an overall approach to data protection.

In this tutorial, you will learn about Volume Snapshots and apply them in the context of a simulated application failure scenario.

The Basics

A Kubernetes Volume Snapshot represents a point in time capture of the contents of a PersistentVolume. Volume snapshots must be supported by the Container Storage Interface (CSI) driver that implements the storage in a cluster. A CSI driver provides three Custom Resource Definitions (CRDs) for consumption: VolumeSnapshotClass, VolumeSnapshot, and VolumeSnapshotContent. Not all CSI drivers implement volume snapshots, so you should confirm that your chosen storage provider supports snapshots if you plan to leverage them.

A VolumeSnapshotClass defines a class of storage associated with a snapshot. This is similar to a StorageClass for a PersistentVolumeClaim. A VolumeSnapshotClass allows an administrator to define the level of service that a particular snapshot will receive on the underlying storage.

A VolumeSnapshot is a request to take a snapshot, and it is similar to a PersistentVolumeClaim. CSI controllers watch for the creation of a VolumeSnapshot, and they fulfill the request by interacting with the underlying storage to take a snapshot.

A VolumeSnapshotContent represents the fulfilled claim for a VolumeSnapshot on the underlying storage. It contains information about the snapshot on the storage, such as a snapshot ID or a path. This is analogous to a PersistentVolume.

The following diagram shows the relationships between these resources and their comparison with standard storage resources:

Relationships between storage and snapshot resources

Deploy a Kubernetes Cluster

You must have a Kubernetes cluster with a CSI driver that supports snapshots to perform the rest of this tutorial. Driver support for snapshots can be found in the documentation for the Kubernetes CSI.

This tutorial uses a Kubernetes cluster on Digital Ocean. Digital Ocean clusters are inexpensive and simple to configure. However, any cluster with a supported CSI driver will work.

Deploy a Stateful Workload

Databases are commonly deployed as a stateful workloads on Kubernetes. As stateful workloads, they benefit from volume snapshot features. This tutorial uses a StatefulSet to deploy MariaDB. A PersistentVolumeClaim provides storage for the database, a Secret contains the database root password (simply “password”), and a StatefulSet deploys the workload:

---
# PersistentVolumeClaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mariadb-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
# Secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql-root-password
data:
  MYSQL_ROOT_PASSWORD: cGFzc3dvcmQ=
type: Opaque
---
# StatefulSet.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb:11
        resources:
        ports:
        - containerPort: 3306
        env:
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-root-password
                key: MYSQL_ROOT_PASSWORD
        volumeMounts:
        - name: mariadb-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: mariadb-pv
        persistentVolumeClaim:
          claimName: mariadb-pvc

Create these resources and confirm that the Pod is running:

$ kubectl apply -f PersistentVolumeClaim.yaml
persistentvolumeclaim/mariadb-pvc created

$ kubectl apply -f Secret.yaml
secret/mysql-root-password created

$ kubectl apply -f StatefulSet.yaml
statefulset.apps/mariadb created

$ kubectl get statefulset
NAME      READY   AGE
mariadb   1/1     43m

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
mariadb-0                         1/1     Running   0          43m

Create Test Data

A database failure scenario involves data loss. To simulate this, you must have some data. Connect to the container and create a basic database with a single table. The table has a single column and a few rows of data:

$ kubectl exec -it mariadb-0 -- mariadb -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 11.1.2-MariaDB-1:11.1.2+maria~ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE test_db;
Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> USE test_db;
Database changed

MariaDB [test_db]> CREATE TABLE test(message VARCHAR(255));
Query OK, 0 rows affected (0.014 sec)

MariaDB [test_db]> INSERT INTO test(message) VALUES("hello");
Query OK, 1 row affected (0.003 sec)

MariaDB [test_db]> INSERT INTO test(message) VALUES("world");
Query OK, 1 row affected (0.002 sec)

MariaDB [test_db]> SELECT * FROM test;
+---------+
| message |
+---------+
| hello   |
| world   |
+---------+
2 rows in set (0.000 sec)

Take a Snapshot

You can now capture a snapshot of the known-good state of the database. A VolumeSnapshot is a request for a snapshot on the underlying storage. Once you create the VolumeSnapshot, the CSI driver will create a snapshot on the underlying storage.

---
# VolumeSnapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mariadb-snapshot
spec:
  volumeSnapshotClassName: do-block-storage
  source:
    persistentVolumeClaimName: mariadb-pvc

Create the VolumeSnapshot and confirm it is ready to use:

$ kubectl apply -f Snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/mariadb-snapshot created

$ kubectl get volumesnapshot
NAME               READYTOUSE   SOURCEPVC     SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS      SNAPSHOTCONTENT                                    CREATIONTIME   AGE
mariadb-snapshot   true         mariadb-pvc                           1Gi           do-block-storage   snapcontent-70dcd914-bfc0-4281-97e8-667484de1bfc   117s           118s

Simulate a Data Loss Scenario

You can now be confident that your data is safely protected by a snapshot. To confirm this, create a failure scenario by dropping the table that you previously created:

MariaDB [test_db]> DROP TABLE test;
Query OK, 0 rows affected (0.021 sec)

Restoring a snapshot involves creating a new PersistentVolumeClaim referencing the snapshot as a dataSource. You can then leverage this PersistentVolumeClaim normally, and you will have access to the contents of the snapshot. In this case, you can create an additional deployment of MariaDB so that you can connect to the database and recover the data:

---
# RestorePersistentVolumeClaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mariadb-restore
spec:
  dataSource:
    name: mariadb-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
# RestoreDeployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb-restore
spec:
  selector:
    matchLabels:
      app: mariadb-restore
  template:
    metadata:
      labels:
        app: mariadb-restore
    spec:
      containers:
      - name: mariadb
        image: mariadb:11
        resources:
        ports:
        - containerPort: 3306
        env:
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-root-password
                key: MYSQL_ROOT_PASSWORD
        volumeMounts:
        - name: mariadb-restore-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: mariadb-restore-pv
        persistentVolumeClaim:
          claimName: mariadb-restore

Create the resources and verify that the new Pod is running:

$ kubectl apply -f RestorePersistentVolumeClaim.yaml
persistentvolumeclaim/mariadb-restore created

$ kubectl apply -f RestoreDeployment.yaml
deployment.apps/mariadb-restore created

$ kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
mariadb-0                         1/1     Running   0          42m
mariadb-restore-8c58d7848-nb6hj   1/1     Running   0          55s

Finally, it’s time to confirm that the restore was successful. Connect to the Pod and confirm that the data is intact:

$ kubectl exec -it mariadb-restore-8c58d7848-nb6hj -- mariadb -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 11.1.2-MariaDB-1:11.1.2+maria~ubu2204 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use test_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [test_db]> select * from test;
+---------+
| message |
+---------+
| hello   |
| world   |
+---------+
2 rows in set (0.006 sec)

The snapshot has been successfully restored to the point-in-time when it was taken, and the data is intact.

Inspect the Snapshot

The VolumeSnapshotContents resource represents the snapshot on the underlying storage. Inspecting the VolumeSnapshotContents contains information about the snapshot itself, including a “Volume Handle” that represents the ID of the snapshot on the underlying storage. In this example, the Volume Handle is “4c97d70f-7140–11ee-a4f1–0a58ac14421d”:

$ kubectl describe volumesnapshotcontents.snapshot.storage.k8s.io snapcontent-70dcd914-bfc0-4281-97e8-667484de1bfc
Name:         snapcontent-70dcd914-bfc0-4281-97e8-667484de1bfc
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshotContent
Metadata:
  Creation Timestamp:  2023-10-23T01:42:05Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  Generation:        1
  Resource Version:  63584
  UID:               ae0b1fc3-3b60-4478-9599-8ee9c697ae78
Spec:
  Deletion Policy:  Delete
  Driver:           dobs.csi.digitalocean.com
  Source:
    Volume Handle:             4c97d70f-7140-11ee-a4f1-0a58ac14421d
  Volume Snapshot Class Name:  do-block-storage
  Volume Snapshot Ref:
    API Version:       snapshot.storage.k8s.io/v1
    Kind:              VolumeSnapshot
    Name:              mariadb-snapshot
    Namespace:         default
    Resource Version:  63570
    UID:               70dcd914-bfc0-4281-97e8-667484de1bfc
Status:
  Creation Time:    1698025326000000000
  Ready To Use:     true
  Restore Size:     1073741824
  Snapshot Handle:  5fd096c3-7145-11ee-a4f1-0a58ac14421d
Events:             <none>

Reviewing the list of snapshots in the Digital Ocean account shows that the snapshot ID matches the Volume Handle from the VolumeSnapshotContents:

$ doctl compute snapshot list
ID                                      Name                                             Created at              Regions    Resource ID                             Resource Type    Min Disk Size    Size        Tags
5fd096c3-7145-11ee-a4f1-0a58ac14421d    snapshot-70dcd914-bfc0-4281-97e8-667484de1bfc    2023-10-23T01:42:06Z    [nyc1]     4c97d70f-7140-11ee-a4f1-0a58ac14421d    volume           1                0.10 GiB    k8s:bf14c456-2fc6-4066-be2f-19b341404ea0

Note: These results will vary if you used a different Kubernetes cluster.

Understanding the relationship between the VolumeSnapshotContents and the underlying storage is important, as you may need to know how the snapshot is represented in the storage itself for troubleshooting or management purposes.

Wrapping Up

In this article, you created and restored a Volume Snapshot as part of a failure simulation. Kubernetes Volume Snapshots are useful for backups and point-in-time captures of volume contents. While they do require support for the CSI driver, many common CSI drivers provide snapshot functionality. Understanding the relationship between Volume Snapshot resources and the procedure to restore a snapshot is important when using Volume Snapshots in your cluster.

Remember that snapshots are not complete backup solutions on their own. Rather, snapshots should be part of an overall backup strategy for stateful workloads. This may include shipping the snapshots to another, geographically distributed location. You should also regularly test your snapshots to ensure that you are able to access your data in a timely manner. Overall, Volume Snapshots are a useful tool in a holistic data protection strategy, and every Kubernetes administrator should understand how to properly use them.