Velero is now a CNCF Sandbox project, accepted on July 30, 2026, and the Velero team says everyday usage did not change with the move. That is good news if you already run it, and a decent reason to learn it if you do not. This tutorial walks through a full Velero Kubernetes backup and restore setup in 13 steps, using a local kind cluster and an S3-compatible bucket you can run on a laptop in about 60 minutes.
By the end you will have a working backup of a stateful app, a scheduled backup with retention, a tested restore, a restore into a second cluster, and a small drill script that proves your backups work before you need them. The same flow carries straight over to Amazon EKS, GKE, AKS, or a self-managed cluster. If you have only ever relied on a cloud snapshot service such as AWS Backup, you will see where Velero fits differently: it saves Kubernetes objects and volume data together, so you can restore one namespace instead of a whole node disk.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why Velero Backup Matters After the CNCF Move
Kubernetes keeps its desired state in etcd, but most teams never touch etcd directly. Managed services hide it, and even on self-managed clusters a raw etcd snapshot restores everything or nothing. Velero works one level up. It asks the API server for objects, writes them to object storage as JSON, and handles persistent volume data through snapshots or a file-level copy. That makes it the standard tool for namespace-level restores, cluster migrations, and disaster recovery drills.
The project’s governance changed this year. Broadcom announced on March 23, 2026 that it had contributed Velero to the CNCF as a Sandbox project, framing the move as a step toward open, community-driven governance. The CNCF acceptance followed on July 30, 2026. VMware had acquired Heptio, the company that created Velero, back in 2019, so this is the first time the project sits outside a single vendor. For buyers, that lowers the risk of betting on it. For you as an operator, nothing in the install flow changes.
On versions, the picture is slightly messy. Velero’s own blog documents 1.17 and 1.18. Release 1.17 added VolumeGroupSnapshot support for crash-consistent multi-volume backups, a modernized file-system backup architecture, and Windows cluster support. Release 1.18 added concurrent backup processing, cache volumes for data mover pods, and incremental backup size reporting. Broadcom’s October 12, 2026 KubeCon North America post also lists Velero 1.19 among its open-source contributions, but I could not find an official 1.19 release post, so this tutorial avoids pinning a version. Run the latest release your CLI download page offers and check it with the version command in Step 3.
What Velero backs up, and what it skips
A Velero backup has two halves. The first half is the Kubernetes resource definitions: Deployments, Services, ConfigMaps, Secrets, custom resources, and so on. Velero stores those in your backup storage location, which is a bucket. The second half is the data behind persistent volumes, and here you have a choice of path.
| Data path | How it works | Where volume data lands | Best for |
|---|---|---|---|
| File system backup (Kopia) | A node agent reads files from the mounted volume and uploads them | Your bucket | Any storage, including NFS and local disks, and the kind lab in this guide |
| CSI snapshot | Velero asks the CSI driver for a snapshot | Stays in the storage provider | Fast backups on EBS, Persistent Disk, or Azure Disk |
| CSI snapshot with data movement | Velero snapshots, then uploads the snapshot contents | Your bucket | Portable backups that survive losing the region or the cloud account |
| Native cloud snapshot via plugin | Provider plugin calls the cloud API directly | Stays in the provider | Older setups on a single cloud |
Velero does not back up the nodes, the container images, or the contents of external databases. It also does not touch etcd. If your cluster dies and you need the control plane rebuilt, you first create a fresh cluster, then Velero fills it back up. Teams that already keep manifests in Git and ship with a progressive delivery tool such as Argo Rollouts still need Velero, because Git cannot hold the runtime state: PVC data, Secrets created by operators, and objects generated by controllers.
Prerequisites and Versions to Check First
You need a Linux, macOS, or Windows (WSL2) machine with Docker running and about 8 GB of free RAM. The lab uses three small kind nodes, so a modest laptop works. Install these tools before you start, and use current releases of each.
- Docker Engine or Docker Desktop, running and able to start containers without sudo
- kind, the Kubernetes-in-Docker tool, latest release
- kubectl, within one minor version of the cluster kind creates
- The Velero CLI, latest release from the Velero download page
- A text editor and a shell that supports here-documents (bash or zsh)
A word on the Kubernetes version. Do not pin an old node image in kind. The Kubernetes 1.34 branch reportedly entered maintenance mode on August 27, 2026, and reports put its end of life at October 27, 2026, so a new lab should start on a currently supported branch. The official Kubernetes blog described 1.36 as the current release on October 6, while, according to AWS’s announcement, Kubernetes 1.37 reached Amazon EKS on October 2, 2026. The sources disagree on which one counts as newest, so simply let kind pull its default image and note what kubectl version reports.
Also check your hosts for cgroup v2. Kubernetes moved cgroup v1 into maintenance mode back in release 1.31 and recommends cgroup v2 today, so modern Linux distributions already ship it. If your kind nodes fail to start on an older host, that is the first thing to inspect.
If you run a self-managed cluster instead of kind, a kubeadm cluster built with OpenTofu works fine as a target. Velero needs only API access, a bucket, and credentials.
Plan for about 60 minutes in total. Here is how the time splits across the 13 steps.
- Steps 1 to 3 (10 min): create the cluster and install the CLI
- Steps 4 to 5 (8 min): start S3-compatible storage and create the bucket
- Steps 6 to 7 (10 min): install Velero and verify the storage location
- Steps 8 to 10 (15 min): deploy an app, back it up, destroy it, restore it
- Steps 11 to 12 (8 min): add schedules, retention, and hooks
- Step 13 (10 min): restore into a second cluster
Steps 1 to 3: Build the Lab Cluster and Install the CLI
Step 1: Confirm Docker and the tooling work. Run the checks below. Each command should print a version without an error. If docker ps fails with a permissions message, fix that first, because kind cannot start without it.
docker ps
kind version
kubectl version --clientStep 2: Create a three-node kind cluster. One control plane and two workers gives the node agent something realistic to schedule across. Save the config and create the cluster.
cat > kind-velero.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --name velero-lab --config kind-velero.yaml
kubectl get nodesExpect three nodes in the Ready state after a minute or two. The output looks like this, with your Kubernetes version in the last column.
NAME STATUS ROLES AGE VERSION
velero-lab-control-plane Ready control-plane 75s (your version)
velero-lab-worker Ready <none> 55s (your version)
velero-lab-worker2 Ready <none> 55s (your version)kind also creates a Docker network named kind. You will use it in Step 4 so that storage lives outside the cluster, which is the right habit for any real backup design.
Step 3: Install the Velero CLI. On macOS or Linux with Homebrew, run brew install velero. Otherwise download the archive that matches your OS from the releases page linked from the Velero documentation, unpack it, and move the velero binary into a folder on your PATH. Then confirm the client version.
velero version --client-onlyWrite down the version it prints. The server component you install in Step 6 should match the CLI’s major and minor release, and the AWS plugin you pick must be compatible with that release. The compatibility table sits in the plugin’s README and in the Velero docs, and you should read it rather than guess.
Set one shell variable now so later commands stay readable. Replace the placeholder with the plugin tag the compatibility table lists for your Velero version.
export AWS_PLUGIN_TAG=<tag-from-compatibility-table>Steps 4 to 5: Stand Up S3-Compatible Storage and Credentials
Step 4: Run MinIO outside the cluster. Velero speaks the S3 API, and the AWS plugin works with any S3-compatible service when you set a custom endpoint and path-style addressing. Running MinIO as a plain Docker container on the kind network gives you a bucket that survives deleting the cluster, which is exactly what a disaster recovery test needs.
docker run -d --name velero-minio --network kind \
-e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=minio123 \
-p 9000:9000 -p 9001:9001 \
quay.io/minio/minio server /data --console-address ":9001"The image tag is deliberately left at the default here because this is a throwaway lab. In production you would pin a tested tag, or skip MinIO entirely and use your cloud’s object storage. If the image pull fails on your network, any S3-compatible store works, including Ceph’s object gateway, which Red Hat’s OpenShift backup documentation lists next to MinIO.
Step 5: Create the bucket and a credentials file. The credentials file uses the standard AWS shared-credentials format, which the plugin reads. The bucket creation runs the MinIO client in a short-lived container on the same network.
cat > credentials-velero <<'EOF'
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123
EOF
docker run --rm --network kind --entrypoint sh quay.io/minio/mc -c \
"mc alias set local http://velero-minio:9000 minio minio123 && mc mb --ignore-existing local/velero-lab && mc ls local"You should see the velero-lab bucket listed. If name resolution of velero-minio fails inside the cluster later, the troubleshooting section shows how to swap the hostname for the container’s IP address.
The credentials above are lab values. Never commit a real credentials file to Git, and never reuse the root account of a storage service for backups. Create a dedicated key that can read and write only the Velero bucket.
Steps 6 to 7: Install Velero With the Node Agent and Verify the Storage Location
Step 6: Install the Velero server components. The install command creates the velero namespace, the CRDs, the Velero deployment, and, because of the node-agent flag, a DaemonSet that handles file-level volume backups. Two flags matter most for this lab. The first is --use-volume-snapshots=false, since kind has no snapshot API to call. The second is --default-volumes-to-fs-backup, which makes Velero back up every pod volume by default instead of only the ones you annotate. That opt-out model is safer, because a forgotten annotation is how teams end up with empty restores.
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:${AWS_PLUGIN_TAG} \
--bucket velero-lab \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--use-node-agent \
--default-volumes-to-fs-backup \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://velero-minio:9000 \
--waitThe region value is arbitrary for MinIO, but the plugin requires one. The s3Url points at the container from Step 4. Path-style addressing is required because the bucket name cannot be a subdomain of a local hostname. Provider documentation for services such as STACKIT and NumSpot shows the same pattern of a custom endpoint plus path-style addressing, which is why the AWS plugin has become the common denominator for S3-compatible targets. If you want every option the plugin accepts, the plugin configuration reference lists them in one place.
When the command finishes, it prints a message saying Velero is installed. Check the pods.
kubectl -n velero get podsNAME READY STATUS RESTARTS AGE
node-agent-4kq7d 1/1 Running 0 40s
node-agent-vz9tn 1/1 Running 0 40s
velero-6c9b5d7f4d-xk2lp 1/1 Running 0 40sYou should see one Velero pod and one node-agent pod per worker node. The control-plane node carries a taint, so the agent normally skips it.
Step 7: Verify the backup storage location is Available. Velero checks the bucket on a schedule and records the result on the BackupStorageLocation object. Nothing else in this tutorial works until this reads Available.
velero backup-location getNAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero-lab Available 12s ago ReadWrite trueIf the phase is Unavailable, read the message with kubectl -n velero get backupstoragelocation default -o yaml and jump to the troubleshooting list. The three usual causes are a wrong endpoint, a missing path-style setting, and bad credentials.
Two details about that output are worth understanding before you move on. Velero revalidates the location on a timer, so a bucket that becomes unreachable later flips to Unavailable on its own, and new backups fail fast instead of hanging. The access mode column matters too. ReadWrite means this cluster can create and delete backups. ReadOnly means it can only list and restore, which is the safe setting for a recovery cluster and for any second cluster that shares a bucket with production. You will use that setting in Step 13.
It also helps to know what lives where. Velero stores each backup as a small tree of files under a prefix in the bucket: the resource JSON, logs, and metadata for the backup, plus a separate repository area for the deduplicated volume data. Because the metadata sits in the bucket and not only in the cluster, a brand-new cluster can rebuild its backup list just by pointing at the same location. That design is what makes the cross-cluster restore possible, and it is also why deleting the bucket deletes your only copy.
Steps 8 to 10: Deploy a Stateful App, Back It Up, Destroy It, and Restore
Step 8: Deploy a sample app that writes to a volume. A backup test is only meaningful if the data inside the volume changes over time. This app appends a timestamp to a file every five seconds. It also includes a ConfigMap and a Secret so you can confirm that Velero restores plain Kubernetes objects too.
cat > demo-app.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: demo
---
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: demo
data:
greeting: hello from the demo namespace
---
apiVersion: v1
kind: Secret
metadata:
name: app-secret
namespace: demo
type: Opaque
stringData:
token: lab-only-token
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: notes-data
namespace: demo
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
namespace: demo
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: writer
image: busybox:stable
command: ["/bin/sh", "-c", "while true; do date >> /data/log.txt; sleep 5; done"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: notes-data
EOF
kubectl apply -f demo-app.yaml
kubectl -n demo rollout status deploy/notes
sleep 60
kubectl -n demo exec deploy/notes -- wc -l /data/log.txtAfter a minute you should see roughly a dozen lines in the file. Keep that count in mind, because the restore in Step 10 should bring back at least that many.
Step 9: Take the first backup and read the report. The --wait flag blocks until the backup finishes, so the next command always sees a final state.
velero backup create demo-first --include-namespaces demo --wait
velero backup describe demo-first --detailsA healthy report ends with the phase set to Completed, zero errors, zero warnings, and a Pod Volume Backups section that lists the volume as Completed. Trimmed, it looks like this.
Name: demo-first
Namespace: velero
Phase: Completed
Errors: 0
Warnings: 0
Namespaces:
Included: demo
Backup Volumes:
Pod Volume Backups - kopia:
Completed:
demo/notes-5d8c7b9f6-abcde: dataDo not stop at the phase. Open the logs once so you know what a normal run looks like, because you will compare against them when something fails.
velero backup logs demo-first | head -40Step 10: Destroy the namespace and restore it. This is the step that turns a backup into a recovery plan. Deleting the namespace removes the Deployment, the PVC, and, with the default reclaim policy of Delete, the volume and its data. That is the realistic version of a bad day.
kubectl delete namespace demo
kubectl get namespace demo # expect: NotFound
velero restore create demo-restore-1 --from-backup demo-first --wait
kubectl -n demo rollout status deploy/notes
kubectl -n demo exec deploy/notes -- wc -l /data/log.txt
kubectl -n demo get configmap app-config -o jsonpath='{.data.greeting}'The restored file should contain the lines from before the deletion, plus a few new ones the pod wrote after it came back. The ConfigMap should print its greeting. If both check out, you have a working backup and restore path.
During the restore Velero first recreates the PVC, then runs an init container that waits for the file data to be copied back from the bucket before the main container starts. That is why the pod briefly sits in Init before it turns Running. You can read per-volume progress with velero restore describe demo-restore-1 --details.
Steps 11 to 12: Schedules, Retention, and Backup Hooks
Step 11: Create a schedule with a time-to-live. A single manual backup is a snapshot of one moment. Production needs a schedule and an expiry, so the bucket does not grow forever. The TTL tells Velero when to delete a backup and its data. The default is 30 days if you set nothing.
velero schedule create demo-nightly \
--schedule="0 2 * * *" \
--include-namespaces demo \
--ttl 168h0m0s
velero schedule get
velero backup create --from-schedule demo-nightly --wait
velero backup getThe last command fires the schedule’s template on demand, which is the quickest way to prove it works without waiting for 2 AM. Backups created by a schedule carry the schedule’s name as a prefix and a timestamp suffix. If you prefer to keep this in Git, the same schedule as a manifest looks like this.
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: demo-nightly
namespace: velero
spec:
schedule: "0 2 * * *"
template:
includedNamespaces:
- demo
ttl: 168h0m0s
defaultVolumesToFsBackup: trueChoose the interval from your recovery point objective, not from habit. A nightly backup means you can lose up to a day of writes. If that is too much, schedule every few hours, and remember that each run also costs storage and API calls.
Step 12: Add backup hooks for consistent data. A file copy taken while an application is mid-write can capture a half-written state. Velero lets you run a command inside a container right before and right after the backup, using pod annotations. For a real database you would flush and freeze it here. For the demo, a simple sync shows the mechanism.
kubectl -n demo patch deployment notes --type merge -p '{
"spec": {"template": {"metadata": {"annotations": {
"pre.hook.backup.velero.io/container": "writer",
"pre.hook.backup.velero.io/command": "[\"/bin/sh\", \"-c\", \"sync\"]",
"pre.hook.backup.velero.io/on-error": "Fail",
"pre.hook.backup.velero.io/timeout": "30s"
}}}}
}'
kubectl -n demo rollout status deploy/notes
velero backup create demo-hooked --include-namespaces demo --wait
velero backup describe demo-hooked --details | grep -i hookThe on-error: Fail setting makes the backup report a failure if the hook fails, which is what you want. A silent hook failure produces a backup that looks fine and restores a corrupt database. For PostgreSQL or MySQL, point the hook at a command that runs a checkpoint or takes a logical dump into the volume first, and consider the database operator’s own backup feature for the data itself.
Step 13: Restore Into a Second Cluster
A backup that only restores into the cluster that produced it is half a plan. The point of putting storage outside the cluster is that a brand-new cluster can read the same bucket and rebuild your workloads. This step simulates losing the whole cluster.
Step 13: Create a second kind cluster and point it at the same bucket. Install Velero there with the same flags, then mark the storage location read-only. A read-only location lets the DR cluster read and restore backups but stops it from writing or deleting anything, so a test restore cannot damage the real backup set.
kind create cluster --name velero-dr --config kind-velero.yaml
kubectl config use-context kind-velero-dr
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:${AWS_PLUGIN_TAG} \
--bucket velero-lab \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--use-node-agent \
--default-volumes-to-fs-backup \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://velero-minio:9000 \
--wait
kubectl -n velero patch backupstoragelocation default \
--type merge -p '{"spec":{"accessMode":"ReadOnly"}}'
sleep 70
velero backup getVelero syncs backup metadata from the bucket about once a minute, so after a short wait the new cluster lists demo-first, demo-hooked, and the scheduled backup, even though it has never run them. Restore the namespace.
velero restore create dr-restore-1 --from-backup demo-first --wait
kubectl -n demo get all,pvc
kubectl -n demo exec deploy/notes -- wc -l /data/log.txtThe restored cluster now holds the same app and the same file contents. In a real migration you add a few checks first. Install the same storage classes, ingress controller, and any operators that own custom resources before you restore, because Velero restores objects but does not install the controllers that understand them. If your apps expose traffic through an Ingress, recreate the controller first, as covered in the Kubernetes ingress setup guide. Moving between cloud providers works the same way, for instance from a cluster on Google Kubernetes Engine to a fresh one elsewhere, as long as the volume data travels through the bucket rather than staying inside a provider snapshot.
When you are done testing, switch back with kubectl config use-context kind-velero-lab (kind names contexts with a kind- prefix, so the first cluster is kind-velero-lab).
Choosing a Storage Target and What It Costs
The lab bucket costs nothing, but a production design needs a real target. Velero treats the bucket as dumb storage, so the main questions are where it lives, who can delete from it, and what it costs to keep and to read back. Keep the bucket in a different account or at least a different failure domain than the cluster. A backup that sits in the same account as the cluster it protects will not survive a compromised admin credential.
| Target | How you configure it | What to verify first |
|---|---|---|
| Amazon S3 | AWS plugin with a region, IAM role or access key | S3 Standard in US East (N. Virginia) lists at $0.023 per GB-month for the first 50 TB, with request, retrieval, and transfer charges on top |
| MinIO or Ceph object gateway | AWS plugin with a custom s3Url and s3ForcePathStyle=true | Version support, TLS certificates, and bucket versioning |
| STACKIT Object Storage | AWS plugin with the provider endpoint and path-style addressing | Endpoint region and access key scope |
| NumSpot Object Storage | AWS plugin with a custom endpoint and path-style addressing | Endpoint and key permissions |
| Azure Blob or Google Cloud Storage | Provider-specific plugin or an S3-compatible gateway | Current plugin compatibility, authentication method, lifecycle rules, and egress charges |
Provider documentation backs the pattern in the first rows. STACKIT’s Velero guide, already referenced in Step 6, and NumSpot’s backup and restore guide both configure the AWS plugin against their own S3-compatible endpoints. For Azure Blob, Google Cloud Storage, or any other service, check the provider’s current endpoint, authentication method, lifecycle rules, object lock or versioning support, and egress charges before you commit, because the Velero plugin ecosystem does not guarantee identical behavior across providers.
Running Velero on Amazon EKS
AWS published an official Velero procedure for Amazon EKS on May 12, 2026. It stores Kubernetes resource definitions in Amazon S3 and persistent volume data in Amazon EBS snapshots, and it supports cross-cluster restores and namespace-level operations. The walkthrough runs on a cluster with EKS Auto Mode, but AWS states that Velero does not require Auto Mode and works on any EKS cluster. If you still need a cluster, the EKS setup guide covers the build.
The production install swaps the access key for an IAM role bound to the Velero service account. The policy below follows the shape the AWS plugin documents, with your bucket name in place of the placeholder.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": ["arn:aws:s3:::YOUR-BUCKET/*"]
},
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::YOUR-BUCKET"]
}
]
}velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:${AWS_PLUGIN_TAG} \
--bucket YOUR-BUCKET \
--backup-location-config region=us-east-1 \
--snapshot-location-config region=us-east-1 \
--no-secret \
--sa-annotations eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/velero \
--use-node-agentDo the cost math before you pick a retention window. A backup set of 100 GB held in S3 Standard costs about $2.30 per month in storage at the listed rate, before requests and transfer. That is small next to the control plane of a standby cluster. AWS charges $0.10 per cluster-hour for standard Kubernetes version support on EKS, which works out to about $73 for a 730-hour month before any nodes, and $0.60 per cluster-hour once a version moves into extended support. AWS says standard support lasts 14 months after a version’s EKS release, followed by 12 months of extended support. Most teams keep the DR cluster as code and create it only during a drill or an incident.
AWS Backup also covers EKS, and its pricing page reportedly describes a one-time charge per namespace backed up plus storage charges for cluster state data. It suits teams that want policy-driven protection inside one cloud. Velero suits teams that want portability across clouds and a restore path they can run from a laptop.
Complete Working Project: A Restore Drill You Can Schedule
Backups fail quietly. The only real proof is a restore, so this project automates one. The script takes a fresh backup of a namespace, restores it into a temporary mapped namespace, checks that the data came back, then cleans up and exits with a non-zero code on any failure. Run it from a laptop, a CI job, or a cron host that has kubectl access.
Create the file velero-drill.sh in your project folder.
#!/usr/bin/env bash
set -euo pipefail
SRC_NS="${1:-demo}"
DRILL_NS="${SRC_NS}-drill"
STAMP="$(date +%Y%m%d-%H%M%S)"
BACKUP="drill-${STAMP}"
RESTORE="${BACKUP}-restore"
echo "[1/5] Creating backup ${BACKUP} from namespace ${SRC_NS}"
velero backup create "${BACKUP}" --include-namespaces "${SRC_NS}" --wait
PHASE="$(kubectl -n velero get backup "${BACKUP}" -o jsonpath='{.status.phase}')"
if [ "${PHASE}" != "Completed" ]; then
echo "FAIL: backup phase is ${PHASE}"
exit 1
fi
echo "[2/5] Restoring into ${DRILL_NS}"
velero restore create "${RESTORE}" \
--from-backup "${BACKUP}" \
--namespace-mappings "${SRC_NS}:${DRILL_NS}" \
--wait
echo "[3/5] Waiting for the workload"
kubectl -n "${DRILL_NS}" rollout status deploy/notes --timeout=180s
echo "[4/5] Checking restored data"
LINES="$(kubectl -n "${DRILL_NS}" exec deploy/notes -- wc -l /data/log.txt | awk '{print $1}')"
if [ "${LINES}" -lt 1 ]; then
echo "FAIL: restored volume is empty"
exit 1
fi
echo "[5/5] Cleaning up"
kubectl delete namespace "${DRILL_NS}" --wait=false
velero restore delete "${RESTORE}" --confirm
echo "DRILL OK: ${LINES} lines restored from ${BACKUP}"The script proves four separate things. The backup stage proves Velero can reach the bucket and finish a run. The restore stage proves the stored data is readable and that object definitions still apply cleanly to the cluster. The rollout check proves the restored pods start with their volume attached. The data check proves the volume actually contains your files. A line count is a weak check, so for a real database replace it with a query that compares a row count or a checksum against a known value. Namespace mapping is what keeps the drill safe, because the restored copy lives in demo-drill and never touches the live namespace.
Make it executable and run it once by hand.
chmod +x velero-drill.sh
./velero-drill.sh demoA passing run prints five numbered stages and finishes with a line that starts with DRILL OK. To schedule it, add a weekly cron entry on the machine that runs it, and pipe failures to whatever pages you.
0 6 * * 1 cd /opt/velero-drill && ./velero-drill.sh demo || curl -fsS -X POST -d 'velero drill failed' https://alerts.example.internal/hookNext, alert on the backups themselves. Velero exposes Prometheus metrics, including counters for failed backups and a timestamp of the last successful run per schedule. Check your own /metrics output for the exact names in your version, then adapt this rule.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: velero-backup-alerts
namespace: velero
spec:
groups:
- name: velero
rules:
- alert: VeleroBackupStale
expr: time() - velero_backup_last_successful_timestamp{schedule="demo-nightly"} > 90000
for: 15m
labels:
severity: page
annotations:
summary: No successful demo-nightly backup in over 25 hours
- alert: VeleroBackupFailing
expr: increase(velero_backup_failure_total{schedule="demo-nightly"}[1h]) > 0
labels:
severity: ticket
annotations:
summary: A demo-nightly backup failed in the last hourThe 90,000-second threshold equals 25 hours, which gives a nightly job one hour of slack before it pages someone. Together, the drill and the alert cover both questions that matter: did a backup run, and can it actually be restored.
Velero vs etcd Snapshots, and Planning RPO and RTO
People often ask whether an etcd snapshot makes Velero redundant. It does not, because the two tools answer different questions. An etcd snapshot captures the entire control plane database at one moment. Restoring it rolls the whole cluster back, including every namespace, and it needs access to the control plane nodes, which managed services such as EKS and GKE do not give you. Velero captures API objects and volume data at the namespace or label level, restores selectively, and works the same on managed and self-managed clusters.
A sensible self-managed design uses both. Take etcd snapshots to rebuild a dead control plane quickly, and run Velero to recover workloads and data. For managed clusters, Velero plus infrastructure as code is the realistic pairing. You can read more about the wider set of managed and self-hosted options in the cloud computing section.
Two numbers shape the whole design. Recovery point objective is how much data you can afford to lose, and recovery time objective is how long recovery may take. Your schedule sets the first. If you back up every six hours, the worst case is six hours of lost writes plus however long the backup itself runs. Your data size and bandwidth set the second. As an illustrative example with assumed numbers, restoring 200 GB at a sustained 100 MB per second takes about 2,000 seconds, or roughly 34 minutes, before the pods start and pass health checks. Measure your own throughput in a drill instead of trusting any estimate.
Use the table below as a planning aid. The values are examples you tune, not Velero defaults.
| Tier | Example workload | Backup interval | Retention | Drill frequency |
|---|---|---|---|---|
| Critical | Customer database volumes | Every 1 to 2 hours | 7 to 14 days | Weekly |
| Standard | Internal tools, queues | Every 6 hours | 14 days | Monthly |
| Low | Dev and staging namespaces | Daily | 3 to 7 days | Quarterly |
Common Pitfalls That Break Velero Backups
Most failed recoveries trace back to a handful of avoidable mistakes. Check your setup against this list.
- Never testing a restore. A Completed phase only proves the upload finished. Run the drill from the previous section on a schedule, and treat a missed drill the same way you treat a missed backup. Teams that only restore during an outage discover permission gaps, plugin mismatches, and missing storage classes at the worst possible time.
- Keeping the bucket in the same failure domain. If one account or region loss takes out both the cluster and the bucket, you have no backup. Replicate the bucket across regions or accounts.
- Assuming volumes are included. Without the default fs-backup flag or a per-pod annotation, Velero saves the PVC object but not the data behind it, and the restore comes back empty.
- Mixing up snapshot types. A CSI snapshot that stays inside the provider dies with the account. If you need portability, use data movement so the contents land in your bucket.
- Skipping the TTL. The default retention is 30 days. Without a deliberate TTL you either pay for old backups forever or lose history sooner than you planned.
- Backing up churn. Short-lived objects are noise. A cluster running game servers, such as those in the Agones scaling walkthrough or the dedicated server guide, creates and destroys GameServer objects constantly. Back up the Fleet definitions and exclude the instances with
--exclude-resources gameservers.agones.dev. - Forgetting restore ordering. Velero restores objects but does not install the operators and CRDs behind them. Install controllers and storage classes in the target cluster first.
- Losing the repository password. File-level backups are encrypted with a password stored in a Secret in the velero namespace. Keep an offline copy, because the data is unreadable without it.
Troubleshooting Velero Setup and Restore Problems
Install and storage problems
The backup location stays Unavailable. Read the message on the BackupStorageLocation object. A connection refused or no such host error means the s3Url is wrong or unreachable from the pod. Fix the endpoint and, for MinIO, keep s3ForcePathStyle set to true.
Name resolution for the MinIO container fails inside the cluster. Look up the container address with docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' velero-minio and use http://IP:9000 as the s3Url in the install command or by editing the location.
AccessDenied or SignatureDoesNotMatch errors. The key pair is wrong, the key lacks permission on the bucket, or the system clock is badly skewed. Re-create the credentials Secret from a corrected file, then restart the Velero deployment.
The Velero pod crash-loops after install with a plugin error. The plugin image tag does not match your Velero release. Check the compatibility table and reinstall with the tag it lists.
Node-agent pods fail to start or cannot see pod volumes. The agent mounts the kubelet’s pod directory from the host. Some distributions place it elsewhere, so edit the DaemonSet hostPath to match your kubelet root directory.
A file backup runs out of memory on large volumes. Raise the node-agent memory request and limit with the install flags for node-agent resources, or patch the DaemonSet, and rerun the backup.
Backup and restore problems
The backup finishes as PartiallyFailed. Run velero backup logs NAME and search for error lines. A common cause is a pod volume that could not be read because the pod was not running. Rerun once the workload is healthy.
The restore reports warnings that resources already exist. Velero does not overwrite existing objects by default. Restore into an empty namespace, use a namespace mapping, or set the existing-resource policy to update when you intend to overwrite.
The restored PVC stays Pending in the new cluster. The storage class from the source does not exist in the target. Create a ConfigMap labeled for the storage class change plugin to map the old name to a new one.
apiVersion: v1
kind: ConfigMap
metadata:
name: change-storage-class-config
namespace: velero
labels:
velero.io/plugin-config: ""
velero.io/change-storage-class: RestoreItemAction
data:
old-storage-class: new-storage-classScheduled backups never run. Check that the schedule is not paused and that the cron expression is valid, then look at the Velero pod logs around the expected time.
Backups vanish earlier than expected. The TTL expired. Extend it on the schedule and note that the new value only applies to future backups.
Advanced Tips for Production Velero
Use label selectors when namespaces are too coarse. A schedule with a selector, such as one for every object labeled backup=true, lets application teams opt in without touching your schedule. A Kyverno policy engine rule can require that label on new namespaces, so nothing slips through unprotected.
Use volume policies to choose a method per volume. Velero supports a resource policies ConfigMap that matches volumes by storage class, size, or driver, so databases on fast block storage take CSI snapshots while small config volumes use file-level copies. That keeps backup windows short and costs predictable.
Here is a volume policy that snapshots volumes on a named storage class and sends everything on NFS through the file-level path. Store it as a ConfigMap in the velero namespace and reference it when you create a backup or schedule.
apiVersion: v1
kind: ConfigMap
metadata:
name: volume-policy
namespace: velero
data:
policy.yaml: |
version: v1
volumePolicies:
- conditions:
storageClass:
- fast-ssd
action:
type: snapshot
- conditions:
nfs: {}
action:
type: fs-backupvelero backup create tiered --include-namespaces demo --resource-policies-configmap volume-policy --waitRestore hooks complete the picture. Where backup hooks quiesce an application, restore hooks run after the pod starts, for example to replay a write-ahead log or to wait for a dependency before the app accepts traffic. Use them sparingly and test them in the drill, because a failing restore hook can leave a workload half-started.
Turn on data movement for portability. When you take CSI snapshots, adding snapshot data movement uploads the contents to the bucket, which means a regional outage does not take your only copy with it. Release 1.18’s cache volumes for data mover pods speed up the restore side of this path.
Keep several storage locations. Add a second BackupStorageLocation in another region or provider and point a second schedule at it. Two independent copies cost little compared with a total loss.
Protect the bucket itself. Turn on versioning and object lock where the provider supports them, restrict delete permissions to a break-glass role, and enable server-side encryption. Velero relies on the storage layer for protecting resource data at rest, so the bucket settings are your only control.
Back up the backup tooling. Export the repository password Secret and your Velero install command to a safe place. In a real incident, you want to reinstall Velero in minutes and point it at the existing bucket with a read-only location, exactly as in Step 13.
Finally, consider the Helm chart for repeatable installs. Velero ships a chart, and since governance moved to the CNCF, confirm its current location in the project documentation before you depend on a specific repository URL.
Frequently Asked Questions
Is Velero free to use? Velero is open source, so the software costs nothing. You pay for the storage and transfer where backups live, plus the compute that runs the node agent. Check commercial support terms with your vendor, because the sources I reviewed did not confirm current paid Velero plans.
Does Velero back up etcd? No. It reads objects through the Kubernetes API and stores them in object storage. For control plane recovery on self-managed clusters, keep separate etcd snapshots.
Which Velero version should I install? Install the newest release the project publishes on its download page, and match the AWS plugin to it using the compatibility table. Velero 1.18 is documented on the project blog, and Broadcom’s KubeCon material lists 1.19, so confirm the latest tag before you install.
Can Velero migrate workloads between clouds? Yes, when volume data travels through the bucket. Use file system backup or snapshot data movement so the contents are portable, install matching storage classes in the target, and restore from a read-only location.
How often should I back up? Pick the interval from the data you can afford to lose. A six-hour schedule means up to six hours of lost writes in the worst case. Critical databases usually need shorter intervals plus application-level backups.
Can I restore into a different namespace? Yes. Use the namespace mapping option on the restore command, as the drill script does, to bring a copy up beside the original without touching it.
Does the CNCF Sandbox move change anything for current users? According to the Velero team, everyday usage did not change. Broadcom framed the contribution as a step toward community-driven governance, and the project continues to ship releases.
