Our own, Mateusz Szostok has two sessions at KubeCon around K8s Service CatalogRead more

Mateusz Szostok, Piotr Kopczynski and Pawel Kosiec have several talks at San Diego Cloud Native MeetupRead more

Backup

Overview

Kyma integrates with Velero to provide backup and restore capabilities.

Velero backs up Kubernetes resources and stores them in buckets of supported cloud providers. It triggers physical volume snapshots and includes the snapshot references in the backup. Velero can create scheduled or on-demand backups, filter objects to include in the backup, and set time to live (TTL) for stored backups.

If you configured Velero when installing Kyma as explained here, backup is enabled with the default schedule and runs once a day every day from Monday to Friday. To change the settings of the backup change the schedules configuration in the Velero chart configuration.

For more details, see the official Velero documentation.

Installation

Install Velero

Install and configure Velero to back up and restore your Kyma cluster.

NOTE: To successfully set up Velero, define a supported storage location and credentials to access it. Currently, you can install Velero on GCP and Azure. AWS is not supported.

Follow the instructions to set up Velero:

  1. Enable Velero components in the Kyma Installer configuration file:

    Click to copy
    - name: "backup-init"
    namespace: "kyma-system"
    - name: "backup"
    namespace: "kyma-system"
  2. Override the default configuration by creating a Secret containing the required parameters for a chosen provider.

    See the examples of such Secrets:

    NOTE: The values are provided in plain text only for illustrative purposes. Remember to set them as base64-encoded strings. For details on Kyma overrides, see the this document.

    • Google Cloud Platform
    • Azure
  3. Run the Kyma installation with Velero overrides:

    • Local installation
    • Cluster installation

Configuration

Velero chart

This document lists and describes the parameters that you can configure in velero, split into required and optional parameters.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Required parameters

This table lists the required parameters for velero to work, their descriptions, and default values:

ParameterDescriptionDefault valueRequired
configuration.providerSpecifies the name of the cloud provider where you are deploying Velero to, such as aws, azure, gcp.NoneYes
configuration.backupStorageLocation.nameSpecifies the name of the cloud provider used to store backups, such as aws, gcp, or azure.NoneYes
configuration.backupStorageLocation.bucketSpecifies the storage bucket where backups are uploaded.NoneYes
configuration.backupStorageLocation.config.regionProvides the region in which the bucket is created. It only applies to AWS.NoneYes, if using AWS
configuration.backupStorageLocation.config.resourceGroupSpecifies the name of the resource group which contains the storage account for the backup storage location. It only applies to Azure.noneyes, if using Azure
configuration.backupStorageLocation.config.storageAccountProvides the name of the storage account for the backup storage location. It only applies to Azure.NoneYes, if using Azure
configuration.volumeSnapshotLocation.nameSpecifies the name of the cloud provider the cluster is using for persistent volumes.NoneYes, if using PV snapshots
configuration.volumeSnapshotLocation.config.regionProvides the region in which the bucket is created. It only applies to AWS.NoneYes, if using AWS
configuration.volumeSnapshotLocation.config.apitimeoutDefines the amount of time after which an API request returns a timeout status. It only applies to Azure.NoneYes, if using Azure
credentials.useSecretSpecifies if a secret is required for IAM credentials. Set this to false when using kube2iam.trueYes
credentials.existingSecretIf specified and useSecret is true, uses an existing secret with this name instead of creating one.NoneYes, if useSecret is true and secretContents is empty
credentials.secretContentsIf specified and useSecret is true, provides the content for the credentials secret.NoneYes, if useSecret is true and existingSecret is empty

Configurable parameters

This table lists the non-required configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
schedulesSets up a scheduled backup. By default, a scheduled backup runs at 07:00 daily on Monday through Friday.scheduled-backup
configuration.volumeSnapshotLocation.bucketSpecifies the name of the storage bucket where volume snapshots are uploaded.None
configuration.backupStorageLocation.prefixSpecifies the directory inside a storage bucket where backups are located.None
configuration.backupStorageLocation.config.resourceGroupSpecifies the name of the resource group which contains the storage account for the backup storage location. It only applies to Azure.None
configuration.backupStorageLocation.config.s3ForcePathStyleSpecifies whether to force path style URLs for S3 objects. Set it to true if you use a local storage service like MinIO. It only applies to AWS.None
configuration.backupStorageLocation.config.s3UrlSpecifies the AWS S3 URL. If not provided, Velero generates it from region and bucket. Use this field for local storage services like MinIO.None
configuration.backupStorageLocation.config.kmsKeyIdSpecifies the AWS KMS key ID or alias to enable encryption of the backups stored in S3. It only works with AWS S3 and may require explicitly granting key usage rights.None
configuration.backupStorageLocation.config.publicUrlSpecifies the parameter used instead of 3Url when generating download URLs, for example for logs. Use this field for local storage services like MinIO.None

See the official Velero documentation for examples and the full list of parameters, as well as for VolumeSnapshotLocation and BackupStorageLocation.

Tutorial

Back up a Kyma cluster

Follow this tutorial to learn how to use the backup.yaml specification file to create a manual or scheduled Kyma backup. For details about the file format, see this document.

Prerequisites

Install Velero using these instructions.

Steps

Follow the steps to back up Kyma.

  • Manual backup
  • Scheduled backup

Backup retention period

To set the retention period of a backup, define the ttl parameter in the Backup specification definition:

Click to copy
# The amount of time before this backup is eligible for garbage collection.
ttl: 24h0m0s

Restore a Kyma cluster

Follow this tutorial to restore a backed up Kyma cluster. Start with restoring CRDs, services, and endpoints, then restore other resources.

Prerequisites

To use the restore functionality, download and install the Velero CLI.

Steps

Follow these steps to restore resources:

  1. Install the Velero server. Use the same bucket as for backups:

    Click to copy
    velero install --bucket <BUCKET> --provider <CLOUD_PROVIDER> --secret-file <CREDENTIALS_FILE> --restore-only --wait

    NOTE: Check out this guide to correctly fill the parameters of this command corresponding to the cloud provider in use.

  2. Install Kyma backup plugins:

    Click to copy
    velero plugin add eu.gcr.io/kyma-project/backup-plugins:e7df9098
  3. List available backups:

    Click to copy
    velero get backups
  4. Restore Kyma CRDs, services, and endpoints:

    Click to copy
    velero restore create --from-backup <BACKUP_NAME> --include-resources customresourcedefinitions.apiextensions.k8s.io,services,endpoints --include-cluster-resources --wait
  5. Restore the rest of Kyma resources:

    Click to copy
    velero restore create --from-backup <BACKUP_NAME> --exclude-resources customresourcedefinitions.apiextensions.k8s.io,services,endpoints --include-cluster-resources --restore-volumes --wait

    Once the status of the restore is COMPLETED, perform a Kyma health check by verifying the Pods:

    Click to copy
    kubectl get pods --all-namespaces

    Even if the restore process is complete, it may take some time for the resources to become available again.

    NOTE: Because of this issue in Velero, custom resources may not be properly restored. In this case, run the second restore command again and check if the custom resources are restored. For example, run the following command to print several VirtualService custom resources:

    Click to copy
    kubectl get virtualservices --all-namespaces
  6. Once the restore succeeds, remove the velero namespace:

    Click to copy
    kubectl delete ns velero

Troubleshooting

Restore troubleshooting

Pod stuck in Init phase

In case the service-catalog-addons-service-binding-usage-controller Pod gets stuck in the Init phase, try deleting the Pod:

Click to copy
kubectl delete $(kubectl get pod -l app=service-catalog-addons-service-binding-usage-controller -n kyma-system -o name) -n kyma-system

Different DNS and public IP address

The restore tutorial assumes that the DNS and the public IP values for the new cluster are the same as for the backed up cluster. If they change, check the relevant fields in the Secrets and ConfigMaps overrides in the kyma-installer Namespace and update them with new values. Then run the installer again to propagate them to all the components:

Click to copy
kubectl label installation/kyma-installation action=install

Eventing not working

Check if all NatssChannels are reporting as ready:

Click to copy
kubectl get natsschannels.messaging.knative.dev -n kyma-system

If one or more channels report as not ready, delete their corresponding channel services. These services will be recreated by the controller automatically.

Click to copy
kubectl delete service -l messaging.knative.dev/role=natss-channel -n kyma-system
kubectl annotate natsschannels.messaging.knative.dev -n kyma-system restore=done --all