Release

Overview

Release guidelines describe the release strategy and process in Kyma. This section also provides details on who to contact, how to communicate during the release process, and how to subscribe to email notifications on new releases in GitHub repositories.

Read about:

The release strategy in Kyma
How to create a release using Prow according to the release process
Communication between the Release Master and the rest of the organization
Subscription to releases

Release strategy

Read about the release strategy in Kyma.

Terminology

Release types

For release versioning, Kyma uses the approach similar to Semantic Versioning. However, while Semantic Versioning focuses on APIs, the release process for Kyma refers to the complete Kyma deliverable. Despite this difference, Kyma follows the same release types:

Type	Scope
MAJOR	Introduces breaking changes
MINOR	Introduces a new functionality in a planned schedule
PATCH hot.fix	Introduces a backward-compatible bug fix as an unscheduled release

Function freeze

The function freeze refers to the point in time when a new release branch is created from the master branch. After the function freeze, no new features are included in the release. This is typically on Monday at 9:00 AM CET/CEST every 4 weeks.

Roles

These persons actively participate in release preparation and execution:

Product Lead ensures that all capabilities provide a valuable and consistent product when it comes to its features and usability.
Release Manager is responsible for the overall release:
- Ensures that all persons involved in the release follow the required processes and standards
- Decides when to create a new release candidate
- Makes the final call for publishing the release candidate
- Creates and manages the release schedule and release plan
Product Manager coordinates the process of gathering the input for the release notes.
Technical Writer writes and publishes the release notes as a blog post on the kyma-project.io website, based on the input from Product Owners.
Developer ensures that the release branch contains all required features and fixes.
Release Master prepares all mandatory release artifacts and conducts technical release activities described in the release process.
Release Publisher prepares and publishes the social media content based on the release notes.

Release requirements and acceptance criteria

Epics and user stories

The functional implementations that are included in the release are documented as GitHub issues. A contributor completes an issue bound by the license agreement and contributing guidelines. All proposed changes must pass the obligatory Continuous Integration checks and require the maintainers' review and approval.

You can close an epic or an issue if the new functionality meets the contributing rules. These include, but are not limited to, the following points:

Automated unit, integration and end-to-end tests verify this functionality.
This functionality is documented.

Test coverage

In Kyma, we follow the test strategy according to which you can verify all functionalities through automated testing that you execute either through the CI or the release pipeline.

The Kyma community must provide the highest possible test coverage to minimize any potential risks and ensure the release is functional and stable.

Consider implementing tests that go beyond verifying the functionality of the system. This includes integration, performance, and end-to-end tests.

Compliance rules

All Kyma organization members involved in the release process comply with contribution rules defined for Kyma and ensure that all checks built into the continuous integration process pass. These requirements represent the absolute minimal acceptance criteria needed for a release from the compliance perspective.

Additionally, manual pre-release verification must cover these areas:

the use of the third-party software and components that require licensing
the use and storage of personal data subject to the GDPR requirements

Security

Open-source projects, like any other, must ensure secure development. You can provide it by constantly raising the awareness of everyone involved in the project. In Kyma, we strive to make all developers aware of the secure development requirement. For example, we conduct threat modeling workshops during which the participants proactively think about possible security threats in the architecture, infrastructure, and implementation.

The Release Manager in Kyma takes care of formal security validation activities performed before major releases. The results of these activities influence the release decision. Lack of attention to security topics can result in release delay.

Critical issues

Any issues affecting the expected functionality of Kyma should be documented on GitHub for review and planning. If any of these issues is defined as critical according to the criteria specified in this section, it should be brought to the immediate attention of the Release Manager.

Critical issues are the ones that:

Render a Kyma cluster unstable or unusable and have no workaround.
Cause loss or corruption of data.
Expose critical security vulnerabilities (CVSS 3 Score 9.0 - 10.0).
Cause tests to fail in any supported environment.
Cause a major impact on any production environment.

Critical issues have to be fixed with priority. They should contain information about the timeline for the fix and the risks which the fix may raise.

Based on the critical issue definition and the time when the issue is found, the Release Manager will decide how to proceed:

Wait for the next release for the issue to be included and fixed.
Include the fix in an ongoing release (cherry-pick).
Release a patch to fix the issue.

In each and every case, the following criteria must be met before the fix can be released:

The issue is properly documented on GitHub and labeled priority/critical.
It is a correction, not an enhancement.
It is a critical issue by definition.
Adequate tests have been implemented to prevent the issue from occurring again.
A fix is readily available and complies with the Definition of Done.

Release schedule

A scheduled release follows the planning cycles for the Kyma development that typically take four weeks. At the beginning of each planning cycle, the Product Lead communicates the specific timeframe for a release. After reaching the end of the development cycle, the Release Master creates a release candidate.

All planned releases happen at the end of a planned development cycle. Only the Product Lead and Release Manager can delay or resign from such a release if there are reasons that justify such a decision.

We do not schedule patches but rather provide them on demand if there is a need to address high priority issues before the next planned release. In this context, we explicitly define high priority issues as those which either affect multiple Kyma installations or result in the financial loss for the users. The Release Manager is involved in the decision to execute a patch release.

Nightly and weekly builds

It is our goal to have fully automated builds generated on a daily or nightly basis. Such builds can be made available to the community to validate the newest functionality and corrections implemented in development or testing environments, but are not intended for production use.

Once per week, we should update the installation with the newest, weekly build. During the following week, automated tests must run on the cluster to validate the stability and performance of the environment, and ensure that the results remain unchanged for a longer period of time. This also supports the whole release strategy by improving the quality of the product and increasing the public confidence in it.

Release candidates

After reaching the end of the development cycle, the Kyma developers create a release candidate, published just like any other release in GitHub. Anyone can test it and provide feedback. Depending on whether this is a minor or major release, the period of time for this validation varies:

two working days for minor releases
one week for major releases

All release candidates must always be fully tested by all defined automated and manual tests, following the test strategy. Manual tests are tracked in this sheet for each release candidate.

During the release process, only critical issues will be considered for cherry-picking. The Release Manager approves the final release to be prepared and published only when all tests have passed, and no critical issues are blocking the release.

If any issue is still blocking the release by the time it should be made available according to the release schedule, the Release Manager will decide to:

Remove or rollback the change causing the issue.
Wait for a fix that can lead to a delay in the release.
Skip the release completely.

The Release Manager communicates such a decision to the community on the public release Slack channel.

Planning start

The Product Lead coordinates the product planning, following the planning cycles for the Kyma development.

Planning end

After completing the planning process, the theme and expected scope of the release is clear. All maintainers know what they have to work on for the release.

At this time, the Release Manager publicly communicates the planned release schedule for Kyma. This communication should include key features or fixes expected to fall within the scope of the release. Possible communication channels include blog posts, social media, and Core SIG meetings.

Development start

After completing the planning, the Product Lead explicitly hands over the release to the engineering teams during the handover meeting. At this point, the development phase of the new release officially starts.

During the development phase, the Release Manager keeps track of the development and takes early actions required to keep the release on track. When closer to the release date, the Release Manager and the Product Lead discuss the status of the release and decide if there are any additional checkpoints or actions to address.

The Core SIG leaders include the external communication regarding the progress and status of each release to the agenda of the Core SIG bi-weekly meetings.

Development end

This is the last day of the period planned for the release development.

Release decision

The Release Manager and the Product Leadership decide to publish a release, based on its status, readiness and compliance of the developed functionality. If any issue blocks the release, the Release Manager actively discusses the issue, implications and risks with the relevant developer(s), Product Owner and Release Engineer to make a decision to release or not.

Release execution

The release execution is a manual process and is not part of an automated nightly or weekly build. It begins when the Release Engineer creates a release branch with all release artifacts. The process finishes after publishing both the artifacts and the related documentation as a release candidate that can be promoted to a final release.

During the execution of a release, all automated end-to-end test scenarios are validated. If any scenario fails, the release process automatically stops and no release is possible.

The Release Manager notifies the Kyma community on the public release Slack channel when a release candidate is available, at which time the community can validate it and provide their feedback. If there are release-related issues identified, the Release Manager decides if any of them is a blocker for the release.

TIP: See the Critical issues section for details.

Release publishing

The final release is available in the GitHub releases, including the installation instructions. It also includes the complete changelog that lists all pull requests merged into this release.

A Technical Writer publishes the blog post on the public Kyma website to announce the release. The post includes release notes that describe new features, as well as resolved and known issues.

You can learn about the new release from additional notifications published on social media, Slack channels, or announcements made during the Core SIG meetings. Alternatively, you can subscribe to GitHub notifications and receive updates each time a new release is available.

Release scope

For each of the planning periods described in the release schedule, GitHub epics and issues define and document the release scope with regards to functionality, corrections, and even non-functional requirements. The collection of all documented issues within a release represents the expected scope of that release that the whole organization and all teams define and commit to. The corresponding release in ZenHub identifies all issues and epics that fall within the final release scope.

When planning the release scope, all persons involved in the release must take the GDPR requirements into consideration when processing or storing personal data.

Version maintenance

There is no guaranteed support for the bug fixes in the previous Kyma versions. The default strategy is to upgrade to the latest version. The Kyma team encourages the community to always upgrade to the latest release.

Deprecation and backward-compatibility

There are clear expectations regarding the depreciation and backward-compatibility of Kyma versions to ensure some level of stability for the users. This can mean a period of time in which you should not change the provided functionality. This is the practice that other open-source projects also follow.

Kyma release process

This document describes how to create a Kyma release. Start from defining release jobs as described in the Preparation section. Then proceed to the Steps.

A Kyma release includes the following items:

Docker images for Kyma components
A GitHub release including release artifacts, such as source code and configuration
A Git tag
A release branch

Preparation

NOTE: This section applies only to new major and minor versions. If you release a patch, skip the preparation and go to the Steps section.

To prepare a release:

Create a release branch in the kyma repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.
Click to copy
```
 git fetch upstream
 git checkout -b release-{release_version} upstream/master
 git push upstream release-{release_version}
```
NOTE: If you don't create the Kyma release branch at this point and add a post-rel{release_version}-kyma-release-candidate post-submit job to the test-infra master, then pushing anything to the Kyma release branch, creating or rebasing the branch, triggers a new GitHub release.
Define new release jobs in the test-infra repository.

Define new release jobs

Go to the test-infra repository.
Open templates/config.yaml
Add the new release to global.releases. Remove the oldest release on the list.
Set global.nextRelease to the future release version.
Run dep ensure -v --vendor-only in the development/tools directory to install dependencies.
Run go run development/tools/cmd/rendertemplates/main.go --config templates/config.yaml in the root of the repository to generate jobs.
Run go test ./development/tools/jobs/... in the root of the repository. If any of the tests is marked red, fix it using these guidelines:

For release tests using GetKymaReleasesSince or jobsuite.Since with a release that is no longer supported, change the method to GetAllKymaReleases or jobsuite.AllReleases respectively.
For release tests using GetKymaReleasesUntil or jobsuite.Until with a release that is no longer supported, remove the part of the test which includes the method.

If tests are green, commit all jobs. The new release jobs are ready.

Steps

Follow these steps to release another Kyma version.

kyma-project/test-infra

Create a release branch in the test-infra repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.
NOTE: This step applies only to new major and minor versions.
Click to copy
```
git fetch upstream
git checkout -b {RELEASE_NAME} upstream/master
```
Ensure that the prow/RELEASE_VERSION file from the test-infra repository on a release branch contains the correct version to be created. The file should contain a release version following the {A}.{B}.{C} or {A}.{B}.{C}-rc{D} format, where A,B, C, and D are numbers. If you define a release candidate version, a pre-release is created.
Make sure the RELEASE_VERSION file includes just this single line:
Click to copy
```
echo -n {RELEASE_VERSION} > prow/RELEASE_VERSION
```
Push the branch to the test-infra repository.
Update the RELEASE_VERSION file with the name of the next minor release candidate and merge the pull request to master. For example, if the RELEASE_VERSION on the master branch is set to 1.4.2, then change the version to 1.5.0-rc1.

kyma-project/kyma

Inside the release branch do the following changes.
i. In installation/resources/installer.yaml replace eu.gcr.io/kyma-project/develop/installer:{image_tag} with eu.gcr.io/kyma-project/kyma-installer:{release_version}
ii. Find these lines in tools/kyma-installer/kyma.Dockerfile:
Click to copy
```
ARG INSTALLER_VERSION="{kyma_operator_version}"
ARG INSTALLER_DIR={kyma_operator_path}
FROM $INSTALLER_DIR/kyma-operator:$INSTALLER_VERSION
```
Replace them with:
Click to copy
```
FROM {kyma_operator_path}/kyma-operator:{kyma_operator_version}
```
iii. In the resources/core/values.yaml file, replace the clusterAssetGroupsVersion value with your release branch name. For example, for the 1.4 release, find the following section:
Click to copy
```
docs:
# (...) - truncated
clusterAssetGroupsVersion: master
```
And replace the clusterAssetGroupsVersion value with the following:
Click to copy
```
docs:
# (...)
clusterAssetGroupsVersion: release-1.4
```
iv. Create a pull request with your changes to the release branch.
If pre-release-pr-image-guard fails, ask the owners to change PR-XXX images of the components to the master version.
CAUTION: Never use /test all as it might run tests that you do not want to execute.
Execute remaining tests. There are dependencies between jobs, so follow the provided order of steps.
i. Run kyma-integration by adding the /test pre-rel{release_number}-kyma-integration comment to the PR.
NOTE: You don't have to wait until the pre-rel{release_number}-kyma-integration job finishes to proceed with further jobs.
ii. Run /test pre-rel{release_number}-kyma-installer and wait until it finishes.
iii. Run /test pre-rel{release_number}-kyma-artifacts and wait until it finishes.
iv. Run the following tests in parallel and wait for them to finish:
Click to copy
```
/test pre-rel{release_number}-kyma-gke-integration
/test pre-rel{release_number}-kyma-gke-central-connector
/test pre-rel{release_number}-kyma-gke-upgrade
/test pre-rel{release_number}-kyma-gke-backup
```
If you detect any problems with the release, such as failing tests, wait for the fix that can be either delivered on a PR or cherry-picked to the PR from the master branch. Prow triggers the jobs again. Rerun manual jobs as described in step 3.
After all checks pass, merge the PR, using the rebase and merge option.
CAUTION: By default, the rebase and merge option is disabled. Contact one of the kyma-project/kyma repository admins to enable it.
Merging the PR to the release branch runs the postsubmit jobs, which:
- create a GitHub release and trigger documentation update on the official Kyma website
- trigger provisioning of the cluster from the created release. The cluster name contains the release version with a period . replaced by a dash -. For example: gke-release-1-4-0-rc1. Use the cluster to test the release candidate.
CAUTION: The cluster is automatically generated for you, but removal is a manual action. The release cluster, the IP Addresses, and the DNS records must be deleted manually after tests on the given cluster are done.
If you don't have access to the GCP project, post a request in the Slack team channel.
Click to copy
```
gcloud container clusters get-credentials gke-release-1-4-0-rc2 --zone europe-west4-c --project sap-kyma-prow-workloads
```
Follow these instructions to give Kyma teams access to start testing the release candidate.
The Github release postsumbit job creates a tag in the kyma-project/kyma repository, which triggers the post-kyma-release-upgrade pipeline. The purpose of this job is to test upgradability between the previous Kyma release, i.e. the latest release that is not a release candidate, and the brand new release published by the release postsubmit job.
For example, if 1.7.0-rc2 is released, the pipeline will try to upgrade 1.6.0 to 1.7.0-rc2.
If you detect any problems with the upgrade, contact the teams responsible for failing components.
CAUTION: The job assumes no manual migration is involved. If the upgrade process requires any additional actions, the pipeline is likely to fail. In such case, the owners of the components concerned are responsible for running manual tests or modifying the pipeline.
Update the RELEASE_VERSION file to contain the next patch RC1 version on the release branch. Do it immediately after the release, otherwise, any PR to a release branch overrides the previously published Docker images.
For example, if the RELEASE_VERSION file on the release branch contains 1.4.1, change it to 1.4.2-rc1.
Validate the yaml and changelog files generated under releases.
Update the release content manually with links to:

Instructions on local Kyma installation
Instructions on cluster Kyma installation
Release notes
For installation instructions, use the links from the previous release and update the version number in URLs. If contributors want you to change something in the instructions, they would address you directly. Contact technical writers for the link to release notes.

Create a spreadsheet with all open issues labeled as test-missing. Every team assigned to an issue must cover the outstanding test with manual verification on every release candidate. After the test is finished successfully, the responsible team must mark it as completed in the spreadsheet. Every issue identified during testing must be reported. To make the testing easier, provision a publicly available cluster with the release candidate version after you complete all steps listed in this document.
Notify Team Breaking Pixels that the release is available for integration with Faros.

NOTE: After the Kyma release is complete, proceed with releasing Kyma CLI.

Kyma CLI release process

This document describes how to create a Kyma CLI release.

Release content

A Kyma CLI release consists of:

GitHub release with automated changelog generation
Artifacts, including cli binaries and source code archives
A release tag or branch holding the code

Steps

Make sure Kyma is released and create a release branch in the cli repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.
NOTE: This step applies only to new major and minor versions.
Click to copy
```
git fetch upstream
git checkout -b {RELEASE_NAME} upstream/master
git push upstream {RELEASE_NAME}
```
Ensure that the KYMA_VERSION variables in the Makefile and .goreleaser.yml files from the cli repository on the release branch contain the latest Kyma version.
Create a PR to cli/release-x.y which triggers the presubmit job for cli.
After merging the PR, create a tag on the release branch that has the same version name as Kyma. If you define a release candidate version, a pre-release is created.
Click to copy
```
git tag -a {RELEASE_VERSION} -m "Release {RELEASE_VERSION}"
git push upstream {RELEASE_VERSION}
```
where {RELEASE_VERSION} could be 1.4.0-rc1, 1.4.0-rc2 or 1.4.0.
Push the tag to trigger a postsubmit job that creates the GitHub release. Validate if the release is available under releases.
Once the release is complete, remove the existing stable tag and re-create it pointing to the latest commit on the master branch. Pushing this tag will trigger a postsubmit job that builds and pushes the latest stable binaries to a storage bucket. These are used by the continuous integration jobs.

Release communication

At the beginning of the release process, the team responsible for the release picks a Release Master who will manage the release process. One of the Release Master's responsibilities is to provide updates regarding the progress of the release process through the public release Slack channel. The Release Master is obliged to do the following:

Inform Kyma teams about release deadlines at least one week before the first release candidate (RC1), and pin the message to the channel so that it is easy to find.
Prepare the environment to test a release candidate. Share access to the testing environment with Kyma teams so that they can test their components before merging the RC.
Create an excel sheet where all the manual tests are defined and ensure that all tests are executed by responsible teams.
Inform Kyma teams about deadlines for cherry-picking changes to the release branch.
Inform Kyma teams about all the release-related pull requests that require approvals.
Notify Kyma teams about failing tests to fix them as quickly as possible. Kyma teams responsible for failing tests should communicate with a Release Master about the fixing progress.

Communication channels

The Release Manager must notify the Kyma team about the planned dates of creating both release candidates, as well as the date of the final release. Apart from that, the Release Master must inform about the progress of the release process. After the release is out, the Release Master sends notification about the new release on the public release Slack channel.

Communication persons

The following are contact persons responsible for the whole release process in Kyma:

Jose Cortina (Release Manager)

Release process improvement

To ensure continuous improvement of the release process, during or after each release the Release Master creates and describes Github issues regarding potential enhancements. These issues are tagged with the quality/release label and added to the Release Improvement Backlog.

After the release is complete, the team responsible for the release, the Release Manager and the previous Release Master meet to refine the Release Improvement Backlog. During the meeting, they discuss the issues in detail and estimate time needed for their implementation. Taking into account how much time during the following release process could be saved, the meeting participants make a decision of the issue(s) bringing the most value to be prioritized.

The team responsible for the release invests time to implement the prioritized release improvement. The implementation must be ready before the following release process starts.

Such an approach ensures the release process is better and better with every release. Also, both the responsibility for the release improvements and the workload are distributed among all Kyma teams instead of a single person, team or SIG.

Subscription to releases

GitHub offers an option to subscribe to email notifications on new releases in repositories. To learn how to watch the given repository only for the releases, read this official GitHub article.

Alternatively, use free online tools, such as CodeRelease.io, GitPunch, or releases, to get e-mail notifications when the GitHub projects release a new version in the GitHub repositories that you watched and starred.