Release

Overview

Release guidelines describe the release strategy and process in Kyma. This section also provides details on who to contact, how to communicate during the release process, and how to subscribe to email notifications on new releases in GitHub repositories.

Read about:

Release strategy

Read about the release strategy in Kyma.

Terminology

Release types

For release versioning, Kyma uses the approach similar to Semantic Versioning. However, while Semantic Versioning focuses on APIs, the release process for Kyma refers to the complete Kyma deliverable. Despite this difference, Kyma follows the same release types:

TypeScope
MAJORIntroduces breaking changes
MINORIntroduces a new functionality in a planned schedule
PATCH hot.fixIntroduces a backward-compatible bug fix as an unscheduled release

NOTE: Major version zero refers to software that is under constant, intensive development. Such software is not production-ready and can include breaking changes.

Function freeze

The function freeze term refers to the point in time after which you cannot merge new features to the release branch.

Roles

These persons actively participate in release preparation and execution:

  • Head of Capabilities ensures that all capabilities provide a valuable and consistent product when it comes to its features and usability.

  • Release Manager is responsible for the overall release:

    • Ensures that all persons involved in the release follow the required processes and standards
    • Decides when to create a new release candidate
    • Makes the final call for publishing the release candidate
    • Creates and manages the release schedule and release plan

  • Product Manager coordinates the process of gathering the input for the release notes.

  • Technical Writer writes and publishes the release notes as a blog post on the kyma-project.io website, based on the input from Product Owners.

  • Developer ensures that the release branch contains all required features and fixes.

  • Release Engineer prepares all mandatory release artifacts and conducts technical release activities described in the release process.

  • Release Publisher prepares and publishes the social media content based on the release notes.

Release requirements and acceptance criteria

Epics and user stories

The functional implementations that are included in the release are documented as GitHub issues. A contributor completes an issue bound by the license agreement and contributing guidelines. All proposed changes must pass the obligatory Continuous Integration checks and require the maintainers' review and approval.

You can close an epic or an issue if the new functionality meets the contributing rules. These include, but are not limited to, the following points:

  • Automated unit and end-to-end tests verify this functionality.
  • This functionality is documented.

Test coverage

In Kyma, we follow the test strategy according to which you can verify all functionalities through automated testing that you execute either through the CI or the release pipeline.

The Kyma community must provide the highest possible test coverage to minimize any potential risks and ensure the release is functional and stable.

Whenever possible, consider implementing tests that go beyond verifying the functionality of the system. This includes integration, performance, and end-to-end tests.

Compliance rules

All Kyma organization members involved in the release process comply with contribution rules defined for Kyma and ensure that all checks built into the continuous integration process pass. These requirements represent the absolute minimal acceptance criteria needed for a release from a compliance perspective. These rules can change as Kyma approaches the 1.0 release.

Additionally, manual pre-release verification must cover these areas:

  • the use of the third-party software and components that require licensing
  • the use and storage of personal data subject to the GDPR requirements

Security

Open-source projects, like any other, must ensure secure development. You can provide it by constantly raising the awareness of everyone involved in the project. In Kyma, we strive to make all developers aware of the secure development requirement. For example, we conduct threat modeling workshops during which the participants proactively think about possible security threats in the architecture, infrastructure, and implementation.

The Release Manager in Kyma takes care of formal security validation activities performed before major releases. The results of these activities influence the release decision. Lack of attention to security topics can result in release delay.

Release schedule

A scheduled release follows the planning cycles for the Kyma development that typically take four weeks. At the beginning of each planning cycle, the Head of Capabilities communicates the specific time plan for a release. After reaching the end of the development cycle, Kyma developers create a release candidate which in most cases is identical to the release published on GitHub.

All planned releases happen at the end of a planned development cycle. Only the Product Lead and Release Manager can delay or resign from such a release if there are reasons that justify such a decision.

We do not schedule patches but rather provide them on-demand if there is a need to address high priority issues before the next planned release. In this context, we explicitly define high priority issues as those which either affect multiple Kyma installations or result in the financial loss for the users. The Release Manager is involved in the decision to execute a patch release. However, we do not expect to have patch releases before the official Kyma 1.0 version.

Nightly and weekly builds

It is our goal to have fully automated builds generated on a daily or nightly basis. Such builds can be made available to the community to validate the newest functionality and corrections implemented in development or testing environments, but are not intended for production use.

Once per week, we should update the installation with the newest, weekly build. During the following week, automated tests must run on the cluster to validate the stability and performance of the environment, and ensure that the results remain unchanged for a longer period of time. This also supports the whole release strategy by improving the quality of the product and increasing the public confidence in it.

Release candidates

After reaching the end of the development cycle, the Kyma developers create a release candidate, published just like any other release in GitHub. Anyone can test it and provide feedback. Depending on whether this is a minor or major release, the period of time for this validation varies:

  • two working days for minor releases
  • one week for major releases

After this time, and ideally with no issues identified, we can promote the release candidate or, depending on the technical implementation, rebuild it and use it as the final release.

Planning start

The Head of Capabilities coordinates the product planning, following the planning cycles for the Kyma development.

Planning end

After completing the planning process, the theme and expected scope of the release is clear. All maintainers know what they have to work on for the release.

At this time, the Release Manager publicly communicates the planned release schedule for Kyma. This communication should include key features or fixes expected to fall within the scope of the release. Possible communication channels include blog posts, social media, and Core SIG meetings.

Development start

After completing the planning, the Head of Capabilities explicitly hands over the release to the engineering teams during the handover meeting. At this point, the development phase of the new release officially starts.

During the development phase, the Release Manager keeps track of the development with the help of the Scrum of Scrums (SoS) meetings held twice a week. This allows the Release Manager to take early actions required to keep the release on track. When closer to the release date, the Release Manager, the Head of Capabilities, and the Lead Product Engineer discuss the status of the release and decide if there are any additional checkpoints or actions to address.

The Core SIG leaders include the external communication regarding the progress and status of each release to the agenda of the Core SIG bi-weekly meetings. The Release Manager, or a person appointed as a stand-in, presents the release status during those meetings.

Development end

This is the last day of the period planned for the release development.

Release decision

The Release Manager and the Product Leadership decide to publish the release, based on the status and readiness of the developed functionality. The Release Manager communicates this decision before the release execution starts.

Release execution

The release execution is a manual process and is not part of an automated nightly or weekly build. It begins when the Release Engineer creates a release branch with all release artifacts. The process finishes after publishing both the artifacts and the related documentation as a release candidate that can be promoted to a final release.

During the execution of a release, all automated end-to-end test scenarios are validated. If any scenario fails, the release process automatically stops and no release is possible.

The Release Manager notifies the Kyma community on a public Slack channel that the release candidate is available, requesting the community to validate it and provide their short-term feedback. The deadline for the feedback depends on the release type. Ideally, the final release is the same as the last release candidate, avoiding any additional changes. If there are release-related issues identified, the Release Manager decides if they are blockers for the release.

Release publishing

The final release is available in the GitHub releases, including the installation instructions. It also includes the complete changelog that lists all pull requests merged into this release.

A Technical Writer publishes the blog post on the public Kyma website to announce the release. The post includes release notes that describe new features, as well as resolved and known issues.

You can learn about the new release from additional notifications published on social media, Slack channels, or announcements made during the Core SIG meetings. Alternatively, you can subscribe to GitHub notifications and receive updates each time a new release is available.

Release scope

For each of the planning periods described in the release schedule, GitHub epics and issues define and document the release scope with regards to functionality, corrections, and even non-functional requirements. The collection of all documented issues within a release represents the expected scope of that release that the whole organization and all teams define and commit to. The corresponding release in ZenHub identifies all issues and epics that fall within the final release scope.

When planning the release scope, all persons involved in the release must take the GDPR requirements into consideration when processing or storing personal data.

Version maintenance

There is no guaranteed support for the bug fixes in the previous Kyma versions. The default strategy is to upgrade to the latest version.

The Kyma project does not plan to provide any patch releases before the 1.0 production release and encourages the community to always upgrade to the latest release.

Deprecation and backward-compatibility

The 1.0 release and further release versions impose clear expectations regarding the depreciation and backward-compatibility of Kyma versions to ensure some level of stability for the users. This can mean a period of time in which you should not change the provided functionality. This is the practice that other open-source projects also follow.

Release process

This document describes how to create Kyma and Kyma CLI releases.

Kyma

Release contents

A Kyma release includes the following items:

  • Docker images for Kyma components
  • A GitHub release including release artifacts, such as source code and configuration
  • A Git tag
  • A release branch

Release process

TIP: Start a few days before creating the first release candidate to prepare the test-infra and Kyma PRs.

These are the steps you must follow to create the first release candidate.

  1. Create the Kyma release branch.
  2. In the test-infra repository, prepare the PR to the master branch.
  3. In the test-infra repository, prepare the PR to the release branch.
  4. Rebase the Kyma release branch to add all RC1 changes.
  5. Merge the PRs created in the test-infra repository.
  6. In the kyma repository, prepare and merge the PR to the release branch.

Create Kyma release branch

NOTE: This section applies only to new major and minor versions. It does not apply to release candidates, such as RC2.

Create a release branch in the kyma repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.

Click to copy
git fetch upstream
git checkout -b release-{release_version} upstream/master
git push upstream release-{release_version}

NOTE: If you don't create the kyma release branch now and add a post-submit job post-rel{release_version}-kyma-release-candidate to test-infra master, then pushing anything to the kyma release branch (includes creation of the branch), or if you rebase the branch, a new Github release will be created.

test-infra repository

Steps for new major/minor version

NOTE: This section applies only to new major and minor versions. It does not apply to release candidates, such as RC2.

Define release jobs on the master branch in the test-infra repository. To ensure every job name is unique, prefix it with pre-rel{versionNumber}. Remember to provide the version number without any periods. For example, to find all jobs for the 1.4 release, look for job names with the pre-rel14 prefix. To learn how to define a release job for a component, read the following document.

  1. Navigate to the test-infra repository.
  2. Define release jobs in the prow/jobs/test-infra directory in the watch-pods.yaml file.

  3. Define release jobs in the prow/jobs/kyma directory in the following files:

    • every .yaml in components
    • every .yaml in tests
    • every .yaml in tools/ except for the tools/failery/failery.yaml file
    • kyma-integration.yaml
    • kyma-artifacts.yaml
    • kyma-installer.yaml
    • kyma-release-candidate.yaml

    • kyma-github-release.yaml For this file also do the following steps:

      Click to copy
      		  - Modify the `presets` array. This is an example of the 1.4 release `preset`:
      	
      			```yaml
      			- labels:
      			    preset-target-commit-1.4: "true"
      			    env:
      			    - name: RELEASE_TARGET_COMMIT
      			        value: release-1.4
      			```
      		  -  Remember to also set the `preset-target-commit-1.4` label for the `post-rel14-kyma-github-release` job. This is an example of the 1.4 release `postsubmit` job:
      Click to copy
      postsubmits: 
        kyma-project/kyma:
        - name: post-rel14-kyma-github-release
          branches:
          - release-1.4
          <<: *job_template
          extra_refs:
          - <<: *test_infra_ref
            base_ref: release-1.4
          labels:
            <<: *job_labels_template
            preset-build-release: "true"
            preset-target-commit-1.4: "true"

  4. Ensure that tests for the release jobs exist. Release tests usually iterate through all release versions and run tests for them. See the TestBucReleases test defined in development/tools/jobs/kyma/service_binding_usage_controller_test.go as a reference.

  5. Update the GetAllKymaReleaseBranches() function defined in the development/tools/jobs/tester/tester.go file under the test-infra repository.

  6. Define branch protection rules for the release branch in the prow/config.yaml file. For example, see the release-1.4 definition:

    Click to copy
    release-1.4:
      protect: true
      required_status_checks:
        contexts:
          - pre-rel14-kyma-integration
          - pre-rel14-kyma-gke-integration
          - pre-rel14-kyma-gke-upgrade
          - pre-rel14-kyma-gke-central-connector
          - pre-rel14-kyma-artifacts
          - pre-rel14-kyma-installer
          - pre-rel14-kyma-gke-minio-gateway

  7. Remove the now unsupported release in tester.go and all references of it. E.g Release12 SupportedRelease = "release-1.2" when releasing 1.5.

  8. Remove previous release jobs.

    After adding new release jobs, remove the old ones. Remember to leave jobs for three latest releases. For example, during the preparation for the 1.4 release, add pre-rel14 jobs and remove all pre-rel11 jobs. Make sure that the only defined jobs are those with pre-rel12, pre-rel13, and pre-rel14 prefixes.

  1. Merge the PR to the current master branch.

NOTE: Before a release, make sure there is no mismatch between source code and .yaml files. All components are rebuilt from source code, which requires all Helm charts to be up to date.

Create release branch

  1. Create a release branch in the test-infra repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.

    Click to copy
    git fetch upstream
    git checkout -b {RELEASE_NAME} upstream/master

Steps for new release candidate

NOTE: This section applies to new release candidates, such as RC1, RC2, and the final release version.

  1. Ensure that the prow/RELEASE_VERSION file from the test-infra repository on a release branch contains the correct version to be created. The file should contain a release version following the {A}.{B}.{C} or {A}.{B}.{C}-rc{D} format, where A,B, C, and D are numbers. If you define a release candidate version, a pre-release is created.

    Make sure the RELEASE_VERSION file includes just this single line:

    Click to copy
    echo -n {RELEASE_VERSION} > prow/RELEASE_VERSION

  2. Push the branch to the test-infra repository.

  3. Create a PR to test-infra/release-x.y. This triggers the pre-release job for watch-pods.

    NOTE: To trigger the watch-pods build without introducing any changes, edit any file within the test-infra repository and create a pull request. You don't need to merge it as the job is triggered anyway. After a successful watch-pods image build, close the pull request.

  4. Update the RELEASE_VERSION file with the name of the next minor release candidate and merge the pull request to master. For example, if the RELEASE_VERSION on the master branch is set to 1.4.2, then change the version to 1.5.0-rc1.

kyma repository

Steps for new major/minor version

In the resources/core/values.yaml file, replace the clusterDocsTopicsVersion value with your release branch name. For example, for the 1.4.1 release, find the following section:

Click to copy
docs:
    # (...) - truncated
    clusterDocsTopicsVersion: master

And replace the clusterDocsTopicsVersion value with the following:

Click to copy
docs:
    # (...)
    clusterDocsTopicsVersion: release-1.4

Steps for new release candidate

NOTE: This section applies to new release candidates such as RC1, RC2, and the final release version.

  1. Inside the release branch do the following changes.

    1. Update your PR with the version and the directory of components used in values.yaml files.

      Find these values in the files:

      Click to copy
      dir: develop/
      version: {current_version}
      Click to copy
      dir: pr/
      version: {current_version}

      Replace them with:

      Click to copy
      dir:
      version: {release_version}

      NOTE: Replace only develop/ so develop/tests becomes tests/.

      Every component image is published with a version defined in the RELEASE_VERSION file stored in the test-infra repository on the given release branch. Test scripts for integration jobs like GKE Integration or GKE Upgrade are also loaded from the test-infra release branch.

      For example, for the first release candidate of 1.1.0, the release version will be 1.1.0-rc1 and the yaml files should be modified as follows:

      Click to copy
      dir:
      version: 1.1.0-rc1

      CAUTION: Do not update the version of components whose dir section does not contain develop, as is the case with Console-related components. Also do not change octopus version in kyma/resources/testing/values.yaml even though the directory is develop .

    2. Check all yaml files in the kyma repository for references of the following Docker image:

      Click to copy
      image: eu.gcr.io/kyma-project/develop/{IMAGE_NAME}:{SOME_SHA}

      Change the Docker image to:

      Click to copy
      image: eu.gcr.io/kyma-project/{IMAGE_NAME}:{release_version}

      CAUTION: In installation/resources/installer.yaml replace eu.gcr.io/kyma-project/develop/installer:{image_tag} with eu.gcr.io/kyma-project/kyma-installer:{release_version}

    3. In the resources/core/values.yaml file, replace the clusterDocsTopicsVersion value with your release branch name. For example, for the 0.9.1 release, find the following section:

      Click to copy
      docs:
          # (...) - truncated
          clusterDocsTopicsVersion: master

      And replace the clusterDocsTopicsVersion value with the following:

      Click to copy
      docs:
          # (...)
          clusterDocsTopicsVersion: release-1.4

    4. Ensure that in the resources/compass/values.yaml there are no PR-XXX values. All image versions should be in a form of commit hashes.

    5. Create a pull request with your changes to the release branch. It triggers all jobs for components.

      PullRequest

  2. If any job fails, trigger it again by adding the following comment to the PR:

    Click to copy
    /test {job_name}

    CAUTION: Never use /test all as it might run tests that you do not want to execute.

  3. Wait until all jobs for components and tools finish.

  4. Execute remaining tests. The diagram shows you the jobs and dependencies between them. JobDependencies

    1. Run kyma-integration by adding the /test pre-{release_number}-kyma-integration comment to the PR.

      NOTE: You don't have to wait until the pre-{release_number}-kyma-integration job finishes to proceed with further jobs.

    2. Run /test pre-{release_number}-kyma-installer and wait until it finishes.

    3. Run /test pre-{release_number}-kyma-artifacts and wait until it finishes.

    4. Run the following tests in parallel:

      Click to copy
      /test pre-rel14-kyma-gke-integration
      /test pre-rel14-kyma-gke-upgrade
      /test pre-rel14-kyma-gke-backup

    5. Wait for the jobs to finish:

      • pre-{release_number}-kyma-integration
      • pre-{release_number}-kyma-gke-integration
      • pre-{release_number}-kyma-gke-upgrade
      • pre-{release_number}-kyma-gke-backup

  5. If you detect any problems with the release, such as failing tests, wait for the fix that can be delivered either on a PR or cherry-picked to the PR from the master branch.
    Prow triggers the jobs again. Rerun manual jobs as described in step 5.

  6. After all checks pass, merge the PR, using the rebase and merge option. To merge the PR to the release branch, you must receive approvals from all teams.

    CAUTION: By default, the rebase and merge option is disabled. Contact one of the kyma-project/kyma repository admins to enable it.

  7. Merging the PR to the release branch runs the postsubmit jobs, which: - create a GitHub release and trigger documentation update on the official Kyma website - trigger provisioning of the cluster from the created release The cluster name contains the release version with a period . replaced by a dash -. For example: gke-release-1-1-0-rc1. Use the cluster to test the release candidate.

    Click to copy
    	> **CAUTION**: The cluster is automatically generated for you, so you don't need to create it. The release cluster, the IP Addresses, and the DNS records must be deleted manually after tests on the RC2 cluster are done.

    • Get access to the release cluster. If you don't have access to the GCP project, post a request in the Slack team channel.

    Click to copy
    gcloud container clusters get-credentials gke-release-1-4-0-rc2 --zone europe-west4-c --project sap-hybris-sf-playground

    • Follow these instructions to give Kyma teams access to start testing the release candidate.

  8. Update the RELEASE_VERSION file to contain the next patch RC1 version on the release branch. Do it immediately after the release, otherwise, any PR to a release branch overrides the previously published Docker images.

    For example, if the RELEASE_VERSION file on the release branch contains 1.4.1, change it to 1.4.2-rc1.

  9. Validate the yaml and changelog files generated under releases.

  10. Update the release content manually with links to the instruction on how to install the latest Kyma release.
    Currently, this means to grab the links from the previous release and update the version number in URLs. If contributors want you to change something in the instruction, they would address you directly.

  11. Create a spreadsheet with all open issues labeled as test-missing. Every team assigned to an issue must cover the outstanding test with manual verification on every release candidate. After the test is finished successfully, the responsible team must mark it as completed in the spreadsheet. Every issue identified during testing must be reported. To make the testing easier, provision a publicly available cluster with the release candidate version after you complete all steps listed in this document.

  12. Notify Team Breaking Pixels that the release is available for integration with Faros.

CLI

Release contents

A Kyma CLI release consists of:

  • A release tag or branch holding the code.
  • Artifacts, including cli binaries and source code archives.
  • GitHub release with automated changelog generation.

In the release process, the current code is pinned in a release branch first. Then the Kyma version is updated to test the integration with the latest Kyma version. Finally, Kyma CLI is released via a git tag which triggers a GitHub release.

Steps for new major/minor version

NOTE: This section applies only to new major and minor versions. It does not apply to release candidates, such as RC2.

After making sure that Kyma is released, create a release branch in the cli repository. The name of this branch should follow the release-x.y pattern, such as release-1.4.

Click to copy
git fetch upstream
git checkout -b {RELEASE_NAME} upstream/master
git push upstream {RELEASE_NAME}

Steps for new release candidate

NOTE: This section applies only to new release candidates, such as RC1, RC2, and the final release version.

  1. Ensure that the KYMA_VERSION variables on Makefile and .goreleaser.yml file from the cli repository on the release branch contains the latest Kyma version that you just released.

  2. Create a PR to cli/release-x.y. This triggers the presubmit job for cli.

    NOTE: A CLI release requires Kyma to be released first. Otherwise integration jobs will fail. However anything can be prepared. Just do not merge the PR until Kyma is relesed. Then do a retest and merge the PR.

  3. After merging the PR, create a tag on the release branch that has the same version name as Kyma. If you define a release candidate version, a pre-release is created.

    Click to copy
    git tag -a {RELEASE_VERSION} -m "Release {RELEASE_VERSION}"
    git push upstream {RELEASE_VERSION}

    where {RELEASE_VERSION} could be e.g. 1.4.0-rc1, 1.4.0-rc2 or 1.4.0.

  4. Pushing the tag triggers the postsubmit job that creates the GitHub release. Validate if the release is available under releases.

Release process improvement

To ensure continuous improvement of the release process, during or after each release the Release Master creates and describes Github issues regarding potential enhancements. These issues are tagged with the quality/release label and added to the Release Improvement Backlog.

After the release is complete, the team responsible for the release, the Release Manager and the previous Release Master meet to refine the Release Improvement Backlog. During the meeting, they discuss the issues in detail and estimate time needed for their implementation. Taking into account how much time during the following release process could be saved, they make a decision of the issue(s) bringing the most value to be prioritized.

The team invests time to implement the prioritized release improvement. The implementation must be ready before the following release process starts.

Such an approach ensures the release process is better and better with every release. Also, both the responsibility for the release improvements and the workload are distributed among all Kyma teams instead of a single person, team or SIG.

Release communication

At the beginning of the release process, the team responsible for the release picks a Release Master who will manage the release process. One of the Release Master's responsibilities is to update the rest of the organization on the progress of the release process. The Release Master is obliged to do the following:

  • Inform Kyma teams about release deadlines on Slack channels at least one week before the first release candidate (RC1), and pin the message so that it is easy to find.

  • Prepare the environment to test a release candidate. Share access to the testing environment with Kyma teams so that they can test their components before merging the RC.

  • Create an excel sheet where all the manual tests are defined and ensure that all tests are executed by responsible teams.

  • Inform Kyma teams about deadlines for cherry-picking changes to the release branch.

  • Inform Kyma teams about all the release-related pull requests that require approvals.

  • Notify Kyma teams about failing tests to fix them as quickly as possible. Kyma teams responsible for failing tests should communicate with a Release Master about the fixing progress.

Communication channels

The Kyma team must be notified about the planned dates of both release candidates, as well as the date of the final release. Apart from that, the Release Master must inform about any occurrences regarding the release process. After the release is out, the Release Master sends notification about the new release on the external Kyma community Slack channel.

Communication persons

The following are contact persons responsible for the whole release process in Kyma:

Scrum of Scrums meetings

A Release Master should cooperate with Scrum Masters regarding the release process communication. Either the Release Master or a Scrum Master dedicated to the releasing team must inform other Scrum Masters on the release progress during the biweekly Scrum of Scrums (SoS) meetings.

A dedicated Scrum Master is also responsible for updating the SoS MagicBox under the Wiki page for a given release. The MagicBox must contain all the information and deadlines that affect all Kyma teams.

Subscription to releases

GitHub offers an option to subscribe to email notifications on new releases in repositories. To learn how to watch the given repository only for the releases, read this official GitHub article.

Alternatively, use free online tools, such as CodeRelease.io, GitPunch, or releases, to get e-mail notifications when the GitHub projects release a new version in the GitHub repositories that you watched and starred.