Release
Overview
Release guidelines describe the release strategy and process in Kyma. This section also provides details on who to contact, how to communicate during the release process, and how to subscribe to email notifications on new releases in GitHub repositories.
Read about:
- The release strategy in Kyma
- How to create a release using Prow according to the release process
- Communication between the Release Master and the rest of the organization
- Subscription to releases
Release strategy
Read about the release strategy in Kyma.
Terminology
Release types
For release versioning, Kyma uses the approach similar to Semantic Versioning. However, while Semantic Versioning focuses on APIs, the release process for Kyma refers to the complete Kyma deliverable. Despite this difference, Kyma follows the same release types:
Type | Scope |
---|---|
MAJOR | Introduces breaking changes |
MINOR | Introduces a new functionality in a planned schedule |
PATCH hot.fix | Introduces a backward-compatible bug fix as an unscheduled release |
Function freeze
The function freeze refers to the point in time when a new release branch is created from the main
branch. After the function freeze, no new features are included in the release. This is typically on Monday at 9:00 AM CET/CEST every 4 weeks.
Roles
These persons actively participate in release preparation and execution:
Product Lead ensures that all capabilities provide a valuable and consistent product when it comes to its features and usability.
Release Manager is responsible for the overall release:
- Ensures that all persons involved in the release follow the required processes and standards
- Decides when to create a new release candidate
- Makes the final call for publishing the release candidate
- Creates and manages the release schedule and release plan
Product Manager coordinates the process of gathering the input for the release notes.
Technical Writer writes and publishes the release notes as a blog post on the
kyma-project.io
website, based on the input from Product Owners.Developer ensures that the release branch contains all required features and fixes.
Release Master prepares all mandatory release artifacts and conducts technical release activities described in the release process.
Release Publisher prepares and publishes the social media content based on the release notes.
Release requirements and acceptance criteria
Epics and user stories
The functional implementations that are included in the release are documented as GitHub issues. A contributor completes an issue bound by the license agreement and contributing guidelines. All proposed changes must pass the obligatory Continuous Integration checks and require the maintainers' review and approval.
You can close an epic or an issue if the new functionality meets the contributing rules. These include, but are not limited to, the following points:
- Automated unit, integration and end-to-end tests verify this functionality.
- This functionality is documented.
Test coverage
In Kyma, we follow the test strategy according to which you can verify all functionalities through automated testing that you execute either through the CI or the release pipeline.
The Kyma community must provide the highest possible test coverage to minimize any potential risks and ensure the release is functional and stable.
Consider implementing tests that go beyond verifying the functionality of the system. This includes integration, performance, and end-to-end tests.
Compliance rules
All Kyma organization members involved in the release process comply with contribution rules defined for Kyma and ensure that all checks built into the continuous integration process pass. These requirements represent the absolute minimal acceptance criteria needed for a release from the compliance perspective.
Additionally, manual pre-release verification must cover these areas:
- the use of the third-party software and components that require licensing
- the use and storage of personal data subject to the GDPR requirements
Security
Open-source projects, like any other, must ensure secure development. You can provide it by constantly raising the awareness of everyone involved in the project. In Kyma, we strive to make all developers aware of the secure development requirement. For example, we conduct threat modeling workshops during which the participants proactively think about possible security threats in the architecture, infrastructure, and implementation.
The Release Manager in Kyma takes care of formal security validation activities performed before major releases. The results of these activities influence the release decision. Lack of attention to security topics can result in release delay.
Critical issues
Any issues affecting the expected functionality of Kyma should be documented on GitHub for review and planning. If any of these issues is defined as critical according to the criteria specified in this section, it should be brought to the immediate attention of the Release Manager.
Critical issues are the ones that:
- Render a Kyma cluster unstable or unusable and have no workaround.
- Cause loss or corruption of data.
- Expose critical security vulnerabilities (CVSS 3 Score 9.0 - 10.0).
- Cause tests to fail in any supported environment.
- Cause a major impact on any production environment.
Critical issues have to be fixed with priority. They should contain information about the timeline for the fix and the risks which the fix may raise.
Based on the critical issue definition and the time when the issue is found, the Release Manager will decide how to proceed:
- Wait for the next release for the issue to be included and fixed.
- Include the fix in an ongoing release (cherry-pick).
- Release a patch to fix the issue.
In each and every case, the following criteria must be met before the fix can be released:
- The issue is properly documented on GitHub and labeled
priority/critical
. - It is a correction, not an enhancement.
- It is a critical issue by definition.
- Adequate tests have been implemented to prevent the issue from occurring again.
- A fix is readily available and complies with the Definition of Done.
Release schedule
A scheduled release follows the planning cycles for the Kyma development that typically take four weeks. At the beginning of each planning cycle, the Product Lead communicates the specific timeframe for a release. After reaching the end of the development cycle, the Release Master creates a release candidate.
All planned releases happen at the end of a planned development cycle. Only the Product Lead and Release Manager can delay or resign from such a release if there are reasons that justify such a decision.
We do not schedule patches but rather provide them on demand if there is a need to address high priority issues before the next planned release. In this context, we explicitly define high priority issues as those which either affect multiple Kyma installations or result in the financial loss for the users. The Release Manager is involved in the decision to execute a patch release.
Nightly and weekly builds
It is our goal to have fully automated builds generated on a daily or nightly basis. Such builds can be made available to the community to validate the newest functionality and corrections implemented in development or testing environments, but are not intended for production use.
Once per week, we should update the installation with the newest, weekly build. During the following week, automated tests must run on the cluster to validate the stability and performance of the environment, and ensure that the results remain unchanged for a longer period of time. This also supports the whole release strategy by improving the quality of the product and increasing the public confidence in it.
Release candidates
After reaching the end of the development cycle, the Kyma developers create a release candidate, published just like any other release in GitHub. Anyone can test it and provide feedback. Depending on whether this is a minor or major release, the period of time for this validation varies:
- two working days for minor releases
- one week for major releases
All release candidates must always be fully tested by all defined automated and manual tests, following the test strategy. Manual tests are tracked in this sheet for each release candidate.
During the release process, only critical issues will be considered for cherry-picking. The Release Manager approves the final release to be prepared and published only when all tests have passed, and no critical issues are blocking the release.
If any issue is still blocking the release by the time it should be made available according to the release schedule, the Release Manager will decide to:
- Remove or rollback the change causing the issue.
- Wait for a fix that can lead to a delay in the release.
- Skip the release completely.
The Release Manager communicates such a decision to the community on the public release Slack channel.
Planning start
The Product Lead coordinates the product planning, following the planning cycles for the Kyma development.
Planning end
After completing the planning process, the theme and expected scope of the release is clear. All maintainers know what they have to work on for the release.
At this time, the Release Manager publicly communicates the planned release schedule for Kyma. This communication should include key features or fixes expected to fall within the scope of the release. Possible communication channels include blog posts, social media, and Core SIG meetings.
Development start
After completing the planning, the Product Lead explicitly hands over the release to the engineering teams during the handover meeting. At this point, the development phase of the new release officially starts.
During the development phase, the Release Manager keeps track of the development and takes early actions required to keep the release on track. When closer to the release date, the Release Manager and the Product Lead discuss the status of the release and decide if there are any additional checkpoints or actions to address.
The Core SIG leaders include the external communication regarding the progress and status of each release to the agenda of the Core SIG bi-weekly meetings.
Development end
This is the last day of the period planned for the release development.
Release decision
The Release Manager and the Product Leadership decide to publish a release, based on its status, readiness and compliance of the developed functionality. If any issue blocks the release, the Release Manager actively discusses the issue, implications and risks with the relevant developer(s), Product Owner and Release Engineer to make a decision to release or not.
Release execution
The release execution is a manual process and is not part of an automated nightly or weekly build. It begins when the Release Engineer creates a release branch with all release artifacts. The process finishes after publishing both the artifacts and the related documentation as a release candidate that can be promoted to a final release.
During the execution of a release, all automated end-to-end test scenarios are validated. If any scenario fails, the release process automatically stops and no release is possible.
The Release Manager notifies the Kyma community on the public release Slack channel when a release candidate is available, at which time the community can validate it and provide their feedback. If there are release-related issues identified, the Release Manager decides if any of them is a blocker for the release.
TIP: See the Critical issues section for details.
Release publishing
The final release is available in the GitHub releases, including the installation instructions. It also includes the complete changelog that lists all pull requests merged into this release.
A Technical Writer publishes the blog post on the public Kyma website to announce the release. The post includes release notes that describe new features, as well as resolved and known issues.
You can learn about the new release from additional notifications published on social media, Slack channels, or announcements made during the Core SIG meetings. Alternatively, you can subscribe to GitHub notifications and receive updates each time a new release is available.
Release scope
For each of the planning periods described in the release schedule, GitHub epics and issues define and document the release scope with regards to functionality, corrections, and even non-functional requirements. The collection of all documented issues within a release represents the expected scope of that release that the whole organization and all teams define and commit to. The corresponding release in ZenHub identifies all issues and epics that fall within the final release scope.
When planning the release scope, all persons involved in the release must take the GDPR requirements into consideration when processing or storing personal data.
Version maintenance
There is no guaranteed support for the bug fixes in the previous Kyma versions. The default strategy is to upgrade to the latest version. The Kyma team encourages the community to always upgrade to the latest release.
Deprecation and backward-compatibility
There are clear expectations regarding the depreciation and backward-compatibility of Kyma versions to ensure some level of stability for the users. This can mean a period of time in which you should not change the provided functionality. This is the practice that other open-source projects also follow.
Kyma release process
This document describes how to create a Kyma release. Start from defining release jobs as described in the Preparation section. Then proceed to the Steps.
A Kyma release includes the following items:
- Docker images for Kyma components
- A GitHub release including release artifacts, such as source code and configuration
- A Git tag
- A release branch
Definitions
The table below includes placeholders used throughout this document. When executing the commands, replace each of them with a suitable release number or version.
Placeholders | Description | Pattern | Example |
---|---|---|---|
RELEASE | Release number | {major}.{minor} | 1.13 |
RELEASE_VERSION | Release version | {major}.{minor}.{patch} or {major}.{minor}.{patch}-rc{candidate} | 1.13.0 or 1.13.0-rc1 |
RELEASE_VERSION_SHORT | Release version without additional characters | {major}{minor} | 113 |
RELEASE_VERSION_DASH | Release version with dashes | {major}-{minor}-{patch} or {major}-{minor}-{patch}-rc{candidate} | 1-13-0 or 1-13-0-rc1 |
Preparation
NOTE: This section applies only to new major and minor versions. If you release a patch, skip the preparation and go to the Steps section.
kyma-project/kyma
Perform initial checks
Check if the main
branch contains any PR-images:
git grep -e 'version:\s.*[Pp][Rr]-.*' -e 'image:.*:[Pp][Rr]-.*' -e 'tag:\s.*[Pp][Rr]-.*' --before-context=2 resources tests
Ask the teams for fixes if this command returns any output.
Create a release branch
NOTE: a release branch needs to be created per new major / minor version. Patch releases and release candidates do not have a dedicated release branch.
Create a release branch in the kyma
repository. The name of this branch should follow the release-{major}.{minor}
pattern, such as release-1.4
.
```bashgit fetch upstreamgit checkout --no-track -b release-{RELEASE} upstream/maingit push -u upstream release-{RELEASE}```
kyma-project/test-infra
Update the jobs on the main branch
Create a PR to
main
containing the following changes to create the new job definitions:- Open
templates/config.yaml
- Add the new release to
global.releases
. Remove the oldest release on the list. - Set
global.nextRelease
to the future release version. - Run
make
in the root of the repository to generate jobs and run tests. If any of the tests is marked red, fix it using these guidelines:
- For release tests using
GetKymaReleasesSince
orjobsuite.Since
with a release that is no longer supported, change the method toGetAllKymaReleases
orjobsuite.AllReleases
respectively. - For release tests using
GetKymaReleasesUntil
orjobsuite.Until
with a release that is no longer supported, remove the part of the test which includes the method.
- If tests are green, commit all jobs.
- Open
Once the PR is merged to
main
, you can proceed.
Create a release branch
NOTE: a release branch needs to be created per new major / minor version. Patch releases and release candidates do not have a dedicated release branch. If this branch already exists this step will be skipped.
Create a release branch in the test-infra
repository
```bashgit fetch upstreamgit checkout --no-track -b release-{RELEASE} upstream/maingit push -u upstream release-{RELEASE}```
Steps
Follow these steps to release another Kyma version. Execute these steps for every patch release or release candidate.
kyma-project/test-infra
Ensure that the prow/RELEASE_VERSION
file from the test-infra
repository on a release branch contains the correct version to be created. If you define a release candidate version, a pre-release is created.
Make sure the
prow/RELEASE_VERSION
file includes just a single line, without the newline character at the end:Click to copyecho -n {RELEASE_VERSION} > prow/RELEASE_VERSIONIf you had to change the RELEASE_VERSION, create a PR to update it on the release branch.
- Once this PR is merged you can proceed.
kyma-project/kyma
Create a PR to the release branch
Inside the release branch do the following changes.
In
installation/resources/installer.yaml
replaceeu.gcr.io/kyma-project/develop/installer:{image_tag}
witheu.gcr.io/kyma-project/kyma-installer:{RELEASE_VERSION}
In the
resources/core/values.yaml
file, findclusterAssetGroupsVersion
.Click to copydocs:# (...) - truncatedclusterAssetGroupsVersion: mainAnd replace the
clusterAssetGroupsVersion
value with the following:Click to copydocs:# (...)clusterAssetGroupsVersion: release-{RELEASE}
Create a pull request with your changes to the release branch.
If
pre-release-pr-image-guard
fails, ask the owners to change PR-XXX images of the components to themain
version.- If the checks are green, merge the PR and proceed to the next step.
Development process towards the release
NOTE: Every developer who is introducing changes to the specific version can perform steps 1-4.
Create a feature-branch based on the given
release-{RELEASE}
branch you want to extend. Add your changes and create a Pull Request.Once you create a Pull Request to the release branch, the set of checks is triggered. These jobs run in the same way as jobs that run on every Pull Request to the
main
branch. If you create a Pull Request that contains changes to the components, the component-building job is triggered. If you make any changes in the charts, the integration tests are triggered.If you detect any problems with your PR, fix the issues until your checks pass.
After all checks pass, merge your PR to the release branch. Merging the PR triggers the post-submit integration tests automatically. The jobs' status will be visible on the Kyma TestGrid in the corresponding dashboard tab.
If there's a need for additional changes in the release branch during the development process, open a new PR to the release branch. Repeat steps 1-4 for this PR.
Create a release
- Once the release process is finished and the release branch is complete, create a new tag in the repository that points to your release branch. To create a tag, run this command:
CAUTION: Make sure you are working on the most up-to-date
release-{RELEASE}
branch for a given release.
git tag -a {RELEASE_VERSION} -m "Release {RELEASE_VERSION}"git push upstream {RELEASE_VERSION}
The tag must have the same name as in the RELEASE_VERSION
file. Creating a new tag triggers the following actions:
- Create a GitHub release and trigger documentation update on the official Kyma website.
- Create a new release cluster for the given Kyma
RELEASE_VERSION
. If you don't have access to the GCP project, post a request in the Slack team channel.CAUTION: The cluster is automatically generated for you, and it is automatically removed after 7 days.
The Github release post-submit job creates a release in the
kyma-project/kyma
repository, which triggers thepost-rel{RELEASE_VERSION_SHORT}-kyma-release-upgrade
pipeline. The purpose of this job is to test upgradability between the latest Kyma release that is not a release candidate and the brand new release published by the release post-submit job. For example, if1.7.0-rc2
is released, the pipeline will try to upgrade1.6.0
to1.7.0-rc2
.If you detect any problems with the upgrade, contact the teams responsible for failing components.
CAUTION: The job assumes no manual migration is involved. If the upgrade process requires any additional actions, the pipeline is likely to fail. In such case, the owners of the components concerned are responsible for running manual tests or modifying the pipeline.
On the release branch, update the
RELEASE_VERSION
file located in theprow
folder of thekyma-project/test-infra
repository. It must contain the next release candidate version. Do it immediately after the release, otherwise, any PR to a release branch overrides the previously published Docker images.For example, if the
RELEASE_VERSION
file on the release branch contains1.4.1
, change it to1.4.2-rc1
.Validate the
yaml
and changelog files generated under releases.Update the release content manually with links to:
- Instructions on local Kyma installation
- Instructions on cluster Kyma installation
- Release notes
For installation instructions, use the links from the previous release and update the version number in URLs. If contributors want you to change something in the instructions, they would address you directly. Contact technical writers for the link to release notes.
NOTE: After the Kyma release is complete, proceed with releasing Kyma CLI.
Post-release tasks
Ask the Huskies team to upgrade the Kyma Helm chart on AppHub.
Click to copy> **NOTE:** Because of a limitation on the AppHub side, only a few people are allowed to create such a PR, which currently includes the members of the Huskies team.Update
prow/RELEASE_VERSION
in themain
branch of thetest-infra
repository with the name of the next minor release candidate, and merge the pull request tomain
. For example, if theRELEASE_VERSION
on themain
branch is set to1.4.2
, change the version to1.5.0-rc1
.
Kyma CLI release process
This document describes how to create a Kyma CLI release.
Release content
A Kyma CLI release consists of:
- GitHub release with automated changelog generation
- Artifacts, including
cli
binaries and source code archives - A release tag or branch holding the code
Steps
Make sure Kyma is released and create a release branch in the
cli
repository. The name of this branch should follow therelease-x.y
pattern, such asrelease-1.4
.NOTE: This step applies only to new major and minor versions.
Click to copygit fetch upstreamgit checkout --no-track -b {RELEASE_NAME} upstream/maingit push upstream {RELEASE_NAME}Ensure that the
KYMA_VERSION
variables in theMakefile
and.goreleaser.yml
files from thecli
repository on the release branch contain the latest Kyma version.Create a PR to
cli/release-x.y
which triggers the presubmit job forcli
.After merging the PR, create a tag on the release branch that has the same version name as Kyma. If you define a release candidate version, a pre-release is created.
Click to copygit tag -a {RELEASE_VERSION} -m "Release {RELEASE_VERSION}"where {RELEASE_VERSION} could be
1.4.0-rc1
,1.4.0-rc2
or1.4.0
.Create the new version
Push the tag to trigger a postsubmit job that creates the GitHub release. Validate if the release is available under releases.
Click to copygit push upstream {RELEASE_VERSION}Verify the Prow Status of the matching revision ({RELEASE_VERSION}).
If the post submit job failed, you can re-trigger it by removing the tag from upstream and pushing it again.
Click to copygit push --delete upstream {RELEASE_VERSION}git push upstream {RELEASE_VERSION}
Release communication
At the beginning of the release process, the team responsible for the release picks a Release Master who will manage the release process. One of the Release Master's responsibilities is to provide updates regarding the progress of the release process through the public release Slack channel. The Release Master is obliged to do the following:
Inform Kyma teams about release deadlines at least one week before the first release candidate (RC1), and pin the message to the channel so that it is easy to find.
Prepare the environment to test a release candidate. Share access to the testing environment with Kyma teams so that they can test their components before merging the RC.
Create an excel sheet where all the manual tests are defined and ensure that all tests are executed by responsible teams.
Inform Kyma teams about deadlines for cherry-picking changes to the release branch.
Inform Kyma teams about all the release-related pull requests that require approvals.
Notify Kyma teams about failing tests to fix them as quickly as possible. Kyma teams responsible for failing tests should communicate with a Release Master about the fixing progress.
Communication channels
The Release Manager must notify the Kyma team about the planned dates of creating both release candidates, as well as the date of the final release. Apart from that, the Release Master must inform about the progress of the release process. After the release is out, the Release Master sends notification about the new release on the public release Slack channel.
Communication persons
The following are contact persons responsible for the whole release process in Kyma:
- Jose Cortina (Release Manager)
Release process improvement
To ensure continuous improvement of the release process, during or after each release the Release Master creates and describes Github issues regarding potential enhancements. These issues are tagged with the quality/release
label and added to the Release Improvement Backlog.
After the release is complete, the team responsible for the release, the Release Manager and the previous Release Master meet to refine the Release Improvement Backlog. During the meeting, they discuss the issues in detail and estimate time needed for their implementation. Taking into account how much time during the following release process could be saved, the meeting participants make a decision of the issue(s) bringing the most value to be prioritized.
The team responsible for the release invests time to implement the prioritized release improvement. The implementation must be ready before the following release process starts.
Such an approach ensures the release process is better and better with every release. Also, both the responsibility for the release improvements and the workload are distributed among all Kyma teams instead of a single person, team or SIG.
Subscription to releases
GitHub offers an option to subscribe to email notifications on new releases in repositories. To learn how to watch the given repository only for the releases, read the GitHub article.
Alternatively, use free online tools, such as CodeRelease.io, GitPunch, or releases, to get e-mail notifications when the GitHub projects release a new version in the GitHub repositories that you watched and starred.