Kyma Environment Broker

Overview

Kyma Environment Broker is a component that allows you to provision Kyma as a Runtime on clusters provided by third-party providers. It uses the Runtime Provisioner's API to install Kyma on a given cluster.

Architecture

Overview

The diagram and steps describe the Kyma Environment Broker (KEB) workflow and the roles of specific components in this process:

KEB diagram

  1. The user sends a request to create a new cluster with Kyma Runtime.

    a. The user sends a request to KEB through Istio VirtualService.

    b. Istio redirects the request to the Ory Oathkeeper, which authorizes the request.

    c. If the authorization ends with success, the request is redirected to KEB.

  2. KEB proxies the request to create a new cluster to the Runtime Provisioner component.

  3. Provisioner registers a new cluster in the Director component.

  4. Provisioner creates a new cluster and installs Kyma Runtime.

  5. When the operation is successful, KEB keeps sending a request for a Dashboard URL to Director:

    a. KEB sends a request to Hydra to refresh the OAuth token, if needed.

    b. KEB passes the OAuth token to Director through Gateway.

    c. Director returns the Dashboard URL to KEB through Gateway. The Dashboard URL is the URL to the newly created cluster.

KEB Endpoints

Kyma Environment Broker (KEB) implements the Open Service Broker API (OSB API). All the OSB API endpoints are served with the following prefixes:

PrefixDescription
/oauthDefines a prefix for the endpoint secured with the OAuth2 authorization. EDP is configured with a region whose default value is specified under the broker.defaultRequestRegion parameter in the values.yaml file.
/oauth/{region}Defines a prefix for the endpoint secured with the OAuth2 authorization. EDP is configured with the region value specified in the request.

NOTE: KEB does not implement the OSB API update operation.

Besides OSB API endpoints, KEB exposes the REST /info/runtimes endpoint that provides information about all created Runtimes, both succeeded and failed. This endpoint is secured with the OAuth2 authorization.

Details

Service description

Kyma Environment Broker (KEB) is compatible with the Open Service Broker API (OSBA) specification. It provides a ServiceClass that provisions Kyma Runtime on a cluster.

Service plans

The supported plans are as follows:

Plan nameDescription
azureInstalls Kyma Runtime on the Azure cluster.
azure_liteInstalls Kyma Lite on the Azure cluster.
gcpInstalls Kyma Runtime on the GCP cluster.
trialInstalls Kyma Trial on Azure or GCP.

Provisioning parameters

There are two types of configurable provisioning parameters: the ones that are compliant for all providers and provider-specific ones.

Parameters compliant for all providers

These are the provisioning parameters that you can configure:

Parameter nameTypeDescriptionRequiredDefault value
namestringSpecifies the name of the cluster.YesNone
nodeCountintSpecifies the number of Nodes in a cluster.No3
componentsarrayDefines optional components that are installed in a Kyma Runtime. The possible values are kiali and tracing.No[]
kymaVersionstringProvides a Kyma version on demand.NoNone

Provider-specific parameters

These are the provisioning parameters for Azure that you can configure:

  • Azure
  • Azure Lite

These are the provisioning parameters for GCP that you can configure:

  • GCP

Trial plan

Trial plan allows you to install Kyma either on Azure or GCP. The Trial plan assumptions are as follows:

  • Kyma is uninstalled after 30 days and the Kyma cluster is deprovisioned after this time.
  • It's possible to provision only one Kyma Runtime per global account.

To reduce the costs, the Trial plan skips some of the provisioning steps. These are the steps that are skipped during the Trial plan provisioning:

  • Create_LMS_Tenant
  • Provision_Azure_Event_Hubs
  • Request_LMS_Certificates
  • AVS External Evaluation (part of the post actions during the Initialisation step)

Provisioning parameters

These are the provisioning parameters for the Trial plan that you can configure:

  • Trial plan

Runtime components

Kyma Environment Broker (KEB) serves the functionality of composing the list of components that are installed in a Runtime. The diagram and steps describe the KEB workflow in terms of calculating and processing Runtime components:

runtime-components-architecture

  1. During KEB initialization, the broker reads two files that contain lists of components to be installed in a Runtime:

    • kyma-installer-cluster.yaml file with the given Kyma version
    • managed-offering-components.yaml file with additional managed components that are added at the end of the base components list
  2. The user provisions a Runtime and selects optional components that they want to install.

  3. KEB composes the final list of components by removing components that were not selected by the user. It also adds the proper global and components overrides and sends the whole provisioning information to the Runtime Provisioner.

There is a defined list of the component names. Use these names in your implementation.

Disabled components

To disable a component for a specific plan, add it to the disabled components list. To disable a component for all plans, add its name under the AllPlansSelector parameter.

Optional components

An optional component is a component that is disabled by default but can be enabled in the provisioning request. Currently, the optional components are:

  • Kiali
  • Tracing

Add the optional component

If you want to add the optional component, you can do it in two ways.

  • If disabling a given component only means to remove it from the installation list, use the generic disabler:
Click to copy
runtime.NewGenericComponentDisabler("component-name", "component-namespace")
  • If disabling a given component requires more complex logic, create a new file called internal/runtime/{compoent-name}_disabler.go and implement a service which fulfills the following interface:
Click to copy
// OptionalComponentDisabler disables component form the given list and returns a modified list
type OptionalComponentDisabler interface {
Disable(components internal.ComponentConfigurationInputList) internal.ComponentConfigurationInputList

NOTE: Check the CustomDisablerExample as an example of custom service for disabling components.

In each method, the framework injects the components parameter which is a list of components that are sent to the Runtime Provisioner. The implemented method is responsible for disabling component and as a result, returns a modified list.

This interface allows you to easily register the disabler in the cmd/broker/main.go file by adding a new entry in the optionalComponentsDisablers list:

Click to copy
// Register disabler. Convention:
// {component-name} : {component-disabler-service}
//
// Using map is intentional - we ensure that component name is not duplicated.
optionalComponentsDisablers := runtime.ComponentsDisablers{
"Kiali": runtime.NewGenericComponentDisabler(components.Kiali),
"Tracing": runtime.NewGenericComponentDisabler(components.Tracing),
}

Remove the optional component

If you want to remove the option to disable components and make them required during Kyma installation, remove a given entry from the optionalComponentsDisablers list in the cmd/broker/main.go file.

Runtime operations

Kyma Environment Broker allows you to configure operations that you can run on a Runtime. Each operation consists of several steps and each step is represented by a separate file. As every step can be re-launched multiple times, for each step, you should determine a behavior in case of a processing failure. It can either:

  • Return an error, which interrupts the entire process, or
  • Repeat the entire operation after the specified period.

NOTE: It's important to set lower timeouts for the Kyma installation in the Runtime Provisioner.

Provisioning

Each provisioning step is responsible for a separate part of preparing Runtime parameters. For example, in a step you can provide tokens, credentials, or URLs to integrate Kyma Runtime with external systems. All data collected in provisioning steps are used in the step called create_runtime which transforms the data into a request input. The request is sent to the Runtime Provisioner component which provisions a Runtime. The provisioning process contains the following steps:

NameDomainDescriptionOwner
InitializationProvisioningStarts the provisioning process and asks the Director for the Dashboard URL if the provisioning in Gardener is finished.@jasiu001 (Team Gopher)
Resolve_Target_SecretHyperscaler Account PoolProvides the name of a Gardener Secret that contains Hypescaler account credentials used during cluster provisioning.@koala7659 (Team Framefrog)
AVS_Configuration_StepAvSSets up external and internal monitoring of Kyma Runtime.@jasiu001 (Team Gopher)
Create_LMS_TenantLMSRequests a tenant in the LMS system or provides a tenant ID if it was created before.@piotrmiskiewicz (Team Gopher)
IAS_RegistrationIdentity Authentication ServiceRegisters a new ServiceProvider on IAS, generates client ID and Secret, and inserts them to Grafana overrides. This step is not required and can be disabled.@jasiu001 (Team Gopher)
EDP_RegistrationEvent Data PlatformRegisters an SKR on Event Data Platform with the necessary parameters. This step is not required and can be disabled.@jasiu001 (Team Gopher)
Provision Azure Event HubsEvent HubCreates the Azure Event Hub Namespace which is a managed Kafka cluster for a Kyma Runtime.@anishj0shi (Team SkyDivingTunas)
Overrides_From_Secrets_And_Config_StepKyma overridesConfigures default overrides for Kyma.@jasiu001 (Team Gopher)
ServiceManagerOverridesService ManagerConfigures overrides with Service Manager credentials.@mszostok (Team Gopher)
Request_LMS_CertificatesLMSChecks if the LMS tenant is ready and requests certificates. The step configures Fluent Bit in a Kyma Runtime. It requires the Create_LMS_Tenant step to be completed beforehand. The step does not fail the provisioning operation.@piotrmiskiewicz (Team Gopher)
Create_RuntimeProvisioningTriggers provisioning of a Runtime in the Runtime Provisioner.@jasiu001 (Team Gopher)

NOTE: The timeout for processing this operation is set to 24h.

Deprovisioning

Each deprovisioning step is responsible for a separate part of cleaning Runtime dependencies. To properly deprovision all Runtime dependencies, you need the data used during the Runtime provisioning. You can fetch this data from the ProvisioningOperation struct in the initialization step.

Any deprovisioning step shouldn't block the entire deprovisioning operation. Use the RetryOperationWithoutFail function from the DeprovisionOperationManager struct to skip your step in case of retry timeout. Set at most 5min timeout for retries in your step.

The deprovisioning process contains the following steps:

NameDomainStatusDescriptionOwner
Deprovision_InitializationDeprovisioningDoneInitializes the DeprovisioningOperation instance with data fetched from the ProvisioningOperation.@polskikiel (Team Gopher)
Deprovision Azure Event HubsEvent HubIn progressDeletes the Azure Event Hub Namespace.@montaro (Team SkyDivingTunas)
De-provision_AVS_EvaluationsAvSDoneRemoves external and internal monitoring of Kyma Runtime.@jasiu001 (Team Gopher)
IAS_DeregistrationIdentity Authentication ServiceDoneRemoves the ServiceProvider from IAS.@jasiu001 (Team Gopher)
EDP_DeregistrationEvent Data PlatformDoneRemoves all entries about SKR from Event Data Platform.@jasiu001 (Team Gopher)
Remove_RuntimeDeprovisioningDoneTriggers deprovisioning of a Runtime in the Runtime Provisioner.@polskikiel (Team Gopher)

NOTE: The timeout for processing this operation is set to 24h.

Upgrade

Each upgrade step is responsible for a separate part of upgrading Runtime dependencies. To properly upgrade the Runtime, you need the data used during the Runtime provisioning. You can fetch this data from the ProvisioningOperation struct in the initialization step.

The upgrade process contains the following steps:

NameDomainStatusDescriptionOwner
Upgrade_Kyma_InitialisationUpgradeDoneInitializes the UpgradeOperation instance with data fetched from the ProvisioningOperation.@ksputo (Team Gopher)
Overrides_From_Secrets_And_Config_StepUpgradeDoneBuilds an input configuration that is passed as overrides to Runtime Provisioner.@ksputo (Team Gopher)
Upgrade_RuntimeUpgradeDoneTriggers the upgrade of a Runtime in Runtime Provisioner.@ksputo (Team Gopher)

NOTE: The timeout for processing this operation is set to 3h.

Provide additional steps

You can configure Runtime operations by providing additional steps. To add a new step, follow these tutorials:

  • Provisioning
  • Deprovisioning
  • Upgrade

Hyperscaler Account Pool

To provision clusters through Gardener using the Runtime Provisioner, the Kyma Environment Broker (KEB) requires a hyperscaler (GCP, Azure, AWS, etc.) account/subscription. Managing the available hyperscaler accounts is not in the scope of the KEB. Instead, the available accounts are handled by the Hyperscaler Account Pool (HAP).

The HAP stores credentials for the hyperscaler accounts that have been set up in advance in Kubernetes Secrets. The credentials are stored separately for each provider and tenant. The content of the credentials Secrets may vary for different use cases. The Secrets are labeled with the hyperscaler-type and tenant-name labels to manage pools of credentials for use by the provisioning process. This way, the in-use credentials and unassigned credentials available for use are tracked. Only the hyperscaler-type label is added during Secret creation, and the tenant-name label is added when the account respective for a given Secret is claimed. The content of the Secrets is opaque to the HAP.

The Secrets are stored in a Gardener seed cluster pointed to by the HAP. They are available within a given Gardener project specified in the KEB and Runtime Provisioner configuration. This configuration uses a kubeconfig that gives the KEB and the Runtime Provisioner access to a specific Gardener seed cluster, which, in turn, enables access to those Secrets.

This diagram shows the HAP workflow:

hap-workflow

Before a new cluster is provisioned, the KEB queries for a Secret based on the tenant-name and hyperscaler-type labels. If a Secret is found, the KEB uses the credentials stored in this Secret. If a matching Secret is not found, the KEB queries again for an unassigned Secret for a given hyperscaler and adds the tenant-name label to claim the account and use the credentials for provisioning.

One tenant can use only one account per given hyperscaler type.

This is an example of a Kubernetes Secret that stores hyperscaler credentials:

Click to copy
apiVersion: v1
kind: Secret
metadata:
name: {SECRET_NAME}
labels:
# tenant-name is omitted for new, not yet claimed account credentials
tenant-name: {TENANT_NAME}
hyperscaler-type: {HYPERSCALER_TYPE}

Shared credentials

For a certain type of Runtimes, KEB can use the same credentials for multiple tenants. In such a case, the Secret with credentials must be labeled differently by adding the shared label set to true. Shared credentials will not be assigned to any tenant.

This is an example of a Kubernetes Secret that stores shared credentials:

Click to copy
apiVersion: v1
kind: Secret
metadata:
name: {SECRET_NAME}
labels:
hyperscaler-type: {HYPERSCALER_TYPE}
shared: "true"

Authorization

Kyma Environment Broker provides OAuth2 authorization. For this purpose, Kyma Environment Broker uses the ApiRule custom resource which generates a VirtualService and uses Oathkeeper Access Rules to allow or deny access. To authorize with the Kyma Environment Broker, use an OAuth2 client registered through the Hydra Maester controller.

To access the Kyma Environment Broker endpoints, use the /oauth prefix before OSB API paths. For example:

Click to copy
/oauth/{region}/v2/catalog

You must also specify the Authorization: Bearer request header:

Click to copy
Authorization: Bearer {ACCESS_TOKEN}

Get the access token

Follow these steps to obtain a new access token:

  1. Export these values as environment variables:
  • The name of your client and the Secret which stores the client credentials:

    Click to copy
    export CLIENT_NAME={YOUR_CLIENT_NAME}
  • The Namespace in which you want to create the client and the Secret that stores its credentials:

    Click to copy
    export CLIENT_NAMESPACE={YOUR_CLIENT_NAMESPACE}
  • The domain of your cluster:

    Click to copy
    export DOMAIN={CLUSTER_DOMAIN}
  1. Create an OAuth2 client:
Click to copy
cat <<EOF | kubectl apply -f -
apiVersion: hydra.ory.sh/v1alpha1
kind: OAuth2Client
metadata:
name: $CLIENT_NAME
namespace: $CLIENT_NAMESPACE
spec:
grantTypes:
- "client_credentials"
scope: "$SCOPE"
secretName: $CLIENT_NAME
EOF

NOTE: The valid scopes are broker:write, broker-upgrade:read, broker-upgrade:write, and cld:read.

  1. Export the credentials of the created client as environment variables. Run:
Click to copy
export CLIENT_ID="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_id}' | base64 --decode)"
export CLIENT_SECRET="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_secret}' | base64 --decode)"
  1. Encode your client credentials and export them as an environment variable:
Click to copy
export ENCODED_CREDENTIALS=$(echo -n "$CLIENT_ID:$CLIENT_SECRET" | base64)
  1. Get the access token:
Click to copy
curl -ik -X POST "https://oauth2.$DOMAIN/oauth2/token" -H "Authorization: Basic $ENCODED_CREDENTIALS" -F "grant_type=client_credentials" -F "scope=broker:write"

Set overrides for Kyma Runtime

You can set overrides to customize your Kyma Runtime. To provision a cluster with custom overrides, add a Secret or a ConfigMap with a specific label. Kyma Environment Broker uses this Secret and/or ConfigMap to prepare a request to the Runtime Provisioner.

Overrides can be either global or specified for a given component. In the second case, use the component: {"COMPONENT_NAME"} label to indicate the component. Create all overrides in the kcp-system Namespace.

See the examples:

  • ConfigMap with global overrides:

    Click to copy
    apiVersion: v1
    kind: ConfigMap
    metadata:
    labels:
    provisioning-runtime-override: "true"
    name: global-overrides
    namespace: kcp-system
    data:
    global.disableLegacyConnectivity: "true"
  • Secret with overrides for the core component:

    Click to copy
    apiVersion: v1
    kind: Secret
    metadata:
    labels:
    component: "core"
    provisioning-runtime-override: "true"
    name: core-overrides
    namespace: kcp-system
    data:
    database.password: YWRtaW4xMjMK

Disable overrides for specific plans

ConfigMaps and Secrets overrides work for all plans, however, you can disable overrides for specific lite plans, such as AzureLite. To disable a specific override for a lite plan, use the default-for-lite: "true" label.

See the example:

Click to copy
apiVersion: v1
kind: ConfigMap
metadata:
labels:
provisioning-runtime-override: "true"
default-for-lite: "true"
name: global-overrides
namespace: compass-system
data:
global.disableLegacyConnectivity: "true"

This ConfigMap activates a global override for all plans except SKRs provisioned with a special plan marked as lite.

Environments Cleanup

Environments Cleanup is an application that cleans up environments that do not meet requirements in a given Gardener project.

Prerequisites

Environments Cleanup requires access to:

  • Gardener project of choice to filter Shoots without a proper label
  • Database to get an Instance ID for each Runtime marked for deletion
  • Kyma Environment Broker to trigger Runtime deprovisioning

Configuration

The Environments Cleanup binary allows you to override some configuration parameters. You can specify the following environment variables:

Environment variableDescriptionDefault value
APP_MAX_AGE_HOURSDefines the maximum time a Shoot can live without deletion in case the label is not specified. The Shoot age is provided in hours.24h
APP_LABEL_SELECTORDefines the label selector to filter out Shoots for deletion.owner.do-not-delete!=true
APP_GARDENER_PROJECTSpecifies the name of a Gardener project.kyma-dev
APP_GARDENER_KUBECONFIG_PATHSpecifies the kubeconfig path to a Gardener cluster./gardener/kubeconfig/kubeconfig
APP_DATABASE_USERSpecifies the username for the database.postgres
APP_DATABASE_PASSWORDSpecifies the user password for the database.password
APP_DATABASE_HOSTSpecifies the host of the database.localhost
APP_DATABASE_PORTSpecifies the port for the database.5432
APP_DATABASE_NAMESpecifies the name of the database.provisioner
APP_DATABASE_SSL_MODEActivates the SSL mode for PostgrSQL. See all the possible values here.disable
APP_BROKER_URLSpecifies the Kyma Environment Broker URL.https://kyma-env-broker.kyma.local
APP_BROKER_TOKEN_URLSpecifies the Kyma Environment Broker OAuth token endpoint.https://oauth.2kyma.local/oauth2/token
APP_BROKER_CLIENT_IDSpecifies the username for the OAuth2 authentication in KEB.None
APP_BROKER_CLIENT_SECRETSpecifies the password for the OAuth2 authentication in KEB.None
APP_BROKER_SCOPESpecifies the scope for the OAuth2 authentication in KEB.None

Configure Kyma version

Kyma Environment Broker is configured with a default Kyma version specified in the APP_KYMA_VERSION environment variable. This means that each Kyma Runtime provisioned by Kyma Environment Broker in a given global account is installed in the default Kyma version. You can also specify a different Kyma version for a global account using a ConfigMap. See the example:

Click to copy
apiVersion: v1
kind: ConfigMap
metadata:
name: kyma-versions
namespace: "kcp-system"
data:
3e64ebae-38b5-46a0-b1ed-9ccee153a0ae: "1.15.0-rc1"

This ConfigMap contains a default version for a global account. The 3e64ebae-38b5-46a0-b1ed-9ccee153a0ae parameter is a global account ID, and the value is the Kyma version specified for this global account.

You can also specify a Kyma version using the kymaVersion provisioning parameter, for example:

Click to copy
export VERSION=1.15
curl --request PUT "https://$BROKER_URL/oauth/v2/service_instances/$INSTANCE_ID?accepts_incomplete=true" \
--header 'X-Broker-API-Version: 2.14' \
--header 'Content-Type: application/json' \
--header "$AUTHORIZATION_HEADER" \
--header 'Content-Type: application/json' \
--data-raw "{
\"service_id\": \"47c9dcbf-ff30-448e-ab36-d3bad66ba281\",
\"plan_id\": \"4deee563-e5ec-4731-b9b1-53b42d855f0c\",
\"context\": {
\"globalaccount_id\": \"$GLOBAL_ACCOUNT_ID\"
},
\"parameters\": {
\"name\": \"$NAME\",
\"kymaVersion\": \"$VERSION"
}
}"

The kymaVersion provisioning parameter overrides the default settings. To enable this feature, set the APP_ENABLE_ON_DEMAND_VERSION environment variable to true.

Subaccount Cleanup CronJob

Each SKR instance in Kyma Environment Broker (KEB) database belongs to a global account and to a subaccount. Subaccount Cleanup is an application that periodically calls the CIS service and notifies about SUBACCOUNT_DELETE events. Based on these events, Subaccount Cleanup triggers the deprovisioning action on the SKR instance to which a given subaccount belongs.

Details

The Subaccount Cleanup workflow is divided into several steps:

  1. Fetch SUBACCOUNT_DELETE events from the CIS service.

    a. CIS client makes a call to the CIS service and as a response, it gets a list of events divided into pages.

    b. CIS client fetches the rest of the events by making a call to each page one by one.

    c. A subaccount ID is taken from each event and kept in an array.

    d. When the CIS client ends its workflow, it displays logs with information on how many subaccounts were fetched.

  2. Find all instances in the KEB database based on the fetched subaccount IDs. The subaccounts pool is divided into pieces. For each piece, a query is made to the database to fetch instances.

  3. Trigger the deprovisioning operation for each instance found in step 2.

    Logs inform about the status of each triggered action:

    Click to copy
    deprovisioning for instance <InstanceID> (SubAccountID: <SubAccountID>) was triggered, operation: <OperationID>

    Subaccount Cleanup also uses logs to inform about the end of the deprovisioning operation.

Prerequisites

Subaccount Cleanup requires access to:

  • CIS service to receive all SUBACCOUNT_DELETE events
  • Database to get the instance ID for each subaccount ID from the SUBACCOUNT_DELETE event
  • Kyma Environment Broker to trigger SKR instance deprovisioning

Configuration

Use the following environment variables to configure the application:

Environment variableDescription
APP_CLIENT_VERSIONSpecifies the service version from which events are fetched. The possible values are v1.0 or v2.0.
APP_CIS_CLIENT_IDSpecifies the client ID for the OAuth2 authentication in CIS.
APP_CIS_CLIENT_SECRETSpecifies the client secret for the OAuth2 authentication in CIS.
APP_CIS_AUTH_URLSpecifies the endpoint for the CIS OAuth token.
APP_CIS_EVENT_SERVICE_URLSpecifies the endpoint for CIS events.
APP_DATABASE_USERSpecifies the username for the database.
APP_DATABASE_PASSWORDSpecifies the user password for the database.
APP_DATABASE_HOSTSpecifies the host of the database.
APP_DATABASE_PORTSpecifies the port for the database.
APP_DATABASE_NAMESpecifies the name of the database.
APP_DATABASE_SSL_MODEActivates the SSL mode for PostgreSQL. For reference, see the list of all the possible values.
APP_BROKER_URLSpecifies the Kyma Environment Broker URL.
APP_BROKER_TOKEN_URLSpecifies the endpoint for Kyma Environment Broker OAuth token.
APP_BROKER_CLIENT_IDSpecifies the username for the OAuth2 authentication in KEB.
APP_BROKER_CLIENT_SECRETSpecifies the password for the OAuth2 authentication in KEB.
APP_BROKER_SCOPESpecifies the scope of the OAuth2 authentication in KEB.

Orchestration

Orchestration is a mechanism that allows you to upgrade Kyma Runtimes. To create an orchestration, follow this tutorial. After sending the request, the orchestration is processed by KymaUpgradeManager. It lists Shoots (Kyma Runtimes) in the Gardener cluster and narrows them to the IDs that you have specified in the request body. Then, KymaUpgradeManager performs the upgrade steps logic on the selected Runtimes.

If Kyma Environment Broker is restarted, it reprocesses the orchestration with the IN PROGRESS state.

NOTE: You need a token with the broker-upgrade:write authorization scope to create an orchestration, and a token with the broker-upgrade:read scope to fetch the orchestrations.

Orchestration API consist of the following handlers:

  • GET /orchestrations - exposes data about all orchestrations.
  • GET /orchestrations/{orchestration_id} - exposes data about a single orchestration status.
  • GET /orchestrations/{orchestration_id}/operations - exposes data about operations scheduled by the orchestration with a given ID.
  • GET /orchestrations/{orchestration_id}/operations/{operation_id} - exposes the detailed data about a single operation with a given ID.
  • POST /upgrade/kyma - schedules the orchestration. It requires specifying a request body.

For more details about the API, check the Swagger schema.

Strategies

To change the behavior of the orchestration, you can specify a strategy in the request body. For now, there is only one parallel strategy with two types of schedule:

  • Immediate - schedules the upgrade operations instantly.
  • MaintenanceWindow - schedules the upgrade operations with the maintenance time windows specified for a given Runtime.

You can also configure how many upgrade operations can be executed in parallel to accelerate the process. Specify the parallel object in the request body with workers field set to the number of concurrent executions for the upgrade operations.

The example strategy configuration looks as follows:

Click to copy
{
"strategy": {
"type": "parallel",
"schedule": "maintenanceWindow",
"parallel": {
"workers": 5
}
}
}

Tutorials

Provision Kyma Runtime using KEB

This tutorial shows how to provision Kyma Runtime on Azure using Kyma Environment Broker.

Prerequisites

  • Compass with:
    • Runtime Provisioner configured for Azure provider
    • Kyma Environment Broker configured and chosen overrides set up

Steps

  1. Export these values as environment variables:

    Click to copy
    export BROKER_URL={KYMA_ENVIRONMENT_BROKER_URL}
    export INSTANCE_ID={INSTANCE_ID}
    export GLOBAL_ACCOUNT_ID={GLOBAL_ACCOUNT_ID}
    export NAME={RUNTIME_NAME}

    NOTE: INSTANCE_ID and NAME must be unique. It is recommended to use UUID as an INSTANCE_ID.

  2. Get the access token. Export this variable based on the token you got from the OAuth client:

    Click to copy
    export AUTHORIZATION_HEADER="Authorization: Bearer $ACCESS_TOKEN"
  3. Make a call to the Kyma Environment Broker to create a Runtime on Azure. Find the list of possible request parameters here.

    Click to copy
    curl --request PUT "https://$BROKER_URL/oauth/v2/service_instances/$INSTANCE_ID?accepts_incomplete=true" \
    --header 'X-Broker-API-Version: 2.14' \
    --header 'Content-Type: application/json' \
    --header "$AUTHORIZATION_HEADER" \
    --data-raw "{
    \"service_id\": \"47c9dcbf-ff30-448e-ab36-d3bad66ba281\",
    \"plan_id\": \"4deee563-e5ec-4731-b9b1-53b42d855f0c\",
    \"context\": {
    \"globalaccount_id\": \"$GLOBAL_ACCOUNT_ID\"
    },
    \"parameters\": {
    \"name\": \"$NAME\"
    }
    }"

A successful call returns the operation ID:

Click to copy
{
"operation":"8a7bfd9b-f2f5-43d1-bb67-177d2434053c"
}
  1. Check the operation status as described here.

Deprovision Kyma Runtime using KEB

This tutorial shows how to deprovision Kyma Runtime on Azure using Kyma Environment Broker.

Steps

  1. Ensure that these environment variables are exported:

    Click to copy
    export BROKER_URL={KYMA_ENVIRONMENT_BROKER_URL}
    export INSTANCE_ID={INSTANCE_ID_FROM_PROVISIONING_CALL}
  2. Get the access token. Export this variable based on the token you got from the OAuth client:

    Click to copy
    export AUTHORIZATION_HEADER="Authorization: Bearer $ACCESS_TOKEN"
  3. Make a call to the Kyma Environment Broker to delete a Runtime on Azure.

    Click to copy
    curl --request DELETE "https://$BROKER_URL/oauth/v2/service_instances/$INSTANCE_ID?accepts_incomplete=true&service_id=47c9dcbf-ff30-448e-ab36-d3bad66ba281&plan_id=4deee563-e5ec-4731-b9b1-53b42d855f0c" \
    --header 'X-Broker-API-Version: 2.13' \
    --header "$AUTHORIZATION_HEADER"

A successful call returns the operation ID:

Click to copy
{
"operation":"8a7bfd9b-f2f5-43d1-bb67-177d2434053c"
}
  1. Check the operation status as described here.

Check operation status

This tutorial shows how to check the operation status for the provisioning and deprovisioning operations.

Steps

  1. Export the operation ID that you obtained during provisioning or deprovisioning as an environment variable:

    Click to copy
    export OPERATION_ID={OBTAINED_OPERATION_ID}

    NOTE: Ensure that the BROKER_URL and INSTANCE_ID environment variables are exported as well before you proceed.

  2. Make a call to the Kyma Environment Broker with a proper Authorization request header to verify that provisioning/deprovisioning succeeded.

    Click to copy
    curl --request GET "https://$BROKER_URL/oauth/v2/service_instances/$INSTANCE_ID/last_operation?operation=$OPERATION_ID&service_id=47c9dcbf-ff30-448e-ab36-d3bad66ba281&plan_id=4deee563-e5ec-4731-b9b1-53b42d855f0c" \
    --header 'X-Broker-API-Version: 2.13' \
    --header "$AUTHORIZATION_HEADER"

A successful call returns the operation status and description:

Click to copy
{
"state": "succeeded",
"description": "Operation created : Operation succeeded."
}

Check SKR instance details

This tutorial shows how to get the SKR instance details.

Steps

  1. Export the instance ID that you set during provisioning:

    Click to copy
    export INSTANCE_ID={SET_INSTANCE_ID}

NOTE: Ensure that the BROKER_URL and INSTANCE_ID environment variables are exported as well before you proceed.

  1. Make a call to the Kyma Environment Broker with a proper Authorization request header to verify that provisioning/deprovisioning succeeded:

    Click to copy
    curl --request GET "https://$BROKER_URL/oauth/v2/service_instances/$INSTANCE_ID" \
    --header 'X-Broker-API-Version: 2.14' \
    --header "$AUTHORIZATION_HEADER"

A successful call returns the instance details:

Click to copy
{
"service_id": "47c9dcbf-ff30-448e-ab36-d3bad66ba281",
"plan_id": "4deee563-e5ec-4731-b9b1-53b42d855f0c",
"dashboard_url": "https://console.{DOMAIN}",
"parameters": {
"autoScalerMax": 3,
"autoScalerMin": 1,
"components": [],
"machineType": null,
"maxSurge": 1,
"maxUnavailable": 1,
"name": "test",
"region": "westeurope",
"targetSecret": "azrspn-ce-skr-dev-00001",
"volumeSizeGb": 50,
"zones": ["1", "2", "3"]
}
}

NOTE: The dashboard_url field is available only if the Runtime was provisioned successfully and the Runtime Agent registered the Runtime in the Director. Fields under the parameters field can differ depending on the provisioning input.

Orchestrate Kyma upgrade

This tutorial shows how to upgrade Kyma Runtime using Kyma Environment Broker.

Prerequisites

  • Compass with:
    • Runtime Provisioner configured for Azure
    • Kyma Environment Broker configured and chosen overrides set up

Steps

  1. Get the access token. Export this variable based on the token you got from the OAuth client:

    Click to copy
    export AUTHORIZATION_HEADER="Authorization: Bearer $ACCESS_TOKEN"
  2. Make a call to the Kyma Environment Broker to orchestrate the upgrade. You can select specific Runtimes to upgrade using the following selectors:

  • target - use the target: "all" selector to select all Runtimes
  • globalAccount - use it to select Runtimes with the specified global account ID
  • subAccount - use it to select Runtimes with the specified subaccount ID
  • runtimeID - use it to select Runtimes with the specified Runtime ID
  • planName - use it to select Runtimes with the specified plan name
  • region - use it to select Runtimes located in the specified region

    Click to copy
    curl --request POST "https://$BROKER_URL/upgrade/kyma" \
    --header "$AUTHORIZATION_HEADER" \
    --header 'Content-Type: application/json' \
    --data-raw "{\
    \"targets\": {\
    \"include\": {\
    \"runtimeID\": \"uuid-sdasd-sda23t-efs\",\
    \"globalAccount\": \"uuid-sdasd-sda23t-efs\",\
    \"subAccount\": \"uuid-sdasd-sda23t-efs\",\
    \"planName\": \"azure\",\
    \"region\": \"europewest\",\
    },\
    },\
    \"dryRun\": false\
    }"

NOTE: If the dryRun parameter specified in the request body is set to true, the upgrade is executed but the upgrade request is not sent to Runtime Provisioner.

  1. If you want to configure the strategy of your orchestration, use the following request example:
Click to copy
curl --request POST "https://$BROKER_URL/upgrade/kyma" \
--header "$AUTHORIZATION_HEADER" \
--header 'Content-Type: application/json' \
--data-raw "{\
\"targets\": {\
\"include\": {\
\"runtimeID\": \"uuid-sdasd-sda23t-efs\",\
\"globalAccount\": \"uuid-sdasd-sda23t-efs\",\
\"subAccount\": \"uuid-sdasd-sda23t-efs\",\
\"planName\": \"azure\",\
\"region\": \"europewest\",\
},\
},\
\"strategies\": {\
\"type\": \"parallel\",\
\"schedule\": \"maintenanceWindow\",\
\"parallel\": {\
\"workers\": 5\
},\
},\
\"dryRun\": false\
}"

NOTE: By default, the orchestration is configured with the parallel strategy, using the immediate type of schedule with only one worker.

A successful call returns the orchestration ID:

Click to copy
{
"orchestrationID":"8a7bfd9b-f2f5-43d1-bb67-177d2434053c"
}
  1. Check the orchestration status.

NOTE: Only one orchestration request can be processed at the same time. If KEB is already processing an orchestration, the newly created request waits for processing with the PENDING state.

Check orchestration status

This tutorial shows how to check the orchestration status. Using this API, you can fetch data about:

  • A single orchestration
  • All orchestrations
  • Upgrade operations scheduled by a given orchestration
  • A single operation with details, such as parameters sent to Runtime Provisioner

Fetch a single orchestration status

  1. Export the orchestration ID that you obtained during the upgrade call as an environment variable:

    Click to copy
    export ORCHESTRATION_ID={OBTAINED_ORCHESTRATION_ID}
  2. Make a call to the Kyma Environment Broker with a proper Authorization request header to verify that the orchestration succeeded.

    Click to copy
    curl --request GET "https://$BROKER_URL/orchestrations/$ORCHESTRATION_ID --header "$AUTHORIZATION_HEADER""

    A successful call returns the orchestration object:

    Click to copy
    {
    "orchestrationID": "07089b96-8e31-49a4-96d0-f8288253c804",
    "state": "in progress",
    "description": "Scheduled 2 operations",
    "createdAt": "2020-10-12T19:27:22.281515Z",
    "updatedAt": "2020-10-12T19:27:22.281515Z",
    "parameters": {
    "targets": {
    "include": [
    {
    "target": "all"
    }
    ]
    },
    "strategy": {
    "type": "parallel",
    "schedule": "immediate",
    "parallel": {
    "workers": 1
    }
    },
    "dryRun": true
    }
    }

Fetch all orchestrations status

Make a call to the Kyma Environment Broker with a proper Authorization request header to verify that the orchestration succeeded.

Click to copy
curl --request GET "https://$BROKER_URL/orchestrations --header "$AUTHORIZATION_HEADER""

A successful call returns the list of all orchestrations:

Click to copy
[
{
"orchestrationID": "07089b96-8e31-49a4-96d0-f8288253c804",
"state": "in progress",
"description": "Scheduled 2 operations",
"createdAt": "2020-10-12T19:27:22.281515Z",
"updatedAt": "2020-10-12T19:27:22.281515Z",
"parameters": {
"targets": {
"include": [
{
"target": "all"
}
]
},
"strategy": {
"type": "parallel",
"schedule": "immediate",
"parallel": {
"workers": 1
}
},
"dryRun": true
}
}
]

List upgrade operations scheduled by an orchestration

  1. Export the orchestration ID that you obtained during the upgrade call as an environment variable:

    Click to copy
    export ORCHESTRATION_ID={OBTAINED_ORCHESTRATION_ID}
  2. Make a call to the Kyma Environment Broker with a proper Authorization request header to fetch the list of the upgrade operations for a given orchestration.

    Click to copy
    curl --request GET "https://$BROKER_URL/orchestrations/$ORCHESTRATION_ID/operations --header "$AUTHORIZATION_HEADER""

    A successful call returns the list of upgrade operations:

    Click to copy
    {
    "data": [
    {
    "operationID": "c4aa1f4b-be2a-4e8d-90e6-edd00194aaa9",
    "runtimeID": "5791e81d-8959-4b78-82e4-7e4edea45683",
    "globalAccountID": "3e64ebae-38b5-46a0-b1ed-9ccee153a0ae",
    "subAccountID": "39b19a66-2c1a-4fe4-a28e-6e5db434084e",
    "orchestrationID": "17089b96-8e31-49a4-96d0-f8288253c804",
    "servicePlanID": "ca1e5357-707f-4565-bbbd-b3ab732597c6",
    "servicePlanName": "gcp",
    "dryRun": true,
    "shootName": "c-3a3xdaf",
    "maintenanceWindowBegin": "0000-01-01T04:00:00Z",
    "maintenanceWindowEnd": "0000-01-01T08:00:00Z"
    },
    {
    "operationID": "669c1644-44c2-349d-a3c5-8bc63dceff93",
    "runtimeID": "8071f119-29a5-3e81-bae3-04751881f317",
    "globalAccountID": "3e64ebae-68b5-46a0-b1ed-9ccee153a0ae",
    "subAccountID": "A791EFE6-6121-1714-9933-E2D3D8CA2992",
    "orchestrationID": "a7089b96-8e31-49a4-96d0-f8288253c804",
    "servicePlanID": "4deee563-e5ec-4731-b9b1-53b42d855f0c",
    "servicePlanName": "azure",
    "dryRun": true,
    "shootName": "c-5d2xd83",
    "maintenanceWindowBegin": "0000-01-01T22:00:00Z",
    "maintenanceWindowEnd": "0000-01-01T02:00:00Z"
    }
    ],
    "count": 2,
    "totalCount": 2
    }

Fetch the detailed operation status

  1. Export the following values as the environment variables:

    Click to copy
    export ORCHESTRATION_ID={OBTAINED_ORCHESTRATION_ID}
    export OPERATION_ID={SCHEDULED_OPERATION_ID}
  2. Make a call to the Kyma Environment Broker with a proper Authorization request header to fetch the details for given operation.

    Click to copy
    curl --request GET "https://$BROKER_URL/orchestrations/$ORCHESTRATION_ID/operations/$OPERATION_ID --header "$AUTHORIZATION_HEADER""

    A successful call returns the upgrade operation object with the kymaConfig and clusterConfig fields:

    Click to copy
    {
    "operationID": "c4aadf4b-be2a-4e8d-90e6-edd00194aaa9",
    "runtimeID": "5791e84d-8959-4b78-82e4-7e4edea45683",
    "globalAccountID": "3e64ebae-38b5-46a0-b1ed-9ccee153a0ae",
    "subAccountID": "39ba9a66-2c1a-4fe4-a28e-6e5db434084e",
    "orchestrationID": "07089b96-8e31-49a4-96d0-f8288253c804",
    "servicePlanID": "ca6e5357-707f-4565-bbbd-b3ab732597c6",
    "servicePlanName": "gcp",
    "dryRun": true,
    "shootName": "c-3a3e0af",
    "maintenanceWindowBegin": "0000-01-01T04:00:00Z",
    "maintenanceWindowEnd": "0000-01-01T08:00:00Z",
    "kymaConfig": {
    "version": "1.15.1",
    "components": [],
    "configuration": []
    },
    "clusterConfig": {}
    }