Our own, Mateusz Szostok has two sessions at KubeCon around K8s Service CatalogRead more

Mateusz Szostok, Piotr Kopczynski and Pawel Kosiec have several talks at San Diego Cloud Native MeetupRead more

Event Bus

Overview

Kyma Event Bus allows you to integrate various external solutions with Kyma. To achieve successful integration, the Event Bus uses the publish-subscribe messaging pattern that allows Kyma to receive business Events from different solutions, enrich them, and trigger business flows using lambdas or services defined in Kyma.

The Event Bus is based on the NATS Streaming open-source log-based streaming system for cloud-native applications, which is a brokered messaging middleware and Knative Eventing. The Event Bus provides an at-least-once delivery guarantee meaning the messages are retransmitted to assure they are successfully delivered at least once.

Architecture

Event consumption

When you create a lambda or a service to perform a given business functionality, you must define which Events trigger it. Define triggers by creating the Subscription CR where you instruct the Event Bus to forward the Events of a particular type to your lambda. For example, whenever the order-created Event comes in, the Event Bus stores it in NATS Streaming. It then dispatches it to the receiver specified in the Subscription CR definition.

NOTE: The Event Bus creates a separate Event trigger for each Subscription.

Configure and Consume Events

  1. A user creates a lambda or a service that an Event coming from an external solution triggers.

    NOTE: When creating a service, the user must create a Kyma Subscription resource manually. If the user uses Kyma Console UI to create a lambda, the Subscription resource is created automatically.

  2. Event Bus Subscription Controller reacts to the creation of Kyma Subscription. It verifies if the Event type from the application can be consumed in the Namespace where the Kyma Subscription has been created. If so, it creates the Knative Channel and Knative Subscription resources.

  3. Knative Eventing Controller reacts to the creation of Knative Channel and creates a NatssChannel resource with all the necessary specification about the subscriber.
  4. NatssChannel Controller reacts to the creation of a NatssChannel and creates the required Kubernetes services. NatssChannel Dispatcher also reacts to the creation of a NatssChannel and creates the NATSS Streaming Subscription.
  5. NatssChannel Dispatcher picks the Event and dispatches it to the configured lambda or the service URL as an HTTP POST request. The lambda reacts to the received Event.

Event publishing

Publish Events

  1. The external application integrated with Kyma makes a REST API request to the Application Connector's Events Gateway to indicate that a new Event is available. The request provides the Application Connector with the Event metadata.
  2. The Application Connector enriches the Event with the details of its source.

    NOTE: There is always one dedicated instance of the Application Connector for every instance of an external solution connected to Kyma.

  3. The Application Connector makes a REST API call to event-publish-service and sends the enriched Event.

  4. event-publish-service makes the HTTP payload compatible with Knative and sends the Event to the relevant knative-channel service URL which is inferred based on source id, event type, and event type version parameters.
  5. Kubernetes service forwards the Event to the nats-dispatcher service served by the nats-dispatcher Pod.
  6. nats-dispatcher saves the Event in NATS Streaming which stores the Event details in the Persistence storage volume.

Event validation

subscription-controller checks if the Namespace can receive Events from the application. It performs the check for each Kyma Subscription created in a Namespace for a particular Event type with a version for a specific application.

Validation flow

See the diagram and a step-by-step description of the Event verification process.

Event validation process

  1. The Kyma user defines a lambda or a service.
  2. The Kyma user creates a Subscription custom resource.
  3. subscription-controller reads the new Subscription.
  4. subscription-controller reads the EventActivation CR to verify if it exists in the Namespace for a certain application.
  5. subscription-controller updates the Subscription resource accordingly with the activation status true or false. The Event Bus uses this status to allow or prohibit Event delivery.

Details

Basic concepts

The following resources are involved in Event transfer and validation in Kyma:

  • EventActivation is a custom resource controller that the Application Broker (AB) creates. Its purpose is to define Event availability in a given Namespace.

  • NATS Streaming is an open source, log-based streaming system that serves as a database allowing the Event Bus to store and transfer the Events on a large scale.

  • Persistence is a back-end storage volume for NATS Streaming that stores Events. When the Event flow fails, the Event Bus can resume the process using the Events saved in Persistence.

  • Publish is an internal Event Bus service that transfers the enriched Event from a given external solution to NATS Streaming.

  • Subscription is a custom resource that the lambda or service creator defines to subscribe a given lambda or a service to particular types of Events.

Event flow requirements

Overview

The Event Bus ensures a successful flow of Events in Kyma when:

  • You have enabled the EventActivation controller.
  • You have created a Subscription custom resource and registered the webhook for a lambda or service to consume the Events.
  • The Events are published.

Create the EventActivation controller

Enable the EventActivation controller to ensure the Event flow between the Namespace and the Application (App). Alternatively, bind the App to the Namespace and create a Service Instance.

The diagram shows you the Event activation flow for a given Namespace.

EventActivation.png

  • The App sends the Events to the Event Bus and uses the EventApplication controller to ensure the Namespace receives the Events.
  • If you define a lambda in the prod123 Namespace, it receives the order.created Event from the App through the Event Bus.
  • The lambda in the test123 Namespace does not receive any Events since you have not enabled the EventActivation controller.

Create Subscription

Configure lambdas and services to use the push command for consuming Events coming from the App. To make sure the lambda or the service receive the Events, register a webhook for them by creating a Subscription custom resource.

Publish Events

Make sure that the external solution sends Events to Kyma. For details of the Event flow, see this document.

Event delivery

Events are delivered as HTTP POST requests. The Event metadata is a part of an HTTP request headers, whereas the Event payload is the body of the request.

The Event delivery workflow is as follows:

eventdelivery

  1. An external system uses the bound Application to publish an Event to the Event Bus.
  2. The Event Bus checks for the Event subscription and activation. It creates a POST request using Event payload and metadata.
  3. The service receives the POST request. The HTTP request headers represent Event metadata and HTTP request body represents the Event payload.

Event metadata

The following HTTP headers provide information about the Event metadata. The headers follow the format defined in the CloudEvents specification.

HeaderDescription
ce-idBusiness Event's ID delivered to the microservice.
ce-timeBusiness Event's time delivered to the microservice.
ce-typeBusiness Event's type delivered to the microservice.
ce-specversionBusiness Cloud Event's specification version delivered to the microservice.
ce-eventtypeversionBusiness Event's version delivered to the microservice.
ce-sourceIdentifies the origin of Events. This can be an external solution or a defined identifier for internally generated Events.
content-typeBusiness Cloud Event's content type delivered to the microservice.
kyma-subscriptionSubscription name defined in the subscription contract, or in a CRD. This business Event is published to its subscribers.
x-b3-flagsHeader used by the Zipkin tracer in Envoy. It encodes one or more options. See more on Zipkin tracing here.
x-b3-parentspanidHeader used by the Zipkin tracer in Envoy. The ParentSpanId is 64-bit in length and indicates the position of the parent operation in the trace tree. When the span is the root of the trace tree, the ParentSpanId is absent.
x-b3-sampledHeader used by the Zipkin tracer in Envoy. When the Sampled flag is either not specified or set to 1, the span is reported to the tracing system. Once Sampled is set to 0 or 1, the same value should be consistently sent downstream.
x-b3-spanidHeader used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree. The value should not be interpreted. It may or may not be derived from the value of the TraceId.
x-b3-traceidHeader used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID.
x-request-idRandomly generated ID which identifies the HTTP request delivering the business Event.
x-envoy-decorator-operationIf this header is present in Ingress requests, its value overrides any locally defined operation (span) name on the server span generated by the tracing mechanism. If this header is present in an Egress response, its value overrides any locally defined operation (span) name on the client span.
x-envoy-expected-rq-timeout-msTime in milliseconds in which the router expects the request to be completed. Envoy sets this header so that the upstream host receiving the request can make decisions based on the request timeout. It is set on internal requests and is either taken from the x-envoy-upstream-rq-timeout-ms header or from the route timeout.
x-istio-attributesIstio-specific metadata.

Event payload

The Event payload is delivered as the body of the HTTP Request in JSON format. The JSON schema is available in the Service Catalog in the registered service for the remote Events.

This example shows the order.created Event published by the external solution service:

Click to copy
{
"example": {
"orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e"
},
"properties": {
"orderCode": {
"description": "Resource identifier",
"title": "OrderCode",
"type": "string"
}
},
"type": "object"
}

The HTTP POST request payload is a JSON object:

Click to copy
{"orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e"}

Responses

The service's HTTP response status code 2xx means that the message was successfully delivered to the subscriber. For example:

Click to copy
{
"event-id": "22ae22a4-f5b7-4fa1-ada9-558a10a96f3d",
"status": "published",
"reason": "Message successfully published to the channel"
}

If the status code is not 2xx (< 200 or >= 300), it means that the delivery was unsuccessful and that the Event Bus will try to deliver it again. This implies At-least-once delivery guarantee.

If there were no subscriptions or consumers to this event-type, the message is ignored and the response will look like this:

Click to copy
{
"event-id": "22ae22a4-f5b7-4fa1-ada9-558a10a96f3d",
"status": "ignored",
"reason": "Event was ignored as there are no subscriptions or consumers configured for this event"
}

Event Subscription service example

See this example to find a complete scenario for implementing a subscriber service to a business Event.

Configure default Knative Channel

Overview

Kyma comes with NATS Streaming as its default Channel. You can see the configuration details in the default-ch-webhook ConfigMap.

You can use a different messaging middleware, other than NATS Streaming, as the Kyma eventing operator. To achieve that:

  • Apply the Channel Resources for the messaging middleware you want to use. These resources connect with the running messaging middleware.
  • Configure the default-ch-webhook ConfigMap in the knative-eventing Namespace to use that particular Channel.

If you want to edit the default-ch-webhook ConfigMap, run:

Click to copy
kubectl -n knative-eventing edit configmaps default-ch-webhook

Read about the examples and the configuration details.

In-memory channel

Follow this guide to add the InMemoryChannel resources. This will deploy the InMemoryChannel CRD, Controller, and Dispatcher.

NOTE: Before installing this provisioner, add the following annotation to the podTemplate.Spec in the in-memory-channel-controller Deployment to remove the Istio sidecar.

Click to copy
template:
annotations:
sidecar.istio.io/inject: "false"
metadata:
labels: *labels

You can change the default channel configuration by editing the ConfigMap default-ch-webhook in knative-eventing Namespace. For example, if you want to set In-Memory Channels as default provisioner, include the following data in the ConfigMap:

Click to copy
apiVersion: v1
kind: ConfigMap
metadata:
name: default-ch-webhook
namespace: knative-eventing
data:
default-ch-config: |
clusterDefault:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
namespaceDefaults:
some-namespace:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel

NOTE: This ConfigMap may specify a cluster-wide default channel and/or namespace-specific channel implementations.

Google PubSub

Follow this guide to install GCP PubSub Channel resources. This will apply the GCP PubSub Channel CRDs and deploy the cloud-run-events-controller controller.

Edit the default-ch-webhook ConfigMap located in the knative-eventing Namespace to include the following data.

Click to copy
apiVersion: v1
kind: ConfigMap
metadata:
name: default-ch-webhook
namespace: knative-eventing
data:
default-ch-config: |
clusterDefault:
apiVersion: messaging.cloud.run/v1alpha1
kind: Channel
spec:
project: <GCP Project Name>

NOTE: You need to mention the GCP Project Name in the specification which will be used as the reference GCP project to create GCP PubSub Topics.

Configuration

Event Bus chart

To configure the Event Bus chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
global.eventPublishService.maxRequestsSpecifies the maximum number of parallel Event requests that eventPublishService can process. If you raise this value, you may also have to increase memory resources for the Event Bus to handle the higher load.16
global.eventPublishService.maxRequestSizeSpecifies the maximum size of one Event. If you raise this value, you may also have to increase memory resources for the Event Bus to handle the higher load.65536
global.eventPublishService.resources.limits.memorySpecifies memory limits set for eventPublishService.32M
global.subscriptionController.resources.limits.memorySpecifies memory limits set for subscriptionController.32M

NATS Streaming chart

To configure NATS Streaming chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
global.natsStreaming.persistence.maxAgeSpecifies the time for which the given Event is stored in NATS Streaming.24h
global.natsStreaming.persistence.sizeSpecifies the size of the persistence volume in NATS Streaming.1Gi
global.natsStreaming.resources.limits.memorySpecifies the memory limits for NATS Streaming.256M
global.natsStreaming.channel.maxInactivitySpecifies the time after which the autocleaner removes all backing resources related to a given Event type from the NATS Streaming database if there is no activity for this Event type.48h

Custom Resource

Subscription

The subscriptions.eventing.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to create an Event trigger for a lambda or microservice in Kyma. After creating a new custom resource, the Event trigger is registered in the Event Bus and Events are delivered to the endpoint specified in the custom resource.

To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd subscriptions.eventing.kyma-project.io -o yaml

Sample custom resource

This is a sample resource that creates an order.created Event trigger for a lambda.

Click to copy
apiVersion: eventing.kyma-project.io/v1alpha1
kind: Subscription
metadata:
name: hello-with-data-subscription
labels:
example: event-bus-lambda-subscription
spec:
endpoint: http://hello-with-data.{NAMESPACE}:8080/
include_subscription_name_header: true
event_type: order.created
event_type_version: v1
source_id: stage.commerce.kyma.local

Custom resource parameters

This table lists all the possible parameters of a given resource together with their descriptions:

ParameterRequiredDescription
metadata.nameYesSpecifies the name of the CR.
spec.endpointYesThe HTTP endpoint to which events are delivered as a POST request.
spec.include_subscription_name_headerYesThe boolean flag indicating if the name of the Subscription should be included in the HTTP headers while delivering the Event.
spec.event_typeYesThe Event type to which the Event trigger is registered. For example, order.created.
spec.event_type_versionYesThe version of the Event type.
spec.source_idYesIdentifies the origin of events. This can be an external solution or a defined identifier for internally generated events.

Update Subscription CRD

To update the Subscription CRD, run this command:

kubectl edit crd subscriptions.eventing.kyma-project.io

The Event Bus reacts to the changes in the CRD and updates the corresponding NATS-Streaming subscription accordingly.

CAUTION: The current subscription update mechanism recreates a Subscription custom resource with new specifications. This may result in the loss of messages delivered during the recreation process.

CLI reference

Management of the Event Bus is based on the custom resources specifically defined for Kyma. Manage all of these resources through kubectl.

Details

This section describes the resource names to use in kubectl, the command syntax, and examples of use.

Resource types

Event Bus operations use the following resources:

Singular namePlural name
subscriptionsubscriptions

Syntax

Follow the kubectl syntax, kubectl {command} {type} {name} {flags}, where:

  • {command} is any command, such as describe.
  • {type} is a resource type, such as clusterserviceclass.
  • {name} is the name of a given resource type. Use {name} to make the command return the details of a given resource.
  • {flags} specifies the scope of the information. For example, use flags to define the Namespace from which to get the information.

Examples

The following examples show how to create new Subscriptions, list them, and obtain detailed information on their statuses.

  • Create a new Subscription directly from the terminal:

    Click to copy
    cat <<EOF | kubectl create -f -
    apiVersion: eventing.kyma-project.io/v1alpha1
    kind: Subscription
    metadata:
    name: my-subscription
    namespace: stage
    spec:
    endpoint: http://testjs.default:8080/
    include_subscription_name_header: true
    event_type: order_created
    event_type_version: v1
    source_id: stage.commerce.kyma.local
    EOF
  • Get the list of all Subscriptions:

    Click to copy
    kubectl get subscription --all-namespaces
  • Get the list of all Subscriptions with detailed information on the Subscription status:

    Click to copy
    kubectl get subscriptions -n stage -o=custom-columns=NAME:.metadata.name,STATUS:.status.conditions[*].status,STATUS\ TYPE:.status.conditions[*].type

Troubleshooting

Basic troubleshooting

Lambda does not receive Events

If the lambda or the service does not receive any Events, do the following:

  • Confirm that the EventActivation custom resource is in place.
  • Ensure that the webhook defined for the lambda or the service is up and running.
  • Make sure the Events are published.

Errors while sending Events

If errors appear while sending Events, do the following:

  • Check if the publish application is up and running.
  • Make sure that NATS Streaming is up and running.

Troubleshooting using tracing

The tracing functionality available in Kyma can help you to pinpoint the root cause of an error and solve the problem. See this document to learn more about tracing.

No microservice or lambda configured to receive an Event

In this case, an external system sends the Event, but a lambda or microservice with an Event trigger does not exist.

As a result, you can see traces for publish only. When you access the trace details, they only show you tags for event-publish-service.

Configured microservice or lambda returns an error

In this case, a microservice or lambda exists and reacts to the Event trigger. However, due to a code issue, the microservice or lambda fails to process the Event.

As a result, the webhook, push, and name-of-lambda services in the trace are marked with error.

To see the error details, click one of the service spans, such as the one for the push service.

Since the Event Bus keeps on retrying to deliver the Event until it is successful, you can see multiple spans for webhook-service.