Event Bus
Overview
Kyma Event Bus allows you to integrate various external solutions with Kyma. To achieve successful integration, the Event Bus uses the publish-subscribe
messaging pattern that allows Kyma to receive business Events from different solutions, enrich them, and trigger business flows using lambdas or services defined in Kyma.
The Event Bus is based on the NATS Streaming open-source log-based streaming system for cloud-native applications, which is a brokered messaging middleware and Knative Eventing. The Event Bus provides an at-least-once delivery guarantee meaning the messages are retransmitted to assure they are successfully delivered at least once.
Architecture
Event consumption
When you create a lambda or a service to perform a given business functionality, you must define which Events trigger it. Define triggers by creating the Subscription CR where you instruct the Event Bus to forward the Events of a particular type to your lambda.
For example, whenever the order-created
Event comes in, the Event Bus stores it in NATS Streaming. It then dispatches it to the receiver specified in the Subscription CR definition.
NOTE: The Event Bus creates a separate Event trigger for each Subscription.
A user creates a lambda or a service that an Event coming from an external solution triggers.
NOTE: When creating a service, the user must create a Kyma Subscription resource manually. If the user uses Kyma Console UI to create a lambda, the Subscription resource is created automatically.
Event Bus Subscription Controller reacts to the creation of Kyma Subscription. It verifies if the Event type from the application can be consumed in the Namespace where the Kyma Subscription has been created. If so, it creates the Knative Channel and Knative Subscription resources.
- Knative Eventing Controller reacts to the creation of Knative Channel and creates a NatssChannel resource with all the necessary specification about the subscriber.
- NatssChannel Controller reacts to the creation of a NatssChannel and creates the required Kubernetes services. NatssChannel Dispatcher also reacts to the creation of a NatssChannel and creates the NATSS Streaming Subscription.
- NatssChannel Dispatcher picks the Event and dispatches it to the configured lambda or the service URL as an HTTP POST request. The lambda reacts to the received Event.
Event publishing
- The external application integrated with Kyma makes a REST API request to the Application Connector's Events Gateway to indicate that a new Event is available. The request provides the Application Connector with the Event metadata.
The Application Connector enriches the Event with the details of its source.
NOTE: There is always one dedicated instance of the Application Connector for every instance of an external solution connected to Kyma.
The Application Connector makes a REST API call to event-publish-service and sends the enriched Event.
- event-publish-service makes the HTTP payload compatible with Knative and sends the Event to the relevant knative-channel service URL which is inferred based on source id, event type, and event type version parameters.
- Kubernetes service forwards the Event to the nats-dispatcher service served by the nats-dispatcher Pod.
- nats-dispatcher saves the Event in NATS Streaming which stores the Event details in the Persistence storage volume.
Event validation
subscription-controller checks if the Namespace can receive Events from the application. It performs the check for each Kyma Subscription created in a Namespace for a particular Event type with a version for a specific application.
Validation flow
See the diagram and a step-by-step description of the Event verification process.
- The Kyma user defines a lambda or a service.
- The Kyma user creates a Subscription custom resource.
- subscription-controller reads the new Subscription.
- subscription-controller reads the EventActivation CR to verify if it exists in the Namespace for a certain application.
- subscription-controller updates the Subscription resource accordingly with the activation status
true
orfalse
. The Event Bus uses this status to allow or prohibit Event delivery.
Details
Basic concepts
The following resources are involved in Event transfer and validation in Kyma:
EventActivation is a custom resource controller that the Application Broker (AB) creates. Its purpose is to define Event availability in a given Namespace.
NATS Streaming is an open source, log-based streaming system that serves as a database allowing the Event Bus to store and transfer the Events on a large scale.
Persistence is a back-end storage volume for NATS Streaming that stores Events. When the Event flow fails, the Event Bus can resume the process using the Events saved in Persistence.
Publish is an internal Event Bus service that transfers the enriched Event from a given external solution to NATS Streaming.
Subscription is a custom resource that the lambda or service creator defines to subscribe a given lambda or a service to particular types of Events.
Event flow requirements
Overview
The Event Bus ensures a successful flow of Events in Kyma when:
- You have enabled the EventActivation controller.
- You have created a Subscription custom resource and registered the webhook for a lambda or service to consume the Events.
- The Events are published.
Create the EventActivation controller
Enable the EventActivation controller to ensure the Event flow between the Namespace and the Application (App). Alternatively, bind the App to the Namespace and create a Service Instance.
The diagram shows you the Event activation flow for a given Namespace.
- The App sends the Events to the Event Bus and uses the EventApplication controller to ensure the Namespace receives the Events.
- If you define a lambda in the
prod123
Namespace, it receives the order.created Event from the App through the Event Bus. - The lambda in the
test123
Namespace does not receive any Events since you have not enabled the EventActivation controller.
Create Subscription
Configure lambdas and services to use the push
command for consuming Events coming from the App. To make sure the lambda or the service receive the Events, register a webhook for them by creating a Subscription custom resource.
Publish Events
Make sure that the external solution sends Events to Kyma. For details of the Event flow, see this document.
Event delivery
Events are delivered as HTTP POST requests. The Event metadata is a part of an HTTP request headers, whereas the Event payload is the body of the request.
The Event delivery workflow is as follows:
- An external system uses the bound Application to publish an Event to the Event Bus.
- The Event Bus checks for the Event subscription and activation. It creates a POST request using Event payload and metadata.
- The service receives the POST request. The HTTP request headers represent Event metadata and HTTP request body represents the Event payload.
Event metadata
The following HTTP headers provide information about the Event metadata. The headers follow the format defined in the CloudEvents specification.
Header | Description |
---|---|
ce-id | Business Event's ID delivered to the microservice. |
ce-time | Business Event's time delivered to the microservice. |
ce-type | Business Event's type delivered to the microservice. |
ce-specversion | Business Cloud Event's specification version delivered to the microservice. |
ce-eventtypeversion | Business Event's version delivered to the microservice. |
ce-source | Identifies the origin of Events. This can be an external solution or a defined identifier for internally generated Events. |
content-type | Business Cloud Event's content type delivered to the microservice. |
kyma-subscription | Subscription name defined in the subscription contract, or in a CRD. This business Event is published to its subscribers. |
x-b3-flags | Header used by the Zipkin tracer in Envoy. It encodes one or more options. See more on Zipkin tracing here. |
x-b3-parentspanid | Header used by the Zipkin tracer in Envoy. The ParentSpanId is 64-bit in length and indicates the position of the parent operation in the trace tree. When the span is the root of the trace tree, the ParentSpanId is absent. |
x-b3-sampled | Header used by the Zipkin tracer in Envoy. When the Sampled flag is either not specified or set to 1 , the span is reported to the tracing system. Once Sampled is set to 0 or 1 , the same value should be consistently sent downstream. |
x-b3-spanid | Header used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree. The value should not be interpreted. It may or may not be derived from the value of the TraceId. |
x-b3-traceid | Header used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID. |
x-request-id | Randomly generated ID which identifies the HTTP request delivering the business Event. |
x-envoy-decorator-operation | If this header is present in Ingress requests, its value overrides any locally defined operation (span) name on the server span generated by the tracing mechanism. If this header is present in an Egress response, its value overrides any locally defined operation (span) name on the client span. |
x-envoy-expected-rq-timeout-ms | Time in milliseconds in which the router expects the request to be completed. Envoy sets this header so that the upstream host receiving the request can make decisions based on the request timeout. It is set on internal requests and is either taken from the x-envoy-upstream-rq-timeout-ms header or from the route timeout. |
x-istio-attributes | Istio-specific metadata. |
Event payload
The Event payload is delivered as the body of the HTTP Request in JSON
format. The JSON
schema is available in the Service Catalog in the registered service for the remote Events.
This example shows the order.created
Event published by the external solution service:
{ "example": { "orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e" }, "properties": { "orderCode": { "description": "Resource identifier", "title": "OrderCode", "type": "string" } }, "type": "object"}
The HTTP POST request payload is a JSON object:
{"orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e"}
Responses
The service's HTTP response status code 2xx
means that the message was successfully delivered to the subscriber.
For example:
{ "event-id": "22ae22a4-f5b7-4fa1-ada9-558a10a96f3d", "status": "published", "reason": "Message successfully published to the channel"}
If the status code is not 2xx
(< 200
or >= 300
), it means that the delivery was unsuccessful and that the Event Bus will try to deliver it again. This implies At-least-once delivery guarantee.
If there were no subscriptions or consumers to this event-type
, the message is ignored and the response will look like this:
{ "event-id": "22ae22a4-f5b7-4fa1-ada9-558a10a96f3d", "status": "ignored", "reason": "Event was ignored as there are no subscriptions or consumers configured for this event"}
Event Subscription service example
See this example to find a complete scenario for implementing a subscriber service to a business Event.
Configure default Knative Channel
Overview
Kyma comes with NATS Streaming as its default Channel. You can see the configuration details in the default-ch-webhook
ConfigMap.
You can use a different messaging middleware, other than NATS Streaming, as the Kyma eventing operator. To achieve that:
- Apply the Channel Resources for the messaging middleware you want to use. These resources connect with the running messaging middleware.
- Configure the
default-ch-webhook
ConfigMap in theknative-eventing
Namespace to use that particular Channel.
If you want to edit the default-ch-webhook
ConfigMap, run:
kubectl -n knative-eventing edit configmaps default-ch-webhook
Read about the examples and the configuration details.
In-memory channel
Follow this guide to add the InMemoryChannel resources. This will deploy the InMemoryChannel CRD, Controller, and Dispatcher.
NOTE: Before installing this provisioner, add the following annotation to the
podTemplate.Spec
in thein-memory-channel-controller
Deployment to remove the Istio sidecar.
template: annotations: sidecar.istio.io/inject: "false" metadata: labels: *labels
You can change the default channel configuration by editing the ConfigMap default-ch-webhook
in knative-eventing
Namespace. For example, if you want to set In-Memory Channels as default provisioner, include the following data in the ConfigMap:
apiVersion: v1kind: ConfigMapmetadata: name: default-ch-webhook namespace: knative-eventingdata: default-ch-config: | clusterDefault: apiVersion: messaging.knative.dev/v1alpha1 kind: InMemoryChannel namespaceDefaults: some-namespace: apiVersion: messaging.knative.dev/v1alpha1 kind: InMemoryChannel
NOTE: This ConfigMap may specify a cluster-wide default channel and/or namespace-specific channel implementations.
Google PubSub
Follow this guide to install GCP PubSub Channel resources. This will apply the GCP PubSub Channel CRDs and deploy the cloud-run-events-controller
controller.
Edit the default-ch-webhook
ConfigMap located in the knative-eventing
Namespace to include the following data.
apiVersion: v1kind: ConfigMapmetadata: name: default-ch-webhook namespace: knative-eventingdata: default-ch-config: | clusterDefault: apiVersion: messaging.cloud.run/v1alpha1 kind: Channel spec: project: <GCP Project Name>
NOTE: You need to mention the GCP Project Name in the specification which will be used as the reference GCP project to create GCP PubSub Topics.
Configuration
Event Bus chart
To configure the Event Bus chart, override the default values of its values.yaml
file. This document describes parameters that you can configure.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
Configurable parameters
This table lists the configurable parameters, their descriptions, and default values:
Parameter | Description | Default value |
---|---|---|
global.eventPublishService.maxRequests | Specifies the maximum number of parallel Event requests that eventPublishService can process. If you raise this value, you may also have to increase memory resources for the Event Bus to handle the higher load. | 16 |
global.eventPublishService.maxRequestSize | Specifies the maximum size of one Event. If you raise this value, you may also have to increase memory resources for the Event Bus to handle the higher load. | 65536 |
global.eventPublishService.resources.limits.memory | Specifies memory limits set for eventPublishService. | 32M |
global.subscriptionController.resources.limits.memory | Specifies memory limits set for subscriptionController. | 32M |
NATS Streaming chart
To configure NATS Streaming chart, override the default values of its values.yaml
file. This document describes parameters that you can configure.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
Configurable parameters
This table lists the configurable parameters, their descriptions, and default values:
Parameter | Description | Default value |
---|---|---|
global.natsStreaming.persistence.maxAge | Specifies the time for which the given Event is stored in NATS Streaming. | 24h |
global.natsStreaming.persistence.size | Specifies the size of the persistence volume in NATS Streaming. | 1Gi |
global.natsStreaming.resources.limits.memory | Specifies the memory limits for NATS Streaming. | 256M |
global.natsStreaming.channel.maxInactivity | Specifies the time after which the autocleaner removes all backing resources related to a given Event type from the NATS Streaming database if there is no activity for this Event type. | 48h |
Custom Resource
Subscription
The subscriptions.eventing.kyma-project.io
CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to create an Event trigger for a lambda or microservice in Kyma. After creating a new custom resource, the Event trigger is registered in the Event Bus and Events are delivered to the endpoint specified in the custom resource.
To get the up-to-date CRD and show the output in the yaml
format, run this command:
kubectl get crd subscriptions.eventing.kyma-project.io -o yaml
Sample custom resource
This is a sample resource that creates an order.created
Event trigger for a lambda.
apiVersion: eventing.kyma-project.io/v1alpha1kind: Subscriptionmetadata: name: hello-with-data-subscription labels: example: event-bus-lambda-subscriptionspec: endpoint: http://hello-with-data.{NAMESPACE}:8080/ include_subscription_name_header: true event_type: order.created event_type_version: v1 source_id: stage.commerce.kyma.local
Custom resource parameters
This table lists all the possible parameters of a given resource together with their descriptions:
Parameter | Required | Description |
---|---|---|
metadata.name | Yes | Specifies the name of the CR. |
spec.endpoint | Yes | The HTTP endpoint to which events are delivered as a POST request. |
spec.include_subscription_name_header | Yes | The boolean flag indicating if the name of the Subscription should be included in the HTTP headers while delivering the Event. |
spec.event_type | Yes | The Event type to which the Event trigger is registered. For example, order.created. |
spec.event_type_version | Yes | The version of the Event type. |
spec.source_id | Yes | Identifies the origin of events. This can be an external solution or a defined identifier for internally generated events. |
Update Subscription CRD
To update the Subscription CRD, run this command:
kubectl edit crd subscriptions.eventing.kyma-project.io
The Event Bus reacts to the changes in the CRD and updates the corresponding NATS-Streaming subscription accordingly.
CAUTION: The current subscription update mechanism recreates a Subscription custom resource with new specifications. This may result in the loss of messages delivered during the recreation process.
CLI reference
Management of the Event Bus is based on the custom resources specifically defined for Kyma. Manage all of these resources through kubectl.
Details
This section describes the resource names to use in kubectl, the command syntax, and examples of use.
Resource types
Event Bus operations use the following resources:
Singular name | Plural name |
---|---|
subscription | subscriptions |
Syntax
Follow the kubectl syntax, kubectl {command} {type} {name} {flags}
, where:
- {command} is any command, such as
describe
. - {type} is a resource type, such as
clusterserviceclass
. - {name} is the name of a given resource type. Use {name} to make the command return the details of a given resource.
- {flags} specifies the scope of the information. For example, use flags to define the Namespace from which to get the information.
Examples
The following examples show how to create new Subscriptions, list them, and obtain detailed information on their statuses.
Create a new Subscription directly from the terminal:
Click to copycat <<EOF | kubectl create -f -apiVersion: eventing.kyma-project.io/v1alpha1kind: Subscriptionmetadata:name: my-subscriptionnamespace: stagespec:endpoint: http://testjs.default:8080/include_subscription_name_header: trueevent_type: order_createdevent_type_version: v1source_id: stage.commerce.kyma.localEOFGet the list of all Subscriptions:
Click to copykubectl get subscription --all-namespacesGet the list of all Subscriptions with detailed information on the Subscription status:
Click to copykubectl get subscriptions -n stage -o=custom-columns=NAME:.metadata.name,STATUS:.status.conditions[*].status,STATUS\ TYPE:.status.conditions[*].type
Troubleshooting
Basic troubleshooting
Lambda does not receive Events
If the lambda or the service does not receive any Events, do the following:
- Confirm that the EventActivation custom resource is in place.
- Ensure that the webhook defined for the lambda or the service is up and running.
- Make sure the Events are published.
Errors while sending Events
If errors appear while sending Events, do the following:
- Check if the
publish
application is up and running. - Make sure that NATS Streaming is up and running.
Troubleshooting using tracing
The tracing functionality available in Kyma can help you to pinpoint the root cause of an error and solve the problem. See this document to learn more about tracing.
No microservice or lambda configured to receive an Event
In this case, an external system sends the Event, but a lambda or microservice with an Event trigger does not exist.
As a result, you can see traces for publish
only. When you access the trace details, they only show you tags for event-publish-service
.
Configured microservice or lambda returns an error
In this case, a microservice or lambda exists and reacts to the Event trigger. However, due to a code issue, the microservice or lambda fails to process the Event.
As a result, the webhook
, push
, and name-of-lambda
services in the trace are marked with error.
To see the error details, click one of the service spans, such as the one for the push
service.
Since the Event Bus keeps on retrying to deliver the Event until it is successful, you
can see multiple spans for webhook-service
.