Kyma Event Bus enables the integration of various external solutions with Kyma. The integration is achieved using the publish-subscribe messaging pattern that allows Kyma to receive business Events from different solutions, enrich them, and trigger business flows using lambdas or services defined in Kyma.

To learn how to write an HTTP service or Lambda in Kyma, and handle the Event Bus published Events, check the Services Programming Model guide and the Lambda Programming Model guide.

NOTE: The Event Bus is based on the NATS Streaming open source log-based streaming system for cloud-native applications, which is a brokered messaging middleware. The Event Bus provides at-least-once delivery guarantees.

The following resources are involved in Event transfer and validation in Kyma:

• EventActivation is a custom resource controller that the Remote Environment Broker (REB) creates. Its purpose is to define Event availability in a given Environment.

• NATS Streaming is an open source, log-based streaming system that serves as a database allowing the Event Bus to store and transfer the Events on a large scale.

• Persistence is a back-end storage volume for NATS Streaming that stores Events. When the Event flow fails, the Event Bus can resume the process using the Events saved in Persistence.

• Publish is an internal Event Bus service that transfers the enriched Event from a given external solution to NATS Streaming.

• Push is an application responsible for receiving Events from NATS Streaming in the Event Bus. Additionally, it delivers the validated Events to the lambda or the service, following the trigger from the Subscription custom resource. The Events are delivered to the lambda or the service through the Envoy proxy sidecar with mTLS enabled.

• Subscription is a custom resource that the lambda or service creator defines to subscribe a given lambda or a service to particular types of Events.

• Sub-validator is a Kubernetes deployment. It updates the status of the Subscription custom resource with the EventActivation status. Depending on the status, push starts or stops delivering Events to the lambda or the service webhook.

The Event Bus enables a successful flow of the Events in Kyma when:

• The EventActivation is in place.
• You create a Subscription Kubernetes custom resource and register the webhook for the lambda or a service to consume the Events.
• The Events are published.

Details

See the following subsections for details on each requirement.

Activate Events

To receive Events, use EventActivation between the Environment and the Remote Environment.

For example, if you define the lambda in the test123 Environment and want to receive the order-created Event type from the ec-qa Remote Environment, you need to enable the EventActivation between the test123 Environment and the ec-qa Remote Environment. Otherwise, the lambda cannot receive the order-created Event type.

Consume Events

Enable lambdas and services to consume Events in Kyma between any Environment and the Remote Environment using push. Deliver Events to the lambda or the service by registering a webhook for it. Create a Subscription Kubernetes custom resource to register the webhook.

See the table for the explanation of parameters in the Subscription custom resource.

Parameter	Description
include_subscription_name_header	It indicates whether the lambda or the service includes the name of the Subscription when receiving an Event.
max_inflight	It indicates the maximum number of Events which can be delivered concurrently. The final value is the max_inflight number multiplied by the number of the push applications.
push_request_timeout_ms	It indicates the time for which the push waits for the response when delivering an Event to the lambda or the service. After the specified time passes, the request times out and the Event Bus retries delivering the Event. Setting the minimum parameter to 0 applies the default value of 1000ms.
event_type	The name of the Event type. For example, order-created.
event_type_version	The version of the Event type. For example, v1.
source	Details of the remote environment that the Event originates from.
source_environment	The environment of the Event source. For example, ec-qa.
source_namespace	The parameter that uniquely identifies the organization publishing the Event.
source_type	The type of the Event source. For example, commerce.

Event publishing

Make sure that the external solution sends Events to Kyma.

You can configure a microservice deployed in Kyma to receive Events from the Event Bus by creating a subscription custom resource. Find various configuration options in the Consume Events section.

Event Delivery

The Event is delivered as an HTTP POST request. Event Metadata is a part of an HTTP request Headers. Event Payload is the body of the request.

The Event Delivery workflow is as follows:

1. The Event is published to the Kyma Event Bus from an external system instance in a bound Remote Environment.
2. The Event Bus checks for the Event subscription and activation. It creates an HTTP POST request using Event Payload and Metadata.
3. The Service receives the HTTP POST request. The Event Metadata is represented in the HTTP Request Headers request and the Event Payload is represented in the HTTP Request Body.

Event Metadata

The following HTTP Headers provide information about the Event Metadata.

Header	Description
kyma-event-id	Business Event's ID delivered to the microservice.
kyma-event-time	Business Event's time delivered to the microservice.
kyma-event-type	Business Event's type delivered to the microservice.
kyma-event-type-version	Business Event's version delivered to the microservice.
kyma-source-environment	Business Event's source environment delivered to the microservice.
kyma-source-namespace	Business Event's source Namespace delivered to the microservice.
kyma-source-type	Business Event's source type delivered to the microservice.
kyma-subscription	Subscription name defined in the subscription contract, or in a CRD. This business Event is published to its subscribers.
x-b3-flags	Header used by the Zipkin tracer in Envoy. It encodes one or more options. See more on Zipkin tracing here.
x-b3-parentspanid	Header used by the Zipkin tracer in Envoy. The ParentSpanId is 64-bit in length and indicates the position of the parent operation in the trace tree. When the span is the root of the trace tree, the ParentSpanId is absent.
x-b3-sampled	Header used by the Zipkin tracer in Envoy. When the Sampled flag is either not specified or set to 1, the span is reported to the tracing system. Once Sampled is set to 0 or 1, the same value should be consistently sent downstream.
x-b3-spanid	Header used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree. The value should not be interpreted. It may or may not be derived from the value of the TraceId.
x-b3-traceid	Header used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID.
x-request-id	Randomly generated ID which identifies the HTTP request delivering the business Event.
x-envoy-decorator-operation	If this header is present in Ingress requests, its value overrides any locally defined operation (span) name on the server span generated by the tracing mechanism. If this header is present in an Egress response, its value overrides any locally defined operation (span) name on the client span.
x-envoy-expected-rq-timeout-ms	Time in milliseconds in which the router expects the request to be completed. Envoy sets this header so that the upstream host receiving the request can make decisions based on the request timeout. It is set on internal requests and is either taken from the x-envoy-upstream-rq-timeout-ms header or from the route timeout.
x-istio-attributes	Istio specific metadata.

Event Payload

The Event Payload is delivered as the body of the HTTP Request in JSON format. The JSON schema is available in the Service Catalog in the registered service for the remote Events.

Event Payload Example

In this example, you write a service for an order.created Event published by the external solution service. The published Event schema looks as follows:

{
  "example": {
    "orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e"
  },
  "properties": {
    "orderCode": {
      "description": "Resource identifier",
      "title": "OrderCode",
      "type": "string"
    }
  },
  "type": "object"
}

The HTTP POST request payload is a JSON object:

{"orderCode": "4caad296-e0c5-491e-98ac-0ed118f9474e"}

Successful Delivery

A message delivered to a subscriber is considered successfully consumed if the service's HTTP response status code is 2xx. If the status code is not 2xx (< 200 or >= 300), it means that a message consumption is not successful and that the message delivery is re-tried. This implies At-least-once delivery guarantee.

Event Subscription Service Example

Refer to this example to find a complete scenario for implementing a subscriber service to a business Event.

See the diagram and steps for an overview of the basic Event Bus flow:

Event flow

1. The external solution integrated with Kyma makes a REST API call to the Application Connector to indicate that a new Event is available.

2. The Application Connector enriches the Event with the details of its source.

NOTE: There is always one dedicated instance of the Application Connector for every instance of an external solution connected to Kyma.

3. The Application Connector makes a REST API call to publish and sends the enriched Event.

4. publish saves the information in the NATS Streaming database.

5. NATS Streaming stores the Event details in the Persistence storage volume to ensure the data is not lost if the NATS Streaming crashes.

6. If the Subscription validation process completes successfully, push consumes the Event from NATS Streaming.

7. push delivers the Event to the lambda or the service.

Event validation

The Event Bus performs Event validation before it allows Event consumption.

Validation details

When you create a lambda or a service to perform a given business functionality, you also need to define which Events trigger it. Define triggers by creating the Subscription custom resource in which you register with the Event Bus to forward the Events of a particular type, such as order-created, to your lambda or a service. Whenever the order-created Event comes in, the Event Bus consumes it by saving it in NATS Streaming and Persistence, and sends it to the correct receiver specified in the Subscription definition.

NOTE: The Event Bus creates a separate Event Trigger for each Subscription.

Before the Event Bus forwards the Event to the receiver, the sub-validator performs a security check to verify the permissions for this Event in a given Environment. It reads all new Subscription resources and refers to the EventActivation resource to check whether a particular Event type is enabled in a given Environment. If the Event is enabled for an Environment, it updates the Subscription resource with the information. Based on the information, push sends the Event to the lambda or the service.

Validation flow

See the diagram and a step-by-step description of the Event verification process.

1. Kyma user defines a lambda or a service.
2. Kyma user creates a Subscription custom resource.
3. The sub-validator reads the new Subscription.
4. The sub-validator refers to the EventActivation resource to check if the Event in the Subscription is activated for the given Environment.
5. The sub-validator updates the Subscription resource accordingly.

The subscriptions.eventing.kyma.cx Custom Resource Definition (CRD) is a detailed description of the kind of data and the format used to create an event trigger for lambda or microservice in Kyma. After creating a new custom resource, the event trigger is registered in the event bus and events are delivered to the endpoint specified in the custom resource.

To get the up-to-date CRD and show the output in the yaml format, run this command:

kubectl get crd subscriptions.eventing.kyma.cx -o yaml

Sample custom resource

This is a sample resource that creates an event trigger for a lambda with order.created event.

apiVersion: eventing.kyma.cx/v1alpha1
kind: Subscription
metadata:
  name: hello-with-data-subscription
  labels:
    example: event-bus-lambda-subscription
spec:
  endpoint: http://hello-with-data.<environment>:8080/
  push_request_timeout_ms: 2000
  max_inflight: 400
  include_subscription_name_header: true
  event_type: order.created
  event_type_version: v1
  source_id: stage.commerce.kyma.local

Custom resource parameters

This table lists all the possible parameters of a given resource together with their descriptions:

Parameter	Mandatory	Description
metadata.name	YES	Specifies the name of the CR.
spec.endpoint	YES	The HTTP endpoint to which events are delivered as a POST request.
spec.push_request_timeout_ms	YES	The HTTP request timeout. After the timeout has expired, event are redelivered.
spec.max_inflight	YES	The maximum number of concurrent HTTP requests to deliver events.
spec.include_subscription_name_header	YES	Boolean flag indicating if the name of the subscription should be included in the HTTP headers while delivering the event.
spec.event_type	YES	The event type to which the event trigger is registered.
spec.event_type_version	YES	The version of the event type.
spec.source_id	YES	Identifies the external the external solution which sent the event to Kyma.

Management of the Event Bus is based on the custom resources specifically defined for Kyma. Manage all of these resources through kubectl.

Details

This section describes the resource names to use in the kubectl command line, the command syntax, and examples of use.

Resource types

Event Bus operations use the following resources:

Singular name	Plural name
subscription	subscriptions

Syntax

Follow the kubectl syntax, kubectl {command} {type} {name} {flags}, where:

• {command} is any command, such as describe.
• {type} is a resource type, such as clusterserviceclass.
• {name} is the name of a given resource type. Use {name} to make the command return the details of a given resource.
• {flags} specifies the scope of the information. For example, use flags to define the Namespace from which to get the information.

Examples

The following examples show how to create new Subscriptions, list them, and obtain detailed information on their statuses.

• Create a new Subscription directly from the terminal:

   cat <<EOF | kubectl create -f -
   apiVersion: eventing.kyma.cx/v1alpha1
   kind: Subscription
   metadata:
     name: my-subscription
     namespace: stage
   spec:
     endpoint: http://testjs.default:8080/
     push_request_timeout_ms: 2000
     max_inflight: 400
     include_subscription_name_header: true
     event_type: order_created
     event_type_version: v1
     source_id: stage.commerce.kyma.local
EOF

• Get the list of all Subscriptions:

kubectl get subscription --all-namespaces

• Get the list of all Subscriptions with detailed information on the Subscription status:

kubectl get subscriptions -n stage -o=custom-columns=NAME:.metadata.name,STATUS:.status.conditions[*].status,STATUS TYPE:.status.conditions[*].type

In some cases, you can encounter some problems related to eventing. This document introduces several ways to troubleshoot such problems.

General Troubleshooting Guidelines

• If the lambda or the service does not receive any Events, check the following:

  • Confirm that the EventActivation custom resource is in place.
  • Ensure that the webhook defined for the lambda or the service is up and running.
  • Make sure the Events are published.

• If errors appear while sending Events:

  • Check if the publish application is up and running.
  • Make sure that NATS Streaming is up and running.

  If these general guidelines do not help, go to the next section of this document.

Search by tags

You can search traces using tags. Tags are key-value pairs configured for each service.

See the full list of tags for a service from the details of that service's span.

For example, these are the tags for the publish-service:

• event-type
• event-type-ver
• event-id
• source-id

To search the traces, you can use either a single tag such as event-type="order.created", or multiple tags such as event-type="order.created" event-type-ver="v1".

Troubleshooting using Kyma Tracing

Tracing allows you to troubleshoot different problems that you might encounter while using Kyma. Understanding the common scenarios and how the expected traces look like gives you a better grasp on how to quickly use Kyma tracing capabilities to pinpoint the root cause. See the exemplary scenario for reference.

Scenario: I have no microservice or lambda configured to receive an Event

This scenario assumes that there is an Event sent from the external system but there is no lambda or microservice configured with the Event trigger.

As a result, only the trace for the publish and initial services are visible.

In the trace details, you can see the tags for the publish-service.

Scenario: Configured microservice or lambda returns an error

This scenario assumes that there is a microservice or lambda configured to recieve the event trigger. However, due to a bug in the code, the microservice or lambda failed to process the Event.

As a result, you can see the webhook, push, and name-of-lambda services in the trace and they are marked with error.

To see the error details, click on one of the service spans. For example, choose the span for the push service.

Since the Event Bus keeps on retrying to deliver the Event until it is successful, you can see multiple spans for the webhook-service.

To update the subscription CRD, run this command:

kubectl edit crd subscriptions.eventing.kyma.cx

The Event Bus reacts to the changes in subscription CRD, and updates the corresponding NATS-Streaming subscription accordingly.

Note: The current subscription update mechanism recreates a subscription with new specifications. This may result in the loss of messages delivered during the recreation process.