API Gateway
Overview
To make your service accessible outside the Kyma cluster, expose it using the Kyma API Gateway Controller, which listens for the custom resource (CR) objects that follow the apirules.gateway.kyma-project.io
CustomResourceDefinition (CRD). Creating a valid CR triggers the API Gateway Controller to create an Istio Virtual Service. Optionally, you can specify the rules attribute of the CR to secure the exposed service with Oathkeeper Access Rules.
The API Gateway Controller allows you to secure the exposed services using JWT tokens issued by Kyma Dex, or OAuth2 tokens issued by the Kyma OAuth2 server. You can secure the entire service, or secure the selected endpoints. Alternatively, you can leave the service unsecured.
NOTE: To learn more, read about the Kyma OAuth2 server.
Architecture
This diagram illustrates the workflow that leads to exposing a service in Kyma:
API Gateway Controller is a component responsible for exposing services. The API Gateway Controller is an application deployed in the
kyma-system
Namespace, implemented according to the Kubernetes Controller principles. The API Gateway Controller listens for newly created custom resources (CR) that follow the setapirule.gateway.kyma-project.io
CustomResourceDefinition (CRD), which describes the details of exposing services in Kyma.Istio Virtual Service specifies the services visible outside the cluster. The API Gateway Controller creates a Virtual Service for the hostname defined in the
apirule.gateway.kyma-project.io
CRD. The convention is to create a hostname using the name of the service as the subdomain, and the domain of the Kyma cluster. To learn more about the Istio Virtual Service concept, read this Istio documentation. To get the list of Virtual Services in Kyma, run:Click to copykubectl get virtualservices.networking.istio.io --all-namespacesOathkeeper Access Rule allows operators to specify authentication requirements for a service. It is an optional resource, created only when the CR specifies the desired authentication method, the trusted token issuer, allowed methods and paths, and required scopes. To learn more, read about ORY Oathkeeper Access Rules.
To get the list of Oathkeeper Access Rules created in Kyma, run:
Click to copykubectl get rules.oathkeeper.ory.sh --all-namespaces
Request flow
This diagram illustrates the request flow for three cases:
- Accessing secured resources with an OAuth2 token
- Accessing secured resources with a JWT token
- Accessing unsecured resources without a token
Accessing secured resources with an OAuth2 token
The developer sends a request to access a secured resource with an OAuth2 access token issued for a registered client. The request is proxied by the Oathkeeper proxy. The proxy identifies the token as an OAuth2 access token and sends it to the registered Token Introspection endpoint in the Hydra OAuth2 server. The OAuth2 server validates the token and returns the outcome of the validation to Oathkeeper. If the validation is successful, Oathkeeper checks the token against the Access Rules that exist for the resource and authorizes the request. Upon successful authorization, the request is forwarded to the resource.
Accessing secured resources with a JWT token
The developer sends a request to access a secured resource with a JWT token. The request is proxied by the Oathkeeper proxy. The proxy identifies the token as a JWT token and fetches the public keys required for token validation from the registered Dex instance. Oathkeeper uses these keys to validate the token. If the validation is successful, Oathkeeper checks the token against the Access Rules that exist for the resource and authorizes the request. Upon successful authorization, the request is forwarded to the resource.
Accessing unsecured resources without a token
The developer sends a request to access a resource without a token. The request is proxied by the Oathkeeper proxy. The proxy checks if there are Access Rules created for the resource, and verifies if it can be accessed without a token. If the resource can be accessed without a token, the request is forwarded to the resource.
Details
Available security options
The API Gateway Controller allows you to secure the services you expose in 3 different ways:
- with OAuth2 tokens
- with JWT tokens
- with both OAuth2 and JWT tokens
If you secure a service with either OAuth2 or JWT tokens, by default you must include a valid OAuth2 or JWT token in the Authorization
header of the call to the service.
If you secure a service with both OAuth2 and JWT, by default the Oathkeeper proxy expects OAuth2 tokens in the Authorization
header of incoming calls. For endpoints secured with JWT, you must define the header from which the system extracts the JWT token for every accessStrategy you define. Set the token_from.location parameter to header:{NAME}
to extract the JWT token from a specific header. You can use any header name different from Authorization
.
Alternatively, you can set the token_from.location parameter to query_parameter:{NAME}
to extract the token from a specific query parameter.
TIP: You can define the location of OAuth2 token through the token_from.location parameter. However, by default, OAuth2 tokens are extracted from the
Authorization
header.
See these sample excerpts from APIRule custom resources that show the rules attribute for services secured with OAuth2, JWT, and an OAuth2 and JWT combination.
- OAuth2
- JWT
- OAuth2 and JWT
TIP: To learn more, read about the APIRule custom resource. You can also follow the tutorial to learn how to expose and secure services.
Whitelisted domains in the API Gateway Controller
The API Gateway Controller uses a whitelist of domains for which it allows to expose services. Every time a user creates a new APIRule custom resource (CR) for a service, the API Gateway Controller checks the domain of the service specified in the CR against the whitelist. If the domain of the service matches a whitelisted entry, the API Gateway Controller creates a Virtual Service and Oathkeeper Access Rules for the service according to the details specified in the CR. If the domain is not whitelisted, the Controller creates neither a Virtual Service nor Oathkeeper Access Rules and, as a result, does not expose the service.
If the domain does not match the whitelist, the API Gateway Controller sets an appropriate validation status on the APIRule CR created for that service.
TIP: For more information, read about the Api CR statuses.
By default, the only whitelisted domain is the domain of the Kyma cluster.
Blacklisted services in the API Gateway Controller
The API Gateway Controller uses a blacklist of services for which it does not create either a Virtual Service or Oathkeeper Access Rules. As a result, these services cannot be exposed. Every time a user creates a new APIRule custom resource (CR) for a service, the API Gateway Controller checks the name of the service specified in the CR against the blacklist. If the name of the service matches a blacklisted entry, the API Gateway Controller sets an appropriate validation status on the APIRule CR created for that service.
TIP: For more information, read about the Api CR statuses.
The blacklist works as a security measure and prevents users from exposing vital internal services of Kubernetes, Istio, and API Server Proxy.
Custom Resource
APIRule
The apirule.gateway.kyma-project.io
CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format the API Gateway Controller listens for. To get the up-to-date CRD and show the output in the yaml
format, run this command:
kubectl get crd apirules.gateway.kyma-project.io -o yaml
Sample custom resource
This is a sample custom resource (CR) that the API Gateway Controller listens for to expose a service. This example has the rules section specified which makes the API Gateway Controller create an Oathkeeper Access Rule for the service.
apiVersion: gateway.kyma-project.io/v1alpha1kind: APIRulemetadata: name: service-securedspec: gateway: kyma-gateway.kyma-system.svc.cluster.local service: name: foo-service port: 8080 host: foo.bar rules: - path: /.* methods: ["GET"] mutators: [] accessStrategies: - handler: oauth2_introspection config: required_scope: ["read"]
This table lists all the possible parameters of a given resource together with their descriptions:
Field | Mandatory | Description |
---|---|---|
metadata.name | YES | Specifies the name of the exposed API. |
spec.gateway | YES | Specifies the Istio Gateway. |
spec.service.name | YES | Specifies the name of the exposed service. |
spec.service.port | YES | Specifies the communication port of the exposed service. |
spec.service.host | YES | Specifies the service's communication address for inbound external traffic. If only the leftmost label is provided, the default domain name will be used. |
spec.rules | YES | Specifies the array of Oathkeeper access rules. |
spec.rules.path | YES | Specifies the path of the exposed service. |
spec.rules.methods | NO | Specifies the list of HTTP request methods available for spec.rules.path. |
spec.rules.mutators | NO | Specifies the array of Oathkeeper mutators. |
spec.rules.accessStrategies | YES | Specifies the array of Oathkeeper authenticators. The supported authenticators are oauth2_introspection , jwt , noop , allow . |
Additional information
When you fetch an existing APIRule CR, the system adds the status section which describes the status of the Virtual Service and the Oathkeeper Access Rule created for this CR. This table lists the fields of the status section.
Field | Description |
---|---|
status.apiRuleStatus | Status code describing the APIRule CR. |
status.virtualServiceStatus.code | Status code describing the Virtual Service. |
status.virtualService.desc | Current state of the Virtual Service. |
status.accessRuleStatus.code | Status code describing the Oathkeeper Rule. |
status.accessRuleStatus.desc | Current state of the Oathkeeper Rule. |
Status codes
These are the status codes used to describe the Virtual Services and Oathkeeper Access Rules:
Code | Description |
---|---|
OK | Resource created. |
SKIPPED | Skipped creating a resource. |
ERROR | Resource not created. |
Tutorials
Expose a service
This tutorial shows how to expose service endpoints and configure different allowed HTTP methods for them using API Gateway Controller.
The tutorial comes with a sample HttpBin service deployment.
Deploy and expose a service
Follow the instruction to deploy an unsecured instance of the HttpBin service and expose it.
Export this value as an environment variable:
Click to copyexport DOMAIN={CLUSTER_DOMAIN}Deploy an instance of the HttpBin service:
Click to copykubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yamlExpose the service by creating an APIRule CR:
Click to copycat <<EOF | kubectl apply -f -apiVersion: gateway.kyma-project.io/v1alpha1kind: APIRulemetadata:name: httpbinspec:service:host: httpbin.$DOMAINname: httpbinport: 8000gateway: kyma-gateway.kyma-system.svc.cluster.localrules:- path: /.*methods: ["GET"]accessStrategies:- handler: noopmutators:- handler: noop- path: /postmethods: ["POST"]accessStrategies:- handler: noopmutators:- handler: noopEOFNOTE: If you are running Kyma on Minikube, add
httpbin.kyma.local
to the entry with Minikube IP in your system's/etc/hosts
file.
Access the exposed resources
Call the endpoint by sending a
GET
request to the HttpBin service:Click to copycurl -ik -X GET https://httpbin.$DOMAIN/ipSend a
POST
request to the HttpBin's/post
endpoint:Click to copycurl -ik -X POST https://httpbin.$DOMAIN/post -d "test data"
These calls return the code 200
response.
Expose and secure a service
This tutorial shows how to expose and secure services or Functions using API Gateway Controller. The controller reacts to an instance of the APIRule custom resource (CR) and creates an Istio Virtual Service and Oathkeeper Access Rules according to the details specified in the CR. To interact with the secured services, the tutorial uses an OAuth2 client registered through the Hydra Maester controller.
The tutorial comes with a sample HttpBin service deployment and a sample Function.
Register an OAuth2 client and get tokens
- Export these values as environment variables:
The name of your client and the Secret which stores the client credentials:
Click to copyexport CLIENT_NAME={YOUR_CLIENT_NAME}The Namespace in which you want to create the client and the Secret that stores its credentials:
Click to copyexport CLIENT_NAMESPACE={YOUR_CLIENT_NAMESPACE}The domain of your cluster:
Click to copyexport DOMAIN={CLUSTER_DOMAIN}
Create an OAuth2 client with
read
andwrite
scopes. Run:Click to copycat <<EOF | kubectl apply -f -apiVersion: hydra.ory.sh/v1alpha1kind: OAuth2Clientmetadata:name: $CLIENT_NAMEnamespace: $CLIENT_NAMESPACEspec:grantTypes:- "client_credentials"scope: "read write"secretName: $CLIENT_NAMEEOFExport the credentials of the created client as environment variables. Run:
Click to copyexport CLIENT_ID="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_id}' | base64 --decode)"export CLIENT_SECRET="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_secret}' | base64 --decode)"Encode your client credentials and export them as an environment variable:
Click to copyexport ENCODED_CREDENTIALS=$(echo -n "$CLIENT_ID:$CLIENT_SECRET" | base64)Get tokens to interact with secured resources using client credentials flow:
- Token with "read" scope
- Token with "write" scope
Deploy, expose, and secure the sample resources
Follow the instructions in the tabs to deploy an instance of the HttpBin service or a sample Function, expose them, and secure them with Oauth2 scopes.
- HttpBin - secure endpoints of a service
- Secure a Function
CAUTION: When you secure a service, don't create overlapping Access Rules for paths. Doing so can cause unexpected behavior and reduce the security of your implementation.
Access the secured resources
Follow the instructions in the tabs to call the secured service or Functions using the tokens issued for the client you registered.
- Call secured endpoints of a service
- Call the secured Function
Troubleshooting
Overview
The troubleshooting section aims to identify the most common recurring issues with the Kyma API Gateway component, as well as the most suitable solutions to these issues.
If you can't find a solution that suits your case, don't hesitate to create a GitHub issue or use the #security Slack channel to get direct support from the community.
Cannot connect to a service exposed by an APIRule
Basic diagnostics
API Gateway is a Kubernetes controller which operates on APIRule custom resources. To diagnose the problems, inspect the status
field of the APIRule CR:
kubectl describe apirules.gateway.kyma-project.io -n {NAMESPACE} {NAME}
If the status is Error
, edit the APIRule and fix issues described in .Status.APIRuleStatus.Desc
. If you still encounter issues, make sure the API Gateway, Hydra and Oathkeeper are running or take a look at other, more specific troubleshooting guides.
401 Unauthorized or 403 Forbidden
If you reach your service and get 401 Unauthorized
or 403 Forbidden
in response, make sure that:
You are using an access token with proper scopes and it is active:
Export the credentials of your OAuth2Client as environment variables:
Click to copyexport CLIENT_ID="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_id}' | base64 --decode)"export CLIENT_SECRET="$(kubectl get secret -n $CLIENT_NAMESPACE $CLIENT_NAME -o jsonpath='{.data.client_secret}' | base64 --decode)"Encode your client credentials and export them as an environment variable:
Click to copyexport ENCODED_CREDENTIALS=$(echo -n "$CLIENT_ID:$CLIENT_SECRET" | base64)Check access token status:
Click to copycurl -X POST "https://oauth2.{DOMAIN}/oauth2/introspect" -H "Authorization: Basic $ENCODED_CREDENTIALS" -F "token={ACCESS_TOKEN}"Generate a new access token if needed.
Your client from OAuth2Client resource is registered properly in Hydra OAuth2 and OpenID Connect server. You need to call the Hydra administrative endpoint
/client
from inside of the cluster. Follow this steps:Fetch the Client ID from Secret specified in the OAuth2Client resource:
Click to copykubectl get secrets {SECRET_NAME} -n {SECRET_NAMESPACE} -o jsonpath='{ .data.client_id }' | base64 --decodeCreate a simple curl Pod:
Click to copy---apiVersion: v1kind: Podmetadata:labels:app: ory-curlname: ory-curlnamespace: {SECRET_NAMESPACE}spec:containers:- name: curlimage: alpineterminationMessagePolicy: "FallbackToLogsOnError"command:- /bin/sh- -c- |apk add curl jqcurl ory-hydra-admin.kyma-system.svc.cluster.local:4445/clients | jq '.'Check logs from the
ory-curl
Pod:Click to copykubectl logs -n {SECRET_NAMESPACE} ory-curl curlIf the Client ID from step 1 is not available on the clients list, make sure Hydra has access to the database and/or restart the Hydra Measter Pod. You can check the logs using the following commands:
Click to copy# Check logs from the Hydra-Maester controller applicationkubectl logs -n kyma-system -l "app.kubernetes.io/name=hydra-maester" -c hydra-maester# Example output2020-05-04T12:19:04.472Z INFO controller-runtime.controller Starting EventSource {"controller": "oauth2client", "source": "kind source: /, Kind="}2020-05-04T12:19:04.472Z INFO setup starting manager2020-05-04T12:19:04.573Z INFO controller-runtime.controller Starting Controller {"controller": "oauth2client"}2020-05-04T12:19:04.673Z INFO controller-runtime.controller Starting workers {"controller": "oauth2client", "worker count": 1}2020-05-04T12:26:30.819Z INFO controllers.OAuth2Client using default client2020-05-04T12:26:30.835Z INFO controllers.OAuth2Client using default client# This log informs that a client has been created, and should be visible within hydra2020-05-04T12:26:31.468Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "oauth2client", "request": "test-ns/test-client"}# Check logs from the Hydra applicationkubectl logs -n kyma-system -l "app.kubernetes.io/name=hydra" -c hydra
404 Not Found
If you reach your service and get 404 Not Found
in response, make sure that:
Proper Oathkeeper Rule has been created:
Click to copykubectl get rules.oathkeeper.ory.sh -n {NAMESPACE}TIP: Name of the Rule consists of the name of the APIRule and a random suffix.
Proper VirtualService has been created:
Click to copykubectl get virtualservices.networking.istio.io -n {NAMESPACE}TIP: Name of the VirtualService consists of the name of the APIRule and a random suffix.