Asset Store

Overview

The Asset Store is a Kubernetes-native solution for storing assets, such as documents, files, images, API specifications, and client-side applications.

This solution has a number of benefits:

  • It is flexible. You can use it for storing various types of assets, such as Markdown documents, ZIP, PNG, or JS files.
  • It is scalable. It allows you to store assets on a production system, using cloud provider storage services. At the same time, you can apply it to local development and use Minio to store assets on-premise.
  • It is multi-cloud and not locked into one vendor. When using the Asset Store in a production system, you can seamlessly switch between different major service providers, such as AWS S3 or Azure Blob.
  • It is location-independent. It allows you to expose files directly to the Internet and replicate them to different regions. This way, you can access them easily, regardless of your location.

Architecture

Resources

The whole concept of the Asset Store relies on the following components:

  • Asset custom resource (CR) is an obligatory CR in which you define the asset you want to store in a given storage bucket. Its definition requires the asset name and mode, the name of the Namespace in which it is available, the address of its web location, and the name of the bucket in which you want to store it. Optionally, you can specify the validation and mutation requirements that the asset must meet before it is stored.

  • Asset Controller (AC) manages the Asset CR lifecycle.

  • Bucket CR is an obligatory CR in which you define the name of the bucket for storing assets.

  • Bucket Controller manages the Bucket CR lifecycle.

  • Validation Service is an optional service which ensures that the asset meets the validation requirements specified in the Asset CR before uploading it to the bucket. The service returns the validation status to the AC.

  • Mutation Service is an optional service which ensures that the asset is modified according to the mutation specification defined in the Asset CR before it is uploaded to the bucket. The service returns the modified asset to the AC.

  • Metadata Service is an optional service which extracts metadata from assets. The metadata information is stored in the CR status. The service returns the asset metadata to the AC.

  • Minio Gateway is a Minio cluster mode which is a production-scalable storage solution. It ensures flexibility of using asset storage services from major cloud providers, including Azure Blob Storage, Amazon S3, and Google Cloud Storage.

Asset flow

This diagram provides an overview of the basic Asset Store workflow and the role of particular components in this process:

  1. The Kyma user creates a bucket through a Bucket CR.
  2. The Bucket Controller listens for new Events and acts upon receiving the Bucket CR creation Event.
  3. The Bucket Controller creates the bucket in the Minio Gateway storage.
  4. The Kyma user creates an Asset CR which specifies the reference to the asset source location and the name of the bucket for storing the asset.
  5. The AC listens for new Events and acts upon receiving the Asset CR creation Event.
  6. The AC reads the CR definition, checks if the Bucket CR is available, and if its name matches the bucket name referenced in the Asset CR. It also verifies if the Bucket CR is in the Ready phase.
  7. If the Bucket CR is available, the AC fetches the asset from the source location provided in the CR. If the asset is a ZIP or TAR file, the AC unpacks and optionally filters the asset before uploading it into the bucket.
  8. Optionally, the AC validates, modifies the asset, or extracts asset's metadata if such a requirement is defined in the Asset CR. The AC communicates with the validation, mutation, and metadata services to validate, modify the asset, or extract asset's metadata according to the specification defined in the Asset CR.
  9. The AC uploads the asset to Minio Gateway, into the bucket specified in the Asset CR.
  10. The AC updates the status of the Asset CR with the storage location of the file in the bucket.

Details

Asset custom resource lifecycle

Learn about the lifecycle of the Asset custom resource (CR) and how its creation, removal, or a change in the bucket reference affects other Asset Store components.

NOTE: This lifecycle also applies to the ClusterAsset CR.

Create an Asset CR

When you create an Asset CR, the Asset Controller (AC) receives a CR creation Event, reads the CR definition, verifies if the bucket exists, downloads the asset, unpacks it, and stores it in Minio Gateway.

Remove an Asset CR

When you remove the Asset CR, the AC receives a CR deletion Event and deletes the CR from Minio Gateway.

Change the bucket reference

When you modify an Asset CR by updating the bucket reference in the Asset CR to a new one while the previous bucket still exists, the lifecycle starts again. The asset is created in a new storage location and this location is updated in the Asset CR.

Unfortunately, this causes duplication of data as the assets from the previous bucket storage are not cleaned up by default. Thus, to avoid multiplication of assets, first remove one Bucket CR and then modify the existing Asset CR with a new bucket reference.

Change the Asset CR specification

When you modify the Asset CR specification, the lifecycle starts again. The previous asset content is removed and no longer available.

Bucket custom resource lifecycle

Learn about the lifecycle of the Bucket custom resource (CR) and how its creation and removal affect other Asset Store components.

NOTE: This lifecycle also applies to the ClusterBucket CR.

Create a Bucket CR

When you create a Bucket CR, the Bucket Controller (BC) receives a CR creation Event and creates a bucket with the name specified in the CR. It is created in the Minio Gateway storage under the {CR_NAME}-{GENERATED_ID} location. The status of the CR contains a reference URL to the created bucket.

Remove a Bucket CR

When you remove the Bucket CR, the BC receives a CR deletion Event and removes the bucket with the whole content from Minio Gateway.

The Asset Controller (AC) also monitors the status of the referenced bucket. The AC checks the Bucket CR status to make sure the bucket exists. If you delete the bucket, the AC receives information that the files are no longer accessible and the bucket was removed. The AC updates the status of the Asset CR to ready: False and removes the asset storage reference. The Asset CR is still available and you can use it later for a new bucket.

Minio and Minio Gateway

The whole concept of the Asset Store relies on Minio as the storage solution. It supports Kyma's manifesto and the "batteries included" rule by providing you with this on-premise solution by default.

Depending on the usage scenario, you can:

  • Use Minio for local development.
  • Store your assets on a production scale using Minio in a Gateway mode.

The Asset Store ensures that both usage scenarios work for Kyma, without additional configuration of the built-in controllers.

Development mode storage

Minio is an open-source asset storage server with Amazon S3 compatible API. You can use it to store various types of assets, such as documents, files, or images.

In the context of the Asset Store, the Asset Controller stores all assets in Minio, in a dedicated storage space.

Access Minio credentials

For security reasons, Minio credentials are generated during Kyma installation and stored inside the Kubernetes Secret object.

  • To get the access key, run:
    Click to copy
    kubectl get secret assetstore-minio -n kyma-system -o jsonpath="{.data.accesskey}" | base64 -D
  • To get the secret key, run:
    Click to copy
    kubectl get secret assetstore-minio -n kyma-system -o jsonpath="{.data.secretkey}" | base64 -D

You can also set Minio credentials directly using values.yaml files. For more details, see the official Minio documentation.

Production storage

For the production purposes, the Asset Store uses Minio Gateway which:

  • Is a multi-cloud solution that offers the flexibility to choose a given cloud provider for the specific Kyma installation, including Azure, Amazon, and Google
  • Allows you to use various cloud providers that support the data replication and CDN configuration
  • Is compatible with Amazon S3 APIs

See this tutorial to learn how to set Minio to the Google Cloud Storage Gateway mode.

Asset Metadata Service

The Asset Metadata Service is an HTTP server that exposes the functionality for extracting metadata from files. It contains a simple HTTP endpoint which accepts multipart/form-data forms. The service extracts front matter YAML metadata from text files of all extensions.

The main purpose of the service is to provide metadata extraction for Asset Store controllers. That's why it is only available inside the cluster. To use it, define metadataWebhookService in Asset and ClusterAsset custom resources.

Front matter metadata

Front matter YAML metadata are YAML properties added at the beginning of a file, between --- lines. The following snippet represents an exemplary Markdown file with metadata specified:

Click to copy
---
title: Example document title
description: Description of the page
order: 3
array:
- foo
- bar
---
## Lorem ipsum
Dolores sit amet

Use the service outside the Kyma cluster

You can expose the service for development purposes. To use the Asset Metadata Service on a local machine, run the following command:

Click to copy
kubectl port-forward deployment/assetstore-asset-metadata-service 3000:3000 -n kyma-system

You can access the service on port 3000.

Metadata files

To extract metadata from files, send the multipart form POST request to the /v1/extract endpoint. Specify the relative or absolute path to the file as a field name. To do the multipart request using curl, run the following command:

Click to copy
curl -v -F foo/foo.md=@foo.md -F bar/bar.yaml=@bar.yaml http://localhost:3000/v1/extract

The result is as follows:

Click to copy
{
"data": [
{
"filePath": "foo/foo.md",
"metadata": {
"no": 3,
"title": "Access logs",
"type": "Details"
}
},
{
"filePath": "bar/bar.yaml",
"metadata": {
"number": 9,
"title": "Hello world",
"url": "https://kyma-project.io"
}
}
]
}

See the OpenAPI specification for the full API documentation. You can use the Swagger Editor to preview and test the API service.

Asset Upload Service

The Asset Upload Service is an HTTP server that exposes the file upload functionality for Minio. It contains a simple HTTP endpoint which accepts multipart/form-data forms. It can upload files to the private and public system buckets.

The main purpose of the service is to provide a solution for hosting static files for components that use the Asset Store, such as the Application Connector. You can also use the Asset Upload Service for development purposes to host files for the Asset Store, without the need to rely on external providers.

System buckets

The Asset Upload Service creates two system buckets, system-private-{generated-suffix} and system-public-{generated-suffix}, where {generated-suffix} is a Unix nano timestamp in the 32-base number system. The public bucket has a read-only policy specified.

To enable the service scaling and to maintain the bucket configuration data between the application restarts, the Asset Upload Service stores its configuration in the assetstore-asset-upload-service ConfigMap.

Once you upload the files, system buckets store them permanently. There is no policy to clean system buckets periodically.

The diagram describes the Asset Upload Service flow:

Use the service outside the Kyma cluster

You can expose the service for development purposes. To use the Asset Upload Service on a local machine, run the following command:

Click to copy
kubectl port-forward deployment/assetstore-asset-upload-service 3000:3000 -n kyma-system

You can access the service on port 3000.

Upload files

To upload files, send the multipart form POST request to the /v1/upload endpoint. The endpoint recognizes the following field names:

  • private that is an array of files to upload to a private system bucket.
  • public that is an array of files to upload to a public system bucket.
  • directory that is an optional directory for storing the uploaded files. If you do not specify it, the service creates a directory with a random name. If the directory and files already exist, the service overwrites them.

To do the multipart request using curl, run the following command:

Click to copy
curl -v -F directory='example' -F private=@sample.md -F private=@text-file.md -F public=@archive.zip http://localhost:3000/v1/upload

The result is as follows:

Click to copy
{
"uploadedFiles": [
{
"fileName": "text-file.md",
"remotePath": "https://minio.kyma.local/private-1b0sjap35m9o0/example/text-file.md",
"bucket": "private-1b0sjap35m9o0",
"size": 212
},
{
"fileName": "archive.zip",
"remotePath": "https://minio.kyma.local/public-1b0sjaq6t6jr8/example/archive.zip",
"bucket": "public-1b0sjaq6t6jr8",
"size": 630
},
{
"fileName": "sample.md",
"remotePath": "https://minio.kyma.local/private-1b0sjap35m9o0/example/sample.md",
"bucket": "private-1b0sjap35m9o0",
"size": 4414
}
]
}

See the OpenAPI specification for the full API documentation. You can use the Swagger Editor to preview and test the API service.

Configuration

Asset Store chart

To configure the Asset Store chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
minio.persistence.enabledEnables Minio persistence. Deactivate it only if you use the Gateway mode. For more details about how to switch to the Minio Gateway mode, see this document.true
minio.environment.MINIO_BROWSEREnables browsing Minio storage. By default, the Minio browser is turned off for security reasons. You can change the value to on to use the browser. If you enable the browser, it is available at https://minio.{DOMAIN}/minio/, for example at https://minio.kyma.local/minio/."off"
minio.resources.requests.memoryDefines requests for memory resources.32Mi
minio.resources.requests.cpuDefines requests for CPU resources.10m
minio.resources.limits.memoryDefines limits for memory resources.128Mi
minio.resources.limits.cpuDefines limits for CPU resources.100m

Asset Store Metadata Service sub-chart

To configure the Asset Store Metadata Service sub-chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
replicaCountDefines the number of service replicas. For more details, see the Kubernetes documentation.1
virtualservice.enabledEnables the use of an external service. If you activate the virtualservice, it is available at https://{VIRTUALSERVICE_NAME}.{DOMAIN}/, for example at https://asset-metadata-service.kyma.local/.false
virtualservice.annotationsDefines the service annotation.{}
virtualservice.nameDefines the service name.asset-metadata-service

Asset Store Controller Manager sub-chart

To configure the Asset Store Controller Manager sub-chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
resources.limits.cpuDefines limits for CPU resources.100m
resources.limits.memoryDefines limits for memory resources.128Mi
resources.requests.cpuDefines requests for CPU resources.100m
resources.requests.memoryDefines requests for memory resources.64Mi
maxClusterAssetConcurrentReconcilesDefines the maximum number of cluster asset concurrent Reconciles which will run.3
maxAssetConcurrentReconcilesDefines the maximum number of asset concurrent Reconciles which will run.3
storeUploadWorkersCountDefines the number of workers used in parallel to upload files to the storage bucket.10
validationWebhookWorkersCountDefines the number of workers used in parallel to validate files.10
mutationWebhookWorkersCountDefines the number of workers used in parallel to mutate files.10

Asset Store Upload Service sub-chart

To configure the Asset Store Upload Service sub-chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
replicaCountDefines the number service replicas.1
virtualservice.enabledEnables the use of an external service. If you activate the virtualservice, it is available at https://{VIRTUALSERVICE_NAME}.{DOMAIN}/, for example at https://asset-upload-service.kyma.local/.false
virtualservice.annotationsDefines the service annotation.{}
virtualservice.nameDefines the service name.asset-upload-service

Minio sub-chart

To configure the Minio sub-chart, override the default values of its values.yaml file.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

As Minio is originally an open-source Helm chart, some of its values in Kyma are already overridden in the main values.yaml of the Asset Store chart.

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
secretkeyDefines the secret key. Add the parameter to set your own secretkey credentials.By default, secretkey is automatically generated.
accesskeyDefines the access key. Add the parameter to set your own accesskey credentials.By default, accesskey is automatically generated.

See the official Minio documentation for a full list of Minio configurable parameters.

Custom Resource

Asset

The assets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define an asset to store in a cloud storage bucket. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd assets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample Asset CR configuration:

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Asset
metadata:
name: my-package-assets
namespace: default
spec:
source:
mode: package
url: https://some.domain.com/structure.zip
filter: .*\.md$
bucketRef:
name: my-bucket

Validation and mutation webhook services

You can also define validation and mutation services:

  • Validation webhook performs the validation of fetched assets before the Asset Controller uploads them into the bucket. It can be a list of several different validation webhooks and all of them should be processed even if one fails. It can refer either to the validation of a specific file against a specification or to the security validation. The validation webhook returns the validation status when the validation completes.
  • Mutation webhook acts similarly to the validation service. The difference is that it mutates the asset instead of just validating it. For example, this can mean asset rewriting through the regex operation or keyvalue, or the modification in the JSON specification. The mutation webhook returns modified files instead of information on the status.
Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Asset
metadata:
name: my-package-assets
namespace: default
spec:
source:
mode: single
parameters:
disableRelativeLinks: "true"
url: https://some.domain.com/main.js
validationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/validate"
filter: \.js$
mutationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/mutate"
filter: \.js$
parameters:
rewrite: keyvalue
pattern: \json|yaml
data:
basePath: /test/v2
metadataWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/extract"
filter: \.js$
bucketRef:
name: my-bucket
status:
phase: Failed
reason: ValidationFailed
message: "The file is not valid against the provided json schema"
lastHeartbeatTime: "2018-01-03T07:38:24Z"
observedGeneration: 1
assetRef:
assets:
- README.md
- directory/subdirectory/file.md
baseUrl: https://minio.kyma.local/test-sample-1b19rnbuc6ir8/asset-sample

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR.
metadata.namespaceYESDefines the Namespace in which the CR is available.
spec.source.modeYESSpecifies if the asset consists of one file or a set of compressed files in the ZIP or TAR formats. Use single for one file and package for a set of files.
spec.source.parametersNOSpecifies a set of parameters for the Asset. For example, use it to define what to render, disable, or modify in the UI. Define it in a valid YAML or JSON format.
spec.source.urlYESSpecifies the location of the file.
spec.source.filterNOSpecifies the regex pattern used to select files to store from the package.
spec.source.validationwebhookserviceNOProvides specification of the validation webhook services.
spec.source.validationwebhookservice.nameYESProvides the name of the validation webhook service.
spec.source.validationwebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.validationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.validationwebhookservice.parametersNOProvides detailed parameters specific for a given validation service and its functionality.
spec.source.validationwebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.source.mutationwebhookserviceNOProvides specification of the mutation webhook services.
spec.source.mutationwebhookservice.nameYESProvides the name of the mutation webhook service.
spec.source.mutationwebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.mutationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookservice.parametersNOProvides detailed parameters specific for a given mutation service and its functionality.
spec.source.mutationwebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.source.metadatawebhookserviceNOProvides specification of the metadata webhook services.
spec.source.metadatawebhookservice.nameYESProvides the name of the metadata webhook service.
spec.source.metadatawebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.metadatawebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.metadatawebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.bucketref.nameYESProvides the name of the bucket for storing the asset.
status.phaseNot applicableThe Asset Controller adds it to the Asset CR. It describes the status of processing the Asset CR by the Asset Controller. It can be Ready, Failed, or Pending.
status.reasonNot applicableProvides the reason why the Asset CR processing failed or is pending.
status.messageNot applicableDescribes a human-readable message on the CR processing progress, success, or failure.
status.lastheartbeattimeNot applicableProvides the last time when the Asset Controller processed the Asset CR.
status.observedGenerationNot applicableSpecifies the most recent generation that the Asset Controller observes.
status.assetrefNot applicableProvides details on the location of the assets stored in the bucket.
status.assetref.assetsNot applicableProvides the relative path to the given asset in the storage bucket.
status.assetref.baseurlNot applicableSpecifies the absolute path to the location of the assets in the storage bucket.

NOTE: The Asset Controller automatically adds all parameters marked as Not applicable to the Asset CR.

These are the resources related to this CR:

Custom resourceDescription
BucketThe Asset CR uses the name of the bucket specified in the definition of the Bucket CR.

These components use this CR:

ComponentDescription
Asset StoreUses the Asset CR for the detailed asset definition, including its location and the name of the bucket in which it is stored.

Bucket

The buckets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define the name of the cloud storage bucket for storing assets. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd buckets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource that defines the storage bucket configuration.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Bucket
metadata:
name: test-sample
namespace: default
spec:
region: "us-east-1"
policy: readonly
status:
lastHeartbeatTime: "2019-02-04T11:50:26Z"
message: Bucket policy has been updated
phase: Ready
reason: BucketPolicyUpdated
remoteName: test-sample-1b19rnbuc6ir8
observedGeneration: 1
url: https://minio.kyma.local/test-sample-1b19rnbuc6ir8

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR which is also used to generate the name of the bucket in the bucket storage.
metadata.namespaceYESSpecifies the Namespace in which the CR is available.
spec.regionNOSpecifies the location of the region under which the Bucket Controller creates the bucket. If the field is empty, the Bucket Controller creates the bucket under the default location.
spec.policyNOSpecifies the type of bucket access. Use none, readonly, writeonly, or readwrite.
status.lastheartbeattimeNot applicableProvides the last time when the Bucket Controller processed the Bucket CR.
status.messageNot applicableDescribes a human-readable message on the CR processing success or failure.
status.phaseNot applicableThe Bucket Controller automatically adds it to the Bucket CR. It describes the status of processing the Bucket CR by the Bucket Controller. It can be Ready or Failed.
status.reasonNot applicableProvides information on the Bucket CR processing success or failure.
status.urlNot applicableProvides the address of the bucket storage under which the asset is available.
status.remoteNameNot applicableProvides the name of the bucket in the storage.
status.observedGenerationNot applicableSpecifies the generation that the Bucket Controller observes.

NOTE: The Bucket Controller automatically adds all parameters marked as Not applicable to the Bucket CR.

These are the resources related to this CR:

Custom resourceDescription
AssetProvides the name of the storage bucket which the Asset CR refers to.

These components use this CR:

ComponentDescription
Asset StoreUses the Bucket CR for the storage bucket definition.

ClusterAsset

The clusterassets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define an asset to store in a cloud storage bucket. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd clusterassets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample ClusterAsset CR configuration:

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterAsset
metadata:
name: my-package-assets
spec:
source:
mode: package
url: https://some.domain.com/structure.zip
filter: .*\.md$
bucketRef:
name: my-bucket

Validation and mutation webhook services

You can also define validation and mutation services:

  • Validation webhook performs the validation of fetched assets before the ClusterAsset Controller uploads them into the bucket. It can be a list of several different validation webhooks and all of them should be processed even if one fails. It can refer either to the validation of a specific file against a specification or to the security validation. The validation webhook returns the validation status when the validation completes.
  • Mutation webhook acts similarly to the validation service. The difference is that it mutates the asset instead of just validating it. For example, this can mean asset rewriting through the regex operation or keyvalue, or the modification in the JSON specification. The mutation webhook returns modified files instead of information on the status.
Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterAsset
metadata:
name: my-package-assets
spec:
source:
mode: single
parameters:
disableRelativeLinks: "true"
url: https://some.domain.com/main.js
validationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/validate"
filter: \.js$
mutationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/mutate"
filter: \.js$
parameters:
rewrite: keyvalue
pattern: \json|yaml
data:
basePath: /test/v2
metadataWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/extract"
filter: \.js$
bucketRef:
name: my-bucket
status:
phase: Failed
reason: ValidationFailed
message: "The file is not valid against the provided json schema"
lastHeartbeatTime: "2018-01-03T07:38:24Z"
observedGeneration: 1
assetRef:
assets:
- README.md
- directory/subdirectory/file.md
baseUrl: https://minio.kyma.local/test-sample-1b19rnbuc6ir8/asset-sample

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR.
spec.source.modeYESSpecifies if the asset consists of one file or a set of compressed files in the ZIP or TAR formats. Use single for one file and package for a set of files.
spec.source.parametersNOSpecifies a set of parameters for the ClusterAsset. For example, use it to define what to render, disable, or modify in the UI. Define it in a valid YAML or JSON format.
spec.source.urlYESSpecifies the location of the file.
spec.source.filterNOSpecifies the regex pattern used to select files to store from the package.
spec.source.validationwebhookserviceNOProvides specification of the validation webhook services.
spec.source.validationwebhookservice.nameYESProvides the name of the validation webhook service.
spec.source.validationwebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.validationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.validationwebhookservice.parametersNOProvides detailed parameters specific for a given validation service and its functionality.
spec.source.validationwebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.source.mutationwebhookserviceNOProvides specification of the mutation webhook services.
spec.source.mutationwebhookservice.nameYESProvides the name of the mutation webhook service.
spec.source.mutationwebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.mutationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookservice.parametersNOProvides detailed parameters specific for a given mutation service and its functionality.
spec.source.mutationwebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.source.metadatawebhookserviceNOProvides specification of the metadata webhook services.
spec.source.metadatawebhookservice.nameYESProvides the name of the metadata webhook service.
spec.source.metadatawebhookservice.namespaceYESProvides the Namespace in which the service is available.
spec.source.metadatawebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.metadatawebhookservice.filterNOSpecifies the regex pattern used to select files sent to the service.
spec.bucketref.nameYESProvides the name of the bucket for storing the asset.
status.phaseNot applicableThe ClusterAsset Controller adds it to the ClusterAsset CR. It describes the status of processing the ClusterAsset CR by the ClusterAsset Controller. It can be Ready, Failed, or Pending.
status.reasonNot applicableProvides the reason why the ClusterAsset CR processing failed or is pending.
status.messageNot applicableDescribes a human-readable message on the CR processing progress, success, or failure.
status.lastheartbeattimeNot applicableProvides the last time when the ClusterAsset Controller processed the ClusterAsset CR.
status.observedGenerationNot applicableSpecifies the most recent generation that the ClusterAsset Controller observes.
status.assetrefNot applicableProvides details on the location of the assets stored in the bucket.
status.assetref.assetsNot applicableProvides the relative path to the given asset in the storage bucket.
status.assetref.baseurlNot applicableSpecifies the absolute path to the location of the assets in the storage bucket.

NOTE: The ClusterAsset Controller automatically adds all parameters marked as Not applicable to the ClusterAsset CR.

These are the resources related to this CR:

Custom resourceDescription
ClusterBucketThe ClusterAsset CR uses the name of the bucket specified in the definition of the ClusterBucket CR.

These components use this CR:

ComponentDescription
Asset StoreUses the ClusterAsset CR for the detailed asset definition, including its location and the name of the bucket in which it is stored.

ClusterBucket

The clusterbuckets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define the name of the cloud storage bucket for storing assets. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd clusterbuckets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource that defines the storage bucket configuration.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterBucket
metadata:
name: test-sample
spec:
region: "us-east-1"
policy: readonly
status:
lastHeartbeatTime: "2019-02-04T11:50:26Z"
message: Bucket policy has been updated
phase: Ready
reason: BucketPolicyUpdated
remoteName: test-sample-1b19rnbuc6ir8
url: https://minio.kyma.local/test-sample-1b19rnbuc6ir8
observedGeneration: 1

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR which is also the prefix of the bucket name in the bucket storage.
spec.regionNOSpecifies the location of the region under which the ClusterBucket Controller creates the bucket. If the field is empty, the ClusterBucket Controller creates the bucket under the default location.
spec.policyNOSpecifies the type of bucket access. Use none, readonly, writeonly, or readwrite.
status.lastheartbeattimeNot applicableProvides the last time when the ClusterBucket Controller processed the ClusterBucket CR.
status.messageNot applicableDescribes a human-readable message on the CR processing success or failure.
status.phaseNot applicableThe ClusterBucket Controller automatically adds it to the ClusterBucket CR. It describes the status of processing the ClusterBucket CR by the ClusterBucket Controller. It can be Ready or Failed.
status.reasonNot applicableProvides information on the ClusterBucket CR processing success or failure.
status.urlNot applicableProvides the address of the bucket storage under which the asset is available.
status.remoteNameNot applicableProvides the name of the bucket in storage.
status.observedGenerationNot applicableSpecifies the most recent generation that the ClusterBucket Controller observes.

NOTE: The ClusterBucket Controller automatically adds all parameters marked as Not applicable to the ClusterBucket CR.

These are the resources related to this CR:

Custom resourceDescription
ClusterAssetProvides the name of the storage bucket which the ClusterAsset CR refers to.

These components use this CR:

ComponentDescription
Asset StoreUses the ClusterBucket CR for the storage bucket definition.

Tutorials

Set Minio to the Google Cloud Storage Gateway mode

By default, you install Kyma with the Asset Store in Minio stand-alone mode. This tutorial shows how to set Minio to the Google Cloud Storage (GCS) Gateway mode using an override.

Prerequisites

Steps

You can set Minio to the GCS Gateway mode both during and after Kyma installation. In both cases, you need to create and configure a Google service account, apply a ConfigMap with an override onto a cluster or Minikube, and trigger the Kyma installation process.

CAUTION: Buckets created in Minio without using Bucket CRs are not recreated or migrated while switching to the Minio Gateway mode.

Create a Google service account

Create a Google service account that has a private key and the Storage Admin role permissions. Follow these steps:

  1. Run the export {VARIABLE}={value} command to set up the following environment variables, where:

    • SA_NAME is the name of the service account.
    • SA_DISPLAY_NAME is the display name of the service account.
    • PROJECT is the GCP project ID.
    • SECRET_FILE is the path to the private key.
    • ROLE is the Storage Admin role bound to the service account.

    Example:

    Click to copy
    export SA_NAME=my-service-account
    export SA_DISPLAY_NAME=service-account
    export PROJECT=service-account-012345
    export SECRET_FILE=my-private-key-path
    export ROLE=roles/storage.admin
  2. When you communicate with Google Cloud for the first time, set context to your Google Cloud project. Run this command:

    Click to copy
    gcloud config set project $PROJECT
  3. Create a service account. Run:
    Click to copy
    gcloud iam service-accounts create $SA_NAME --display-name $SA_DISPLAY_NAME
  4. Add a policy binding for the Storage Admin role to the service account. Run:
    Click to copy
    gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA_NAME@$PROJECT.iam.gserviceaccount.com --role=$ROLE
  5. Create a private key for the service account:
    Click to copy
    gcloud iam service-accounts keys create $SECRET_FILE --iam-account=$SA_NAME@$PROJECT.iam.gserviceaccount.com
  6. Export the private key as an environment variable:
    Click to copy
    export GCS_KEY_JSON=$(< "$SECRET_FILE" base64 | tr -d '\n')

Configure Minio Gateway mode

Apply the following ConfigMap with an override onto a cluster or Minikube. Run:

Click to copy
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: asset-store-overrides
namespace: kyma-installer
labels:
installer: overrides
component: assetstore
kyma-project.io/installation: ""
type: Opaque
data:
minio.gcsgateway.gcsKeyJson: "$GCS_KEY_JSON"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: asset-store-overrides
namespace: kyma-installer
labels:
installer: overrides
component: assetstore
kyma-project.io/installation: ""
data:
minio.persistence.enabled: "false"
minio.gcsgateway.enabled: "true"
minio.gcsgateway.projectId: "$PROJECT"
minio.DeploymentUpdate.type: RollingUpdate
minio.DeploymentUpdate.maxSurge: "0"
minio.DeploymentUpdate.maxUnavailable: "50%"
EOF

CAUTION: When you install Kyma locally from sources, you need to manually add the ConfigMap and the Secret to the installer-config-local.yaml.tpl template located under the installation/resources subfolder before you run the installation script.

Trigger installation

Trigger Kyma installation or update by labeling the Installation custom resource. Run:

Click to copy
kubectl label installation/kyma-installation action=install