Asset Store

Overview

The Asset Store is a Kubernetes-native solution for storing assets, such as documents, files, images, API specifications, and client-side applications.

This solution has a number of benefits:

  • It is flexible. You can use it for storing various types of assets, such as Markdown documents, ZIP, PNG, or JS files.
  • It is scalable. It allows you to store assets on a production system, using cloud provider storage services. At the same time, you can apply it to local development and use Minio to store assets on-premise.
  • It is multi-cloud and not locked into one vendor. When using the Asset Store in a production system, you can seamlessly switch between different major service providers, such as AWS S3 or Azure Blob.
  • It is location-independent. It allows you to expose files directly to the Internet and replicate them to different regions. This way, you can access them easily, regardless of your location.

Architecture

Resources

The whole concept of the Asset Store relies on the following components:

  • Asset custom resource (CR) is an obligatory CR in which you define the asset you want to store in a given storage bucket. Its definition requires the asset name and mode, the name of the Namespace in which it is available, the address of its web location, and the name of the bucket in which you want to store it. Optionally, you can specify the validation and mutation requirements that the asset must meet before it is stored.

  • Asset Controller (AC) manages the Asset CR lifecycle.

  • Bucket CR is an obligatory CR in which you define the name of the bucket for storing assets.

  • Bucket Controller manages the Bucket CR lifecycle.

  • Validation Service is an optional service which ensures that the asset meets the validation requirements specified in the Asset CR before uploading it to the bucket. The service returns the validation status to the AC.

  • Mutation Service is an optional service which ensures that the asset is modified according to the mutation specification defined in the Asset CR before it is uploaded to the bucket. The service returns the modified asset to the AC.

  • Minio Gateway is a Minio cluster mode which is a production-scalable storage solution. It ensures flexibility of using asset storage services from major cloud providers, including Azure Blob Storage, Amazon S3, and Google Cloud Storage.

Asset flow

This diagram provides an overview of the basic Asset Store workflow and the role of particular components in this process:

  1. The Kyma user creates a bucket through a Bucket CR.
  2. The Bucket Controller listens for new Events and acts upon receiving the Bucket CR creation Event.
  3. The Bucket Controller creates the bucket in the Minio Gateway storage.
  4. The Kyma user creates an Asset CR which specifies the reference to the asset source location and the name of the bucket for storing the asset.
  5. The AC listens for new Events and acts upon receiving the Asset CR creation Event.
  6. The AC reads the CR definition, checks if the Bucket CR is available, and if its name matches the bucket name referenced in the Asset CR. It also verifies if the Bucket CR is in the Ready phase.
  7. If the Bucket CR is available, the AC fetches the asset from the source location provided in the CR. If the asset is a ZIP or TAR file, the AC unpacks and optionally filters the asset before uploading it into the bucket.
  8. Optionally, the AC validates or modifies the asset if such a requirement is defined in the Asset CR. The AC communicates with the validation and mutation services and validates or modifies the asset according to the specification defined in the Asset CR.
  9. The AC uploads the asset to Minio Gateway, into the bucket specified in the Asset CR.
  10. The AC updates the status of the Asset CR with the storage location of the file in the bucket.

Details

Asset custom resource lifecycle

Learn about the lifecycle of the Asset custom resource (CR) and how its creation, removal, or a change in the bucket reference affects other Asset Store components.

Create an Asset CR

When you create an Asset CR, the Asset Controller (AC) receives a CR creation Event, reads the CR definition, verifies if the bucket exists, downloads the asset, unpacks it, and stores it in Minio Gateway.

Remove an Asset CR

When you remove the Asset CR, the AC receives a CR deletion Event and deletes the CR from Minio Gateway.

Change the bucket reference

When you modify an Asset CR by updating the bucket reference in the Asset CR to a new one while the previous bucket still exists, the lifecycle starts again. The asset is created in a new storage location and this location is updated in the Asset CR.

Unfortunately, this causes duplication of data as the assets from the previous bucket storage are not cleaned up by default. Thus, to avoid multiplication of assets, first remove one Bucket CR and then modify the existing Asset CR with a new bucket reference.

Change the Asset CR specification

When you modify the Asset CR specification, the lifecycle starts again. The previous asset content is removed and no longer available.

Bucket custom resource lifecycle

Learn about the lifecycle of the Bucket custom resource (CR) and how its creation and removal affect other Asset Store components.

Create a Bucket CR

When you create a Bucket CR, the Bucket Controller (BC) receives a CR creation Event and creates a bucket with the name specified in the CR. It is created in the Minio Gateway storage under the ns-{NAMESPACE_NAME}-{CR_NAME} location. The status of the CR contains a reference URL to the created bucket.

Remove a Bucket CR

When you remove the Bucket CR, the BC receives a CR deletion Event and removes the bucket with the whole content from Minio Gateway.

The Asset Controller (AC) also monitors the status of the referenced bucket. The AC checks the Bucket CR status to make sure the bucket exists. If you delete the bucket, the AC receives information that the files are no longer accessible and the bucket was removed. The AC updates the status of the Asset CR to ready: False and removes the asset storage reference. The Asset CR is still available and you can use it later for a new bucket.

Minio and Minio Gateway

The whole concept of the Asset Store relies on Minio as the storage solution. It supports the Kyma's manifesto and the "batteries included" rule by providing you with this on-premise solution by default.

Depending on the usage scenario, you can:

  • Use Minio for local development.
  • Store your assets on a production scale using Minio in a Gateway mode.

The Asset Store ensures that both usage scenarios work for Kyma, without additional configuration of the in-build controllers.

Development mode storage

Minio is an open-source asset storage server with Amazon S3 compatible API. You can use it to store various types of assets, such as documents, files, or images.

In the context of the Asset Store, the Asset Controller stores all assets in Minio, in a dedicated storage space.

Production storage

For the production purposes, the Asset Store uses Minio Gateway which:

  • Is a multi-cloud solution that offers the flexibility to choose a given cloud provider for the specific Kyma installation, including Azure, Amazon, and Google
  • Allows you to use various cloud providers that support the data replication and CDN configuration
  • Is compatible with Amazon S3 APIs

Asset Upload Service

The Asset Upload Service is an HTTP server that exposes the file upload functionality for Minio. It contains a simple HTTP endpoint which accepts multipart/form-data forms. It can upload files to the private and public system buckets.

The main purpose of the service is to provide a solution for hosting static files for components that use the Asset Store, such as the Application Connector. You can also use the Asset Upload Service for development purposes to host files for the Asset Store, without the need to rely on external providers.

System buckets

The Asset Upload Service creates two system buckets, system-private-{generated-suffix} and system-public-{generated-suffix}, where {generated-suffix} is a Unix nano timestamp in the 32-base number system. The public bucket has a read-only policy specified.

To enable the service scaling and to maintain the bucket configuration data between the application restarts, the Asset Upload Service stores its configuration in the assetstore-asset-upload-service ConfigMap.

Once you upload the files, system buckets store them permanently. There is no policy to clean system buckets periodically.

Use the service outside the Kyma cluster

You can expose the service for development purposes. To use the Asset Upload Service on a local machine, run the following command:

Click to copy
kubectl port-forward deployment/assetstore-asset-upload-service 3000:3000 -n kyma-system

You can access the service on port 3000.

Upload files

To upload files, send the multipart form POST request to the /v1/upload endpoint. The endpoint recognizes the following field names:

  • private that is an array of files to upload to a private system bucket.
  • public that is an array of files to upload to a public system bucket.
  • directory that is an optional directory for storing the uploaded files. If you do not specify it, the service creates a directory with a random name. If the directory and files already exist, the service overwrites them.

To do the multipart request using curl, run the following command in this repository:

Click to copy
curl -v -F directory='example' -F private=@sample.md -F private=@text-file.md -F public=@archive.zip http://localhost:3000/v1/upload

The result is as follows:

Click to copy
{
"uploadedFiles": [
{
"fileName": "text-file.md",
"remotePath": "https://minio.kyma.local/private-1b0sjap35m9o0/example/text-file.md",
"bucket": "private-1b0sjap35m9o0",
"size": 212
},
{
"fileName": "archive.zip",
"remotePath": "https://minio.kyma.local/public-1b0sjaq6t6jr8/example/archive.zip",
"bucket": "public-1b0sjaq6t6jr8",
"size": 630
},
{
"fileName": "sample.md",
"remotePath": "https://minio.kyma.local/private-1b0sjap35m9o0/example/sample.md",
"bucket": "private-1b0sjap35m9o0",
"size": 4414
}
]
}

See the Swagger specification for the full API documentation. You can use the Swagger Editor to preview and test the API service.

Custom Resource

Asset

The assets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define an asset to store in a cloud storage bucket. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd assets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource (CR) that provides details of the bucket for storing assets.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Asset
metadata:
name: my-package-assets
namespace: default
spec:
source:
mode: package
url: https://some.domain.com/structure.zip
filter: .*\.md$
bucketRef:
name: my-bucket

Validation and mutation webhook services

You can also define validation and mutation services:

  • Validation webhook performs the validation of fetched assets before the Asset Controller uploads them into the bucket. It can be a list of several different validation webhooks and all of them should be processed even if one fails. It can refer either to the validation of a specific file against a specification or to the security validation. The validation webhook returns the validation status when the validation completes.
  • Mutation webhook acts similarly to the validation service. The difference is that it mutates the asset instead of just validating it. For example, this can mean asset rewriting through the regex operation or keyvalue, or the modification in the JSON specification. The mutation webhook returns modified files instead of information on the status.
Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Asset
metadata:
name: my-package-assets
namespace: default
spec:
source:
mode: single
url: https://some.domain.com/main.js
validationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/validate"
mutationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/mutate"
metadata:
rewrite: keyvalue
pattern: \json|yaml
data:
basePath: /test/v2
bucketRef:
name: my-bucket
status:
phase: Failed
reason: ValidationFailed
message: "The file is not valid against the provided json schema"
lastHeartbeatTime: "2018-01-03T07:38:24Z"
observedGeneration: 1
assetRef:
assets:
- README.md
- directory/subdirectory/file.md
baseUrl: https://minio.kyma.local/ns-default-test-sample-1b19rnbuc6ir8/asset-sample

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR.
metadata.namespaceYESDefines the Namespace in which the CR is available.
spec.source.modeYESSpecifies if the asset consists of one file or a set of compressed files in the ZIP or TAR formats. Use single for one file and package for a set of files.
spec.source.urlYESSpecifies the location of the file.
spec.source.filterNOSpecifies the regex pattern used to select files to store from the package.
spec.source.validationwebhookserviceNOProvides specification of the validation webhook services.
spec.source.validationwebhookservice.nameNOProvides the name of the validation webhook service.
spec.source.validationwebhookservice.namespaceNOProvides the Namespace in which the service is available. It must be the same as the asset's Namespace.
spec.source.validationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookserviceNOProvides specification of the mutation webhook services.
spec.source.mutationwebhookservice.nameNOProvides the name of the mutation webhook service.
spec.source.mutationwebhookservice.namespaceNOProvides the Namespace in which the service is available. It must be the same as the asset's Namespace.
spec.source.mutationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookservice.metadataNOProvides detailed metadata specific for a given mutation service and its functionality.
spec.bucketref.nameYESProvides the name of the bucket for storing the asset.
status.phaseNot applicableThe Asset Controller adds it to the Asset CR. It describes the status of processing the Asset CR by the Asset Controller. It can be Ready, Failed, or Pending.
status.reasonNot applicableProvides the reason why the Asset CR processing failed or is pending.
status.messageNot applicableDescribes a human-readable message on the CR processing progress, success, or failure.
status.lastheartbeattimeNot applicableProvides the last time when the Asset Controller processed the Asset CR.
status.observedGenerationNot applicableSpecifies the most recent generation that the Asset Controller observes.
status.assetrefNot applicableProvides details on the location of the assets stored in the bucket.
status.assetref.assetsNot applicableProvides the relative path to the given asset in the storage bucket.
status.assetref.baseurlNot applicableSpecifies the absolute path to the location of the assets in the storage bucket.

NOTE: The Asset Controller automatically adds all parameters marked as Not applicable to the Asset CR.

These are the resources related to this CR:

Custom resourceDescription
BucketThe Asset CR uses the name of the bucket specified in the definition of the Bucket CR.

These components use this CR:

ComponentDescription
Asset StoreUses the Asset CR for the detailed asset definition, including its location and the name of the bucket in which it is stored.

Bucket

The buckets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define the name of the cloud storage bucket for storing assets. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd buckets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource that defines the storage bucket configuration.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: Bucket
metadata:
name: test-sample
namespace: default
spec:
region: "us-east-1"
policy: readonly
status:
lastHeartbeatTime: "2019-02-04T11:50:26Z"
message: Bucket policy has been updated
phase: Ready
reason: BucketPolicyUpdated
remoteName: ns-default-test-sample-1b19rnbuc6ir8
observedGeneration: 1
url: https://minio.kyma.local/ns-default-test-sample-1b19rnbuc6ir8

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR which is also used to generate the name of the bucket in the bucket storage.
metadata.namespaceYESSpecifies the Namespace in which the CR is available. It is also used to generate the name of the bucket in the bucket storage.
spec.regionNOSpecifies the location of the region under which the Bucket Controller creates the bucket. If the field is empty, the Bucket Controller creates the bucket under the default location.
spec.policyNOSpecifies the type of bucket access. Use none, readonly, writeonly, or readwrite.
status.lastheartbeattimeNot applicableProvides the last time when the Bucket Controller processed the Bucket CR.
status.messageNot applicableDescribes a human-readable message on the CR processing success or failure.
status.phaseNot applicableThe Bucket Controller automatically adds it to the Bucket CR. It describes the status of processing the Bucket CR by the Bucket Controller. It can be Ready or Failed.
status.reasonNot applicableProvides information on the Bucket CR processing success or failure.
status.urlNot applicableProvides the address of the bucket storage under which the asset is available.
status.remoteNameNot applicableProvides the name of the bucket in the storage.
status.observedGenerationNot applicableSpecifies the generation that the Bucket Controller observes.

NOTE: The Bucket Controller automatically adds all parameters marked as Not applicable to the Bucket CR.

These are the resources related to this CR:

Custom resourceDescription
AssetProvides the name of the storage bucket which the Asset CR refers to.

These components use this CR:

ComponentDescription
Asset StoreUses the Bucket CR for the storage bucket definition.

ClusterAsset

The clusterassets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define an asset to store in a cloud storage bucket. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd clusterassets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource (CR) that provides details of the bucket for storing assets.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterAsset
metadata:
name: my-package-assets
spec:
source:
mode: package
url: https://some.domain.com/structure.zip
filter: .*\.md$
bucketRef:
name: my-bucket

Validation and mutation webhook services

You can also define validation and mutation services:

  • Validation webhook performs the validation of fetched assets before the ClusterAsset Controller uploads them into the bucket. It can be a list of several different validation webhooks and all of them should be processed even if one fails. It can refer either to the validation of a specific file against a specification or to the security validation. The validation webhook returns the validation status when the validation completes.
  • Mutation webhook acts similarly to the validation service. The difference is that it mutates the asset instead of just validating it. For example, this can mean asset rewriting through the regex operation or keyvalue, or the modification in the JSON specification. The mutation webhook returns modified files instead of information on the status.
Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterAsset
metadata:
name: my-package-assets
spec:
source:
mode: single
url: https://some.domain.com/main.js
validationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/validate"
mutationWebhookService:
- name: swagger-operations-svc
namespace: default
endpoint: "/mutate"
metadata:
rewrite: keyvalue
pattern: \json|yaml
data:
basePath: /test/v2
bucketRef:
name: my-bucket
status:
phase: Failed
reason: ValidationFailed
message: "The file is not valid against the provided json schema"
lastHeartbeatTime: "2018-01-03T07:38:24Z"
observedGeneration: 1
assetRef:
assets:
- README.md
- directory/subdirectory/file.md
baseUrl: https://minio.kyma.local/test-sample-1b19rnbuc6ir8/asset-sample

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR.
spec.source.modeYESSpecifies if the asset consists of one file or a set of compressed files in the ZIP or TAR formats. Use single for one file and package for a set of files.
spec.source.urlYESSpecifies the location of the file.
spec.source.filterNOSpecifies the regex pattern used to select files to store from the package.
spec.source.validationwebhookserviceNOProvides specification of the validation webhook services.
spec.source.validationwebhookservice.nameNOProvides the name of the validation webhook service.
spec.source.validationwebhookservice.namespaceNOProvides the Namespace in which the service is available. It must be the same as the asset's Namespace.
spec.source.validationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookserviceNOProvides specification of the mutation webhook services.
spec.source.mutationwebhookservice.nameNOProvides the name of the mutation webhook service.
spec.source.mutationwebhookservice.namespaceNOProvides the Namespace in which the service is available. It must be the same as the asset's Namespace.
spec.source.mutationwebhookservice.endpointNOSpecifies the endpoint to which the service sends calls.
spec.source.mutationwebhookservice.metadataNOProvides detailed metadata specific for a given mutation service and its functionality.
spec.bucketref.nameYESProvides the name of the bucket for storing the asset.
status.phaseNot applicableThe ClusterAsset Controller adds it to the ClusterAsset CR. It describes the status of processing the ClusterAsset CR by the ClusterAsset Controller. It can be Ready, Failed, or Pending.
status.reasonNot applicableProvides the reason why the ClusterAsset CR processing failed or is pending.
status.messageNot applicableDescribes a human-readable message on the CR processing progress, success, or failure.
status.lastheartbeattimeNot applicableProvides the last time when the ClusterAsset Controller processed the ClusterAsset CR.
status.observedGenerationNot applicableSpecifies the most recent generation that the ClusterAsset Controller observes.
status.assetrefNot applicableProvides details on the location of the assets stored in the bucket.
status.assetref.assetsNot applicableProvides the relative path to the given asset in the storage bucket.
status.assetref.baseurlNot applicableSpecifies the absolute path to the location of the assets in the storage bucket.

NOTE: The ClusterAsset Controller automatically adds all parameters marked as Not applicable to the ClusterAsset CR.

These are the resources related to this CR:

Custom resourceDescription
ClusterBucketThe ClusterAsset CR uses the name of the bucket specified in the definition of the ClusterBucket CR.

These components use this CR:

ComponentDescription
Asset StoreUses the ClusterAsset CR for the detailed asset definition, including its location and the name of the bucket in which it is stored.

ClusterBucket

The clusterbuckets.assetstore.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define the name of the cloud storage bucket for storing assets. To get the up-to-date CRD and show the output in the yaml format, run this command:

Click to copy
kubectl get crd clusterbuckets.assetstore.kyma-project.io -o yaml

Sample custom resource

This is a sample resource that defines the storage bucket configuration.

Click to copy
apiVersion: assetstore.kyma-project.io/v1alpha2
kind: ClusterBucket
metadata:
name: test-sample
spec:
region: "us-east-1"
policy: readonly
status:
lastHeartbeatTime: "2019-02-04T11:50:26Z"
message: Bucket policy has been updated
phase: Ready
reason: BucketPolicyUpdated
remoteName: test-sample-1b19rnbuc6ir8
url: https://minio.kyma.local/test-sample-1b19rnbuc6ir8
observedGeneration: 1

Custom resource parameters

This table lists all possible parameters of a given resource together with their descriptions:

ParameterMandatoryDescription
metadata.nameYESSpecifies the name of the CR which is also the prefix of the bucket name in the bucket storage.
spec.regionNOSpecifies the location of the region under which the ClusterBucket Controller creates the bucket. If the field is empty, the ClusterBucket Controller creates the bucket under the default location.
spec.policyNOSpecifies the type of bucket access. Use none, readonly, writeonly, or readwrite.
status.lastheartbeattimeNot applicableProvides the last time when the ClusterBucket Controller processed the ClusterBucket CR.
status.messageNot applicableDescribes a human-readable message on the CR processing success or failure.
status.phaseNot applicableThe ClusterBucket Controller automatically adds it to the ClusterBucket CR. It describes the status of processing the ClusterBucket CR by the ClusterBucket Controller. It can be Ready or Failed.
status.reasonNot applicableProvides information on the ClusterBucket CR processing success or failure.
status.urlNot applicableProvides the address of the bucket storage under which the asset is available.
status.remoteNameNot applicableProvides the name of the bucket in storage.
status.observedGenerationNot applicableSpecifies the most recent generation that the ClusterBucket Controller observes.

NOTE: The ClusterBucket Controller automatically adds all parameters marked as Not applicable to the ClusterBucket CR.

These are the resources related to this CR:

Custom resourceDescription
ClusterAssetProvides the name of the storage bucket which the ClusterAsset CR refers to.

These components use this CR:

ComponentDescription
Asset StoreUses the ClusterBucket CR for the storage bucket definition.