Supply Chain Security on Amazon Elastic Kubernetes Service (Amazon EKS) using AWS Key Management Service (AWS KMS), Kyverno, and Cosign

In this fast paced world, any software can introduce vulnerabilities into a supply chain. This is especially true for modern software applications that can have many dependencies and are built for distributed systems like Kubernetes. As these systems get more complex, it is critical to put best practices and checks in place to ensure artifact integrity. In this post we will demonstrate how you can implement supply chain security using open source tools on Amazon Elastic Kubernetes Service (Amazon EKS) with AWS Key Management Service (AWS KMS) and Cosign with Kyverno.

Supply Chain Overview

A supply chain is a system of activities involved in handling, distributing, manufacturing and processing goods in order to move resources from a vendor into the hands of the final consumer. Supply chain attack is a broad term, however according to Wikipedia:

“A supply chain attack involves physically tampering with electronics (computers, ATMs, power systems, factory data networks) in order to install undetectable malware for the purpose of bringing harm to a player further down the supply chain network.”

Supply chains require more than one linked process, and their security relies on the validation and verification of each process. Preventing supply chain attacks is a rapidly developing field and consensus on “best practices” and security has been lacking. The Cloud Native Computing Foundation (CNCF) Security Technical Advisory Group (TAG) has published a new paper, Software Supply Chain Security Best Practices, designed to provide the cloud native and open source communities with a holistic approach to architect a secure supply chain regardless of whether they are a software producer or consumer.

The approach outlined by the CNCF Security TAG group has four key elements:

1. Verification – Establishing and verifying “trust” at every step in the process through a combination of code-signing, metadata, and cryptographic validation.
2. Automation – Leveraging automation helps to ensure that processes are deterministic, reinforcing the attestation and verification mechanisms we rely on for supply chain security. Everything that can be automated should be automated and documented.
3. Authorization in Controlled Environments – Every step in the software build and supply chain process should be clearly defined with a limited scope. Next, deriving from this design, every operator within the supply chain (human or machine) must have a clearly defined role with minimum permission.
4. Secure Authentication – Finally, every entity in our system must engage in “mutual authentication.” This means that no human, software process, or machine should be trusted to be who they say they are. They must demonstrate through a hardened technique (such as multi-factor authentication) that they are in fact who they purport to be.

These principles can be operationalized across first-party source code repositories, third-party dependencies, build pipelines, artifact repositories, and deployments. Each of these stages involves different considerations and requirements, and comes with its own set of best practices.

In this post, using Cosign with AWS KMS we are first going to generate a signed and verified public/private key. Then we will deploy Kyverno ImageVerify policy on an existing Amazon EKS cluster. This policy will check the signature of container images and ensure it’s only allowed to deploy/run containers that have been signed against the provided AWS KMS public key.

Kyverno Overview

Kyverno is an open source policy engine designed for Kubernetes. Kyverno enables users to manage policies as Kubernetes resources without needing them to learn a new language to write policies. This enables users to keep using familiar tools such as kubectl, git, and kustomize for policies management. It can validate, mutate, and generate Kubernetes resources plus ensure Open Container Initiative (OCI)-compliant image supply chain security.

Kyverno runs as a dynamic admission controller in a Kubernetes cluster. It receives validating and mutating admission webhook callbacks from the API server and applies matching policies to return results that enforce admission policies or reject requests.

Cosign Overview

Cosign is a new open source tool to manage the process of signing and verifying container images. It is developed as part of the sigstore project and aims “to make signatures invisible infrastructure”. With images signed by Cosign, users do not need to change their infrastructure to store the public signing key. With Cosign, the signatures directly appear as tags of the image, linked to the associated image via the digest:

Cosign supports:

• Hardware and KMS signing
• Bring-your-own Public Key Infrastructure (PKI)
• Fulcio
• Rekor.

Since Cosign supports using a KMS provider to generate and sign keys, in this blog we will use AWS KMS.

Prerequisites

• An AWS account with admin privileges: We will assume you already have an AWS account that has an Amazon EKS cluster and an Amazon Elastic Container Registry (Amazon ECR) with admin privileges.
• Command line tools. Mac/Linux users need to install the latest version of AWS Command Line Interface (AWS CLI), kubectl, and Cosign on their workstation. Windows users need to create an AWS Cloud9 environment and then install these CLI tools inside it.

Steps to Implement Supply Chain security on Amazon EKS using AWS KMS and Cosign with Kyverno :

Step 1: Generate keys using AWS KMS

To generate keys using a KMS provider, use the cosign generate-key-pair command with the --kms flag.

For example:

$ cosign generate-key-pair --kms awskms:///container-image-verification

The above command will store the public key in the cosign.pub file. You can verify the key generation by running this command:

aws kms list-aliases | grep -i container-image verification

The public key can be retrieved later with:

$ cosign public-key --key awskms:///alias/container-image-verification

Step 2: Build and Sign a Container Image using Cosign

For the purpose of this blog we will use an existing Amazon Elastic Container Registry (ECR) named “supply-chain-security”. You can create a new one by following the steps outlined here.

Use the following steps to authenticate and push an image to your repository.

• Retrieve an authentication token and authenticate your Docker client to your registry.

aws ecr get-login-password —region us-east-1 | docker login —username AWS —password-stdin <AWS Account ID>.dkr.ecr.us-east-1.amazonaws.com

Note: If you receive an error using the AWS CLI, make sure that you have the latest version of the AWS CLI and Docker installed.

• For this example, we will use an “nginx” image, If you would like you can build your own custom Docker image from scratch by following the instructions from here. You can skip this step if your image is already built.
• Tag your image so you can push the image to the repository and then push it, by running the following commands (substituting your AWS Account ID as indicated):

docker tag nginx:latest <AWS Account ID>.dkr.ecr.us-east-1.amazonaws.com/supply-chain-security:latest
docker push <AWS Account ID>.dkr.ecr.us-east-1.amazonaws.com/supply-chain-security:latest

• After pushing the image to Amazon ECR, sign the image using the public key named “container-image-verification” we created in step 1. Run the following cosign command to sign the “supply-chain-security” image and re-upload it to your AWS repository:

cosign sign --key awskms:///alias/container-image-verification --upload=true <AWS Account ID>.dkr.ecr.us-east-1.amazonaws.com/supply-chain-security:latest

Note: If you get “kms: AccessDeniedException” error, ensure User running the command is authorized to perform kms:Sign action.

Next, run this command to list your container images and verify your container against the public key:

aws ecr list-images --repository-name supply-chain-security

Step 3: Apply Image Verification Policy using Kyverno

For this we first need to install the latest version of Kyverno (for detailed installation instructions, refer to the Kyverno docs). To start this process, access your Amazon EKS cluster and run the below kubectl command:

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml

This will create various Kubernetes resources (shown below) needed for Kyverno to work properly.

Out of the many available Kyverno rules, we will be using Kyverno “verifyImages”, which performs the following actions:

1. It validates signatures for matching images using Cosign.
2. It mutates image references with the digest returned by Cosign.

Using an image digest guarantees immutability of images and hence improves security.

The rule is executed in the mutating admission controller, but runs after resources are mutated to allow policies to mutate image registries and other configurations, before the image signature is verified.

Here is a policy that verifies that all images from the repository “public.ecr.aws/b3u2a5x0” that start with the name “supply-chain-security” are signed with the provided public key:

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata:  name: check-image spec:  validationFailureAction: enforce  background: false  rules:  - name: check-image  match:  resources:  kinds:  - Pod  verifyImages:  - image: "*"  key: |-  -----BEGIN PUBLIC KEY-----  MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEnlPDpSnZqceVm4eQy+mzCV7LFrLh  nUFmvi4SO5uhY6OOWBLbAyo5c7xoz7VyDgRxxIDSJRkWoAqozlLbUNOIyg==  -----END PUBLIC KEY-----   

Let’s take a look at this in action with a step-by-step demonstration:

1. Patch the Kyverno webhook, to allow time for calling the OCI registry (future releases of Kyverno will automatically adjust the defaults):kubectl patch mutatingwebhookconfigurations kyverno-resource-mutating-webhook-cfg \
--type json \
-p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"},{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 15}]'
2. Install an image validation policy:Copy the content of the below policy into verify_image.yaml. Be sure to make sure that you have added the correct public key.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: check-image spec: validationFailureAction: enforce background: false rules: - name: check-image match: resources: kinds: - Pod verifyImages: - image: "*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEnlPDpSnZqceVm4eQy+mzCV7LFrLh nUFmvi4SO5uhY6OOWBLbAyo5c7xoz7VyDgRxxIDSJRkWoAqozlLbUNOIyg== -----END PUBLIC KEY-----

kubectl apply -f verify_image.yaml

3. Try running a signed test image as a deployment from the Amazon ECR repository:

kubectl run signed \ --image=public.ecr.aws/b3u2a5x0/supply-chain-security:latest

4. Try running an unsigned image that matches the configured rule:

kubectl run deployment unsigned \ --image=public.ecr.aws/b3u2a5x0/nginx:latest

And you can see from the image shown here that the admission controller will block this image from getting deployed to our cluster:

Cleanup

To clean everything up, follow these steps:

• Delete the image verification policy from the Amazon EKS cluster by running:

kubectl delete -f verify_image.yaml

• Uninstall Kyverno by running:

kubectl delete -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml

• Delete the sign and verify key from AWS KMS.

aws kms delete-alias \
--alias-name "alias/container-image-verification"

Conclusion

In this post, we outlined how to integrate Cosign with AWS KMS. Then we ensured supply chain security is maintained using Kyverno ImageVerify policy on an existing Amazon EKS cluster. Kyverno policy checks the signature of public.ecr.aws/b3u2a5x0/supply-chain-security repository to ensure it has been signed by verifying its signature against the provided AWS KMS public key.