In a thought-provoking presentation by Kelsey Hightower, he compares the act of plugging in a random USB key discovered at a coffee shop to the common practice of pulling code from GitHub.

What's funny is that while people might give a suspicious look to someone plugging in a random USB key, they often don't think twice about grabbing a container image from a public registry and tossing it into production. But they really should. Lately, there have been a bunch of security breaches, and guess what? Many of them worked because they went after the weak spots in the software supply chain that no one bothered to check.

Recent supply chain attacks like SolarWinds, Log4j, and Kaseya have underscored the vulnerability of software supply chains. In response, US Federal agencies, under the influence of Executive Order 14028 and similar directives, are taking proactive steps to enhance the security of software supply chains. This has led to the emergence of an entirely new category of tools aimed at securing the software supply chain.

In this article, you'll be introduced to four open-source tools that, when integrated into your CI/CD pipeline, can significantly enhance the security of your container deployments, helping you avoid potential disasters arising from deploying compromised container images.

Overview

Before diving into the specifics of these open-source tools, let's talk about the high-level picture of how they can seamlessly integrate into your pipelines.

During the build stage, Trivy, Copacetic, and Notation play pivotal roles in automating vulnerability scanning, patching, and image signing. Once the build phase concludes, you'll have a trustworthy container image hosted in a registry, ready for deployment.

On the deployment end, two more tools, Gatekeeper and Ratify, are introduced to enforce policies that allow only signed container images to run on Kubernetes clusters.

Trivy: Vulnerability Scanner for Container Images

Trivy is CLI scanner tool and is a lifesaver for developers and DevOps teams navigating the intricate world of container security.

With its user-friendly command-line interface, Trivy makes it a breeze to detect vulnerabilities lurking within container images.

It has several different flags and options you can use to filter the results of the scanner to ensure the reports it generates remain actionable.

IMAGE=azure-voting-app-rust:v0.1-alpha
trivy image $IMAGE
trivy image --severity CRITICAL $IMAGE
trivy image --vuln-type os --severity CRITICAL $IMAGE

You can also export the vulnerability report generated by Trivy to a file, which is especially useful for the next tool in the lineup.

trivy image --exit-code 0 --format json --output ./patch.json --scanners vuln --vuln-type os --ignore-unfixed $IMAGE

Copacetic: Patching Container Images

Copacetic, another open-source gem, works in tandem with Trivy to tackle vulnerabilities in container images.

It utilizes Trivy's vulnerability reports to identify weak spots and then introduces patched layers to rectify these vulnerabilities. It's worth noting that Copacetic currently only works with remote container registries.

Here's an example of how it works:

# Push unpatched image to a remote registry
IMAGE=azure-voting-app-rust:v0.1-alpha
ACR_IMAGE=$ACR_NAME.azurecr.io/azure-voting-app-rust:v0.1-alpha
docker tag $IMAGE $ACR_IMAGE
docker push $ACR_IMAGE

#Start buildkit & run copa patch command
sudo ./bin/buildkitd &> /dev/null & 
sudo copa patch -i ${ACR_IMAGE} -r ./patch.json -t v0.1-alpha-1

#Rescan the newly patched image to verify a reduction in vulnerabilities
trivy image --severity CRITICAL --scanners vuln ${ACR_IMAGE_PATCHED}

#Push the patched image to the registry
ACR_IMAGE_PATCHED=${ACR_NAME}.azurecr.io/azure-voting-app-rust:v0.1-alpha-1

docker push ${ACR_IMAGE_PATCHED}

Notation: Image Signing for Enhanced Security

Notation is another command-line too that lets you digitally sign artifacts. And those signatures essentially become the stamps of approval for the different things in your software supply chain. For example, container images.

Using Notation is simple and straight forward. Once an image has gone through the scanning and patching steps, you sign the image with the notation sign command.

Once signed the container registry has an image index artifact with the signature. That signature will also be listed in the referrers of the container image.

APP_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${ACR_IMAGE_PATCHED})
notation sign $APP_DIGEST

NOTE: It's a best practice to use the container image digest for signing because tags are mutable, meaning they can be overwritten. Digests on the other hand are unique and change each time a layer is added to the container. By using digests for signing, an additional layer of security is added, ensuring the integrity and authenticity of container images.

Ratify: Ensuring Policy Compliance

Ratify is an admission controller. It's available both as a binary and as a Kubernetes tool installed via a Helm Chart. It ensures that only signed images are deployed. It's an invaluable tool for safeguarding your AKS cluster by preventing unsigned container images from being deployed.

To install it on an existing Kubernetes cluster, run the following commands:

curl -L https://raw.githubusercontent.com/deislabs/ratify/main/helmfile.yaml | helmfile sync -f - 

Once the installation is complete, any pods that use an unsigned image will be prevented from deploying.

Conclusion

As software supply chain security takes center stage, the importance of integrating open-source tools into your CI/CD pipeline cannot be overstated. By adopting these tools, you'll shield your deployments against potential threats, contributing to a more secure software supply chain.

Learn how to add these tools to your existing pipelines, with this step-by-step workshop!