Max AllanImages are a really easy way to distribute software. If you're not careful you might inadvertently distribute your secrets as well. I'm going to run through some ways you can make this mistake and how to do it right.
I'm going to use examples that don't require you to create any files wherever possible. You can copy/paste a few commands and there is minimal clean up afterwards other than a few images based on alpine. Because of the way Docker layers work, they should only have a tiny footprint and use a few extra bytes. I've also slightly shortened the build output text to make it easier to read.
All my example commands on the host are run as a normal user with Docker privileges and I use [max]$ as the shell prompt. If it's in a running container the prompt may vary but will usually end with # when running as root and $ when not root. These simple examples should work with Docker or Podman and almost any shell on a Unix-like OS.
Environment variables
Environment variables are convenient for passing secrets between processes. And you can easily use them as inputs to a Docker build process. Environment variables are not preserved from one shell session to another, so you may already be using variables to store credentials in your environment. You might think that environment variables could be a safe method for passing in passwords to the build.
Sadly no. "Environment variables" in Docker builds are used specifically when you want them to be in the final running container.
Here is a short demo of using a variable in a docker build.
[max]$ echo "FROM alpine" | PASSWORD=superSecret docker build --env PASSWORD -t envs -
STEP 1/2: FROM alpine
STEP 2/2: ENV "PASSWORD"="superSecret"
[max]$ docker inspect --format '{{.Config.Env}}' envs
[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PASSWORD=superSecret]
[max]$ docker run -it envs
/ # echo $PASSWORD
superSecret
/ #
Simply do not use --env for secrets! Anyone who gets a copy of your image can trivially extract your secrets.
Build Arguments
These are kind of like environment variables but they're specifically intended only to be present during the build process and not in the final running image. Sounds like they could be useful for secrets. But again: no.
A slightly more complex docker build is needed to demonstrate this.
[max]$ printf 'FROM alpine \nARG PASSWORD \nRUN echo $PASSWORD' | docker build --build-arg PASSWORD=superSecret -t args --progress=plain -
STEP 1/3: FROM alpine
STEP 2/3: ARG PASSWORD
STEP 3/3: RUN echo $PASSWORD
superSecret
OK, so the password shows up during the build, because I told it to. That is not the problem. (Normally that would be piped into another command or used as a variable without echoing.) If we look at the built image, what do we see:
[max]$ docker run -it args
/ # echo $PASSWORD
So, looks like we're in the clear.
But no:
[max]$ docker inspect args --format '{{index .History 3}}'
{2024-09-28 13:02:47.50065912 +0000 UTC |1 PASSWORD=superSecret /bin/sh -c echo $PASSWORD FROM e247e9b4c8fc false}
The way Docker sets the variables on each command is to quietly prefix the command with setting the variable for you. And that appears in the history of the build.
So, can you use build arguments safely? Some people would say "Yes, do a multi stage build". In this example, our docker build is a little bit more complex than before:
docker build -t staged --build-arg PASSWORD=differentSecret - << "EOF"
FROM alpine as build
ARG PASSWORD
RUN echo $PASSWORD | rev > /secret
FROM alpine
COPY --from=build /secret /secret
EOF
[1/2] STEP 1/3: FROM alpine AS build
[1/2] STEP 2/3: ARG PASSWORD
[1/2] STEP 3/3: RUN echo | rev > /secret
[2/2] STEP 1/2: FROM alpine
[2/2] STEP 2/2: COPY --from=build /secret /secret
[max]$ docker inspect staged --format '{{.History}}' | grep PASSWORD
And here we see the password is no longer included in the history. The command to put the secret in the history was run in a temporary stage that does not get recorded.
Personally I would advise against doing this. Not because it doesn't work, but because not everyone knows what they are doing. How long before someone new comes along and follows your expertly crafted Dockerfile and cuts out the second stage because "they don't need it"?
Secrets in files
It is quite common to store passwords for dev tools in files. Like your private npm/maven/python registry. They're usually one of the "secrets" that everyone knows. So you COPY in the file with the password during the build. And then delete it. So that it is gone.
Sadly Docker builds use "layers" and the quick explanation of that is: each saved layer is from a different statement in the Dockerfile.
ss
In this example, we need a file to copy in. echo somepassword > .credentials and remember to delete it when you're done.
[max]$ docker build -t layers -f- . << EOF
FROM alpine
COPY .credentials .credentials
RUN rev .credentials
RUN rm .credentials
EOF
STEP 1/4: FROM alpine
STEP 2/4: COPY .credentials .credentials
STEP 3/4: RUN rev .credentials
drowssapemos
STEP 4/4: RUN rm .credentials
[max]$ docker run layers find / -name .credentials
[max]$
And we can see: No .credentials file.
However, if we dig into the layers of the image we can find it. If you want to try this yourself, create a temporary directory to extract the image into. It will create many tar files.
[max]$ docker save layers -o layers.tar
[max]$ tar xf layers.tar
[max]$ for n in *tar ; do tar tf $n | grep -q '^.credentials' && echo $n ; done
87a5f56e584e6f90c691a0cf2c990ad6b4cf3ede2ca34c9b897c072dc657b7a8.tar
[max]$ tar xf 87a5f56e584e6f90c691a0cf2c990ad6b4cf3ede2ca34c9b897c072dc657b7a8.tar .credentials
[max]$ more .credentials
somepassword
A quick explanation of those commands is: Save the image as a tar file. Extract the content of the tar file (which is more tar files). Loop through the file list for each tar file and look for the file we want. Then extract the file from the tar we found to recover the content of the file.)
Again, if you use a multistage build, you may appear to be safe from this leak. Layers created in other stages are not in the final image. But don't forget the new person who doesn't really know what they're doing! Also, depending on your setup, the build layers may be cached. Meaning your .credentials file is possibly in Docker's build cache somewhere and someone else with access to your build machine could find it. Find files in the build cache is beyond the scope of this short article.
While we're discussing these shared secrets, it is a good idea to ensure that the shared value is a read only account. To prevent any exposure giving someone bad the ability to overwrite all your trusted, internal, repository content with malware. And ensure that during a build, everyone uses the read-only password when they need to read. The read/write password is only used when the build is complete and needs to be stored in the registry. Do not be lazy!
Labels and annotations
This is a more obscure problem. And someone maybe doing it for a good reason, but it could expose you to deeper attacks if part of your infrastructure is compromised.
Each image has metadata associated with it. If you include helpful information like the name of the person who built the image then you could be at risk from social engineering attacks. If you include details of the build server, then maybe an attacker will know somewhere they can focus their next attack.
There are slightly different commands to view annotations and labels:
[max]$ docker inspect bitnami/redis | jq .[].Config.Labels
{
"com.vmware.cp.artifact.flavor": "sha256:c50c90cfd9d12b445b011e6ad529f1ad3daea45c26d20b00732fae3cd71f6a83",
"org.opencontainers.image.base.name": "docker.io/bitnami/minideb:bookworm",
"org.opencontainers.image.created": "2024-09-05T07:05:22Z",
"org.opencontainers.image.description": "Application packaged by Broadcom, Inc.",
....
[max]$ docker buildx imagetools inspect chainguard/wolfi-base --raw | jq .annotations
{
"org.opencontainers.image.authors": "Chainguard Team https://www.chainguard.dev/",
"org.opencontainers.image.source": "https://github.com/chainguard-images/images/tree/main/images/wolfi-base",
"org.opencontainers.image.url": "https://images.chainguard.dev/directory/image/wolfi-base/overview",
"org.opencontainers.image.vendor": "Chainguard"
}
There is no specific defence against this sort of leak, other than checking what is being set and making sure it would not provide any advantage to an attacker who may already have a shell inside your firewall. If you've helpfully annotated your image with Built on https://jenkins1.internal.url your hacker is going to be looking for any Jenkins vulnerabilities and will know where to send them.
How to do Secrets properly
I've shown you a load of bad practice, but what about the good practice? Docker provides a "secrets" feature. Use it!
https://docs.docker.com/build/building/secrets/
I'm not going to repeat their documentation here, if you are struggling to understand it, please ask in the comments.
