Configuring HashiCorp Vault In AWS For High Availability In Kubernetes

Michael Levan - May 9 '23 - - Dev Community

Regardless of what your Kubernetes environment looks like, whether it’s one cluster or fifty clusters, at some point you will have a secret, password, or API key that you need to store in an encrypted fashion for one of your containerized workloads.

Because Kubernetes Secrets are stored as Base64 plain text in Etcd (the Kubernetes data store), you’ll see a lot of engineers going the third-party route.

In this blog post, you’ll learn about one of the most popular third-party routes, Vault.

Prerequisites

To follow along with this blog post in a hands-on fashion, you should have the following:

  • An EKS cluster running.
  • A KMS key.
  • Access to create IAM users.

If you don’t have these prerequisites, that’s okay! You can still read, follow along, and understand how it all works.

What Is Vault

Before diving into the Vault, implementation, let’s discuss what Vault is.

Vault is a secret manager from HashiCorp. Outside of it being so popular in the Kubernetes space, HashiCorp Vault has very much been the go-to secret store for all types of workloads. Whether you’re running containerized environments or running on VMs or even running on bare-metal, you can store your secrets in Vault and utilize them within your environment.

When you’re thinking of a “secret”, it’s anything that you want to be encrypted at rest and in transit. You don’t want the contents of the secret to ever be in plain text (other than when the endpoint is reading the value).

A secret could be anything from an API key to a password to even a username. Regardless of what the content of the secret is, it’s essentially anything that you wish to have as an encrypted value.

AWS KMS and Auto Unseal

One of the key aspects of HA is the fact that there will be several servers, or in the case of Kubernetes, Pods, that are running Vault. Because of that, you very rarely want to manually go through and unseal the instance of Vault.

What’s the unseal process?

When Vault is started, it starts as sealed. This means Vault knows how to access the storage for the secrets, but it doesn’t know how to unencrypt it. You can think of it like a bank vault. You need a certain access key to access the bank vault. When you unseal Vault, you’re getting the “key” (like a bank vault) to unencrypt Vault itself so you can begin to store and utilize secrets.

Why does this matter? Because if you have five Pods running Vault, that means you’d have to perform the unseal process manually across all of them.

Instead, you can use the Auto Unseal. When you use the Auto Unseal, you’ll just need to manually unseal one Vault Pod. After that one Vault Pod is unsealed, the rest of the Pods get automatically unsealed.

The best way in AWS to Auto Unseal is by using AWS KMS (in the prerequisites of this blog post).

To set up what’s needed for the Unseal process, you’ll need a KMS key and an IAM user to authenticate to AWS from Vault.

First, create the Vault Namespace.

kubectl create namespace vault
Enter fullscreen mode Exit fullscreen mode

Next, create the Kubernetes Secret with the IAM user's access key and secret key to authenticate to AWS.

kubectl create secret generic -n vault eks-creds \
    --from-literal=AWS_ACCESS_KEY_ID="" \
    --from-literal=AWS_SECRET_ACCESS_KEY=""
Enter fullscreen mode Exit fullscreen mode

Once complete, you can set up the Vault configuration.

Vault Helm Config

Now that the Namespace and eks-creds Kubernetes Secret are created, let’s learn how to implement Vault in an HA fashion.

For the purposes of utilizing Kubernetes, the best way to go about this implementation is by using Helm. Because the values.yaml for the Vault Helm config is so large, let’s break it down into chunks below.

First, you’d set the configuration to global and ensure that the injector exists so Vault can be injected into the Pods as a sidecar container.

# Vault Helm Chart Value Overrides
global:
  enabled: true

injector:
  enabled: true
  # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"
Enter fullscreen mode Exit fullscreen mode

Next, create the storage for the Raft algorithm backend.

server:
  # This configures the Vault Statefulset to create a PVC for data
  # storage when using the file or raft backend storage engines.
  # See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
  dataStorage:
    enabled: true
    # Size of the PVC created
    size: 20Gi
    # Location where the PVC will be mounted.
    mountPath: "/vault/data"
    # Name of the storage class to use.  If null it will use the
    # configured default Storage Class.
    storageClass: null
    # Access Mode of the storage device being used for the PVC
    accessMode: ReadWriteOnce
    # Annotations to apply to the PVC
    annotations: {}
Enter fullscreen mode Exit fullscreen mode

Set the resource limits and requests along with the readiness probe to ensure that the Vault Pods are getting the resources (CPU, memory) that they need along with confirming that they’re running as expected.

  # These Resource Limits are in line with node requirements in the
  # Vault Reference Architecture for a Small Cluster
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m

  # For HA configuration and because we need to manually init the vault,
  # we need to define custom readiness/liveness Probe settings
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60
Enter fullscreen mode Exit fullscreen mode

Create the audit storage and add the environment variables for the Kubernetes Secret you created in the previous section to authenticate to AWS for the purposes of utilizing AWS KMS for the unsealing process.

  # This configures the Vault Statefulset to create a PVC for audit logs.
  # See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  # Authentication to AWS for auto unseal
  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY
Enter fullscreen mode Exit fullscreen mode

Next, create the HA configuration for Vault.

Notice in the seal block that the AWS KMS key ID (kms_key_id) is blank. You’ll have to input this for your environment.

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false

      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        seal "awskms" {
          region     = "us-east-1"
          kms_key_id = ""
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
          leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }

        service_registration "kubernetes" {}
Enter fullscreen mode Exit fullscreen mode

Lastly, enable the Vault UI so you can access the Vault dashboard.

# Vault UI
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200
Enter fullscreen mode Exit fullscreen mode

All together, the override-values.yaml Helm config should look like the below.

# Vault Helm Chart Value Overrides
global:
  enabled: true

injector:
  enabled: true
  # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"

  resources:
      requests:
        memory: 256Mi
        cpu: 250m
      limits:
        memory: 256Mi
        cpu: 250m

server:
  # This configures the Vault Statefulset to create a PVC for data
  # storage when using the file or raft backend storage engines.
  # See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
  dataStorage:
    enabled: true
    # Size of the PVC created
    size: 20Gi
    # Location where the PVC will be mounted.
    mountPath: "/vault/data"
    # Name of the storage class to use.  If null it will use the
    # configured default Storage Class.
    storageClass: null
    # Access Mode of the storage device being used for the PVC
    accessMode: ReadWriteOnce
    # Annotations to apply to the PVC
    annotations: {}

  # Use the Enterprise Image
  image:
    repository: "hashicorp/vault"
    tag: "latest"

  # These Resource Limits are in line with node requirements in the
  # Vault Reference Architecture for a Small Cluster
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m

  # For HA configuration and because we need to manually init the vault,
  # we need to define custom readiness/liveness Probe settings
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60

  # This configures the Vault Statefulset to create a PVC for audit logs.
  # See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  # Authentication to AWS for auto unseal
  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false

      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        seal "awskms" {
          region     = "us-east-1"
          kms_key_id = ""
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
          leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }

        service_registration "kubernetes" {}

# Vault UI
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200
Enter fullscreen mode Exit fullscreen mode

Once you save the override-values.yaml file, run the helm installation with the following.

helm install vault hashicorp/vault \
    -f ./override-values.yaml \
    --namespace vault
Enter fullscreen mode Exit fullscreen mode

Vault Configuration

Now that Vault is running, you’ll have to take two steps:

  • Initialize Vault
  • Unseal one Vault Pod

Run the following command to initialize Vault.

kubectl exec --stdin=true --tty=true vault-0 -n vault -- vault operator init
Enter fullscreen mode Exit fullscreen mode

Once the command runs, you’ll see five unseal keys that get printed to the terminal.

Run the following command THREE (3) times inputting a new unseal key from the command above. For example, the command above will output five keys, so you can use keys 1, 2, and 3.

kubectl exec --stdin=true --tty=true vault-0 -n vault -- vault operator unseal
Enter fullscreen mode Exit fullscreen mode

Once complete, Vault will be unsealed and the other Pods will be auto-unsealed with KMS.

Image
A Developer’s Guide to Choosing the Best Notification Platform

webdev, javascript, programming
Image
12 Companies Leading The Way In Car Keys Repair
Image
20 Quotes That Will Help You Understand Key Fob Repair Near Me
Image
플러스카지노: 새로운 차원의 게임을 경험하세요!
Image
The Most Popular Truck Key Cover The Gurus Have Been Doing 3 Things
Image
Self Propelled Lightweight Folding Wheelchair: 11 Things You've Forgotten To Do
Image
10 Healthy Habits For Self Propelled Wheel Chair
Image
Day 24: Gradient Accumulation

llm, 75daysofllm, nlp
Image
Check Out The Inattentive ADHD Medication Tricks That The Celebs Are Using
Image
Are You In Search Of Inspiration? Look Up Spare Car Keys Near Me
Image
5 Wall Hanging Electric Fires Myths You Should Avoid
Image
플러스카지노: 운과 전략이 만나는 신세계
Image
The Most Underrated Companies To Follow In The Spare Van Keys Industry
Image
How SEO Services Drive Revenue for Small Businesses

seoservices, seocompany, seoagency, webdev
Image
The 3 Largest Disasters In Car Key Remote Repair Near Me The Car Key Remote Repair Near Me's 3 Biggest Disasters In History
Image
Adding a Cat Flap to Your Bifold Doors
Image
The Main Issue With Bronx Accident Lawyer, And How You Can Fix It
Image
3 Reasons Three Reasons Your Mesothelioma Non Asbestos Exposure Is Broken (And How To Repair It)
Image
5 Common Myths About Filter Coffee Maker You Should Stay Clear Of
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player