HashiCorp Vault: Kubernetes Secrets with External Secrets Operator (ESO)

Table of Contents

HashiCorp-Vault - This article is part of a series.

Part 1: HashiCorp Vault: Docker Compose Deployment with TLS Encryption and Web-UI, Initial Setup via Vault CLI, Key/Value Secret Example

Part 2: This Article

Vault Prerequisites
#

Create Key/Value Secret Engine
#

# Enable a Key/Value (KV) secrets engine
vault secrets enable -path=project-1 kv-v2

# Shell output:
Success! Enabled the kv-v2 secrets engine at: project-1/

Verify Key/Value Secret Engine
#

# List secrets
vault secrets list

# Shell output:
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_92bb725d    per-token private secret storage
identity/     identity     identity_002dc7ca     identity store
project-1/    kv           kv_91e9cd44           n/a
sys/          system       system_4aa0af3b       system endpoints used for control, policy and debugging

Create a Secret
#

Note: Tokens can not read secrets that were created with the root token!

# Create a secret in Vault
vault kv put project-1/development/user-2 username="user-2" password="my-secure-pw"

Create Vault Token
#

Create a dedicated token for the Kubernetes cluster external secret provider.

Create a Policy
#

Create a policy in Vault that specifies what actions the token can perform:

# Create a policy file
vi external-secrets-policy.hcl

# Example: Read only permission
path "project-1/*" {
  capabilities = ["read"]
}

# Example: Full permissions
path "project-1/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Write the policy to Vault
vault policy write external-secrets external-secrets-policy.hcl

# Shell output:
Success! Uploaded policy: external-secrets

Create the Token
#

# Create a Vault token for the "external-secrets" policy
vault token create -policy="external-secrets" -period="720h" -orphan -display-name="k8s-vault-secrets"

# Shell output:
Key                  Value
---                  -----
token                hvs.CAESIE0sxNfDl6qtrvnDjVoNeS84p7X5x6A6nCaq-Eaz9vLdGh4KHGh2cy5DYWdWTXFWOUx5MkUwb09sVVlkQkxzZVU
token_accessor       7k2K3tEg7hMk5OPGgA4FjRPc
token_duration       720h
token_renewable      true
token_policies       ["default" "external-secrets"]
identity_policies    []
policies             ["default" "external-secrets"]

Verify the Token
#

# List tokens
vault list auth/token/accessors

# Shell output:
Keys
----
7k2K3tEg7hMk5OPGgA4FjRPc
dONSpNlr4AiawUyOS1ioITGT

# List token details
vault token lookup -accessor 7k2K3tEg7hMk5OPGgA4FjRPc

# Shell output:
Key                 Value
---                 -----
accessor            7k2K3tEg7hMk5OPGgA4FjRPc
creation_time       1722610472
creation_ttl        720h
display_name        token-k8s-vault-secrets
entity_id           n/a
expire_time         2024-09-01T14:54:32.208539362Z
explicit_max_ttl    0s
id                  n/a
issue_time          2024-08-02T14:54:32.208541572Z
meta                <nil>
num_uses            0
orphan              true
path                auth/token/create
period              720h
policies            [default external-secrets]
renewable           true
ttl                 719h48m48s
type                service

Revoke the Token
#

# If necessary, revoke the token
vault token revoke hvs.CAESIE0sxNfDl6qtrvnDjVoNeS84p7X5x6A6nCaq-Eaz9vLdGh4KHGh2cy5DYWdWTXFWOUx5MkUwb09sVVlkQkxzZVU

Kubernetes Cluster
#

CoreDNS DNS Entry
#

Create a DNS entry for Vault, in my example it’s:
192.168.30.19 vault.jklug.work

# Edit the CoreDNS ConfigMap
kubectl edit cm coredns -n kube-system

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        hosts {
            192.168.30.19 vault.jklug.work
            fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }    
kind: ConfigMap
metadata:
  creationTimestamp: "2024-07-05T17:36:16Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "224"
  uid: 0c9d9c5a-c49f-4392-a465-ffb1d047811c

Verify the DNS resolution

# Run pod for network troubleshooting 
kubectl run busybox --image=busybox --restart=Never --stdin --tty

# Run nslookup
nslookup vault.jklug.work

External Secrets Operator (ESO)
#

Helm Repository
#

# Add Helm repository & update the index
helm repo add external-secrets https://charts.external-secrets.io &&
helm repo update

Install External Secrets Operator
#

# Install external secrets operator
helm install external-secrets \
    external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

# Shell output:
NAME: external-secrets
LAST DEPLOYED: Fri Aug  2 13:42:11 2024
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully in namespace external-secrets!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets

Create a Secret Store
#

The Secret Store is used by the External Secrets Operator to store information about how to communicate with the Vault secrets provider.

Create Secret
#

Add the Vault token as Kubernetes secret, so that the External Secrets Operator can communicate with the Vault secrets provider.

# Create Kubernetes secret with the previously created Vault token
kubectl create secret generic vault-token --from-literal=token=hvs.CAESIE0sxNfDl6qtrvnDjVoNeS84p7X5x6A6nCaq-Eaz9vLdGh4KHGh2cy5DYWdWTXFWOUx5MkUwb09sVVlkQkxzZVU

# Optional, in an non production setup the Initial Root Token can be used
kubectl create secret generic vault-token --from-literal=token=hvs.kG9fJG4s56A6vIxDTPpm0i3l

# Shell output:
secret/vault-token created

Verify Secret
#

# List secrets
kubectl get secrets

# Shell output:
NAME          TYPE     DATA   AGE
vault-token   Opaque   1      7s

Setup Secret Store
#

Set up the Secret Store with the details for reaching the external secret provider:

# Create a manifest for the Secret Store
vi secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.jklug.work:8200"
      path: "project-1" # Define Kev/Value secret path
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"

# Deploy the Secret Store
kubectl apply -f secret-store.yaml

Verify Store Secret
#

Verify the SecretStore is validated and shows: message: store validated

# List SecretStore details
kubectl get SecretStore vault-backend -o yaml

# Shell output:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"external-secrets.io/v1beta1","kind":"SecretStore","metadata":{"annotations":{},"name":"vault-backend","namespace":"default"},"spec":{"provider":{"vault":{"auth":{"tokenSecretRef":{"key":"token","name":"vault-token"}},"path":"project-1","server":"https://vault.jklug.work:8200","version":"v2"}}}}
  creationTimestamp: "2024-08-02T14:56:22Z"
  generation: 1
  name: vault-backend
  namespace: default
  resourceVersion: "16207"
  uid: 61d1649e-5437-4603-a4f9-a2fbf8336346
spec:
  provider:
    vault:
      auth:
        tokenSecretRef:
          key: token
          name: vault-token
      path: project-1
      server: https://vault.jklug.work:8200
      version: v2
status:
  capabilities: ReadWrite
  conditions:
  - lastTransitionTime: "2024-08-02T14:56:22Z"
    message: store validated # Verify the SecretStore is validated
    reason: Valid
    status: "True"
    type: Ready

Example: External Secret
#

Overview
#

The following ExternalSecret syncs the project-1/development/user-2 secret with the username="user-2" and password="my-secure-pw" values from the Vault, into the Kubernetes cluster as native Kubernetes secrets.

This allows the Kubernetes applications to access the secrets without directly interacting with Vault.

Create External Secret
#

# Create ExternalSecret Manifest
vi external-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: example-external-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: project-1/development/user-2
        property: username
    - secretKey: password
      remoteRef:
        key: project-1/development/user-2
        property: password

# Apply the External Secret
kubectl apply -f external-secret.yaml

Verify the Secret
#

Check Sync Status
#

# List ExternalSecret
kubectl get ExternalSecret external-secret

# Shell output:
NAME              STORE           REFRESH INTERVAL   STATUS         READY
external-secret   vault-backend   15s                SecretSynced   True

# List ExternalSecret details
kubectl describe ExternalSecret external-secret

# Shell output:
Name:         external-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  Creation Timestamp:  2024-08-02T15:00:25Z
  Generation:          1
  Resource Version:    16748
  UID:                 9e060e8d-2573-43bd-a764-1b2be4f277d3
Spec:
  Data:
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  project-1/development/user-2
      Metadata Policy:      None
      Property:             username
    Secret Key:             username
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  project-1/development/user-2
      Metadata Policy:      None
      Property:             password
    Secret Key:             password
  Refresh Interval:         15s
  Secret Store Ref:
    Kind:  SecretStore
    Name:  vault-backend
  Target:
    Creation Policy:  Owner
    Deletion Policy:  Retain
    Name:             example-external-secret
Status:
  Binding:
    Name:  example-external-secret
  Conditions:
    Last Transition Time:   2024-08-02T15:00:25Z
    Message:                Secret was synced
    Reason:                 SecretSynced
    Status:                 True
    Type:                   Ready
  Refresh Time:             2024-08-02T15:00:25Z
  Synced Resource Version:  1-33a3cd1f082890d5a8e161d68d5ffb8d
Events:
  Type    Reason   Age   From              Message
  ----    ------   ----  ----              -------
  Normal  Created  10s   external-secrets  Created Secret

Verify the Secret Value
#

# List decoded values of the secret
kubectl get secret example-external-secret -o jsonpath="{.data}" | jq 'map_values(@base64d)'

# Shell outout:
{
  "password": "my-secure-pw",
  "username": "user-2"
}

HashiCorp-Vault - This article is part of a series.

Part 1: HashiCorp Vault: Docker Compose Deployment with TLS Encryption and Web-UI, Initial Setup via Vault CLI, Key/Value Secret Example

Part 2: This Article

Vault Prerequisites #

Create Key/Value Secret Engine #

Verify Key/Value Secret Engine #

Create a Secret #

Create Vault Token #

Create a Policy #

Create the Token #

Verify the Token #

Revoke the Token #

Kubernetes Cluster #

CoreDNS DNS Entry #

External Secrets Operator (ESO) #

Helm Repository #

Install External Secrets Operator #

Create a Secret Store #

Create Secret #

Verify Secret #

Setup Secret Store #

Verify Store Secret #

Example: External Secret #

Overview #

Create External Secret #

Verify the Secret #

Check Sync Status #

Verify the Secret Value #