Install Hashicorp/Vault Server on K8S with Helm in Production

• Note: Currently, hashicorop recommend use vault-secret-operator, which has the same functions as vault-k8s. Comparison refer to footer FAQ#3.

Requesting the Certificate

• Prerequisite refer to: @Solution 2: Self-Hosting (PKI) CA Issuer

kubectl -n vault-system apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-server-cert
spec:
  isCA: false
  commonName: "Vault Server Certificate"
  secretName: vault-server-tls
  duration: 24h
  renewBefore: 144m # roughly 10% of 24h
  privateKey:
    algorithm: ECDSA
    size: 384
  issuerRef:
    name: root-ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io
  subject:
    organizations:
    - "Vault Team of Example Enterprise, Inc."
    organizationalUnits:
    - "Security Operations"
    countries:
    - "US"
    - "CN"
    localities:
    - "San Francisco"
    provinces:
    - "California"
    - "Hongkong"
  dnsNames:
  - localhost
  - vault
  - vault.vault-system
  - vault.vault-system.svc
  - vault.vault-system.svc.cluster.local
  ipAddresses:
  - 127.0.0.1
  emailAddresses:
  - securityadmin@myapp.com
EOF

kubectl -n vault-system get certificate,secret

Deployment the Vault Server

• Refer to: https://developer.hashicorp.com/vault/docs/platform/k8s/helm
• Refer to: https://github.com/hashicorp/vault-helm
• Add the hashicorp helm repo if necessary.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm search repo hashicorp/vault --version 0.29.1

• Installation

helm -n vault-system uninstall vault
helm -n vault-system upgrade --create-namespace -i \
vault hashicorp/vault \
--version 0.29.1 \
--set server.enabled=true \
--set server.image.repository=registry.cn-shenzhen.aliyuncs.com/wl4g-k8s/hashicorp_vault \
--set server.image.tag=1.18.1 \
--set server.logLevel=trace \
--set ui.enabled=true \
--set ui.serviceType=NodePort \
--set injector.enabled=false \
--set installCRDs=true \
--set 'server.volumes[0].name=vault-server-tls' \
--set 'server.volumes[0].secret.secretName=vault-server-tls' \
--set 'server.volumeMounts[0].name=vault-server-tls' \
--set 'server.volumeMounts[0].mountPath=/vault/config/tls' \
--set 'server.readinessProbe.exec.command[0]=/bin/sh' \
--set 'server.readinessProbe.exec.command[1]=-c' \
--set 'server.readinessProbe.exec.command[2]=export VAULT_SKIP_VERIFY=true && vault status -tls-skip-verify' \
--set 'server.readinessProbe.initialDelaySeconds=5' \
--set 'server.readinessProbe.periodSeconds=5' \
--set 'server.extraEnvironmentVars.VAULT_ADDR=https://127.0.0.1:8200' \
--set 'server.standalone.config=
ui = true
listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/config/tls/tls.crt"
  tls_key_file = "/vault/config/tls/tls.key"
  tls_client_ca_file = "/vault/config/tls/ca.crt"
}
storage "raft" {
  path = "/vault/data"
  node_id = "vault-0"
}'

helm -n vault-system list
kubectl -n vault-system get sts,deploy,svc,secret,pvc
echo "http://localhost:$(kubectl -n vault-system get svc/vault-ui -ojson \
| jq '.spec.ports[] | select(.port = "8200") | .nodePort')"

Uninstallation (If necessary)

helm -n vault-system uninstall vault
#kubectl -n vault-system delete pvc/data-vault-0 # IMPORTANT!!!

Initializing & Unseal

• Note: Here we only use the simple mode for the development/generally production environment. In strict productionmust use PGP, AWS Secret, GCP Secret Providerand other secure modes to prevent the creator from obtaining all the unsealed keys.
• Initializing (If Necessary)

kubectl -n vault-system exec -it vault-0 -- vault operator init

export VAULT_CACERT=/vault/config/tls/ca.crt

vault status -address=https://localhost:8200
#Key                Value
#---                -----
#Seal Type          shamir
#Initialized        false
#Sealed             true
#Total Shares       0
#Threshold          0
#Unseal Progress    0/0
#Unseal Nonce       n/a
#Version            1.18.1
#Build Date         2024-10-29T14:21:31Z
#Storage Type       raft
#HA Enabled         true

vault operator init -address=https://localhost:8200
#Unseal Key 1: sAKWWiEpwVxSe/y7CnvlzEO6Ybf4jwAznQLGDMzqBtWi
#Unseal Key 2: vnQ7WoLWZCzB3ARwGekTQrJzdPtCTgW8LZZLUD0jbhzF
#Unseal Key 3: C1VjdbnWfLpBIXT1megDkxJn+0YlnIOVKQuwIeHG7tOw
#Unseal Key 4: 4iiKiB5NhPFHja0+B+JvBpjyPLUyxIBnMnxypBZ+MYCO
#Unseal Key 5: qeUvlAJVRllApssfhHam/tgka1OeCiixy1DhcNHfwgep
#
#Initial Root Token: hvs.Rl3nh3fC6WZf8xkmE2T4wVME
#
#Vault initialized with 5 key shares and a key threshold of 3. Please securely
#distribute the key shares printed above. When the Vault is re-sealed,
#restarted, or stopped, you must supply at least 3 of these keys to unseal it
#before it can start servicing requests.
#
#Vault does not store the generated root key. Without at least 3 keys to
#reconstruct the root key, Vault will remain permanently sealed!
#
#It is possible to generate new unseal keys, provided you have a quorum of
#existing unseal keys shares. See "vault operator rekey" for more information.

• Unseal

export VAULT_CACERT=/vault/config/tls/ca.crt

# Repeat the requests 3 times to unseal.
vault operator unseal -address=https://localhost:8200

# Check the vault access.
export VAULT_ADDR='https://localhost:8200'
export VAULT_CACERT='/vault/config/tls/ca.crt'
export VAULT_TOKEN='hvs.Rl3nh3fC6WZf8xkmE2T4wVME'

vault secrets list
#Path          Type         Accessor              Description
#----          ----         --------              -----------
#cubbyhole/    cubbyhole    cubbyhole_79d88e50    per-token private secret storage
#identity/     identity     identity_803c0a51     identity store
#sys/          system       system_5ba706f3       system endpoints used for control, policy and debugging

Accessing

echo "https://localhost:$(kubectl -n vault-system get svc/vault-ui -ojson \
| jq '.spec.ports[] | select(.port = "8200") | .nodePort')"

Next step to: Vault Inject Secret to Application Pods