--- title: Using Secrets Store CSI Driver with HashiCorp Vault description: This walkthrough deploys PingDirectory and PingFederate using the ping-devops Helm chart with all secrets sourced from HashiCorp Vault via the Secrets Store CSI Driver. component: helm page_id: helm::examples/vault-spc-walkthrough canonical_url: https://developer.pingidentity.com/helm/examples/vault-spc-walkthrough.html section_ids: devops-vault-spc-overview: Overview devops-vault-spc-prerequisites: Prerequisites devops-vault-spc-infra-setup: Infrastructure Setup (Steps 1–4) devops-vault-spc-step1: "Step 1: Install the Secrets Store CSI Driver" devops-vault-spc-step2: "Step 2: Install Vault with the CSI Provider" devops-vault-spc-step3: "Step 3: Seed Vault with Secrets" 3-1-authenticate-and-enable-the-kv-engine: 3.1 Authenticate and enable the KV engine 3-2-pingdirectory-devops-credentials: 3.2 PingDirectory DevOps credentials 3-3-pingdirectory-password-and-license-secrets: 3.3 PingDirectory password and license secrets 3-4-pingfederate-secrets-not-for-production-use: 3.4 PingFederate secrets (not for production use!) devops-vault-spc-step4: "Step 4: Configure Kubernetes Auth in Vault" 4-1-create-the-namespace-and-serviceaccount: 4.1 Create the namespace and ServiceAccount 4-2-enable-kubernetes-auth: 4.2 Enable Kubernetes auth 4-3-write-a-policy-granting-read-access-to-the-ping-secrets: 4.3 Write a policy granting read access to the ping secrets 4-4-bind-the-policy-to-the-serviceaccount: 4.4 Bind the policy to the ServiceAccount devops-vault-spc-step5: "Step 5: PingDirectory Values" devops-vault-spc-step6: "Step 6: PingFederate Values" devops-vault-spc-step7: "Step 7: Deploy" devops-vault-spc-verify: Verify verify-the-env-file-is-mounted: Verify the .env file is mounted verify-environment-variables-were-sourced: Verify environment variables were sourced verify-pingfederate: Verify PingFederate devops-vault-spc-alternative: "Alternative: Kubernetes Secret Sync" devops-vault-spc-preexisting: "Optional: Using a Pre-Existing SecretProviderClass" devops-vault-spc-rotation: Secret Rotation before-you-rotate-add-pd_rebuild_on_restart: "Before you rotate: add PD_REBUILD_ON_RESTART" step-1-update-the-secret-in-vault: "Step 1: Update the secret in Vault" step-2-confirm-the-csi-driver-has-propagated-the-new-value: "Step 2: Confirm the CSI driver has propagated the new value" step-3-restart-the-pod: "Step 3: Restart the pod" step-4-verify-the-new-password-is-active: "Step 4: Verify the new password is active" devops-vault-spc-troubleshooting: Troubleshooting pod-stuck-in-containercreating: Pod stuck in ContainerCreating vault-authentication-errors: Vault authentication errors environment-variables-not-set-after-pod-starts: Environment variables not set after pod starts kubernetes-secret-not-created-alternative-path-only: Kubernetes Secret not created (Alternative path only) verifying-the-csi-driver-and-vault-provider-daemonsets: Verifying the CSI driver and Vault provider DaemonSets devops-vault-spc-cleanup: Clean Up --- # Using Secrets Store CSI Driver with HashiCorp Vault This walkthrough deploys PingDirectory and PingFederate using the `ping-devops` Helm chart with all secrets sourced from HashiCorp Vault via the Secrets Store CSI Driver. | | | | - | -------------------------------------------------------------------------------------------------------------------------------- | | | In order for this example to work, you must be on the `ping-devops` Helm chart version `0.12.3` or later, released in June 2026. | | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | This walkthrough uses Vault in development mode — in-memory storage, auto-unsealed, single node. Do not use this configuration in production. Production deployments should use Vault HA with TLS and a dedicated namespace. In addition, the credentials shown here are samples and should not be used in production. | The [Getting Started](../getting-started/getting-started.html) page has instructions on configuring your environment for using Ping `ping-devops` Helm charts. For more examples, see [Helm Chart Example Configurations](../helm-charts-landing-page.html). ## Overview The Secrets Store CSI Driver fetches secrets from Vault and makes them available inside pods as mounted files. Ping product containers already know how to consume secrets this way: the `source_secret_envs` hook function sources every `*.env` file found in `SECRETS_DIR` and exports its `KEY=VALUE` pairs as environment variables. This walkthrough uses that native mechanism as the primary delivery path: 1. Each product's secrets are stored in Vault as a single `content` key containing `KEY=VALUE` pairs. 2. The CSI driver mounts that content as a `product.env` file at a configurable path. 3. Setting `SECRETS_DIR` to the mount path tells the Ping startup hooks where to find the file. 4. No Kubernetes `Secret` objects are created — secret values exist only as files inside the pod. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The Secrets Store CSI Driver fetches one Vault key per `objectName` entry and writes its value directly to a file. If you store secrets as individual keys in Vault (e.g. `ROOT_USER_PASSWORD`, `PING_IDENTITY_DEVOPS_USER`), the CSI driver fetches each value as a plain string — which is what you want.However, if you attempt to fetch the **entire secret** in one shot without specifying a `secretKey`, the CSI driver writes the raw KV v2 API envelope JSON (`{"data":{"ROOT_USER_PASSWORD":"…​"},"metadata":{…​}}`) to the file. The Ping startup hooks source that file expecting `KEY=VALUE` lines — the JSON envelope is unparseable and credentials are silently ignored.The fix is a storage convention: put all of a product's credentials into a single Vault key named `content` whose value is a `KEY=VALUE` block. The CSI driver then writes that block verbatim to the mounted file, and `source_secret_envs` picks it up natively. | The `ping-devops` chart integrates with the CSI driver through the `secretProviderClass` values block: set `enabled: true` and the chart auto-wires a CSI volume and mount into every pod spec, and optionally creates the `SecretProviderClass` resource itself. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | Ping container images default `SECRETS_DIR` to `/run/secrets`, and `/var/run` is a symlink to `/run` inside those images. Mounting a read-only CSI volume at `/run/secrets` conflicts with the Kubernetes ServiceAccount token mount at `/var/run/secrets/kubernetes.io/serviceaccount`. This walkthrough mounts at `/run/vault-secrets` and overrides `SECRETS_DIR` to match. | ## Prerequisites * `kubectl` configured to a running cluster (Docker Desktop or kind works for this walkthrough) * `helm` v3 * A `PING_IDENTITY_DEVOPS_USER` and `PING_IDENTITY_DEVOPS_KEY` ## Infrastructure Setup (Steps 1–4) | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Steps 1–4 install and configure the Secrets Store CSI Driver, HashiCorp Vault, and the Kubernetes auth plumbing those components require. In a real environment these would already be managed by your platform team. If you have them in place, skip to [Step 5](#devops-vault-spc-step5) where the `ping-devops` Helm chart configuration begins. | ## Step 1: Install the Secrets Store CSI Driver The CSI driver runs as a DaemonSet on every node. ```shell helm repo add secrets-store-csi-driver \ https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm repo update helm install csi-secrets-store \ secrets-store-csi-driver/secrets-store-csi-driver \ --namespace kube-system \ --set syncSecret.enabled=true \ --set enableSecretRotation=true ``` Verify the DaemonSet is running: ```shell kubectl get daemonset -n kube-system \ -l app.kubernetes.io/instance=csi-secrets-store ``` `DESIRED` and `READY` should match (one pod per schedulable node). ## Step 2: Install Vault with the CSI Provider The HashiCorp Helm chart installs both the Vault server and the Vault CSI provider DaemonSet in a single release. The injector (sidecar-based secret delivery) is disabled because we are using the CSI path instead. ```shell helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update helm install vault hashicorp/vault \ --namespace vault \ --create-namespace \ --set "server.dev.enabled=true" \ --set "server.dev.devRootToken=root" \ --set "csi.enabled=true" \ --set "injector.enabled=false" ``` Wait for Vault to be ready: ```shell kubectl wait pod vault-0 \ -n vault \ --for=condition=Ready \ --timeout=120s ``` ## Step 3: Seed Vault with Secrets Open a shell into the Vault pod: ```shell kubectl exec -it vault-0 -n vault -- /bin/sh ``` The following commands are run **inside** that shell. ### 3.1 Authenticate and enable the KV engine ```shell export VAULT_TOKEN=root export VAULT_ADDR=http://127.0.0.1:8200 vault secrets enable -path=ping kv-v2 ``` ### 3.2 PingDirectory DevOps credentials DevOps credentials and EULA acceptance are stored as a `content` key sourced by the Ping startup hooks. Replace the placeholder values with your credentials. ```shell vault kv put ping/pd-env content="PING_IDENTITY_DEVOPS_USER= PING_IDENTITY_DEVOPS_KEY= PING_IDENTITY_ACCEPT_EULA=YES" ``` ### 3.3 PingDirectory password and license secrets PingDirectory resolves credentials from files at startup via `*_FILE` environment variables. Store the passwords — and optionally the license file — as individual keys in a separate Vault secret so each can be mounted as its own file. ```shell vault kv put ping/pdpwd - <" } EOF ``` | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | This walkthrough uses `PING_IDENTITY_DEVOPS_USER` and `PING_IDENTITY_DEVOPS_KEY` to obtain a development license from the Ping DevOps license server at startup, so no static license file is required. The `pd-license` key and `PingDirectory.lic` object entry in Step 5 illustrate the pattern for production deployments where a static license file is managed centrally in Vault — replace the placeholder with the base64-encoded content of your `PingDirectory.lic` file. | ### 3.4 PingFederate secrets (not for production use!) ```shell vault kv put ping/pf-env content="PING_IDENTITY_DEVOPS_USER= PING_IDENTITY_DEVOPS_KEY= PING_IDENTITY_ACCEPT_EULA=YES PING_IDENTITY_PASSWORD= PF_RUN_PF_CLUSTER_AUTH_PWD=" ``` | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | `PF_RUN_PF_CLUSTER_AUTH_PWD` sets the cluster authentication password shared between the admin and engine nodes. If you rotate this value in Vault, you must restart **both** the admin and engine pods — restarting only the admin leaves the engine using the old cluster password and it will fail to rejoin the cluster. | Exit the shell: ```shell exit ``` ## Step 4: Configure Kubernetes Auth in Vault The Vault Kubernetes auth method lets pods authenticate to Vault using the ServiceAccount token Kubernetes automatically mounts into every pod. When a pod requests a secret via the CSI driver, the Vault provider presents that token; Vault validates it against the cluster and, if the token's ServiceAccount matches a configured role, issues a short-lived Vault token scoped to the policy you define. ### 4.1 Create the namespace and ServiceAccount ```shell kubectl create namespace ping kubectl create serviceaccount ping-vault-auth -n ping \ --dry-run=client -o json \ | jq '.automountServiceAccountToken = false' \ | kubectl apply -f - ``` Setting `automountServiceAccountToken: false` prevents Kubernetes from mounting the SA token at the default `/var/run/secrets/kubernetes.io/serviceaccount` path. The Vault CSI provider does not use that auto-mounted token — it mounts its own projected token — so disabling automounting removes an unnecessary credential from every pod. ### 4.2 Enable Kubernetes auth ```shell kubectl exec -it vault-0 -n vault -- vault auth enable kubernetes ``` Configure the auth method using the Vault pod's own cluster credentials: ```shell kubectl exec -it vault-0 -n vault -- /bin/sh -c ' vault write auth/kubernetes/config \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt ' ``` ### 4.3 Write a policy granting read access to the ping secrets ```shell kubectl exec -it vault-0 -n vault -- vault policy write ping-policy - <<'EOF' path "ping/data/pd-env" { capabilities = ["read"] } path "ping/data/pdpwd" { capabilities = ["read"] } path "ping/data/pf-env" { capabilities = ["read"] } EOF ``` ### 4.4 Bind the policy to the ServiceAccount ```shell kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/role/ping-role \ bound_service_account_names=ping-vault-auth \ bound_service_account_namespaces=ping \ policies=ping-policy \ ttl=1h ``` Any pod in the `ping` namespace using the `ping-vault-auth` ServiceAccount can now request secrets. ## Step 5: PingDirectory Values | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The next two sections provide snippets from a values file for exploration and discussion. Refer to [Step 7](#devops-vault-spc-step7) for the location of this file. | The `objects` block requests three files from Vault: the `content` key of `ping/pd-env` (mounted as `pingdirectory.env` and sourced by the Ping startup hooks), the `root-user-password` key from `ping/pdpwd` (mounted as a standalone file), and optionally `PingDirectory.lic` for static-license deployments (see the note in Step 3.3). Setting `SECRETS_DIR` injects the `.env` file's key-value pairs as environment variables; `ROOT_USER_PASSWORD_FILE` tells PingDirectory to read its root password from the mounted file. | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | `ROOT_USER_PASSWORD_FILE` delivers the root password via a mounted file rather than a static environment variable, which is what enables Vault secret rotation to work. However, PingData products (PingDirectory, PingDirectoryProxy, PingDataSync) write credentials into an internal configuration database at first startup and do not re-read them on subsequent restarts unless explicitly told to do so. If you intend to rotate the `root-user-password` secret in Vault, you must also set `PD_REBUILD_ON_RESTART: "true"` in the `envs` block — without it, `manage-profile replace-profile` detects no profile changes and exits without re-applying the rotated password. See the [Secret Rotation](#devops-vault-spc-rotation) section for full details. | pingdirectory portion of ping-values.yaml ```yaml pingdirectory: enabled: true envs: SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git SERVER_PROFILE_PATH: getting-started/pingdirectory SECRETS_DIR: /run/vault-secrets ROOT_USER_PASSWORD_FILE: "/run/vault-secrets/root-user-password" secretProviderClass: enabled: true create: true provider: vault mountPath: /run/vault-secrets parameters: vaultAddress: http://vault.vault:8200 roleName: ping-role objects: | - objectName: "pingdirectory.env" secretPath: "ping/data/pd-env" secretKey: "content" - objectName: "root-user-password" secretPath: "ping/data/pdpwd" secretKey: "root-user-password" - objectName: "PingDirectory.lic" secretPath: "ping/data/pdpwd" secretKey: "pd-license" ``` ## Step 6: PingFederate Values PingFederate follows the same pattern — one `*.env` file per pod type. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | The CSI driver propagates updated Vault secrets to the mounted file automatically, but PingFederate does not re-read environment variables from that file while running. A pod restart is required to pick up rotated secrets. Depending on your [operating pattern](https://devops.pingidentity.com/how-to/operatingPatterns/) (Deployment vs. StatefulSet, single node vs. clustered), a rolling restart of the affected pods is typically sufficient. If you rotate `PF_RUN_PF_CLUSTER_AUTH_PWD`, restart **both** admin and engine pods — the cluster auth password must match across all members. | pingfederate-admin portion of ping-values.yaml ```yaml pingfederate-admin: enabled: true envs: SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git SERVER_PROFILE_PATH: getting-started/pingfederate SECRETS_DIR: /run/vault-secrets secretProviderClass: enabled: true create: true provider: vault mountPath: /run/vault-secrets parameters: vaultAddress: http://vault.vault:8200 roleName: ping-role objects: | - objectName: "pingfederate.env" secretPath: "ping/data/pf-env" secretKey: "content" ``` pingfederate-engine portion of ping-values.yaml ```yaml pingfederate-engine: enabled: true envs: SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git SERVER_PROFILE_PATH: getting-started/pingfederate SECRETS_DIR: /run/vault-secrets secretProviderClass: enabled: true create: true provider: vault mountPath: /run/vault-secrets parameters: vaultAddress: http://vault.vault:8200 roleName: ping-role objects: | - objectName: "pingfederate.env" secretPath: "ping/data/pf-env" secretKey: "content" ``` ## Step 7: Deploy The complete values file combining Steps 5 and 6 is in the [getting-started repository](https://github.com/pingidentity/pingidentity-devops-getting-started) at `30-helm/vault-spc/ping-values.yaml`. Clone it if you haven't already, then deploy. The `global.rbac` block tells the chart to assign the `ping-vault-auth` ServiceAccount — which is bound to the Vault role — to every workload pod. Without this, pods authenticate with the `default` ServiceAccount, which is not authorized in Vault. ```shell git clone https://github.com/pingidentity/pingidentity-devops-getting-started.git cd pingidentity-devops-getting-started helm install ping pingidentity/ping-devops \ -n ping \ --set global.rbac.serviceAccountName=ping-vault-auth \ --set global.rbac.applyServiceAccountToWorkload=true \ -f 30-helm/vault-spc/ping-values.yaml ``` Watch the pods reach `Running`: ```shell kubectl get pods -n ping -w ``` ## Verify ### Verify the .env file is mounted Confirm the CSI driver wrote the secret file into the pod: ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1) \ -- ls /run/vault-secrets ``` Expected: ``` PingDirectory.lic pingdirectory.env root-user-password ``` ### Verify environment variables were sourced Confirm `ROOT_USER_PASSWORD_FILE` is set and the `.env` file was consumed at startup: ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1) \ -- env | grep ROOT_USER_PASSWORD_FILE ``` Expected: ``` ROOT_USER_PASSWORD_FILE=/run/vault-secrets/root-user-password ``` | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | `PING_IDENTITY_DEVOPS_USER` and `PING_IDENTITY_DEVOPS_KEY` are read from the mounted `.env` file at startup to obtain the development license, but are not exported into the container's environment. If the server started successfully, the credentials were consumed correctly. | Confirm the password file itself is present: ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1) \ -- cat /run/vault-secrets/root-user-password ``` ### Verify PingFederate ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingfederate-admin -o name | head -1) \ -- env | grep -E 'PING_IDENTITY_PASSWORD|PING_IDENTITY_DEVOPS_USER' ``` ## Alternative: Kubernetes Secret Sync If you cannot set `SECRETS_DIR` on the container — for example when a platform team manages the deployment spec — you can use the Kubernetes Secret sync mechanism instead. The `secretObjects` block instructs the CSI driver to create a Kubernetes `Secret` from the mounted files; `container.envFrom` then injects that Secret's keys as environment variables. This approach requires one `objectName` entry per secret key (rather than one per product) and creates Kubernetes `Secret` objects in the cluster. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The Kubernetes `Secret` created by `secretObjects` is only created after the first pod successfully mounts the CSI volume. It does not exist before the first pod starts. | For this path, seed Vault with individual per-key secrets instead of the `content` key format used in Step 3. You will also need to update the Vault policy in Step 4.3 to cover these paths: ```shell vault kv put ping/devops \ PING_IDENTITY_DEVOPS_USER="" \ PING_IDENTITY_DEVOPS_KEY="" \ PING_IDENTITY_ACCEPT_EULA="YES" vault kv put ping/pingdirectory \ ROOT_USER_PASSWORD="3FederateMuchMore" \ ROOT_USER_DN="cn=administrator" vault kv put ping/pingfederate \ PING_IDENTITY_PASSWORD="" ``` The values configuration for PingDirectory then becomes: pingdirectory with secretObjects/envFrom ```yaml pingdirectory: enabled: true envs: SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git SERVER_PROFILE_PATH: getting-started/pingdirectory secretProviderClass: enabled: true create: true provider: vault mountPath: /run/vault-secrets parameters: vaultAddress: http://vault.vault:8200 roleName: ping-role objects: | - objectName: "PING_IDENTITY_DEVOPS_USER" secretPath: "ping/data/devops" secretKey: "PING_IDENTITY_DEVOPS_USER" - objectName: "PING_IDENTITY_DEVOPS_KEY" secretPath: "ping/data/devops" secretKey: "PING_IDENTITY_DEVOPS_KEY" - objectName: "PING_IDENTITY_ACCEPT_EULA" secretPath: "ping/data/devops" secretKey: "PING_IDENTITY_ACCEPT_EULA" - objectName: "ROOT_USER_PASSWORD" secretPath: "ping/data/pingdirectory" secretKey: "ROOT_USER_PASSWORD" - objectName: "ROOT_USER_DN" secretPath: "ping/data/pingdirectory" secretKey: "ROOT_USER_DN" secretObjects: - secretName: ping-pd-env type: Opaque data: - objectName: PING_IDENTITY_DEVOPS_USER key: PING_IDENTITY_DEVOPS_USER - objectName: PING_IDENTITY_DEVOPS_KEY key: PING_IDENTITY_DEVOPS_KEY - objectName: PING_IDENTITY_ACCEPT_EULA key: PING_IDENTITY_ACCEPT_EULA - objectName: ROOT_USER_PASSWORD key: ROOT_USER_PASSWORD - objectName: ROOT_USER_DN key: ROOT_USER_DN container: envFrom: - secretRef: name: ping-pd-env ``` ## Optional: Using a Pre-Existing SecretProviderClass If your platform team manages `SecretProviderClass` resources centrally, set `create: false` and provide the name. The chart will wire up the CSI volume pointing at that resource without attempting to create or own it. For example: ```yaml pingdirectory: secretProviderClass: enabled: true create: false name: platform-team-pd-spc ``` ## Secret Rotation This section walks through rotating the PingDirectory root user password end-to-end: updating it in Vault, confirming the CSI driver propagates it to the pod, and restarting the pod so the product picks it up. ### Before you rotate: add `PD_REBUILD_ON_RESTART` PingData products (PingDirectory, PingDirectoryProxy, PingDataSync) write credentials into an internal configuration database at first startup. On subsequent restarts, `manage-profile replace-profile` checks whether the server profile has changed. Secret files delivered via `*_FILE` environment variables are not tracked in the profile manifest, so `replace-profile` detects "no changes" and exits without re-applying credentials — even though the mounted file now contains the rotated value. The pod starts successfully using the old internally-stored password; the rotation silently fails. Setting `PD_REBUILD_ON_RESTART: "true"` forces `replace-profile` to run in full replace mode, re-applying all credentials from the mounted files on every restart. Add this to the `pingdirectory` `envs` block before rotating any credential: ```yaml pingdirectory: envs: PD_REBUILD_ON_RESTART: "true" ``` Then apply it: ```shell helm upgrade ping pingidentity/ping-devops \ -n ping \ --reuse-values \ --set-string pingdirectory.envs.PD_REBUILD_ON_RESTART=true ``` | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Use `--set-string` rather than `--set` for this value. `--set` without a type hint sends `true` as a boolean, which Kubernetes rejects when writing it to a ConfigMap string field. `--set-string` forces the value to a string. If you prefer to manage this in your values file, `PD_REBUILD_ON_RESTART: "true"` in YAML is equivalent — the YAML quotes are syntax, not value characters. | ### Step 1: Update the secret in Vault ```shell kubectl exec -it vault-0 -n vault -- /bin/sh -c ' export VAULT_TOKEN=root export VAULT_ADDR=http://127.0.0.1:8200 vault kv patch ping/pdpwd root-user-password="R0tatedNewPa55word!" ' ``` ### Step 2: Confirm the CSI driver has propagated the new value The CSI driver polls Vault on a rotation interval (default: two minutes). Wait for the mounted file to update before restarting the pod: ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1) \ -- cat /run/vault-secrets/root-user-password ``` Repeat until the output shows the new password. ### Step 3: Restart the pod Delete the pod so the StatefulSet recreates it, picking up the rotated secret and running `replace-profile` with `--replaceFullProfile`: ```shell kubectl delete pod \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1 | sed 's|pod/||') \ -n ping ``` | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | In this single-pod walkthrough, deleting the pod is safe — the StatefulSet recreates it immediately. In a production deployment with multiple replicas, rotate one pod at a time: delete it, wait for `1/1 Running`, then proceed to the next. Deleting all pods simultaneously risks replication quorum loss. | Watch it return to `Running`: ```shell kubectl get pods -n ping -w ``` ### Step 4: Verify the new password is active ```shell kubectl exec -n ping \ $(kubectl get pod -n ping -l app.kubernetes.io/name=pingdirectory -o name | head -1) \ -- ldapsearch --useSSL --trustAll -h localhost -p 1636 \ -D "cn=administrator" -w "R0tatedNewPa55word!" \ -b "" -s base "(objectClass=*)" namingContexts ``` A `Result Code: 0 (success)` response confirms the rotated password is active. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | A rotated `PingDirectory.lic` written to the mount point has no effect until the pod restarts and re-runs the Profile stage. The same `PD_REBUILD_ON_RESTART` flag and restart procedure applies. | | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Environment variables injected via `envFrom` are set at container startup from the synced Kubernetes `Secret` and are not re-read at runtime. A Vault secret rotation updates the mounted file and the Kubernetes `Secret`, but the running container's environment does not change until the pod is restarted. Unlike the primary path, there is no `PD_REBUILD_ON_RESTART` equivalent — a restart is always required and is sufficient. | ## Troubleshooting ### Pod stuck in `ContainerCreating` ```shell kubectl describe pod -n ping ``` Common events and their causes: * `failed to get secretproviderclass ping/…​-spc` — the `SecretProviderClass` resource does not exist or is in the wrong namespace. Check `kubectl get secretproviderclass -n ping`. * `failed to mount secrets store objects` — the Vault provider could not authenticate. The most common cause is a mismatch between the pod's ServiceAccount name or namespace and the Vault role's `bound_service_account_names` / `bound_service_account_namespaces`. Verify the pod is using the correct ServiceAccount with `kubectl get pod -n ping -o jsonpath='{.spec.serviceAccountName}'` and confirm the Vault role with `kubectl exec -it vault-0 -n vault — vault read auth/kubernetes/role/ping-role`. * `secrets-store.csi.x-k8s.io/v1 is not available in Capabilities.APIVersions` — the CSI driver CRDs are not installed. Re-run [Step 1](#devops-vault-spc-step1). * `make mountpoint …​ read-only file system` on the SA token path — the CSI mountPath conflicts with the Kubernetes ServiceAccount token path. Set `mountPath: /run/vault-secrets` in the `secretProviderClass` block. ### Vault authentication errors ```shell kubectl logs -n vault -l app.kubernetes.io/name=vault-csi-provider ``` Verify the role binding matches the pod's ServiceAccount: ```shell kubectl exec -it vault-0 -n vault -- vault read auth/kubernetes/role/ping-role ``` ### Environment variables not set after pod starts If the pod reaches `Running` but expected env vars are missing, confirm the startup hooks found the `.env` file: ```shell kubectl logs -n ping | grep -E "SECRETS_DIR|\.env" ``` The log should show `SECRETS_DIR : /run/vault-secrets`. If it shows the default `/run/secrets` instead, the `envs.SECRETS_DIR` override was not applied — check the values file. ### Kubernetes Secret not created (Alternative path only) The synced Secret (`secretObjects`) is only created after the first pod successfully mounts the CSI volume. If the pod never reaches `Running`, the Secret will not exist. Fix the pod startup issue first, then check: ```shell kubectl get secret -n ping ``` ### Verifying the CSI driver and Vault provider DaemonSets ```shell kubectl get daemonset -n kube-system -l app.kubernetes.io/instance=csi-secrets-store kubectl get daemonset -n vault -l app.kubernetes.io/instance=vault ``` Both should show one pod per schedulable node. ## Clean Up ```shell helm uninstall ping -n ping kubectl delete namespace ping helm uninstall vault -n vault kubectl delete namespace vault helm uninstall csi-secrets-store -n kube-system ```