PingDirectoryProxy Automatic Server Discovery Demo

Purpose

This example demonstrates PingDirectoryProxy automatic server discovery from a typical multi-region environment.

Prequisites:

Access to the published pingidentity/ping-devops Helm chart. Ensure you are at version 0.12.1 or later
Access to the pingidentity-server-profiles GitHub repository

The demonstrations below illustrate one method of deploying PingDirectoryProxy to use location-aware load-balancing algorithms without requiring manual post-deployment dsconfig operations. The hook provided in the example server profile automatically configures the proxy’s local location and preferred failover locations based on environment variables.

The server profile hook script and other files provided for this example are for demonstration purposes only. They should be considered only as a starting point for similar functionality in a production environment.

Expected Deployment Flow

The west deployment is the seed side of the example and is applied first. The west PingDirectory pods start first, with west-pingdirectory-0 acting as the seed server for topology operations.

The west PingDirectoryProxy pod is also created by the west Helm release, but its init container waits for the east PingDirectory pods to converge before it starts. This action prevents PingDirectoryProxy from joining the topology while the PingDirectory servers are still converging.

After west PingDirectory pods are running, apply the east deployment. The east PingDirectory pods use the west seed server when they join the topology. The east PingDirectoryProxy pod then waits for the west PingDirectoryProxy pod, so proxy topology operations happen after the directory topology is established.

To summarize, the intended readiness order is:

West PingDirectory pods start and establish the seed topology.
East PingDirectory pods start and join the west seed topology.
West PingDirectoryProxy pod starts, joins the PingDirectory topology, and configures the local west location with east as failover.
East PingDirectoryProxy pod starts, joins the PingDirectory topology, and configures the local east location with west as failover.

Local Working Directory

For this walkthrough, run the commands from a local directory. Create a local demo directory for generated values and DNS files:

mkdir pdproxy-discovery-demo
cd pdproxy-discovery-demo

Helm Repository

Add or refresh the published Ping Identity Helm repository:

helm repo add pingidentity https://helm.pingidentity.com/
helm repo update pingidentity
helm search repo pingidentity/ping-devops --versions | head

Expected Behavior

Without PREFERRED_FAILOVER_LOCATIONS:

K8S_CLUSTER becomes the local proxy location
All non-local entries from K8S_CLUSTERS become preferred failover locations
Ordering follows K8S_CLUSTERS

With PREFERRED_FAILOVER_LOCATIONS:

The hook uses that value as an explicit ordered subset
Each value must exist in K8S_CLUSTERS
The local K8S_CLUSTER must not be included
Duplicate preferred failover values are ignored

Example:

K8S_CLUSTERS="west east"
K8S_CLUSTER="west"
PREFERRED_FAILOVER_LOCATIONS="east"

Expected local proxy location result:

west preferred-failover-location: east

Common Validation Commands

Run these in a session established through kubectl exec in a proxy pod:

dsconfig list-locations
dsconfig get-location-prop --location-name <local-location>
dsconfig list-server-instances --property load-balancing-algorithm-name
ldapsearch -b cn=monitor "(objectclass=ds-load-balancing-algorithm-monitor-entry)"

Useful filtered monitor command:

ldapsearch \
  -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers ldap-external-server

Expected success:

dsconfig list-locations includes local and configured failover locations
dsconfig get-location-prop --location-name <local> shows expected preferred-failover-location values
dsconfig list-server-instances --property load-balancing-algorithm-name shows PingDirectory instances assigned to the expected LBAs
LBA monitor entries report AVAILABLE, not No servers configured

Demo 1: Docker Desktop Namespace Simulation

This first demonstration validates the server-profile hook with minimal infrastructure. It is not a true multi-cluster networking test, but rather simulates two regions with two namespaces in one Kubernetes cluster and uses Kubernetes DNS names instead of external DNS.

Docker Prerequisites

Required:

Docker Desktop with Kubernetes enabled
kubectl
helm
Ping Identity DevOps credentials available as environment variables or in a local .env file

Recommended Docker Desktop resources:

CPUs: 6 or more
Memory: 12 GB or more
Disk: 30 GB free

Verify:

kubectl config current-context
kubectl cluster-info
helm version

Create Namespaces

kubectl create namespace west
kubectl create namespace east

If they already exist:

kubectl get namespace west east

Create DevOps Secret

Using an existing .env file:

kubectl -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl -n west apply -f -

kubectl -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl -n east apply -f -

Using existing environment variables:

kubectl -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl -n west apply -f -

kubectl -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl -n east apply -f -

Verify:

kubectl -n west get secret devops-secret
kubectl -n east get secret devops-secret

Create Docker Desktop Values

Create ./dd-west.yaml:

cat > ./dd-west.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-east-pd:
    name: wait-for-east-pd
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for east PingDirectory..."
      wait-for east-pingdirectory-1.east-pingdirectory-cluster.east.svc.cluster.local:1636 -t 600 -- echo "east PingDirectory running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-east-pd
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local
    PINGDIRECTORY_LDAPS_PORT: "1636"
YAML

Create ./dd-east.yaml:

cat > ./dd-east.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-west-proxy:
    name: wait-for-west-proxy
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for west PingDirectoryProxy..."
      wait-for west-pingdirectoryproxy-0.west-pingdirectoryproxy-cluster.west.svc.cluster.local:1636 -t 600 -- echo "west PingDirectoryProxy running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectory-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".east-pingdirectory-cluster.east.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-west-proxy
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectoryproxy-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".east-pingdirectoryproxy-cluster.east.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local
    PINGDIRECTORY_LDAPS_PORT: "1636"
YAML

Deploy Docker Desktop Demo

Do not use --wait on the west install. The west proxy intentionally waits for east PingDirectory through an init container.

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml

Wait for west PingDirectory pods to start:

kubectl -n west rollout status statefulset/west-pingdirectory --timeout=15m

Install east:

helm upgrade --install east pingidentity/ping-devops \
  -n east \
  -f ./dd-east.yaml

Wait for all workloads:

kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

Confirm DNS and TCP Reachability

kubectl -n west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east-pingdirectory-cluster.east.svc.cluster.local

kubectl -n east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local

kubectl -n west run net-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for east-pingdirectory-0.east-pingdirectory-cluster.east.svc.cluster.local:1636 -t 30 -- echo ok

kubectl -n east run net-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local:1636 -t 30 -- echo ok

Confirm Default Hook Behavior

West:

kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers

East:

kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers

Expected:

west proxy:
  locations: west, east
  west preferred-failover-location: east

east proxy:
  locations: east, west
  east preferred-failover-location: west

both proxies:
  LBA monitor status: AVAILABLE
  num-available-servers: 4

Restart Validation

kubectl -n west rollout restart statefulset/west-pingdirectoryproxy
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west

The proxy pod can report Ready before backend health checks converge. Wait for LBA convergence:

until kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done

Repeat for east:

kubectl -n east rollout restart statefulset/east-pingdirectoryproxy
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east

until kubectl -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done

Expected:

Hook runs again.
No duplicate location problem occurs.
Preferred failover values remain correct.
LBAs become available.

Explicit Preferred Failover Variation

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east"

helm upgrade --install east pingidentity/ping-devops \
  -n east \
  -f ./dd-east.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west"

kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east

Expected:

west preferred-failover-location: east
east preferred-failover-location: west

Invalid Override Includes Local Cluster

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west east"

Expected:

west proxy startup fails
logs contain PREFERRED_FAILOVER_LOCATIONS
logs contain must not include the local K8S_CLUSTER

Check:

kubectl -n west logs west-pingdirectoryproxy-0

Restore:

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml

kubectl -n west delete pod west-pingdirectoryproxy-0
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m

Invalid Override References Unknown Cluster

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east central"

Expected:

west proxy startup fails
logs contain PREFERRED_FAILOVER_LOCATIONS
logs contain must only include clusters listed in K8S_CLUSTERS

Restore:

helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml

kubectl -n west delete pod west-pingdirectoryproxy-0
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m

Docker Desktop Cleanup

helm uninstall west -n west
helm uninstall east -n east

kubectl delete namespace west
kubectl delete namespace east

Demo 2: Local Multi-Cluster with kind

Use this demo to validate the real cross-cluster hostname model.

This demo uses two separate kind clusters:

kind-west
kind-east

This path exercises separate Kubernetes control planes and requires:

stable per-pod LoadBalancer IPs for PingDirectory
routable LoadBalancer IPs for PingDirectoryProxy
DNS records for external names
CoreDNS forwarding in both clusters
cross-cluster TCP reachability

kind Prerequisites

Required:

Docker
If using Docker Desktop, ensure Kubernetes is disabled so as to not interfere with kind
kind
kubectl
helm
Ping Identity DevOps credentials available as environment variables or in a local .env file

Recommended resources:

CPUs: 8 or more
Memory: 16 GB or more

Create kind Clusters

kind create cluster --name west
kind create cluster --name east

Verify contexts:

kubectl config get-contexts | grep kind-

Expected contexts:

kind-west
kind-east

Inspect Docker Network

Both kind clusters usually use the Docker network named kind.

docker network inspect kind

Record the subnet. Example:

172.19.0.0/16

Choose non-overlapping IP ranges from that subnet:

export DNSMASQ_IP=172.19.0.53
export WEST_LB_RANGE=172.19.10.200-172.19.10.230
export EAST_LB_RANGE=172.19.11.200-172.19.11.230

Adjust these values if the Docker kind network uses a different subnet.

Install MetalLB

kubectl --context kind-west apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml
kubectl --context kind-east apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml

kubectl --context kind-west -n metallb-system wait --for=condition=Available deployment/controller --timeout=180s
kubectl --context kind-east -n metallb-system wait --for=condition=Available deployment/controller --timeout=180s

Create MetalLB pools:

cat > ./kind-west-metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pdproxy-discovery-west
  namespace: metallb-system
spec:
  addresses:
  - ${WEST_LB_RANGE}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: pdproxy-discovery-west
  namespace: metallb-system
spec:
  ipAddressPools:
  - pdproxy-discovery-west
EOF

cat > ./kind-east-metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pdproxy-discovery-east
  namespace: metallb-system
spec:
  addresses:
  - ${EAST_LB_RANGE}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: pdproxy-discovery-east
  namespace: metallb-system
spec:
  ipAddressPools:
  - pdproxy-discovery-east
EOF

kubectl --context kind-west apply -f ./kind-west-metallb.yaml
kubectl --context kind-east apply -f ./kind-east-metallb.yaml

Run dnsmasq

Create a small dnsmasq image:

mkdir -p dnsmasq

cat > ./dnsmasq/Dockerfile <<'EOF'
FROM alpine:3.23
RUN apk add --no-cache dnsmasq
ENTRYPOINT ["dnsmasq"]
EOF

docker build -t pdproxy-discovery-dnsmasq:2.91 ./dnsmasq

Create DNS files:

cat > ./dnsmasq/dnsmasq.conf <<'EOF'
no-daemon
log-queries
log-facility=-
listen-address=0.0.0.0
bind-interfaces
addn-hosts=/etc/dnsmasq.d/hosts.lab
EOF

cat > ./dnsmasq/hosts.lab <<'EOF'
# Filled in after LoadBalancer services receive IPs.
EOF

Start dnsmasq:

docker rm -f pdproxy-discovery-dnsmasq 2>/dev/null || true

docker run -d --name pdproxy-discovery-dnsmasq \
  --network kind \
  --ip "${DNSMASQ_IP}" \
  -v "$PWD/./dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf:ro" \
  -v "$PWD/./dnsmasq/hosts.lab:/etc/dnsmasq.d/hosts.lab:ro" \
  pdproxy-discovery-dnsmasq:2.91

Verify:

docker logs pdproxy-discovery-dnsmasq

Configure CoreDNS Forwarding

Patch both clusters so west.example.com and east.example.com forward to dnsmasq.

kubectl --context kind-west -n kube-system get configmap coredns -o yaml > ./kind-west-coredns.before.yaml
kubectl --context kind-east -n kube-system get configmap coredns -o yaml > ./kind-east-coredns.before.yaml

Patch CoreDNS in west:

If DNSMASQ_IP differs from the example, replace 172.19.0.53 in both patch payloads.

kubectl --context kind-west -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 172.19.0.53\n}\n\neast.example.com:53 {\n    forward . 172.19.0.53\n}\n\n.:53 {\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'

Patch CoreDNS in east with the same forwarding blocks:

kubectl --context kind-east -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 172.19.0.53\n}\n\neast.example.com:53 {\n    forward . 172.19.0.53\n}\n\n.:53 {\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'

Restart CoreDNS:

kubectl --context kind-west -n kube-system rollout restart deployment coredns
kubectl --context kind-east -n kube-system rollout restart deployment coredns
kubectl --context kind-west -n kube-system rollout status deployment coredns --timeout=120s
kubectl --context kind-east -n kube-system rollout status deployment coredns --timeout=120s

Create Namespaces and Secrets

kubectl --context kind-west create namespace west
kubectl --context kind-east create namespace east

Using an existing .env file:

kubectl --context kind-west -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context kind-west -n west apply -f -

kubectl --context kind-east -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context kind-east -n east apply -f -

Using existing environment variables:

kubectl --context kind-west -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context kind-west -n west apply -f -

kubectl --context kind-east -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context kind-east -n east apply -f -

Create kind Values

Create ./kind-west.yaml:

cat > ./kind-west.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-east-pd:
    name: wait-for-east-pd
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for east PingDirectory..."
      wait-for east-pingdirectory-1.east.example.com:1636 -t 900 -- echo "east PingDirectory running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    MAKELDIF_USERS: "2000"
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
  services:
    loadBalancerServicePerPod: true
    loadBalancerExternalDNSHostnameSuffix: .west.example.com

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-east-pd
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west.example.com
    PINGDIRECTORY_LDAPS_PORT: "1636"
  services:
    useLoadBalancerForDataService: true
    dataExternalDNSHostname: west-pingdirectoryproxy-0.west.example.com
    ldaps:
      servicePort: 1636
      containerPort: 1636
      clusterService: true
      dataService: true
YAML

Create ./kind-east.yaml:

cat > ./kind-east.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-west-proxy:
    name: wait-for-west-proxy
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for west PingDirectoryProxy..."
      wait-for west-pingdirectoryproxy-0.west.example.com:1636 -t 900 -- echo "west PingDirectoryProxy running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    MAKELDIF_USERS: "2000"
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectory-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".east.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
  services:
    loadBalancerServicePerPod: true
    loadBalancerExternalDNSHostnameSuffix: .east.example.com

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-west-proxy
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectoryproxy-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".east.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west.example.com
    PINGDIRECTORY_LDAPS_PORT: "1636"
  services:
    useLoadBalancerForDataService: true
    dataExternalDNSHostname: east-pingdirectoryproxy-0.east.example.com
    ldaps:
      servicePort: 1636
      containerPort: 1636
      clusterService: true
      dataService: true
YAML

Deploy West

Do not use --wait. The west proxy waits for east PingDirectory through an init container.

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml

Patch the west proxy LoadBalancer service so the advertised proxy hostname can route to the starting proxy before the proxy pod is marked Ready:

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

This patch is required when the chart does not expose publishNotReadyAddresses for the generated service. The proxy topology join reaches the proxy through its advertised external LoadBalancer hostname before readiness succeeds. This patch is needed for this local example to work.

Wait for west PingDirectory pods to start:

kubectl --context kind-west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context kind-west -n west get svc

Record:

west-pingdirectory-0 external IP
west-pingdirectory-1 external IP
west-pingdirectoryproxy external IP

Update ./dnsmasq/hosts.lab with west records.

Example:

172.19.10.202 west-pingdirectory-0.west.example.com
172.19.10.201 west-pingdirectory-1.west.example.com
172.19.10.200 west-pingdirectoryproxy-0.west.example.com

Restart dnsmasq:

docker restart pdproxy-discovery-dnsmasq

Verify west DNS from both clusters:

kubectl --context kind-west run dns-west-from-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-west-from-west --timeout=120s
kubectl --context kind-west logs dns-west-from-west
kubectl --context kind-west delete pod dns-west-from-west

kubectl --context kind-east run dns-west-from-east --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-west-from-east --timeout=120s
kubectl --context kind-east logs dns-west-from-east
kubectl --context kind-east delete pod dns-west-from-east

Deploy East

helm upgrade --install east pingidentity/ping-devops \
  --kube-context kind-east \
  -n east \
  -f ./kind-east.yaml

Patch the east proxy LoadBalancer service for the same pre-readiness routing requirement:

kubectl --context kind-east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

Get east service IPs:

kubectl --context kind-east -n east get svc

Append east records to ./dnsmasq/hosts.lab.

Example:

172.19.11.201 east-pingdirectory-0.east.example.com
172.19.11.202 east-pingdirectory-1.east.example.com
172.19.11.200 east-pingdirectoryproxy-0.east.example.com

Restart dnsmasq:

docker restart pdproxy-discovery-dnsmasq

Verify east DNS from both clusters:

kubectl --context kind-west run dns-east-from-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-east-from-west --timeout=120s
kubectl --context kind-west logs dns-east-from-west
kubectl --context kind-west delete pod dns-east-from-west

kubectl --context kind-east run dns-east-from-east --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-east-from-east --timeout=120s
kubectl --context kind-east logs dns-east-from-east
kubectl --context kind-east delete pod dns-east-from-east

Confirm Cross-Cluster TCP Reachability

kubectl --context kind-west run net-west-to-east-pd --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for east-pingdirectory-0.east.example.com:1636 -t 60 -- echo ok
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/net-west-to-east-pd --timeout=120s
kubectl --context kind-west logs net-west-to-east-pd
kubectl --context kind-west delete pod net-west-to-east-pd

kubectl --context kind-east run net-east-to-west-pd --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectory-0.west.example.com:1636 -t 60 -- echo ok
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/net-east-to-west-pd --timeout=120s
kubectl --context kind-east logs net-east-to-west-pd
kubectl --context kind-east delete pod net-east-to-west-pd

kubectl --context kind-east run proxy-east-to-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectoryproxy-0.west.example.com:1636 -t 60 -- echo ok
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/proxy-east-to-west --timeout=120s
kubectl --context kind-east logs proxy-east-to-west
kubectl --context kind-east delete pod proxy-east-to-west

Wait for Workloads

kubectl --context kind-west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

Confirm kind Default Hook Behavior

West:

kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers

East:

kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers

Expected:

west proxy:
  locations: west, east
  west preferred-failover-location: east

east proxy:
  locations: east, west
  east preferred-failover-location: west

both proxies:
  LBA monitor status: AVAILABLE
  num-available-servers: 4

kind Explicit Preferred Failover Variation

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east"

helm upgrade --install east pingidentity/ping-devops \
  --kube-context kind-east \
  -n east \
  -f ./kind-east.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west"

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east

Expected:

west preferred-failover-location: east
east preferred-failover-location: west

kind Invalid Local Override

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west east"

Expected:

west proxy fails during startup
logs show local cluster is not allowed in PREFERRED_FAILOVER_LOCATIONS

Check:

kubectl --context kind-west -n west logs west-pingdirectoryproxy-0 -f

Restore:

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west delete pod west-pingdirectoryproxy-0
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m

kind Invalid Unknown Override

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east central"

Expected:

west proxy fails during startup
logs show every preferred failover location must be listed in K8S_CLUSTERS

Restore:

helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west delete pod west-pingdirectoryproxy-0
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m

kind Restart Validation

kubectl --context kind-west -n west rollout restart statefulset/west-pingdirectoryproxy
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west

until kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done

Expected:

hook re-runs
preferred failover remains correct
LBAs become available

kind Cleanup

helm --kube-context kind-west uninstall west -n west
helm --kube-context kind-east uninstall east -n east

kind delete cluster --name west
kind delete cluster --name east

docker rm -f pdproxy-discovery-dnsmasq

minikube Variant

This variant uses two Docker-driver minikube profiles:

west
east

minikube Prerequisites

Required:

Docker Desktop with Kubernetes disabled
kubectl
helm
minikube
Ping Identity DevOps credentials available as environment variables or in a local .env file

The profile subnets vary by machine. Substitute the ranges from docker network inspect.

Create Profiles

minikube start --profile west --driver=docker --cpus=4 --memory=8192
minikube start --profile east --driver=docker --cpus=4 --memory=8192

Verify contexts:

kubectl config get-contexts | grep -E 'west|east'

LoadBalancer Support

Enable MetalLB:

minikube -p west addons enable metallb
minikube -p east addons enable metallb

This step requires Docker API access because minikube inspects and updates the profile containers.

Find Docker network subnets:

docker network inspect west --format '{{json .IPAM.Config}}'
docker network inspect east --format '{{json .IPAM.Config}}'

Example subnets:

west: 192.168.49.0/24
east: 192.168.58.0/24

Patch deterministic MetalLB ranges. Adjust IPs for the actual subnets:

kubectl --context west -n metallb-system patch configmap config --type merge \
  -p '{"data":{"config":"address-pools:\n- name: west\n  protocol: layer2\n  addresses:\n  - 192.168.49.200-192.168.49.209\n"}}'

kubectl --context east -n metallb-system patch configmap config --type merge \
  -p '{"data":{"config":"address-pools:\n- name: east\n  protocol: layer2\n  addresses:\n  - 192.168.58.200-192.168.58.209\n"}}'

Attach each minikube node container to the other profile network:

docker network connect east west || true
docker network connect west east || true

minikube dnsmasq and CoreDNS

Create DNS files:

mkdir -p ./minikube-dnsmasq

cat > ./minikube-dnsmasq/dnsmasq.conf <<'EOF'
no-daemon
log-queries
log-facility=-
listen-address=0.0.0.0
bind-interfaces
addn-hosts=/etc/dnsmasq.d/hosts.lab
EOF

cat > ./minikube-dnsmasq/hosts.lab <<'EOF'
# Filled in after minikube LoadBalancer services receive IPs.
EOF

Build the dnsmasq image if it is not already present:

docker build -t pdproxy-discovery-dnsmasq:2.91 ./dnsmasq

Run dnsmasq on both profile networks. Adjust resolver IPs for the actual subnets:

docker rm -f pdproxy-discovery-minikube-dnsmasq 2>/dev/null || true

docker run -d --name pdproxy-discovery-minikube-dnsmasq \
  --network west \
  --ip 192.168.49.250 \
  -v "$PWD/./minikube-dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf:ro" \
  -v "$PWD/./minikube-dnsmasq/hosts.lab:/etc/dnsmasq.d/hosts.lab:ro" \
  pdproxy-discovery-dnsmasq:2.91 \
  -C /etc/dnsmasq.conf

docker network connect --ip 192.168.58.250 east pdproxy-discovery-minikube-dnsmasq

Patch CoreDNS in each profile. Forward to the dnsmasq IP on that profile’s local Docker network.

If your profile subnets differ from the examples, replace 192.168.49.250 and 192.168.58.250 in the patch payloads.

Back up the existing CoreDNS ConfigMaps:

kubectl --context west -n kube-system get configmap coredns -o yaml > ./minikube-west-coredns.before.yaml
kubectl --context east -n kube-system get configmap coredns -o yaml > ./minikube-east-coredns.before.yaml

Patch the west profile, using the west dnsmasq IP:

kubectl --context west -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 192.168.49.250\n}\n\neast.example.com:53 {\n    forward . 192.168.49.250\n}\n\n.:53 {\n    log\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    hosts {\n       192.168.65.254 host.minikube.internal\n       fallthrough\n    }\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'

Patch the east profile, using the east dnsmasq IP:

kubectl --context east -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 192.168.58.250\n}\n\neast.example.com:53 {\n    forward . 192.168.58.250\n}\n\n.:53 {\n    log\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    hosts {\n       192.168.65.254 host.minikube.internal\n       fallthrough\n    }\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'

Restart CoreDNS:

kubectl --context west -n kube-system rollout restart deployment coredns
kubectl --context east -n kube-system rollout restart deployment coredns
kubectl --context west -n kube-system rollout status deployment coredns --timeout=120s
kubectl --context east -n kube-system rollout status deployment coredns --timeout=120s

minikube Namespaces and Secrets

kubectl --context west create namespace west
kubectl --context east create namespace east

Using an existing .env file:

kubectl --context west -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context west -n west apply -f -

kubectl --context east -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context east -n east apply -f -

Using existing environment variables:

kubectl --context west -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context west -n west apply -f -

kubectl --context east -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context east -n east apply -f -

minikube Deploy West

Use the kind values files created earlier. They use the same west.example.com and east.example.com external hostname model.

helm upgrade --install west pingidentity/ping-devops \
  --kube-context west \
  -n west \
  -f ./kind-west.yaml

kubectl --context west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context west -n west get svc

Add west records to ./minikube-dnsmasq/hosts.lab using actual LoadBalancer IPs.

Example:

192.168.49.201 west-pingdirectory-0.west.example.com
192.168.49.202 west-pingdirectory-1.west.example.com
192.168.49.200 west-pingdirectoryproxy-0.west.example.com

Restart dnsmasq and verify west DNS from both profiles:

docker restart pdproxy-discovery-minikube-dnsmasq

kubectl --context west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com

kubectl --context east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com

minikube Deploy East

helm upgrade --install east pingidentity/ping-devops \
  --kube-context east \
  -n east \
  -f ./kind-east.yaml

kubectl --context east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context east -n east get svc

Add east records to ./minikube-dnsmasq/hosts.lab using actual LoadBalancer IPs.

Example:

192.168.58.201 east-pingdirectory-0.east.example.com
192.168.58.202 east-pingdirectory-1.east.example.com
192.168.58.200 east-pingdirectoryproxy-0.east.example.com

Restart dnsmasq and verify east DNS from both profiles:

docker restart pdproxy-discovery-minikube-dnsmasq

kubectl --context west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com

kubectl --context east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com

Wait for all workloads:

kubectl --context west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context east -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl --context west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

Then use the TCP verification, proxy validation, variation, and restart steps from the kind demo, replacing:

kind-west -> west
kind-east -> east

After restoring from an intentional invalid override, delete the failed proxy pod if it remains on the old invalid checksum:

kubectl --context west -n west delete pod west-pingdirectoryproxy-0
kubectl --context west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m

minikube Cleanup

helm --kube-context west uninstall west -n west
helm --kube-context east uninstall east -n east

minikube delete --profile west
minikube delete --profile east

docker rm -f pdproxy-discovery-minikube-dnsmasq

Troubleshooting

Confirm devops-secret exists in every namespace.
Confirm the Helm command uses pingidentity/ping-devops, not a local chart path.
Confirm SERVER_PROFILE_BRANCH is not set in the values.
Confirm SERVER_PROFILE_PATH is pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy.
Confirm DNS resolves every name used by PINGDIRECTORY_HOSTNAME, K8S_POD_HOSTNAME_PREFIX, and K8S_POD_HOSTNAME_SUFFIX.
Confirm TCP port 1636 is reachable cross-region.
Confirm proxy logs show the location hook ran.
Confirm dsconfig list-locations.
Confirm preferred-failover-location.
Confirm LBA monitor availability.

Useful log checks:

kubectl -n west logs west-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
kubectl -n east logs east-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"

For kind:

kubectl --context kind-west -n west logs west-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
kubectl --context kind-east -n east logs east-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"

Demo Closeout Evidence

Capture these outputs:

kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor "(objectclass=ds-load-balancing-algorithm-monitor-entry)"
kubectl -n west logs west-pingdirectoryproxy-0 | grep "PingDirectoryProxy multi-cluster location configuration completed"

For kind and minikube, include the appropriate --context values.

Ping Identity Devops

PingDirectoryProxy Automatic Server Discovery Demo

Purpose

Expected Deployment Flow

Local Working Directory

Helm Repository

Expected Behavior

Common Validation Commands

Demo 1: Docker Desktop Namespace Simulation

Docker Prerequisites

Create Namespaces

Create DevOps Secret

Create Docker Desktop Values

Deploy Docker Desktop Demo

Confirm DNS and TCP Reachability

Confirm Default Hook Behavior

Restart Validation

Explicit Preferred Failover Variation

Invalid Override Includes Local Cluster

Invalid Override References Unknown Cluster

Docker Desktop Cleanup

Demo 2: Local Multi-Cluster with kind

kind Prerequisites

Create kind Clusters

Inspect Docker Network

Install MetalLB

Run dnsmasq

Configure CoreDNS Forwarding

Create Namespaces and Secrets

Create kind Values

Deploy West

Deploy East

Confirm Cross-Cluster TCP Reachability

Wait for Workloads

Confirm kind Default Hook Behavior

kind Explicit Preferred Failover Variation

kind Invalid Local Override

kind Invalid Unknown Override

kind Restart Validation

kind Cleanup

minikube Variant

minikube Prerequisites

Create Profiles

LoadBalancer Support

minikube dnsmasq and CoreDNS

minikube Namespaces and Secrets

minikube Deploy West

minikube Deploy East

minikube Cleanup

Troubleshooting

Demo Closeout Evidence