---
title: PingDirectoryProxy Automatic Server Discovery Demo
description: This example demonstrates PingDirectoryProxy automatic server discovery from a typical multi-region environment.
component: devops
page_id: devops::deployment/deployPDProxyDiscoveryDemoWalkthrough
canonical_url: https://developer.pingidentity.com/devops/deployment/deployPDProxyDiscoveryDemoWalkthrough.html
section_ids:
  devops-purpose: Purpose
  devops-expected-deployment-flow: Expected Deployment Flow
  devops-local-working-directory: Local Working Directory
  devops-helm-repository: Helm Repository
  devops-expected-behavior: Expected Behavior
  devops-common-validation-commands: Common Validation Commands
  devops-demo-1-docker-desktop-namespace-simulation: "Demo 1: Docker Desktop Namespace Simulation"
  devops-docker-prerequisites: Docker Prerequisites
  devops-create-namespaces: Create Namespaces
  devops-create-devops-secret: Create DevOps Secret
  devops-create-docker-desktop-values: Create Docker Desktop Values
  devops-deploy-docker-desktop-demo: Deploy Docker Desktop Demo
  devops-confirm-dns-and-tcp-reachability: Confirm DNS and TCP Reachability
  devops-confirm-default-hook-behavior: Confirm Default Hook Behavior
  devops-restart-validation: Restart Validation
  devops-explicit-preferred-failover-variation: Explicit Preferred Failover Variation
  devops-invalid-override-includes-local-cluster: Invalid Override Includes Local Cluster
  devops-invalid-override-references-unknown-cluster: Invalid Override References Unknown Cluster
  devops-docker-desktop-cleanup: Docker Desktop Cleanup
  devops-demo-2-local-multi-cluster-with-kind: "Demo 2: Local Multi-Cluster with kind"
  devops-kind-prerequisites: kind Prerequisites
  devops-create-kind-clusters: Create kind Clusters
  devops-inspect-docker-network: Inspect Docker Network
  devops-install-metallb: Install MetalLB
  devops-run-dnsmasq: Run dnsmasq
  devops-configure-coredns-forwarding: Configure CoreDNS Forwarding
  devops-create-namespaces-and-secrets: Create Namespaces and Secrets
  devops-create-kind-values: Create kind Values
  devops-deploy-west: Deploy West
  devops-deploy-east: Deploy East
  devops-confirm-cross-cluster-tcp-reachability: Confirm Cross-Cluster TCP Reachability
  devops-wait-for-workloads: Wait for Workloads
  devops-confirm-kind-default-hook-behavior: Confirm kind Default Hook Behavior
  devops-kind-explicit-preferred-failover-variation: kind Explicit Preferred Failover Variation
  devops-kind-invalid-local-override: kind Invalid Local Override
  devops-kind-invalid-unknown-override: kind Invalid Unknown Override
  devops-kind-restart-validation: kind Restart Validation
  devops-kind-cleanup: kind Cleanup
  devops-minikube-variant: minikube Variant
  devops-minikube-prerequisites: minikube Prerequisites
  devops-create-profiles: Create Profiles
  devops-loadbalancer-support: LoadBalancer Support
  devops-minikube-dnsmasq-and-coredns: minikube dnsmasq and CoreDNS
  devops-minikube-namespaces-and-secrets: minikube Namespaces and Secrets
  devops-minikube-deploy-west: minikube Deploy West
  devops-minikube-deploy-east: minikube Deploy East
  devops-minikube-cleanup: minikube Cleanup
  devops-troubleshooting: Troubleshooting
  devops-demo-closeout-evidence: Demo Closeout Evidence
---

# PingDirectoryProxy Automatic Server Discovery Demo

## Purpose

This example demonstrates PingDirectoryProxy automatic server discovery from a typical multi-region environment.

Prequisites:

* Access to the published `pingidentity/ping-devops` Helm chart. Ensure you are at version 0.12.1 or later

* Access to the `pingidentity-server-profiles` GitHub repository

The demonstrations below illustrate one method of deploying PingDirectoryProxy to use location-aware load-balancing algorithms without requiring manual post-deployment `dsconfig` operations. The hook provided in the example server profile automatically configures the proxy's local location and preferred failover locations based on environment variables.

|   |                                                                                                                                                                                                                         |
| - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|   | The server profile hook script and other files provided for this example are for demonstration purposes only. They should be considered only as a starting point for similar functionality in a production environment. |

## Expected Deployment Flow

The west deployment is the seed side of the example and is applied first. The west PingDirectory pods start first, with `west-pingdirectory-0` acting as the seed server for topology operations.

The west PingDirectoryProxy pod is also created by the west Helm release, but its init container waits for the east PingDirectory pods to converge before it starts. This action prevents PingDirectoryProxy from joining the topology while the PingDirectory servers are still converging.

After west PingDirectory pods are running, apply the east deployment. The east PingDirectory pods use the west seed server when they join the topology. The east PingDirectoryProxy pod then waits for the west PingDirectoryProxy pod, so proxy topology operations happen after the directory topology is established.

To summarize, the intended readiness order is:

1. West PingDirectory pods start and establish the seed topology.

2. East PingDirectory pods start and join the west seed topology.

3. West PingDirectoryProxy pod starts, joins the PingDirectory topology, and configures the local `west` location with `east` as failover.

4. East PingDirectoryProxy pod starts, joins the PingDirectory topology, and configures the local `east` location with `west` as failover.

## Local Working Directory

For this walkthrough, run the commands from a local directory. Create a local demo directory for generated values and DNS files:

```shell
mkdir pdproxy-discovery-demo
cd pdproxy-discovery-demo
```

## Helm Repository

Add or refresh the published Ping Identity Helm repository:

```shell
helm repo add pingidentity https://helm.pingidentity.com/
helm repo update pingidentity
helm search repo pingidentity/ping-devops --versions | head
```

## Expected Behavior

Without `PREFERRED_FAILOVER_LOCATIONS`:

* `K8S_CLUSTER` becomes the local proxy location

* All non-local entries from `K8S_CLUSTERS` become preferred failover locations

* Ordering follows `K8S_CLUSTERS`

With `PREFERRED_FAILOVER_LOCATIONS`:

* The hook uses that value as an explicit ordered subset

* Each value must exist in `K8S_CLUSTERS`

* The local `K8S_CLUSTER` must not be included

* Duplicate preferred failover values are ignored

Example:

```shell
K8S_CLUSTERS="west east"
K8S_CLUSTER="west"
PREFERRED_FAILOVER_LOCATIONS="east"
```

Expected local proxy location result:

```shell
west preferred-failover-location: east
```

## Common Validation Commands

Run these in a session established through `kubectl exec` in a proxy pod:

```shell
dsconfig list-locations
dsconfig get-location-prop --location-name <local-location>
dsconfig list-server-instances --property load-balancing-algorithm-name
ldapsearch -b cn=monitor "(objectclass=ds-load-balancing-algorithm-monitor-entry)"
```

Useful filtered monitor command:

```shell
ldapsearch \
  -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers ldap-external-server
```

Expected success:

* `dsconfig list-locations` includes local and configured failover locations

* `dsconfig get-location-prop --location-name <local>` shows expected `preferred-failover-location` values

* `dsconfig list-server-instances --property load-balancing-algorithm-name` shows PingDirectory instances assigned to the expected LBAs

* LBA monitor entries report `AVAILABLE`, not `No servers configured`

## Demo 1: Docker Desktop Namespace Simulation

This first demonstration validates the server-profile hook with minimal infrastructure. It is not a true multi-cluster networking test, but rather simulates two regions with two namespaces in one Kubernetes cluster and uses Kubernetes DNS names instead of external DNS.

### Docker Prerequisites

Required:

* Docker Desktop with Kubernetes enabled

* `kubectl`

* `helm`

* [Ping Identity DevOps](../how-to/devopsRegistration.html) credentials available as environment variables or in a local `.env` file

Recommended Docker Desktop resources:

* CPUs: 6 or more

* Memory: 12 GB or more

* Disk: 30 GB free

Verify:

```shell
kubectl config current-context
kubectl cluster-info
helm version
```

### Create Namespaces

```shell
kubectl create namespace west
kubectl create namespace east
```

If they already exist:

```shell
kubectl get namespace west east
```

### Create DevOps Secret

Using an existing `.env` file:

```shell
kubectl -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl -n west apply -f -

kubectl -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl -n east apply -f -
```

Using existing environment variables:

```shell
kubectl -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl -n west apply -f -

kubectl -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl -n east apply -f -
```

Verify:

```shell
kubectl -n west get secret devops-secret
kubectl -n east get secret devops-secret
```

### Create Docker Desktop Values

Create `./dd-west.yaml`:

```shell
cat > ./dd-west.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-east-pd:
    name: wait-for-east-pd
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for east PingDirectory..."
      wait-for east-pingdirectory-1.east-pingdirectory-cluster.east.svc.cluster.local:1636 -t 600 -- echo "east PingDirectory running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-east-pd
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local
    PINGDIRECTORY_LDAPS_PORT: "1636"
YAML
```

Create `./dd-east.yaml`:

```shell
cat > ./dd-east.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-west-proxy:
    name: wait-for-west-proxy
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for west PingDirectoryProxy..."
      wait-for west-pingdirectoryproxy-0.west-pingdirectoryproxy-cluster.west.svc.cluster.local:1636 -t 600 -- echo "west PingDirectoryProxy running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectory-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".east-pingdirectory-cluster.east.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectory-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-west-proxy
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectoryproxy-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".east-pingdirectoryproxy-cluster.east.svc.cluster.local"
    K8S_SEED_HOSTNAME_SUFFIX: ".west-pingdirectoryproxy-cluster.west.svc.cluster.local"
    K8S_INCREMENT_PORTS: "false"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local
    PINGDIRECTORY_LDAPS_PORT: "1636"
YAML
```

### Deploy Docker Desktop Demo

Do not use `--wait` on the west install. The west proxy intentionally waits for east PingDirectory through an init container.

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml
```

Wait for west PingDirectory pods to start:

```shell
kubectl -n west rollout status statefulset/west-pingdirectory --timeout=15m
```

Install east:

```shell
helm upgrade --install east pingidentity/ping-devops \
  -n east \
  -f ./dd-east.yaml
```

Wait for all workloads:

```shell
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m
```

### Confirm DNS and TCP Reachability

```shell
kubectl -n west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east-pingdirectory-cluster.east.svc.cluster.local

kubectl -n east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local

kubectl -n west run net-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for east-pingdirectory-0.east-pingdirectory-cluster.east.svc.cluster.local:1636 -t 30 -- echo ok

kubectl -n east run net-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectory-0.west-pingdirectory-cluster.west.svc.cluster.local:1636 -t 30 -- echo ok
```

### Confirm Default Hook Behavior

West:

```shell
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers
```

East:

```shell
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers
```

Expected:

```shell
west proxy:
  locations: west, east
  west preferred-failover-location: east

east proxy:
  locations: east, west
  east preferred-failover-location: west

both proxies:
  LBA monitor status: AVAILABLE
  num-available-servers: 4
```

### Restart Validation

```shell
kubectl -n west rollout restart statefulset/west-pingdirectoryproxy
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
```

The proxy pod can report Ready before backend health checks converge. Wait for LBA convergence:

```shell
until kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done
```

Repeat for east:

```shell
kubectl -n east rollout restart statefulset/east-pingdirectoryproxy
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east

until kubectl -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done
```

Expected:

* Hook runs again.

* No duplicate location problem occurs.

* Preferred failover values remain correct.

* LBAs become available.

### Explicit Preferred Failover Variation

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east"

helm upgrade --install east pingidentity/ping-devops \
  -n east \
  -f ./dd-east.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west"

kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
```

Expected:

```shell
west preferred-failover-location: east
east preferred-failover-location: west
```

### Invalid Override Includes Local Cluster

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west east"
```

Expected:

* west proxy startup fails

* logs contain `PREFERRED_FAILOVER_LOCATIONS`

* logs contain `must not include the local K8S_CLUSTER`

Check:

```shell
kubectl -n west logs west-pingdirectoryproxy-0
```

Restore:

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml

kubectl -n west delete pod west-pingdirectoryproxy-0
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
```

### Invalid Override References Unknown Cluster

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east central"
```

Expected:

* west proxy startup fails

* logs contain `PREFERRED_FAILOVER_LOCATIONS`

* logs contain `must only include clusters listed in K8S_CLUSTERS`

Restore:

```shell
helm upgrade --install west pingidentity/ping-devops \
  -n west \
  -f ./dd-west.yaml

kubectl -n west delete pod west-pingdirectoryproxy-0
kubectl -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
```

### Docker Desktop Cleanup

```shell
helm uninstall west -n west
helm uninstall east -n east

kubectl delete namespace west
kubectl delete namespace east
```

## Demo 2: Local Multi-Cluster with kind

Use this demo to validate the real cross-cluster hostname model.

This demo uses two separate `kind` clusters:

* `kind-west`

* `kind-east`

This path exercises separate Kubernetes control planes and requires:

* stable per-pod `LoadBalancer` IPs for PingDirectory

* routable `LoadBalancer` IPs for PingDirectoryProxy

* DNS records for external names

* CoreDNS forwarding in both clusters

* cross-cluster TCP reachability

### kind Prerequisites

Required:

* Docker

* If using Docker Desktop, ensure Kubernetes is disabled so as to not interfere with kind

* `kind`

* `kubectl`

* `helm`

* [Ping Identity DevOps](../how-to/devopsRegistration.html) credentials available as environment variables or in a local `.env` file

Recommended resources:

* CPUs: 8 or more

* Memory: 16 GB or more

### Create kind Clusters

```shell
kind create cluster --name west
kind create cluster --name east
```

Verify contexts:

```shell
kubectl config get-contexts | grep kind-
```

Expected contexts:

```shell
kind-west
kind-east
```

### Inspect Docker Network

Both kind clusters usually use the Docker network named `kind`.

```shell
docker network inspect kind
```

Record the subnet. Example:

```shell
172.19.0.0/16
```

Choose non-overlapping IP ranges from that subnet:

```shell
export DNSMASQ_IP=172.19.0.53
export WEST_LB_RANGE=172.19.10.200-172.19.10.230
export EAST_LB_RANGE=172.19.11.200-172.19.11.230
```

Adjust these values if the Docker `kind` network uses a different subnet.

### Install MetalLB

```shell
kubectl --context kind-west apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml
kubectl --context kind-east apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml

kubectl --context kind-west -n metallb-system wait --for=condition=Available deployment/controller --timeout=180s
kubectl --context kind-east -n metallb-system wait --for=condition=Available deployment/controller --timeout=180s
```

Create MetalLB pools:

```shell
cat > ./kind-west-metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pdproxy-discovery-west
  namespace: metallb-system
spec:
  addresses:
  - ${WEST_LB_RANGE}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: pdproxy-discovery-west
  namespace: metallb-system
spec:
  ipAddressPools:
  - pdproxy-discovery-west
EOF

cat > ./kind-east-metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pdproxy-discovery-east
  namespace: metallb-system
spec:
  addresses:
  - ${EAST_LB_RANGE}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: pdproxy-discovery-east
  namespace: metallb-system
spec:
  ipAddressPools:
  - pdproxy-discovery-east
EOF

kubectl --context kind-west apply -f ./kind-west-metallb.yaml
kubectl --context kind-east apply -f ./kind-east-metallb.yaml
```

### Run dnsmasq

Create a small dnsmasq image:

```shell
mkdir -p dnsmasq

cat > ./dnsmasq/Dockerfile <<'EOF'
FROM alpine:3.23
RUN apk add --no-cache dnsmasq
ENTRYPOINT ["dnsmasq"]
EOF

docker build -t pdproxy-discovery-dnsmasq:2.91 ./dnsmasq
```

Create DNS files:

```shell
cat > ./dnsmasq/dnsmasq.conf <<'EOF'
no-daemon
log-queries
log-facility=-
listen-address=0.0.0.0
bind-interfaces
addn-hosts=/etc/dnsmasq.d/hosts.lab
EOF

cat > ./dnsmasq/hosts.lab <<'EOF'
# Filled in after LoadBalancer services receive IPs.
EOF
```

Start dnsmasq:

```shell
docker rm -f pdproxy-discovery-dnsmasq 2>/dev/null || true

docker run -d --name pdproxy-discovery-dnsmasq \
  --network kind \
  --ip "${DNSMASQ_IP}" \
  -v "$PWD/./dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf:ro" \
  -v "$PWD/./dnsmasq/hosts.lab:/etc/dnsmasq.d/hosts.lab:ro" \
  pdproxy-discovery-dnsmasq:2.91
```

Verify:

```shell
docker logs pdproxy-discovery-dnsmasq
```

### Configure CoreDNS Forwarding

Patch both clusters so `west.example.com` and `east.example.com` forward to dnsmasq.

```shell
kubectl --context kind-west -n kube-system get configmap coredns -o yaml > ./kind-west-coredns.before.yaml
kubectl --context kind-east -n kube-system get configmap coredns -o yaml > ./kind-east-coredns.before.yaml
```

Patch CoreDNS in west:

If `DNSMASQ_IP` differs from the example, replace `172.19.0.53` in both patch payloads.

```shell
kubectl --context kind-west -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 172.19.0.53\n}\n\neast.example.com:53 {\n    forward . 172.19.0.53\n}\n\n.:53 {\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'
```

Patch CoreDNS in east with the same forwarding blocks:

```shell
kubectl --context kind-east -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 172.19.0.53\n}\n\neast.example.com:53 {\n    forward . 172.19.0.53\n}\n\n.:53 {\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'
```

Restart CoreDNS:

```shell
kubectl --context kind-west -n kube-system rollout restart deployment coredns
kubectl --context kind-east -n kube-system rollout restart deployment coredns
kubectl --context kind-west -n kube-system rollout status deployment coredns --timeout=120s
kubectl --context kind-east -n kube-system rollout status deployment coredns --timeout=120s
```

### Create Namespaces and Secrets

```shell
kubectl --context kind-west create namespace west
kubectl --context kind-east create namespace east
```

Using an existing `.env` file:

```shell
kubectl --context kind-west -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context kind-west -n west apply -f -

kubectl --context kind-east -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context kind-east -n east apply -f -
```

Using existing environment variables:

```shell
kubectl --context kind-west -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context kind-west -n west apply -f -

kubectl --context kind-east -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context kind-east -n east apply -f -
```

### Create kind Values

Create `./kind-west.yaml`:

```shell
cat > ./kind-west.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-east-pd:
    name: wait-for-east-pd
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for east PingDirectory..."
      wait-for east-pingdirectory-1.east.example.com:1636 -t 900 -- echo "east PingDirectory running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    MAKELDIF_USERS: "2000"
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
  services:
    loadBalancerServicePerPod: true
    loadBalancerExternalDNSHostnameSuffix: .west.example.com

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-east-pd
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: west
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west.example.com
    PINGDIRECTORY_LDAPS_PORT: "1636"
  services:
    useLoadBalancerForDataService: true
    dataExternalDNSHostname: west-pingdirectoryproxy-0.west.example.com
    ldaps:
      servicePort: 1636
      containerPort: 1636
      clusterService: true
      dataService: true
YAML
```

Create `./kind-east.yaml`:

```shell
cat > ./kind-east.yaml <<'YAML'
global:
  image:
    tag: "2603"

initContainers:
  wait-for-west-proxy:
    name: wait-for-west-proxy
    image: pingidentity/pingtoolkit:2603
    command:
    - sh
    - -c
    - |
      echo "Waiting for west PingDirectoryProxy..."
      wait-for west-pingdirectoryproxy-0.west.example.com:1636 -t 900 -- echo "west PingDirectoryProxy running"

pingdirectory:
  container:
    replicaCount: 2
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: baseline/pingdirectory
    LOAD_BALANCING_ALGORITHM_NAMES: dc_example_dc_com-fewest-operations;dc_example_dc_com-failover
    MAKELDIF_USERS: "2000"
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "2"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectory-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectory-"
    K8S_POD_HOSTNAME_SUFFIX: ".east.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
  services:
    loadBalancerServicePerPod: true
    loadBalancerExternalDNSHostnameSuffix: .east.example.com

pingdirectoryproxy:
  includeInitContainers:
  - wait-for-west-proxy
  container:
    replicaCount: 1
  enabled: true
  envs:
    SERVER_PROFILE_URL: https://github.com/pingidentity/pingidentity-server-profiles.git
    SERVER_PROFILE_PATH: pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy
    K8S_CLUSTERS: west east
    K8S_CLUSTER: east
    K8S_SEED_CLUSTER: west
    K8S_NUM_REPLICAS: "1"
    K8S_POD_HOSTNAME_PREFIX: "east-pingdirectoryproxy-"
    K8S_SEED_HOSTNAME_PREFIX: "west-pingdirectoryproxy-"
    K8S_POD_HOSTNAME_SUFFIX: ".east.example.com"
    K8S_SEED_HOSTNAME_SUFFIX: ".west.example.com"
    K8S_INCREMENT_PORTS: "false"
    SKIP_WAIT_FOR_DNS: "true"
    JOIN_PD_TOPOLOGY: "true"
    PINGDIRECTORY_HOSTNAME: west-pingdirectory-0.west.example.com
    PINGDIRECTORY_LDAPS_PORT: "1636"
  services:
    useLoadBalancerForDataService: true
    dataExternalDNSHostname: east-pingdirectoryproxy-0.east.example.com
    ldaps:
      servicePort: 1636
      containerPort: 1636
      clusterService: true
      dataService: true
YAML
```

### Deploy West

Do not use `--wait`. The west proxy waits for east PingDirectory through an init container.

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml
```

Patch the west proxy LoadBalancer service so the advertised proxy hostname can route to the starting proxy before the proxy pod is marked Ready:

```shell
kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'
```

This patch is required when the chart does not expose `publishNotReadyAddresses` for the generated service. The proxy topology join reaches the proxy through its advertised external LoadBalancer hostname before readiness succeeds. This patch is needed for this local example to work.

Wait for west PingDirectory pods to start:

```shell
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context kind-west -n west get svc
```

Record:

* `west-pingdirectory-0` external IP

* `west-pingdirectory-1` external IP

* `west-pingdirectoryproxy` external IP

Update `./dnsmasq/hosts.lab` with west records.

Example:

```shell
172.19.10.202 west-pingdirectory-0.west.example.com
172.19.10.201 west-pingdirectory-1.west.example.com
172.19.10.200 west-pingdirectoryproxy-0.west.example.com
```

Restart dnsmasq:

```shell
docker restart pdproxy-discovery-dnsmasq
```

Verify west DNS from both clusters:

```shell
kubectl --context kind-west run dns-west-from-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-west-from-west --timeout=120s
kubectl --context kind-west logs dns-west-from-west
kubectl --context kind-west delete pod dns-west-from-west

kubectl --context kind-east run dns-west-from-east --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-west-from-east --timeout=120s
kubectl --context kind-east logs dns-west-from-east
kubectl --context kind-east delete pod dns-west-from-east
```

### Deploy East

```shell
helm upgrade --install east pingidentity/ping-devops \
  --kube-context kind-east \
  -n east \
  -f ./kind-east.yaml
```

Patch the east proxy LoadBalancer service for the same pre-readiness routing requirement:

```shell
kubectl --context kind-east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'
```

Get east service IPs:

```shell
kubectl --context kind-east -n east get svc
```

Append east records to `./dnsmasq/hosts.lab`.

Example:

```shell
172.19.11.201 east-pingdirectory-0.east.example.com
172.19.11.202 east-pingdirectory-1.east.example.com
172.19.11.200 east-pingdirectoryproxy-0.east.example.com
```

Restart dnsmasq:

```shell
docker restart pdproxy-discovery-dnsmasq
```

Verify east DNS from both clusters:

```shell
kubectl --context kind-west run dns-east-from-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-east-from-west --timeout=120s
kubectl --context kind-west logs dns-east-from-west
kubectl --context kind-west delete pod dns-east-from-west

kubectl --context kind-east run dns-east-from-east --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/dns-east-from-east --timeout=120s
kubectl --context kind-east logs dns-east-from-east
kubectl --context kind-east delete pod dns-east-from-east
```

### Confirm Cross-Cluster TCP Reachability

```shell
kubectl --context kind-west run net-west-to-east-pd --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for east-pingdirectory-0.east.example.com:1636 -t 60 -- echo ok
kubectl --context kind-west wait --for=jsonpath='{.status.phase}'=Succeeded pod/net-west-to-east-pd --timeout=120s
kubectl --context kind-west logs net-west-to-east-pd
kubectl --context kind-west delete pod net-west-to-east-pd

kubectl --context kind-east run net-east-to-west-pd --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectory-0.west.example.com:1636 -t 60 -- echo ok
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/net-east-to-west-pd --timeout=120s
kubectl --context kind-east logs net-east-to-west-pd
kubectl --context kind-east delete pod net-east-to-west-pd

kubectl --context kind-east run proxy-east-to-west --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  wait-for west-pingdirectoryproxy-0.west.example.com:1636 -t 60 -- echo ok
kubectl --context kind-east wait --for=jsonpath='{.status.phase}'=Succeeded pod/proxy-east-to-west --timeout=120s
kubectl --context kind-east logs proxy-east-to-west
kubectl --context kind-east delete pod proxy-east-to-west
```

### Wait for Workloads

```shell
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m
```

### Confirm kind Default Hook Behavior

West:

```shell
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers
```

East:

```shell
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers
```

Expected:

```shell
west proxy:
  locations: west, east
  west preferred-failover-location: east

east proxy:
  locations: east, west
  east preferred-failover-location: west

both proxies:
  LBA monitor status: AVAILABLE
  num-available-servers: 4
```

### kind Explicit Preferred Failover Variation

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east"

helm upgrade --install east pingidentity/ping-devops \
  --kube-context kind-east \
  -n east \
  -f ./kind-east.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west"

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m

kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl --context kind-east -n east exec east-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name east
```

Expected:

```shell
west preferred-failover-location: east
east preferred-failover-location: west
```

### kind Invalid Local Override

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="west east"
```

Expected:

* west proxy fails during startup

* logs show local cluster is not allowed in `PREFERRED_FAILOVER_LOCATIONS`

Check:

```shell
kubectl --context kind-west -n west logs west-pingdirectoryproxy-0 -f
```

Restore:

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west delete pod west-pingdirectoryproxy-0
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
```

### kind Invalid Unknown Override

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml \
  --set-string pingdirectoryproxy.envs.PREFERRED_FAILOVER_LOCATIONS="east central"
```

Expected:

* west proxy fails during startup

* logs show every preferred failover location must be listed in `K8S_CLUSTERS`

Restore:

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context kind-west \
  -n west \
  -f ./kind-west.yaml

kubectl --context kind-west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context kind-west -n west delete pod west-pingdirectoryproxy-0
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
```

### kind Restart Validation

```shell
kubectl --context kind-west -n west rollout restart statefulset/west-pingdirectoryproxy
kubectl --context kind-west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west

until kubectl --context kind-west -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor \
  "(objectclass=ds-load-balancing-algorithm-monitor-entry)" \
  algorithm-name health-check-state local-servers-health-check-state \
  non-local-servers-health-check-state num-available-servers | \
  grep -q "non-local-servers-health-check-state: AVAILABLE"; do
  sleep 10
done
```

Expected:

* hook re-runs

* preferred failover remains correct

* LBAs become available

### kind Cleanup

```shell
helm --kube-context kind-west uninstall west -n west
helm --kube-context kind-east uninstall east -n east

kind delete cluster --name west
kind delete cluster --name east

docker rm -f pdproxy-discovery-dnsmasq
```

## minikube Variant

This variant uses two Docker-driver minikube profiles:

* `west`

* `east`

### minikube Prerequisites

Required:

* Docker Desktop with Kubernetes disabled

* `kubectl`

* `helm`

* `minikube`

* [Ping Identity DevOps](../how-to/devopsRegistration.html) credentials available as environment variables or in a local `.env` file

The profile subnets vary by machine. Substitute the ranges from `docker network inspect`.

### Create Profiles

```shell
minikube start --profile west --driver=docker --cpus=4 --memory=8192
minikube start --profile east --driver=docker --cpus=4 --memory=8192
```

Verify contexts:

```shell
kubectl config get-contexts | grep -E 'west|east'
```

### LoadBalancer Support

Enable MetalLB:

```shell
minikube -p west addons enable metallb
minikube -p east addons enable metallb
```

This step requires Docker API access because minikube inspects and updates the profile containers.

Find Docker network subnets:

```shell
docker network inspect west --format '{{json .IPAM.Config}}'
docker network inspect east --format '{{json .IPAM.Config}}'
```

Example subnets:

```shell
west: 192.168.49.0/24
east: 192.168.58.0/24
```

Patch deterministic MetalLB ranges. Adjust IPs for the actual subnets:

```shell
kubectl --context west -n metallb-system patch configmap config --type merge \
  -p '{"data":{"config":"address-pools:\n- name: west\n  protocol: layer2\n  addresses:\n  - 192.168.49.200-192.168.49.209\n"}}'

kubectl --context east -n metallb-system patch configmap config --type merge \
  -p '{"data":{"config":"address-pools:\n- name: east\n  protocol: layer2\n  addresses:\n  - 192.168.58.200-192.168.58.209\n"}}'
```

Attach each minikube node container to the other profile network:

```shell
docker network connect east west || true
docker network connect west east || true
```

### minikube dnsmasq and CoreDNS

Create DNS files:

```shell
mkdir -p ./minikube-dnsmasq

cat > ./minikube-dnsmasq/dnsmasq.conf <<'EOF'
no-daemon
log-queries
log-facility=-
listen-address=0.0.0.0
bind-interfaces
addn-hosts=/etc/dnsmasq.d/hosts.lab
EOF

cat > ./minikube-dnsmasq/hosts.lab <<'EOF'
# Filled in after minikube LoadBalancer services receive IPs.
EOF
```

Build the dnsmasq image if it is not already present:

```shell
docker build -t pdproxy-discovery-dnsmasq:2.91 ./dnsmasq
```

Run dnsmasq on both profile networks. Adjust resolver IPs for the actual subnets:

```shell
docker rm -f pdproxy-discovery-minikube-dnsmasq 2>/dev/null || true

docker run -d --name pdproxy-discovery-minikube-dnsmasq \
  --network west \
  --ip 192.168.49.250 \
  -v "$PWD/./minikube-dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf:ro" \
  -v "$PWD/./minikube-dnsmasq/hosts.lab:/etc/dnsmasq.d/hosts.lab:ro" \
  pdproxy-discovery-dnsmasq:2.91 \
  -C /etc/dnsmasq.conf

docker network connect --ip 192.168.58.250 east pdproxy-discovery-minikube-dnsmasq
```

Patch CoreDNS in each profile. Forward to the dnsmasq IP on that profile's local Docker network.

If your profile subnets differ from the examples, replace `192.168.49.250` and `192.168.58.250` in the patch payloads.

Back up the existing CoreDNS ConfigMaps:

```shell
kubectl --context west -n kube-system get configmap coredns -o yaml > ./minikube-west-coredns.before.yaml
kubectl --context east -n kube-system get configmap coredns -o yaml > ./minikube-east-coredns.before.yaml
```

Patch the west profile, using the west dnsmasq IP:

```shell
kubectl --context west -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 192.168.49.250\n}\n\neast.example.com:53 {\n    forward . 192.168.49.250\n}\n\n.:53 {\n    log\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    hosts {\n       192.168.65.254 host.minikube.internal\n       fallthrough\n    }\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'
```

Patch the east profile, using the east dnsmasq IP:

```shell
kubectl --context east -n kube-system patch configmap coredns --type merge -p '{
  "data": {
    "Corefile": "west.example.com:53 {\n    forward . 192.168.58.250\n}\n\neast.example.com:53 {\n    forward . 192.168.58.250\n}\n\n.:53 {\n    log\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    hosts {\n       192.168.65.254 host.minikube.internal\n       fallthrough\n    }\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30 {\n       disable success cluster.local\n       disable denial cluster.local\n    }\n    loop\n    reload\n    loadbalance\n}\n"
  }
}'
```

Restart CoreDNS:

```shell
kubectl --context west -n kube-system rollout restart deployment coredns
kubectl --context east -n kube-system rollout restart deployment coredns
kubectl --context west -n kube-system rollout status deployment coredns --timeout=120s
kubectl --context east -n kube-system rollout status deployment coredns --timeout=120s
```

### minikube Namespaces and Secrets

```shell
kubectl --context west create namespace west
kubectl --context east create namespace east
```

Using an existing `.env` file:

```shell
kubectl --context west -n west create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context west -n west apply -f -

kubectl --context east -n east create secret generic devops-secret \
  --from-env-file=.env \
  --dry-run=client -o yaml | kubectl --context east -n east apply -f -
```

Using existing environment variables:

```shell
kubectl --context west -n west create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context west -n west apply -f -

kubectl --context east -n east create secret generic devops-secret \
  --from-literal=PING_IDENTITY_DEVOPS_USER="$PING_IDENTITY_DEVOPS_USER" \
  --from-literal=PING_IDENTITY_DEVOPS_KEY="$PING_IDENTITY_DEVOPS_KEY" \
  --from-literal=PING_IDENTITY_ACCEPT_EULA="YES" \
  --type=Opaque \
  --dry-run=client -o yaml | kubectl --context east -n east apply -f -
```

### minikube Deploy West

Use the `kind` values files created earlier. They use the same `west.example.com` and `east.example.com` external hostname model.

```shell
helm upgrade --install west pingidentity/ping-devops \
  --kube-context west \
  -n west \
  -f ./kind-west.yaml

kubectl --context west -n west patch service west-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context west -n west get svc
```

Add west records to `./minikube-dnsmasq/hosts.lab` using actual LoadBalancer IPs.

Example:

```shell
192.168.49.201 west-pingdirectory-0.west.example.com
192.168.49.202 west-pingdirectory-1.west.example.com
192.168.49.200 west-pingdirectoryproxy-0.west.example.com
```

Restart dnsmasq and verify west DNS from both profiles:

```shell
docker restart pdproxy-discovery-minikube-dnsmasq

kubectl --context west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com

kubectl --context east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup west-pingdirectory-0.west.example.com
```

### minikube Deploy East

```shell
helm upgrade --install east pingidentity/ping-devops \
  --kube-context east \
  -n east \
  -f ./kind-east.yaml

kubectl --context east -n east patch service east-pingdirectoryproxy \
  --type merge \
  -p '{"spec":{"publishNotReadyAddresses":true}}'

kubectl --context east -n east get svc
```

Add east records to `./minikube-dnsmasq/hosts.lab` using actual LoadBalancer IPs.

Example:

```shell
192.168.58.201 east-pingdirectory-0.east.example.com
192.168.58.202 east-pingdirectory-1.east.example.com
192.168.58.200 east-pingdirectoryproxy-0.east.example.com
```

Restart dnsmasq and verify east DNS from both profiles:

```shell
docker restart pdproxy-discovery-minikube-dnsmasq

kubectl --context west run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com

kubectl --context east run dns-test --rm -i --restart=Never --image=pingidentity/pingtoolkit:2603 -- \
  nslookup east-pingdirectory-0.east.example.com
```

Wait for all workloads:

```shell
kubectl --context west -n west rollout status statefulset/west-pingdirectory --timeout=15m
kubectl --context east -n east rollout status statefulset/east-pingdirectory --timeout=15m
kubectl --context west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
kubectl --context east -n east rollout status statefulset/east-pingdirectoryproxy --timeout=15m
```

Then use the TCP verification, proxy validation, variation, and restart steps from the kind demo, replacing:

```shell
kind-west -> west
kind-east -> east
```

After restoring from an intentional invalid override, delete the failed proxy pod if it remains on the old invalid checksum:

```shell
kubectl --context west -n west delete pod west-pingdirectoryproxy-0
kubectl --context west -n west rollout status statefulset/west-pingdirectoryproxy --timeout=15m
```

### minikube Cleanup

```shell
helm --kube-context west uninstall west -n west
helm --kube-context east uninstall east -n east

minikube delete --profile west
minikube delete --profile east

docker rm -f pdproxy-discovery-minikube-dnsmasq
```

## Troubleshooting

1. Confirm `devops-secret` exists in every namespace.

2. Confirm the Helm command uses `pingidentity/ping-devops`, not a local chart path.

3. Confirm `SERVER_PROFILE_BRANCH` is not set in the values.

4. Confirm `SERVER_PROFILE_PATH` is `pingdirectoryproxy-automatic-server-discovery/pingdirectoryproxy`.

5. Confirm DNS resolves every name used by `PINGDIRECTORY_HOSTNAME`, `K8S_POD_HOSTNAME_PREFIX`, and `K8S_POD_HOSTNAME_SUFFIX`.

6. Confirm TCP port `1636` is reachable cross-region.

7. Confirm proxy logs show the location hook ran.

8. Confirm `dsconfig list-locations`.

9. Confirm `preferred-failover-location`.

10. Confirm LBA monitor availability.

Useful log checks:

```shell
kubectl -n west logs west-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
kubectl -n east logs east-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
```

For kind:

```shell
kubectl --context kind-west -n west logs west-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
kubectl --context kind-east -n east logs east-pingdirectoryproxy-0 | grep -E "PingDirectoryProxy multi-cluster|preferred failover|PREFERRED_FAILOVER_LOCATIONS|CONTAINER FAILURE"
```

## Demo Closeout Evidence

Capture these outputs:

```shell
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-locations
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig get-location-prop --location-name west
kubectl -n west exec west-pingdirectoryproxy-0 -- dsconfig list-server-instances --property load-balancing-algorithm-name
kubectl -n west exec west-pingdirectoryproxy-0 -- ldapsearch -b cn=monitor "(objectclass=ds-load-balancing-algorithm-monitor-entry)"
kubectl -n west logs west-pingdirectoryproxy-0 | grep "PingDirectoryProxy multi-cluster location configuration completed"
```

For kind and minikube, include the appropriate `--context` values.