--- title: PingFederate Multi-Region Active/Passive Administrative Consoles description: This walkthrough demonstrates the deployment of multiple PingFederate administrative consoles in an active/passive multi-cluster configuration. Cross-cluster networking for this demonstration is handled by Submariner. Submariner exports services from one cluster into another via the svc.clusterset.local DNS namespace. With this communication in place, all PingFederate administrative pods use standard JGroups DNS_PING with a single DNS_QUERY_LOCATION pointing at the clusterset headless service. component: devops page_id: devops::deployment/deployPFAdminSubmarinerWalkthrough canonical_url: https://developer.pingidentity.com/devops/deployment/deployPFAdminSubmarinerWalkthrough.html section_ids: devops-pf-submariner-purpose: Purpose devops-pf-submariner-topology: Cluster Topology Overview devops-pf-submariner-discovery: How Discovery Works in This Approach devops-pf-submariner-prerequisites: Prerequisites devops-pf-submariner-deploy-kind: Deploy Submariner Kind Clusters devops-pf-submariner-install-subctl: Install subctl devops-pf-submariner-create-clusters: Create kind clusters devops-pf-submariner-proxy-settings: Fix proxy settings on kind nodes devops-pf-submariner-preload-images: Pre-load Submariner images devops-pf-submariner-broker-join: Deploy Submariner broker and join clusters devops-pf-submariner-namespaces: Prepare Namespaces and Secrets devops-pf-submariner-deployment-files: Deployment Files devops-pf-submariner-scripts-yaml: scripts.yaml devops-pf-submariner-base-yaml: base.yaml devops-pf-submariner-east-yaml: east-pf.yaml devops-pf-submariner-kustomize: Kustomize Post-Renderer devops-pf-submariner-deploy: Deploy devops-pf-submariner-console: Accessing the Admin Console devops-pf-submariner-nodeport-rationale: Why NodePort instead of port-forward devops-pf-submariner-active-node: Determining which cluster holds the active node devops-pf-submariner-browser: Open the console in a browser devops-pf-submariner-failover: After failover devops-pf-submariner-validation: Validation devops-pf-submariner-cluster-status: Cluster Status devops-pf-submariner-active-endpoint: Active Service Endpoint devops-pf-submariner-scale-test: Scale Test devops-pf-submariner-cold-start: Simultaneous Cold Start Validation devops-pf-submariner-dns-verify: Cross-Cluster DNS Verification devops-pf-submariner-cleanup: Cleanup --- # PingFederate Multi-Region Active/Passive Administrative Consoles ## Purpose This walkthrough demonstrates the deployment of multiple PingFederate administrative consoles in an active/passive multi-cluster configuration. Cross-cluster networking for this demonstration is handled by [Submariner](https://multicluster.sigs.k8s.io/implementations/mcs-implementations/). Submariner exports services from one cluster into another via the `svc.clusterset.local` DNS namespace. With this communication in place, all PingFederate administrative pods use standard JGroups `DNS_PING` with a single `DNS_QUERY_LOCATION` pointing at the clusterset headless service. | | | | - | -------------------------------------------------------------------------------------------------------------------------- | | | The configuration shown here is a starting point for demonstration purposes. It is not intended to be a production design. | ## Cluster Topology Overview When deployed in a typical scenario, two distinct Kubernetes clusters have their own isolated Kubernetes control plane and DNS domain (`svc.cluster.local`). With this isolation, a pod in one cluster cannot resolve any name in another cluster. In this demonstration, two kind clusters (`kind-west`, `kind-east`) will illustrate how the Multicluster Services (MCS) API overcomes this limitation and can be used by PingFederate. Submariner, implementing MCS, joins the clusters at the network layer: * A Submariner gateway pod on each cluster establishes a cross-cluster tunnel over the Docker bridge network (no NAT required in a local kind setup). * The Submariner operator installs a Lighthouse CoreDNS plugin in each cluster that intercepts `*.svc.clusterset.local` queries. After joining, any pod in either cluster can reach pods in the other cluster by their pod CIDR addresses (east pods: `10.245.x.x`, west pods: `10.244.x.x`). A `ServiceExport` resource instructs the Submariner lighthouse agent to advertise that service into the clusterset. A corresponding `ServiceImport` is created in the same namespace on every other cluster. The Lighthouse CoreDNS plugin answers `..svc.clusterset.local` queries by returning all endpoint pod IPs across all clusters. For a headless service (like `pingfederate-cluster`), the DNS response contains one A-record per pod across both `kind-west` and `kind-east`. ``` kind-west kind-east ----------------------------- ----------------------------- pf-mr/pingfederate-admin-0 pf-mr/pingfederate-admin-0 DNS_QUERY_LOCATION -> DNS_QUERY_LOCATION -> pingfederate-cluster.pf-mr.svc.clusterset.local pingfederate-cluster.pf-mr.svc.clusterset.local | | +---------------- Submariner tunnel -------------------+ (Docker bridge) Lighthouse CoreDNS returns: 10.244.x.x (west admin-0 pod IP) 10.245.x.x (east admin-0 pod IP) ``` The `pingfederate-admin-active` Service selects pods with label `pf.admin.active: "true"`. The `pfadmin-check-active.sh` hook, called from `liveness.sh` at each liveness interval, queries the admin heartbeat with `?checkActive=true` and patches the pod's label via the Kubernetes API. Only the active node carries the label; the service endpoint changes automatically on failover. Because `pingfederate-admin-active` is also exported via `ServiceExport`, its clusterset FQDN routes to the active pod regardless in which cluster it is running. ## How Discovery Works in This Approach JGroups `DNS_PING` resolves `DNS_QUERY_LOCATION` to find cluster members. In a single cluster, `DNS_QUERY_LOCATION` points at the local headless Service (`svc.cluster.local`). In a multi-cluster setup without shared DNS, that resolution fails across cluster boundaries. With Submariner in place, setting `DNS_QUERY_LOCATION` to the clusterset address gives JGroups a single DNS name that always returns the full member list regardless in which cluster the pod is located. When `PF_ADMIN_SEED` is not set, the image uses `DNS_PING` — this value is the default and resolves as expected. ## Prerequisites * Docker (Docker Desktop Kubernetes **disabled** to avoid `kind` conflicts) * `kind` — `brew install kind` * `kubectl` — `brew install kubectl` * `kubectx` - a convenience tool for managing contexts () * `helm` — `brew install helm` * `kustomize` — `brew install kustomize` * `subctl` — `curl -Ls https://get.submariner.io | bash` (installs to `~/.local/bin`) * [Ping Identity DevOps](../how-to/devopsRegistration.html) credentials * PingFederate image `2605` or later (required for the multi-cluster hook logic in `scripts.yaml`) Recommended resources: 8 or more CPUs, 16 GB or more RAM. ## Deploy Submariner Kind Clusters Install `subctl` and create two kind clusters with non-overlapping CIDRs, then join them via Submariner. ### Install subctl ```shell curl -Ls https://get.submariner.io | bash export PATH="$HOME/.local/bin:$PATH" subctl version ``` ### Create kind clusters The kind cluster configurations used here include `extraPortMappings` that bind host ports directly to a NodePort on each cluster's control-plane node. This configuration enables the user to access the PingFederate admin console from a host browser without a running `kubectl port-forward` session. See [Why NodePort instead of port-forward](#devops-pf-submariner-nodeport-rationale) for details. Create `kind-west.yaml`: ```yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: podSubnet: "10.244.0.0/16" serviceSubnet: "10.96.0.0/16" nodes: - role: control-plane extraPortMappings: - containerPort: 30999 hostPort: 9999 protocol: TCP - role: worker ``` Create `kind-east.yaml`: ```yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: podSubnet: "10.245.0.0/16" serviceSubnet: "10.97.0.0/16" nodes: - role: control-plane extraPortMappings: - containerPort: 30999 hostPort: 19999 protocol: TCP - role: worker ``` Create the clusters: ```shell kind create cluster --name west --config kind-west.yaml kind create cluster --name east --config kind-east.yaml kubectx # Expected: kind-west, kind-east ``` ### Fix proxy settings on kind nodes If your Docker Desktop has a proxy configured, kind nodes inherit it and `containerd` cannot pull images from `quay.io`. Override `NO_PROXY` to bypass the proxy for registry traffic: ```shell for node in west-control-plane west-worker east-control-plane east-worker; do docker exec $node sh -c ' mkdir -p /etc/systemd/system/containerd.service.d cat > /etc/systemd/system/containerd.service.d/no-proxy.conf << EOF [Service] Environment="NO_PROXY=quay.io,registry.access.redhat.com,gcr.io,k8s.gcr.io,registry.k8s.io,docker.io" EOF systemctl daemon-reload && systemctl restart containerd ' done ``` ### Pre-load Submariner images Submariner 0.24.0 uses images from `quay.io`. Pull them to the host and load into each node to avoid pull failures: ```shell IMAGES=( quay.io/submariner/submariner-operator:0.24.0 quay.io/submariner/submariner-gateway:0.24.0 quay.io/submariner/submariner-route-agent:0.24.0 quay.io/submariner/lighthouse-agent:0.24.0 quay.io/submariner/lighthouse-coredns:0.24.0 ) for img in "${IMAGES[@]}"; do docker pull "$img"; done for cluster in west east; do for node in $(kubectl --context kind-${cluster} get nodes -o jsonpath='{.items[*].metadata.name}'); do for img in "${IMAGES[@]}"; do docker save "$img" | docker exec -i $node ctr -n k8s.io images import - done done done ``` ### Deploy Submariner broker and join clusters ```shell # Deploy broker on west (it becomes the broker host) subctl deploy-broker --context kind-west # Label gateway nodes kubectl --context kind-west label node west-worker submariner.io/gateway=true kubectl --context kind-east label node east-worker submariner.io/gateway=true # Join both clusters subctl join broker-info.subm --context kind-west --clusterid west \ --natt=false --label-gateway=false --check-broker-certificate=false subctl join broker-info.subm --context kind-east --clusterid east \ --natt=false --label-gateway=false --check-broker-certificate=false ``` After `subctl join`, the operator on each cluster tries to reach the broker using `127.0.0.1` (the host port-forward address), which is unreachable from inside the pod. Patch the broker URL to use in-cluster addresses: ```shell # west operator reaches broker via its own in-cluster kubernetes service # 10.96.0.1 is the ClusterIP of the kubernetes service in the default namespace: # kubectl --context kind-west get svc kubernetes -n default kubectl --context kind-west patch submariner -n submariner-operator submariner \ --type=json -p='[{"op":"replace","path":"/spec/brokerK8sApiServer","value":"10.96.0.1:443"}]' # east operator must reach west's API server via the docker bridge network C1_IP=$(docker inspect west-control-plane \ --format '{{(index .NetworkSettings.Networks "kind").IPAddress}}') kubectl --context kind-east patch submariner -n submariner-operator submariner \ --type=json -p="[{\"op\":\"replace\",\"path\":\"/spec/brokerK8sApiServer\",\"value\":\"${C1_IP}:6443\"}]" ``` Verify connectivity: ```shell subctl show connections --context kind-west # GATEWAY CLUSTER REMOTE IP NAT STATUS # east-worker east 172.19.x.x no connected ``` ## Prepare Namespaces and Secrets ```shell DEVOPS_USER= DEVOPS_KEY= for ctx in kind-west kind-east; do kubectl --context $ctx create namespace pf-mr kubectl --context $ctx create secret generic devops-secret -n pf-mr \ --from-literal=PING_IDENTITY_DEVOPS_USER=$DEVOPS_USER \ --from-literal=PING_IDENTITY_DEVOPS_KEY=$DEVOPS_KEY done ``` ## Deployment Files This deployment uses four files. Three are Helm values files consumed directly by `helm upgrade --install`; the fourth is a Kustomize post-renderer that injects additional Kubernetes resources that the `ping-devops` chart does not generate natively. ### scripts.yaml This file defines two scripts mounted into the admin pods as ConfigMap entries. Starting with the `2605` release of the PingFederate image, there is new hook logic that uses `403 + license_agreement_not_accepted` detection to determine whether a node needs seeding, which works correctly in both single- and multi-cluster deployments with a shared DNS. The two scripts here: * `liveness.sh` calls `pfadmin-check-active.sh` on each liveness check to keep the pod label in sync with the current active/passive role. * `pfadmin-check-active.sh` queries the active heartbeat endpoint and patches the pod's `pf.admin.active` label accordingly. The `pingfederate-admin-active` Service selector uses this label to route traffic only to the currently active node. ```yaml configMaps: custom-hooks: data: liveness.sh: |- #!/usr/bin/env sh URL="https://127.0.0.1:${PF_ENGINE_PORT}/pf/heartbeat.ping" if test "${OPERATIONAL_MODE}" = "CLUSTERED_CONSOLE" -o "${OPERATIONAL_MODE}" = "STANDALONE"; then test -f /tmp/ready || exit 1 if test "${PF_CLUSTER_ADMIN_NODES_SYNC_ENABLED}" = "true"; then . "${HOOKS_DIR}/pfadmin-check-active.sh" setPFActive fi test "${OPERATIONAL_MODE}" = "CLUSTERED_CONSOLE" && URL="https://127.0.0.1:${PF_ADMIN_PORT}/pf/heartbeat.ping" fi curl -sSk -o /dev/null "${URL}" || exit 1 pfadmin-check-active.sh: |- #!/usr/bin/env sh setPFActive() { APISERVER="https://kubernetes.default.svc" SERVICEACCOUNT="/var/run/secrets/kubernetes.io/serviceaccount" NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) TOKEN=$(cat ${SERVICEACCOUNT}/token) CACERT=${SERVICEACCOUNT}/ca.crt POD_NAME=$(hostname) URL="https://127.0.0.1:${PF_ADMIN_PORT}/pf/heartbeat.ping?checkActive=true" _activelabel="false" _curlres=$(curl -sSk -o /dev/null -w "%{response_code}" "${URL}" 2>/dev/null) test "${_curlres}" -eq "200" && _activelabel="true" curl -s --cacert ${CACERT} -X PATCH -o /dev/null -w "%{http_code}" \ -H "Content-Type: application/strategic-merge-patch+json" \ -H "Authorization: Bearer ${TOKEN}" \ "${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods/${POD_NAME}" \ --data "{\"metadata\":{\"labels\":{\"pf.admin.active\":\"${_activelabel}\"}}}" } ``` ### base.yaml The west cluster Helm values. `global.image.tag` sets the image version for all products. `PF_NODE_GROUP_ID` labels the cluster in the PingFederate console and is used by the `east-pf.yaml` override to distinguish the east cluster. ```yaml global: addReleaseNameToResource: none image: tag: "2605" envs: PING_IDENTITY_ACCEPT_EULA: "YES" PING_IDENTITY_PASSWORD: 2FederateM0re pingfederate-admin: enabled: true envs: OPERATIONAL_MODE: CLUSTERED_CONSOLE # clusterset.local resolves across both clusters via Submariner Lighthouse CoreDNS DNS_QUERY_LOCATION: "pingfederate-cluster.pf-mr.svc.clusterset.local" PF_NODE_GROUP_ID: west PF_CLUSTER_ADMIN_NODES_SYNC_ENABLED: "true" volumeMounts: - mountPath: "/opt/liveness.sh" name: custom-vol subPath: liveness.sh - mountPath: "/opt/in/hooks/pfadmin-check-active.sh" name: custom-vol subPath: pfadmin-check-active.sh volumes: - name: custom-vol configMap: name: custom-hooks defaultMode: 0777 rbac: generateServiceAccount: true serviceAccountName: pf-mc-admin-sa generateRoleAndRoleBinding: true applyServiceAccountToWorkload: true role: rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list", "patch"] workload: labels: active: "false" type: StatefulSet statefulSet: persistentvolume: enabled: true pingfederate-engine: enabled: false ``` ### east-pf.yaml Applied on top of `base.yaml` for the east cluster deployment. Overrides the cluster identity. ```yaml pingfederate-admin: enabled: true envs: PF_NODE_GROUP_ID: east container: replicaCount: 1 ``` ### Kustomize Post-Renderer The `ping-devops` chart does not generate per-instance `ServiceExport` resources or the selector-based active-admin Services. A Kustomize post-renderer injects these after Helm renders its manifests. Create a `kustomize/` subdirectory with these five files. `kustomize/kustomize` (executable script): ```bash #!/bin/bash cd "$(dirname "$0")" cat <&0 > all.yaml kustomize build . && rm all.yaml ``` ```shell chmod +x kustomize/kustomize ``` `kustomize/kustomization.yaml`: ```yaml resources: - all.yaml - pf-admin-active-svc.yaml - pf-admin-active-nodeport.yaml - ServiceExport.yaml ``` `kustomize/pf-admin-active-svc.yaml` — a ClusterIP Service that selects only pods with label `pf.admin.active: "true"`: ```yaml apiVersion: v1 kind: Service metadata: name: pingfederate-admin-active namespace: pf-mr labels: app.kubernetes.io/instance: pf-mr app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: pingfederate-admin spec: ports: - name: https port: 9999 protocol: TCP targetPort: 9999 selector: app.kubernetes.io/instance: pf-mr app.kubernetes.io/name: pingfederate-admin pf.admin.active: "true" ``` `kustomize/pf-admin-active-nodeport.yaml` — a NodePort Service using the same `pf.admin.active: "true"` selector, bound to `nodePort: 30999` which maps to host ports via `extraPortMappings` in the kind cluster configuration: ```yaml apiVersion: v1 kind: Service metadata: name: pingfederate-admin-active-nodeport namespace: pf-mr spec: type: NodePort ports: - name: https port: 9999 targetPort: 9999 nodePort: 30999 protocol: TCP selector: app.kubernetes.io/instance: pf-mr app.kubernetes.io/name: pingfederate-admin pf.admin.active: "true" ``` `kustomize/ServiceExport.yaml` — exports all four PingFederate services into the Submariner clusterset so they are accessible from both clusters via `svc.clusterset.local`: ```yaml apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: pingfederate-admin namespace: pf-mr --- apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: pingfederate-admin-active namespace: pf-mr --- apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: pingfederate-cluster namespace: pf-mr --- apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: pingfederate-engine namespace: pf-mr ``` ## Deploy The `kustomize/kustomize` script acts as a Helm post-renderer: it receives the rendered manifests on stdin and emits them plus the additional resources that the `ping-devops` chart does not generate natively. Helm v4 requires post-renderers to be registered as plugins — passing a script path directly does not work. Create a `plugin.yaml` file in your working directory, replacing `` with the absolute path to your `kustomize/` directory: ```yaml name: "kustomize-renderer" apiVersion: "v1" version: "0.1.0" type: "postrenderer/v1" runtime: "subprocess" runtimeConfig: platformCommand: - command: "/kustomize/kustomize" ``` Register the plugin (this is a one-time step per machine): ```shell mkdir -p "$(helm env HELM_PLUGINS)/kustomize-renderer" cp plugin.yaml "$(helm env HELM_PLUGINS)/kustomize-renderer/" ``` Verify: `helm plugin list` should show `kustomize-renderer` with type `postrenderer/v1`. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | On Helm v3, plugin registration is not required. You can pass the script path directly with `--post-renderer ./kustomize/kustomize`. See [helm/helm#31340](https://github.com/helm/helm/issues/31340). | Deploy to the west cluster: ```shell helm upgrade --install pf-mr pingidentity/ping-devops \ --kube-context kind-west \ --namespace pf-mr \ -f scripts.yaml -f base.yaml \ --post-renderer kustomize-renderer ``` Wait for `pingfederate-admin-0` to come up and verify ServiceExports and ServiceImports: ```shell kubectl --context kind-west get serviceexport -n pf-mr # NAME AGE # pingfederate-admin ... # pingfederate-admin-active ... # pingfederate-cluster ... kubectl --context kind-west get serviceimport -n pf-mr # NAME TYPE IP AGE # pingfederate-admin ClusterSetIP ... ... # pingfederate-admin-active ClusterSetIP ... ... # pingfederate-cluster Headless ... ``` Deploy to the east cluster: ```shell helm upgrade --install pf-mr pingidentity/ping-devops \ --kube-context kind-east \ --namespace pf-mr \ -f scripts.yaml -f base.yaml -f east-pf.yaml \ --post-renderer kustomize-renderer ``` ## Accessing the Admin Console The standard `pingfederate-admin` Service includes all admin pods — active and passive. Passive nodes return `403` for most admin operations and give severely limited console access. Always use `pingfederate-admin-active` to reach the admin console or admin API. With the scripts in place, the `pf.admin.active` label is refreshed on each pod at every liveness interval. When this label is in place, the `pingfederate-admin-active` Service's selector matches only the active pod so its endpoint tracks the currently active node automatically — including across a failover from one cluster to the other. ### Why NodePort instead of port-forward On macOS, Docker Desktop runs kind nodes inside a Linux VM. MetalLB LoadBalancer IPs live on the Docker bridge inside that VM and are not reachable from the host browser. `kubectl port-forward` works but requires a live terminal session, breaks on pod restart, and can only target one cluster at a time. kind `extraPortMappings` bind a host port to a container port on the kind control-plane node at cluster creation time. This gives each cluster a stable `localhost` URL with no running process required. The `pingfederate-admin-active-nodeport` Service uses the same `pf.admin.active: "true"` selector as the ClusterIP service — it tracks the active node automatically and follows failover to the other cluster. In a production environment, this port configuration would likely not be necessary. However, in any case, there would need to be some means of tracking the active pod and dynamically or manually updating DNS or other configurations to get to the active console. In this demonstration, we rely on using the assigned port of the active cluster service as our indicator for which pod we will be accessing. | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | If host port `9999` is already in use on your machine, change `hostPort` in the `kind-west.yaml` configuration and update the browser URL accordingly. The same applies to `19999` for the east cluster. | ### Determining which cluster holds the active node ```shell kubectl --context kind-west -n pf-mr get pods \ -l pf.admin.active=true -o wide kubectl --context kind-east -n pf-mr get pods \ -l pf.admin.active=true -o wide ``` Exactly one pod across both clusters should show `pf.admin.active=true`. ### Open the console in a browser Open the PingFederate admin console using the host port for the cluster that holds the active node: * West active: `https://localhost:9999/pingfederate/app` * East active: `https://localhost:19999/pingfederate/app` Accept the self-signed certificate warning. Log in with username `administrator` and password `2FederateM0re`. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The NodePort on the cluster that does not hold the active node has no endpoint and returns no response. This is expected — it also serves as a quick indicator of which cluster is currently active. | ### After failover When the active node changes, liveness checks on the surviving pods update their labels within one liveness period. The `pingfederate-admin-active-nodeport` endpoint shifts accordingly. Re-run the active node check above to confirm which cluster holds the new active pod, then open the corresponding browser URL. ## Validation ### Cluster Status Query from whichever cluster holds the active node (see [Determining which cluster holds the active node](#devops-pf-submariner-active-node)): ```shell kubectl --context kind-west -n pf-mr exec pingfederate-admin-0 -- \ curl --insecure --silent \ -H 'X-XSRF-Header: PingFederate' \ --user "administrator:2FederateM0re" \ 'https://localhost:9999/pf-admin-api/v1/cluster/status' | jq . ``` Expected: 1 `ACTIVE` node, all others `PASSIVE`. ### Active Service Endpoint ```shell kubectl --context kind-west -n pf-mr get endpoints pingfederate-admin-active # NAME ENDPOINTS AGE # pingfederate-admin-active 10.130.1.57:9999 ... # (single endpoint — the active pod only) kubectl --context kind-west -n pf-mr get endpoints pingfederate-admin # NAME ENDPOINTS AGE # pingfederate-admin 10.130.1.57:9999,10.130.1.59:9999 ... # (all admin pods) ``` ### Scale Test ```shell kubectl --context kind-west -n pf-mr scale sts pingfederate-admin --replicas=2 ``` Expected log from `pingfederate-admin-1`: ```shell INFO: Node is already seeded (HTTP 200). Skipping initial setup. INFO: Querying cluster for existing active admin node... INFO: Active admin already present in cluster. This node will remain passive. ``` The `pingfederate-admin-active` endpoint must still point only at the original active pod. ### Simultaneous Cold Start Validation Both clusters deployed in the preceding steps without sequencing already proves this feature will work. The 403-detection in hook 81 (built into the image) handles the potential race condition: each node queries `/cluster/status`; whichever sees `403 + license_agreement_not_accepted` first seeds itself and promotes; the other sees `200` (state already replicated via DNS\_PING) and stays passive. To repeat the test explicitly: ```shell kubectl --context kind-west -n pf-mr delete pvc --all kubectl --context kind-east -n pf-mr delete pvc --all kubectl --context kind-west delete namespace pf-mr kubectl --context kind-east delete namespace pf-mr kubectl --context kind-west create namespace pf-mr kubectl --context kind-east create namespace pf-mr # (re-create devops-secret in both namespaces) helm upgrade --install pf-mr pingidentity/ping-devops \ --kube-context kind-west --namespace pf-mr \ -f scripts.yaml -f base.yaml \ --post-renderer kustomize-renderer & helm upgrade --install pf-mr pingidentity/ping-devops \ --kube-context kind-east --namespace pf-mr \ -f scripts.yaml -f base.yaml -f east-pf.yaml \ --post-renderer kustomize-renderer & wait kubectl --context kind-west -n pf-mr rollout status statefulset/pingfederate-admin --timeout=15m & kubectl --context kind-east -n pf-mr rollout status statefulset/pingfederate-admin --timeout=15m & wait ``` Expected: 2 nodes in cluster status, exactly 1 `ACTIVE`, 1 `PASSIVE`. ### Cross-Cluster DNS Verification Confirm the clusterset DNS name resolves from an east pod: ```shell kubectl --context kind-east -n pf-mr exec pingfederate-admin-0 -- \ nslookup pingfederate-cluster.pf-mr.svc.clusterset.local # Returns A-records for all pod IPs across both clusters ``` ## Cleanup Delete the PersistentVolumeClaims before deleting the clusters. kind's local-path provisioner binds PVCs to host directories on the kind nodes; deleting the cluster without first removing the PVCs leaves those directories orphaned on the host. ```shell kubectl --context kind-west -n pf-mr delete pvc --all kubectl --context kind-east -n pf-mr delete pvc --all ``` Then delete the kind clusters. All Submariner broker state, gateway state, and `subctl join` artifacts are destroyed with the clusters — no separate Submariner cleanup step is required. ```shell kind delete cluster --name west kind delete cluster --name east ```