Ping Identity Devops

PingFederate Multi-Region Active/Passive Administrative Consoles

Purpose

This walkthrough demonstrates the deployment of multiple PingFederate administrative consoles in an active/passive multi-cluster configuration. Cross-cluster networking for this demonstration is handled by Submariner. Submariner exports services from one cluster into another via the svc.clusterset.local DNS namespace. With this communication in place, all PingFederate administrative pods use standard JGroups DNS_PING with a single DNS_QUERY_LOCATION pointing at the clusterset headless service.

The configuration shown here is a starting point for demonstration purposes. It is not intended to be a production design.

Cluster Topology Overview

When deployed in a typical scenario, two distinct Kubernetes clusters have their own isolated Kubernetes control plane and DNS domain (svc.cluster.local). With this isolation, a pod in one cluster cannot resolve any name in another cluster. In this demonstration, two kind clusters (kind-west, kind-east) will illustrate how the Multicluster Services (MCS) API overcomes this limitation and can be used by PingFederate.

Submariner, implementing MCS, joins the clusters at the network layer:

  • A Submariner gateway pod on each cluster establishes a cross-cluster tunnel over the Docker bridge network (no NAT required in a local kind setup).

  • The Submariner operator installs a Lighthouse CoreDNS plugin in each cluster that intercepts *.svc.clusterset.local queries.

After joining, any pod in either cluster can reach pods in the other cluster by their pod CIDR addresses (east pods: 10.245.x.x, west pods: 10.244.x.x).

A ServiceExport resource instructs the Submariner lighthouse agent to advertise that service into the clusterset. A corresponding ServiceImport is created in the same namespace on every other cluster. The Lighthouse CoreDNS plugin answers <service>.<namespace>.svc.clusterset.local queries by returning all endpoint pod IPs across all clusters. For a headless service (like pingfederate-cluster), the DNS response contains one A-record per pod across both kind-west and kind-east.

         kind-west                                               kind-east
-----------------------------                        -----------------------------
      pf-mr/pingfederate-admin-0                           pf-mr/pingfederate-admin-0
        DNS_QUERY_LOCATION ->                              DNS_QUERY_LOCATION ->
  pingfederate-cluster.pf-mr.svc.clusterset.local    pingfederate-cluster.pf-mr.svc.clusterset.local
              |                                                      |
              +---------------- Submariner tunnel -------------------+
                                 (Docker bridge)

Lighthouse CoreDNS returns:
  10.244.x.x  (west admin-0 pod IP)
  10.245.x.x  (east admin-0 pod IP)

The pingfederate-admin-active Service selects pods with label pf.admin.active: "true". The pfadmin-check-active.sh hook, called from liveness.sh at each liveness interval, queries the admin heartbeat with ?checkActive=true and patches the pod’s label via the Kubernetes API. Only the active node carries the label; the service endpoint changes automatically on failover. Because pingfederate-admin-active is also exported via ServiceExport, its clusterset FQDN routes to the active pod regardless in which cluster it is running.

How Discovery Works in This Approach

JGroups DNS_PING resolves DNS_QUERY_LOCATION to find cluster members. In a single cluster, DNS_QUERY_LOCATION points at the local headless Service (svc.cluster.local). In a multi-cluster setup without shared DNS, that resolution fails across cluster boundaries. With Submariner in place, setting DNS_QUERY_LOCATION to the clusterset address gives JGroups a single DNS name that always returns the full member list regardless in which cluster the pod is located.

When PF_ADMIN_SEED is not set, the image uses DNS_PING — this value is the default and resolves as expected.

Prerequisites

  • Docker (Docker Desktop Kubernetes disabled to avoid kind conflicts)

  • kind — brew install kind

  • kubectl — brew install kubectl

  • kubectx - a convenience tool for managing contexts (https://github.com/ahmetb/kubectx#installation)

  • helm — brew install helm

  • kustomize — brew install kustomize

  • subctl — curl -Ls https://get.submariner.io | bash (installs to ~/.local/bin)

  • Ping Identity DevOps credentials

  • PingFederate image 2605 or later (required for the multi-cluster hook logic in scripts.yaml)

Recommended resources: 8 or more CPUs, 16 GB or more RAM.

Deploy Submariner Kind Clusters

Install subctl and create two kind clusters with non-overlapping CIDRs, then join them via Submariner.

Install subctl

curl -Ls https://get.submariner.io | bash
export PATH="$HOME/.local/bin:$PATH"
subctl version

Create kind clusters

The kind cluster configurations used here include extraPortMappings that bind host ports directly to a NodePort on each cluster’s control-plane node. This configuration enables the user to access the PingFederate admin console from a host browser without a running kubectl port-forward session. See Why NodePort instead of port-forward for details.

Create kind-west.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30999
    hostPort: 9999
    protocol: TCP
- role: worker

Create kind-east.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  podSubnet: "10.245.0.0/16"
  serviceSubnet: "10.97.0.0/16"
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30999
    hostPort: 19999
    protocol: TCP
- role: worker

Create the clusters:

kind create cluster --name west --config kind-west.yaml
kind create cluster --name east --config kind-east.yaml
kubectx
# Expected: kind-west, kind-east

Fix proxy settings on kind nodes

If your Docker Desktop has a proxy configured, kind nodes inherit it and containerd cannot pull images from quay.io. Override NO_PROXY to bypass the proxy for registry traffic:

for node in west-control-plane west-worker east-control-plane east-worker; do
  docker exec $node sh -c '
    mkdir -p /etc/systemd/system/containerd.service.d
    cat > /etc/systemd/system/containerd.service.d/no-proxy.conf << EOF
[Service]
Environment="NO_PROXY=quay.io,registry.access.redhat.com,gcr.io,k8s.gcr.io,registry.k8s.io,docker.io"
EOF
    systemctl daemon-reload && systemctl restart containerd
  '
done

Pre-load Submariner images

Submariner 0.24.0 uses images from quay.io. Pull them to the host and load into each node to avoid pull failures:

IMAGES=(
  quay.io/submariner/submariner-operator:0.24.0
  quay.io/submariner/submariner-gateway:0.24.0
  quay.io/submariner/submariner-route-agent:0.24.0
  quay.io/submariner/lighthouse-agent:0.24.0
  quay.io/submariner/lighthouse-coredns:0.24.0
)

for img in "${IMAGES[@]}"; do docker pull "$img"; done

for cluster in west east; do
  for node in $(kubectl --context kind-${cluster} get nodes -o jsonpath='{.items[*].metadata.name}'); do
    for img in "${IMAGES[@]}"; do
      docker save "$img" | docker exec -i $node ctr -n k8s.io images import -
    done
  done
done

Deploy Submariner broker and join clusters

# Deploy broker on west (it becomes the broker host)
subctl deploy-broker --context kind-west

# Label gateway nodes
kubectl --context kind-west label node west-worker submariner.io/gateway=true
kubectl --context kind-east label node east-worker submariner.io/gateway=true

# Join both clusters
subctl join broker-info.subm --context kind-west --clusterid west \
  --natt=false --label-gateway=false --check-broker-certificate=false

subctl join broker-info.subm --context kind-east --clusterid east \
  --natt=false --label-gateway=false --check-broker-certificate=false

After subctl join, the operator on each cluster tries to reach the broker using 127.0.0.1 (the host port-forward address), which is unreachable from inside the pod. Patch the broker URL to use in-cluster addresses:

# west operator reaches broker via its own in-cluster kubernetes service
# 10.96.0.1 is the ClusterIP of the kubernetes service in the default namespace:
#   kubectl --context kind-west get svc kubernetes -n default
kubectl --context kind-west patch submariner -n submariner-operator submariner \
  --type=json -p='[{"op":"replace","path":"/spec/brokerK8sApiServer","value":"10.96.0.1:443"}]'

# east operator must reach west's API server via the docker bridge network
C1_IP=$(docker inspect west-control-plane \
  --format '{{(index .NetworkSettings.Networks "kind").IPAddress}}')
kubectl --context kind-east patch submariner -n submariner-operator submariner \
  --type=json -p="[{\"op\":\"replace\",\"path\":\"/spec/brokerK8sApiServer\",\"value\":\"${C1_IP}:6443\"}]"

Verify connectivity:

subctl show connections --context kind-west
# GATEWAY           CLUSTER    REMOTE IP   NAT   STATUS
# east-worker       east       172.19.x.x  no    connected

Prepare Namespaces and Secrets

DEVOPS_USER=<your-devops-user>
DEVOPS_KEY=<your-devops-key>

for ctx in kind-west kind-east; do
  kubectl --context $ctx create namespace pf-mr
  kubectl --context $ctx create secret generic devops-secret -n pf-mr \
    --from-literal=PING_IDENTITY_DEVOPS_USER=$DEVOPS_USER \
    --from-literal=PING_IDENTITY_DEVOPS_KEY=$DEVOPS_KEY
done

Deployment Files

This deployment uses four files. Three are Helm values files consumed directly by helm upgrade --install; the fourth is a Kustomize post-renderer that injects additional Kubernetes resources that the ping-devops chart does not generate natively.

scripts.yaml

This file defines two scripts mounted into the admin pods as ConfigMap entries. Starting with the 2605 release of the PingFederate image, there is new hook logic that uses 403 + license_agreement_not_accepted detection to determine whether a node needs seeding, which works correctly in both single- and multi-cluster deployments with a shared DNS.

The two scripts here:

  • liveness.sh calls pfadmin-check-active.sh on each liveness check to keep the pod label in sync with the current active/passive role.

  • pfadmin-check-active.sh queries the active heartbeat endpoint and patches the pod’s pf.admin.active label accordingly. The pingfederate-admin-active Service selector uses this label to route traffic only to the currently active node.

configMaps:
  custom-hooks:
    data:
      liveness.sh: |-
        #!/usr/bin/env sh
        URL="https://127.0.0.1:${PF_ENGINE_PORT}/pf/heartbeat.ping"
        if test "${OPERATIONAL_MODE}" = "CLUSTERED_CONSOLE" -o "${OPERATIONAL_MODE}" = "STANDALONE"; then
            test -f /tmp/ready || exit 1
            if test "${PF_CLUSTER_ADMIN_NODES_SYNC_ENABLED}" = "true"; then
                . "${HOOKS_DIR}/pfadmin-check-active.sh"
                setPFActive
            fi
            test "${OPERATIONAL_MODE}" = "CLUSTERED_CONSOLE" && URL="https://127.0.0.1:${PF_ADMIN_PORT}/pf/heartbeat.ping"
        fi
        curl -sSk -o /dev/null "${URL}" || exit 1
      pfadmin-check-active.sh: |-
        #!/usr/bin/env sh
        setPFActive() {
            APISERVER="https://kubernetes.default.svc"
            SERVICEACCOUNT="/var/run/secrets/kubernetes.io/serviceaccount"
            NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
            TOKEN=$(cat ${SERVICEACCOUNT}/token)
            CACERT=${SERVICEACCOUNT}/ca.crt
            POD_NAME=$(hostname)

            URL="https://127.0.0.1:${PF_ADMIN_PORT}/pf/heartbeat.ping?checkActive=true"
            _activelabel="false"
            _curlres=$(curl -sSk -o /dev/null -w "%{response_code}" "${URL}" 2>/dev/null)
            test "${_curlres}" -eq "200" && _activelabel="true"

            curl -s --cacert ${CACERT} -X PATCH -o /dev/null -w "%{http_code}" \
                -H "Content-Type: application/strategic-merge-patch+json" \
                -H "Authorization: Bearer ${TOKEN}" \
                "${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods/${POD_NAME}" \
                --data "{\"metadata\":{\"labels\":{\"pf.admin.active\":\"${_activelabel}\"}}}"
        }

base.yaml

The west cluster Helm values. global.image.tag sets the image version for all products. PF_NODE_GROUP_ID labels the cluster in the PingFederate console and is used by the east-pf.yaml override to distinguish the east cluster.

global:
  addReleaseNameToResource: none
  image:
    tag: "2605"
  envs:
    PING_IDENTITY_ACCEPT_EULA: "YES"
    PING_IDENTITY_PASSWORD: 2FederateM0re

pingfederate-admin:
  enabled: true
  envs:
    OPERATIONAL_MODE: CLUSTERED_CONSOLE
    # clusterset.local resolves across both clusters via Submariner Lighthouse CoreDNS
    DNS_QUERY_LOCATION: "pingfederate-cluster.pf-mr.svc.clusterset.local"
    PF_NODE_GROUP_ID: west
    PF_CLUSTER_ADMIN_NODES_SYNC_ENABLED: "true"
  volumeMounts:
    - mountPath: "/opt/liveness.sh"
      name: custom-vol
      subPath: liveness.sh
    - mountPath: "/opt/in/hooks/pfadmin-check-active.sh"
      name: custom-vol
      subPath: pfadmin-check-active.sh
  volumes:
    - name: custom-vol
      configMap:
        name: custom-hooks
        defaultMode: 0777
  rbac:
    generateServiceAccount: true
    serviceAccountName: pf-mc-admin-sa
    generateRoleAndRoleBinding: true
    applyServiceAccountToWorkload: true
    role:
      rules:
        - apiGroups: [""]
          resources: ["pods"]
          verbs: ["get", "watch", "list", "patch"]
  workload:
    labels:
      active: "false"
    type: StatefulSet
    statefulSet:
      persistentvolume:
        enabled: true

pingfederate-engine:
  enabled: false

east-pf.yaml

Applied on top of base.yaml for the east cluster deployment. Overrides the cluster identity.

pingfederate-admin:
  enabled: true
  envs:
    PF_NODE_GROUP_ID: east
  container:
    replicaCount: 1

Kustomize Post-Renderer

The ping-devops chart does not generate per-instance ServiceExport resources or the selector-based active-admin Services. A Kustomize post-renderer injects these after Helm renders its manifests.

Create a kustomize/ subdirectory with these five files.

kustomize/kustomize (executable script):

#!/bin/bash
cd "$(dirname "$0")"
cat <&0 > all.yaml
kustomize build . && rm all.yaml
chmod +x kustomize/kustomize

kustomize/kustomization.yaml:

resources:
  - all.yaml
  - pf-admin-active-svc.yaml
  - pf-admin-active-nodeport.yaml
  - ServiceExport.yaml

kustomize/pf-admin-active-svc.yaml — a ClusterIP Service that selects only pods with label pf.admin.active: "true":

apiVersion: v1
kind: Service
metadata:
  name: pingfederate-admin-active
  namespace: pf-mr
  labels:
    app.kubernetes.io/instance: pf-mr
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: pingfederate-admin
spec:
  ports:
    - name: https
      port: 9999
      protocol: TCP
      targetPort: 9999
  selector:
    app.kubernetes.io/instance: pf-mr
    app.kubernetes.io/name: pingfederate-admin
    pf.admin.active: "true"

kustomize/pf-admin-active-nodeport.yaml — a NodePort Service using the same pf.admin.active: "true" selector, bound to nodePort: 30999 which maps to host ports via extraPortMappings in the kind cluster configuration:

apiVersion: v1
kind: Service
metadata:
  name: pingfederate-admin-active-nodeport
  namespace: pf-mr
spec:
  type: NodePort
  ports:
    - name: https
      port: 9999
      targetPort: 9999
      nodePort: 30999
      protocol: TCP
  selector:
    app.kubernetes.io/instance: pf-mr
    app.kubernetes.io/name: pingfederate-admin
    pf.admin.active: "true"

kustomize/ServiceExport.yaml — exports all four PingFederate services into the Submariner clusterset so they are accessible from both clusters via svc.clusterset.local:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: pingfederate-admin
  namespace: pf-mr
---
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: pingfederate-admin-active
  namespace: pf-mr
---
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: pingfederate-cluster
  namespace: pf-mr
---
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: pingfederate-engine
  namespace: pf-mr

Deploy

The kustomize/kustomize script acts as a Helm post-renderer: it receives the rendered manifests on stdin and emits them plus the additional resources that the ping-devops chart does not generate natively.

Helm v4 requires post-renderers to be registered as plugins — passing a script path directly does not work. Create a plugin.yaml file in your working directory, replacing <absolute-path-to> with the absolute path to your kustomize/ directory:

name: "kustomize-renderer"
apiVersion: "v1"
version: "0.1.0"
type: "postrenderer/v1"
runtime: "subprocess"
runtimeConfig:
  platformCommand:
    - command: "<absolute-path-to>/kustomize/kustomize"

Register the plugin (this is a one-time step per machine):

mkdir -p "$(helm env HELM_PLUGINS)/kustomize-renderer"
cp plugin.yaml "$(helm env HELM_PLUGINS)/kustomize-renderer/"

Verify: helm plugin list should show kustomize-renderer with type postrenderer/v1.

On Helm v3, plugin registration is not required. You can pass the script path directly with --post-renderer ./kustomize/kustomize. See helm/helm#31340.

Deploy to the west cluster:

helm upgrade --install pf-mr pingidentity/ping-devops \
  --kube-context kind-west \
  --namespace pf-mr \
  -f scripts.yaml -f base.yaml \
  --post-renderer kustomize-renderer

Wait for pingfederate-admin-0 to come up and verify ServiceExports and ServiceImports:

kubectl --context kind-west get serviceexport -n pf-mr
# NAME                       AGE
# pingfederate-admin         ...
# pingfederate-admin-active  ...
# pingfederate-cluster       ...

kubectl --context kind-west get serviceimport -n pf-mr
# NAME                       TYPE          IP     AGE
# pingfederate-admin         ClusterSetIP  ...    ...
# pingfederate-admin-active  ClusterSetIP  ...    ...
# pingfederate-cluster       Headless             ...

Deploy to the east cluster:

helm upgrade --install pf-mr pingidentity/ping-devops \
  --kube-context kind-east \
  --namespace pf-mr \
  -f scripts.yaml -f base.yaml -f east-pf.yaml \
  --post-renderer kustomize-renderer

Accessing the Admin Console

The standard pingfederate-admin Service includes all admin pods — active and passive. Passive nodes return 403 for most admin operations and give severely limited console access. Always use pingfederate-admin-active to reach the admin console or admin API.

With the scripts in place, the pf.admin.active label is refreshed on each pod at every liveness interval. When this label is in place, the pingfederate-admin-active Service’s selector matches only the active pod so its endpoint tracks the currently active node automatically — including across a failover from one cluster to the other.

Why NodePort instead of port-forward

On macOS, Docker Desktop runs kind nodes inside a Linux VM. MetalLB LoadBalancer IPs live on the Docker bridge inside that VM and are not reachable from the host browser. kubectl port-forward works but requires a live terminal session, breaks on pod restart, and can only target one cluster at a time.

kind extraPortMappings bind a host port to a container port on the kind control-plane node at cluster creation time. This gives each cluster a stable localhost URL with no running process required. The pingfederate-admin-active-nodeport Service uses the same pf.admin.active: "true" selector as the ClusterIP service — it tracks the active node automatically and follows failover to the other cluster.

In a production environment, this port configuration would likely not be necessary. However, in any case, there would need to be some means of tracking the active pod and dynamically or manually updating DNS or other configurations to get to the active console. In this demonstration, we rely on using the assigned port of the active cluster service as our indicator for which pod we will be accessing.

If host port 9999 is already in use on your machine, change hostPort in the kind-west.yaml configuration and update the browser URL accordingly. The same applies to 19999 for the east cluster.

Determining which cluster holds the active node

kubectl --context kind-west -n pf-mr get pods \
  -l pf.admin.active=true -o wide

kubectl --context kind-east -n pf-mr get pods \
  -l pf.admin.active=true -o wide

Exactly one pod across both clusters should show pf.admin.active=true.

Open the console in a browser

Open the PingFederate admin console using the host port for the cluster that holds the active node:

Accept the self-signed certificate warning. Log in with username administrator and password 2FederateM0re.

The NodePort on the cluster that does not hold the active node has no endpoint and returns no response. This is expected — it also serves as a quick indicator of which cluster is currently active.

After failover

When the active node changes, liveness checks on the surviving pods update their labels within one liveness period. The pingfederate-admin-active-nodeport endpoint shifts accordingly. Re-run the active node check above to confirm which cluster holds the new active pod, then open the corresponding browser URL.

Validation

Cluster Status

Query from whichever cluster holds the active node (see Determining which cluster holds the active node):

kubectl --context kind-west -n pf-mr exec pingfederate-admin-0 -- \
  curl --insecure --silent \
    -H 'X-XSRF-Header: PingFederate' \
    --user "administrator:2FederateM0re" \
    'https://localhost:9999/pf-admin-api/v1/cluster/status' | jq .

Expected: 1 ACTIVE node, all others PASSIVE.

Active Service Endpoint

kubectl --context kind-west -n pf-mr get endpoints pingfederate-admin-active
# NAME                       ENDPOINTS           AGE
# pingfederate-admin-active  10.130.1.57:9999    ...
# (single endpoint — the active pod only)

kubectl --context kind-west -n pf-mr get endpoints pingfederate-admin
# NAME                  ENDPOINTS                             AGE
# pingfederate-admin    10.130.1.57:9999,10.130.1.59:9999    ...
# (all admin pods)

Scale Test

kubectl --context kind-west -n pf-mr scale sts pingfederate-admin --replicas=2

Expected log from pingfederate-admin-1:

INFO: Node is already seeded (HTTP 200). Skipping initial setup.
INFO: Querying cluster for existing active admin node...
INFO: Active admin already present in cluster. This node will remain passive.

The pingfederate-admin-active endpoint must still point only at the original active pod.

Simultaneous Cold Start Validation

Both clusters deployed in the preceding steps without sequencing already proves this feature will work. The 403-detection in hook 81 (built into the image) handles the potential race condition: each node queries /cluster/status; whichever sees 403 + license_agreement_not_accepted first seeds itself and promotes; the other sees 200 (state already replicated via DNS_PING) and stays passive.

To repeat the test explicitly:

kubectl --context kind-west -n pf-mr delete pvc --all
kubectl --context kind-east -n pf-mr delete pvc --all
kubectl --context kind-west delete namespace pf-mr
kubectl --context kind-east delete namespace pf-mr

kubectl --context kind-west create namespace pf-mr
kubectl --context kind-east create namespace pf-mr
# (re-create devops-secret in both namespaces)

helm upgrade --install pf-mr pingidentity/ping-devops \
  --kube-context kind-west --namespace pf-mr \
  -f scripts.yaml -f base.yaml \
  --post-renderer kustomize-renderer &

helm upgrade --install pf-mr pingidentity/ping-devops \
  --kube-context kind-east --namespace pf-mr \
  -f scripts.yaml -f base.yaml -f east-pf.yaml \
  --post-renderer kustomize-renderer &
wait

kubectl --context kind-west -n pf-mr rollout status statefulset/pingfederate-admin --timeout=15m &
kubectl --context kind-east -n pf-mr rollout status statefulset/pingfederate-admin --timeout=15m &
wait

Expected: 2 nodes in cluster status, exactly 1 ACTIVE, 1 PASSIVE.

Cross-Cluster DNS Verification

Confirm the clusterset DNS name resolves from an east pod:

kubectl --context kind-east -n pf-mr exec pingfederate-admin-0 -- \
  nslookup pingfederate-cluster.pf-mr.svc.clusterset.local
# Returns A-records for all pod IPs across both clusters

Cleanup

Delete the PersistentVolumeClaims before deleting the clusters. kind’s local-path provisioner binds PVCs to host directories on the kind nodes; deleting the cluster without first removing the PVCs leaves those directories orphaned on the host.

kubectl --context kind-west -n pf-mr delete pvc --all
kubectl --context kind-east -n pf-mr delete pvc --all

Then delete the kind clusters. All Submariner broker state, gateway state, and subctl join artifacts are destroyed with the clusters — no separate Submariner cleanup step is required.

kind delete cluster --name west
kind delete cluster --name east