Custom kubectl Credential Plugin for Browser SSO

In Part 4 you ran vault login -method=oidc role=human, then copied service_account_token from vault write k8s-*/creds/… into kubectl --token …. This final post removes that copy step with a small kubectl credential plugin. It picks the target cluster from your context, reuses a valid Vault token when possible, opens the Keycloak browser login only when needed, mints a short-lived SA token from the right k8s-* mount, and returns a proper ExecCredential to kubectl.

---

Prerequisites

Part 4’s lab must still be running on the EC2 host:

On the laptop:

vault CLI
kubectl
jq
PUBLIC_DNS of the lab host

export PUBLIC_DNS=lab.onlysre.dev
export VAULT_ADDR="http://${PUBLIC_DNS}:8200"

Reachability check:

curl -k https://${PUBLIC_DNS}:6443/version
curl -k https://${PUBLIC_DNS}:6444/version
curl -I ${VAULT_ADDR}/v1/sys/health
vault status

---

The Credential Plugin

One bash script: vault-k8s-creds.sh. It does four jobs:

1. Decide cluster-a vs cluster-b
   (from KUBERNETES_EXEC_INFO server port or current context name)
2. vault token lookup
   → if missing/expired: vault login -method=oidc role=human (browser)
3. vault write -format=json k8s-*/creds/…  for the matching mount/role
4. Print client.authentication.k8s.io/v1 ExecCredential
   with token + expirationTimestamp (15m TTL − 2m safety buffer)

Install it on PATH:

mkdir -p ~/.local/bin
cp scripts/vault-k8s-lab/vault-k8s-creds.sh ~/.local/bin/vault-k8s-creds.sh
chmod +x ~/.local/bin/vault-k8s-creds.sh

export PATH="$HOME/.local/bin:$PATH"
which vault-k8s-creds.sh

/home/you/.local/bin/vault-k8s-creds.sh

---

Kubeconfig with exec

Tell kubectl to call the plugin instead of embedding tokens. Minimal config (back up any existing file first):

apiVersion: v1
kind: Config
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: "https://lab.onlysre.dev:6443"
  name: cluster-a
- cluster:
    insecure-skip-tls-verify: true
    server: "https://lab.onlysre.dev:6444"
  name: cluster-b

users:
- name: vault-cluster-a
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: vault-k8s-creds.sh
      env:
      - name: VAULT_ADDR
        value: "http://lab.onlysre.dev:8200"
      provideClusterInfo: true
      interactiveMode: IfAvailable
- name: vault-cluster-b
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: vault-k8s-creds.sh
      env:
      - name: VAULT_ADDR
        value: "http://lab.onlysre.dev:8200"
      provideClusterInfo: true
      interactiveMode: IfAvailable

contexts:
- context:
    cluster: cluster-a
    user: vault-cluster-a
  name: cluster-a
- context:
    cluster: cluster-b
    user: vault-cluster-b
  name: cluster-b

current-context: cluster-a

Write it to ~/.kube/config or a dedicated file:

export KUBECONFIG=$HOME/.kube/vault-lab.yaml
kubectl config get-contexts

CURRENT   NAME        CLUSTER     AUTHINFO          NAMESPACE
*         cluster-a   cluster-a   vault-cluster-a
          cluster-b   cluster-b   vault-cluster-b

---

Full End-to-End Flow (First Login)

Start with no cached Vault token:

rm -f ~/.vault-token
vault token lookup 2>&1 || true

Error looking up token: Error making API request.

URL: GET http://lab.onlysre.dev:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

First command against cluster-a. The plugin sees no Vault token and starts OIDC:

kubectl get nodes --context cluster-a

vault-k8s-creds: no valid Vault token, starting OIDC login...
Complete the login via your OIDC provider. Launching browser to:

    http://lab.onlysre.dev:8080/realms/sre/protocol/openid-connect/auth?client_id=vault&…

Waiting for OIDC authentication to complete…

While the CLI waits, open the printed URL if the browser did not, sign in as demo / demo123 in the sre realm, and complete consent if prompted. After redirect to http://localhost:8250/oidc/callback, the Vault CLI writes ~/.vault-token and the plugin continues.

You may see a one-shot kubectl JSON parse error while login is still in progress—that is the first process exiting before credentials exist. Re-run the same command after login finishes:

NAME                      STATUS   ROLES           AGE   VERSION
cluster-a-control-plane   Ready    control-plane   66m   v1.31.0

Run it again immediately:

kubectl get nodes --context cluster-a

No browser, no “starting OIDC login” line—the Vault token is still valid and kubectl is honoring the plugin’s expirationTimestamp.

---

Switching Clusters

kubectl config use-context cluster-b
kubectl get nodes --context cluster-b

NAME                      STATUS   ROLES           AGE   VERSION
cluster-b-control-plane   Ready    control-plane   66m   v1.31.0

The plugin resolved cluster-b from the context name (or the :6444 server in KUBERNETES_EXEC_INFO), selected mount k8s-b and role cluster-b-admin, minted a fresh SA JWT, and returned it to kubectl. Day-to-day use is just context switches:

kubectl get pods --context cluster-a
kubectl get pods --context cluster-b

No more hand-copying tokens.

---

The Plugin Source

Complete script (same behavior as above). Save as vault-k8s-creds.sh, chmod +x, put on PATH.

#!/usr/bin/env bash
#
# vault-k8s-creds.sh
#
# kubectl exec credential plugin.
# Derives the target cluster (a or b) from the current kubectl context.
#
# Requirements:
#   - vault CLI configured for OIDC against the lab
#   - jq
#
# Behavior:
#   - Reuses existing Vault token (from `vault token lookup`)
#   - If no/expired Vault token → runs `vault login -method=oidc role=human`
#     (opens browser for Keycloak login)
#   - Fetches short-lived SA token from the correct k8s-* mount
#   - Returns ExecCredential v1 with token + expirationTimestamp (13 min)

set -euo pipefail

# --- Determine cluster from current context ---

get_context() {
  # Best source when provideClusterInfo: true
  if [[ -n "${KUBERNETES_EXEC_INFO:-}" ]]; then
    local server
    server=$(echo "$KUBERNETES_EXEC_INFO" | jq -r '.spec.cluster.server // ""' 2>/dev/null || true)
    if [[ "$server" == *":6443"* ]]; then
      echo "cluster-a"
      return 0
    fi
    if [[ "$server" == *":6444"* ]]; then
      echo "cluster-b"
      return 0
    fi
  fi

  # Fallback to current-context name
  if command -v kubectl >/dev/null 2>&1; then
    kubectl config current-context 2>/dev/null || true
  fi
}

CONTEXT_NAME=$(get_context)

case "$CONTEXT_NAME" in
  *cluster-a*|cluster-a)
    MOUNT="k8s-a"
    ROLE="cluster-a-admin"
    ;;
  *cluster-b*|cluster-b)
    MOUNT="k8s-b"
    ROLE="cluster-b-admin"
    ;;
  *)
    echo "vault-k8s-creds: cannot determine cluster from context '$CONTEXT_NAME'" >&2
    echo "Expected context names containing 'cluster-a' or 'cluster-b'." >&2
    exit 1
    ;;
esac

# --- Vault token handling (reuse or re-login) ---

if ! vault token lookup >/dev/null 2>&1; then
  echo "vault-k8s-creds: no valid Vault token, starting OIDC login..." >&2
  vault login -method=oidc role=human
fi

# --- Get short-lived Kubernetes token ---

OUT=$(vault write -format=json "${MOUNT}/creds/${ROLE}" kubernetes_namespace=default)

TOKEN=$(echo "$OUT" | jq -r '.data.service_account_token')

if [[ -z "$TOKEN" || "$TOKEN" == "null" ]]; then
  echo "vault-k8s-creds: failed to get token from Vault" >&2
  exit 1
fi

# --- Expiration: 15m - 2min safety buffer ---

if EXP=$(date -u -d '+13 minutes' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
  :
elif EXP=$(date -u -v+13M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
  :
else
  EXP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
fi

# --- Emit ExecCredential ---

cat <<EOF
{
  "apiVersion": "client.authentication.k8s.io/v1",
  "kind": "ExecCredential",
  "status": {
    "token": "${TOKEN}",
    "expirationTimestamp": "${EXP}"
  }
}
EOF

---

Why expirationTimestamp Matters

Returning expirationTimestamp tells kubectl the credential is good until that instant. kubectl caches it and skips calling the plugin until the time is near or past. Repeated commands stay fast without hammering Vault or reopening Keycloak on every request.

---

Reset / Logout

Force a fresh OIDC login on the next plugin call:

rm -f ~/.vault-token
vault token revoke -self 2>/dev/null || true

Hard reset including contexts you created for this lab:

rm -f ~/.vault-token
kubectl config delete-context cluster-a 2>/dev/null || true
kubectl config delete-context cluster-b 2>/dev/null || true
# re-apply the kubeconfig snippet or restart the shell

---

Troubleshooting

---

Summary of Values Used in This Post

---

What We Deliberately Simplified

---

Cleanup

# On the EC2
kind delete cluster --name cluster-a
kind delete cluster --name cluster-b
docker stop keycloak || true
# If you ran Vault in a container:
docker stop vault || true

# On your laptop
rm -f ~/.vault-token

---

That's the Series

End-to-end path you built:

kind cluster-a + cluster-b
Keycloak (identity)
Vault OIDC auth  +  k8s-a / k8s-b secrets engines
Laptop: vault login (browser) once, then short-lived SA JWTs
kubectl exec plugin: no manual token paste

Two clusters, one IdP, Vault as the glue, and a tiny plugin so day-to-day access feels like ordinary kubectl with browser SSO on first use.

Thanks for following along.