Keycloak OIDC and Vault Kubernetes Secrets Engines

In Part 2 you provisioned the lab on EC2: Keycloak on 8080, Vault listening on all interfaces at 8200, and two kind clusters. Now you will configure identity in Keycloak and enable the two building blocks in Vault (OIDC auth and the Kubernetes secrets engines) that will later let a human obtain short-lived tokens for either cluster.

---

Prerequisites

The lab from Part 2 must be running on the EC2 instance. Confirm the environment variables are set in your current shell:

source ./scripts/vault-k8s-lab/env.sh

PUBLIC_DNS=lab.onlysre.dev
VAULT_ADDR=http://lab.onlysre.dev:8200
KEYCLOAK_URL=http://lab.onlysre.dev:8080

Verify Keycloak and Vault are reachable on the EC2 host:

curl -sI ${KEYCLOAK_URL} | head -1
vault status

HTTP/1.1 302 Found
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         2.0.3
Build Date      2026-06-17T12:39:45Z
Storage Type    inmem
Cluster Name    vault-cluster-2cad9942
Cluster ID      5c80e347-367a-cb56-c9be-b2233af0f54f
HA Enabled      false

You will use the Keycloak admin CLI (kcadm.sh) by running it inside the running container. No separate install is required.

---

Logging Into Keycloak as Admin

Keycloak is running with the bootstrap admin user admin / admin. Use kcadm.sh inside the container to configure credentials against the master realm:

docker exec keycloak /opt/keycloak/bin/kcadm.sh config credentials \
  --server http://localhost:8080 \
  --realm master \
  --user admin \
  --password admin

Logging into http://localhost:8080 as user admin of realm master

This writes a temporary session file inside the container. All subsequent kcadm.sh commands in this post run through docker exec keycloak.

Briefly inspect the master realm (the default realm that ships with Keycloak):

docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms/master --fields realm,enabled

{
  "realm" : "master",
  "enabled" : true
}

You will not use the master realm for the demo. You will create a dedicated realm called sre.

---

Creating the 'sre' Realm

Create a new realm named sre and enable it:

docker exec keycloak /opt/keycloak/bin/kcadm.sh create realms \
  -s realm=sre \
  -s enabled=true

Created new realm with id 'sre'

List realms to confirm:

docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms --fields realm,enabled

[
  {
    "realm" : "master",
    "enabled" : true
  },
  {
    "realm" : "sre",
    "enabled" : true
  }
]

The sre realm is now ready to hold the OIDC client that Vault will use.

---

Creating the Confidential OIDC Client for Vault

Vault will act as an OIDC client. Create a confidential client named vault inside the sre realm.

The client must:

• Be confidential (has a client secret)
• Have Standard Flow enabled (for browser-based login)
• Allow the redirect URIs that Vault and the future kubectl plugin will use

Set the redirect URIs to cover both laptop testing and the Vault UI on the EC2 public DNS:

docker exec keycloak /opt/keycloak/bin/kcadm.sh create clients -r sre \
  -s clientId=vault \
  -s enabled=true \
  -s clientAuthenticatorType=client-secret \
  -s standardFlowEnabled=true \
  -s 'redirectUris=["http://localhost:8250/oidc/callback","http://lab.onlysre.dev:8200/ui/vault/auth/oidc/oidc/callback","http://lab.onlysre.dev:8200/oidc/callback"]' \
  -s 'webOrigins=["*"]'

Created new client with id '4204d17f-851d-4bc9-96ee-2e94c8dd5633'

Retrieve the generated client secret. You will need this value when configuring Vault later:

CLIENT_ID=$(docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields id --format csv | tail -1 | tr -d '"')
docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients/$CLIENT_ID/client-secret -r sre

{
  "type" : "secret",
  "value" : "FhT8zbYof6TIhXJK2cVXF8vXGCdVaGJS"
}

Save the secret (FhT8zbYof6TIhXJK2cVXF8vXGCdVaGJS in this run) for the next post. Treat it like any other credential.

Verify the client settings:

docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields clientId,enabled,standardFlowEnabled,clientAuthenticatorType

[
  {
    "clientId" : "vault",
    "enabled" : true,
    "clientAuthenticatorType" : "client-secret",
    "standardFlowEnabled" : true
  }
]

Keycloak is now ready to issue tokens for the vault client to users who log into the sre realm.

---

Enabling OIDC Authentication in Vault

With the realm and client in place, enable the OIDC auth method in Vault:

vault auth enable oidc

Success! Enabled oidc auth method at: oidc/

List auth methods to see it alongside the token method:

vault auth list

Path      Type     Accessor               Description                Version
----      ----     --------               -----------                -------
oidc/     oidc     auth_oidc_10b620fc     n/a                        n/a
token/    token    auth_token_9f8ea698    token based credentials    n/a

Understanding the OIDC Auth Method in Vault

OpenID Connect (OIDC) is an identity layer on top of OAuth 2.0. An application (the relying party) verifies who you are by sending you to an external OpenID Provider. The provider returns a signed ID token (a JWT). The application validates that token and never sees your password.

In this lab the mapping is fixed:

Vault never stores Keycloak passwords. It only validates a signed assertion from Keycloak, then issues its own Vault token with policies attached.

Login flow (what actually happens)

The command you will run in Part 4 looks like this:

vault login -method=oidc role=human

Or you pick OIDC in the Vault UI. Either way, the sequence is the same:

You  →  Vault (start OIDC, role=human)
     →  Browser redirect to Keycloak (sre realm)
     →  You authenticate at Keycloak
     →  Keycloak redirects back with authorization code
     →  Vault exchanges code for ID token (back-channel)
     →  Vault validates JWT, maps role, issues Vault token

Start. Vault creates a random state and nonce, then builds an authorization URL for the sre realm. The URL carries client_id=vault, a registered redirect_uri, response_type=code, scopes such as openid profile email, plus state and nonce for CSRF and replay protection.

Keycloak. Your browser hits Keycloak’s authorization endpoint. You sign in against the sre user store. On success Keycloak redirects to the exact redirect_uri Vault requested, with a short-lived authorization code and the same state.

Code exchange. Vault receives the code at its callback. On a back-channel HTTPS POST to Keycloak’s token endpoint it authenticates with client_id + client_secret, sends the code and redirect_uri, and receives an ID token (JWT), optionally with an access token.

Validation and Vault token. Vault loads Keycloak’s JWKS keys from discovery, then checks signature, iss, aud (must include the client id), exp, nonce, and sub. It looks up the OIDC role (here human), maps identity via user_claim (usually sub), attaches that role’s policies, and returns a Vault token. That Vault token is what every later CLI and UI call uses—not the Keycloak password, and not the raw ID token.

The whole dance finishes in a few seconds. The only secrets that leave Keycloak are the short-lived authorization code and the ID token—never your password.

Why redirect URIs are pre-registered

Keycloak only redirects to URIs you listed on the confidential client. That stops an attacker from swapping in their own callback and stealing the authorization code. For the vault client you registered three URIs:

The same three values must appear again on the Vault OIDC role as allowed_redirect_uris (Part 4). Mismatch on either side breaks login with opaque redirect errors.

The two Vault objects you will write in Part 4

Enabling oidc/ only mounts the auth method. Behavior comes from two writes:

Connection to Keycloak — path auth/oidc/config:

oidc_discovery_url  = http://${PUBLIC_DNS}:8080/realms/sre
oidc_client_id      = vault
oidc_client_secret  = <from Keycloak client>
default_role        = human

Discovery expands to the usual well-known document under that realm URL. Vault uses it for authorization, token, and JWKS endpoints.

What a successful login becomes — path auth/oidc/role/human:

user_claim             = sub
allowed_redirect_uris  = (same three URIs as Keycloak)
policies               = (Vault policies for this role)
ttl / max_ttl          = session lifetime of the Vault token

You will write both objects with real values in Part 4. Until then the method is enabled but not wired.

OIDC vs userpass (Part 1)

OIDC turns Vault into a consumer of identity rather than the source of truth. That is the foundation for the single sign-on path the rest of the series builds on.

---

Enabling the Kubernetes Secrets Engines

Vault can generate short-lived Kubernetes ServiceAccount tokens on demand using the Kubernetes secrets engine. You will mount two separate engines so each cluster has its own path:

vault secrets enable -path=k8s-a kubernetes
vault secrets enable -path=k8s-b kubernetes

Success! Enabled the kubernetes secrets engine at: k8s-a/
Success! Enabled the kubernetes secrets engine at: k8s-b/

Confirm:

vault secrets list

Path               Type              Accessor                   Description
----               ----              --------                   -----------
agent-registry/    agent_registry    agent-registry_df7537c4    agent registry
cubbyhole/         cubbyhole         cubbyhole_4e3ffed5         per-token private secret storage
identity/          identity          identity_b02496e4          identity store
k8s-a/             kubernetes        kubernetes_5db4f109        n/a
k8s-b/             kubernetes        kubernetes_17c3096b        n/a
secret/            kv                kv_0b3756a9                key/value secret storage
sys/               system            system_ab89613e            system endpoints used for control, policy and debugging

(The kv/ engine may be present from earlier experiments; it is not used in this series.)

Understanding the Kubernetes Secrets Engine in Depth

The Kubernetes secrets engine is not a password store and not a static secret vault. It is a broker: with a valid Vault identity it asks a real Kubernetes cluster to mint a fresh ServiceAccount token, wraps that token in a short lease, and returns it. When the lease ends, the token dies with it.

Why bother

Human access to clusters usually degrades into long-lived kubeconfig tokens, hard-to-rotate client certs, or shared ServiceAccounts that never expire. Someone who only needed to inspect pods in one namespace ends up holding cluster-admin forever.

This engine replaces that pattern with on-demand generation, short TTLs (you will use 15 minutes), and an explicit map from a Vault identity to a specific Kubernetes ServiceAccount. Least privilege becomes a path + policy problem instead of a “share the admin kubeconfig” problem.

How ServiceAccount tokens are minted

Since Kubernetes 1.21 the supported path is the TokenRequest API—not the legacy secret of type kubernetes.io/service-account-token. Manually it looks like:

kubectl create token vault-demo-sa --namespace default

The API server checks that the caller may mint tokens for that ServiceAccount, signs a JWT with the cluster key, embeds claims such as sub, aud, exp, and iss, and returns a token valid for the requested TTL. Any client that presents that JWT is that ServiceAccount.

Vault’s engine is the same TokenRequest call, automated and gated by Vault policy.

What lives under a mount

When you ran vault secrets enable -path=k8s-a kubernetes, Vault created an isolated mount. Two objects matter:

Cluster connection — path k8s-a/config:

kubernetes_host     = URL Vault uses to reach the API server
kubernetes_ca_cert  = CA bundle to verify that API server
# optional: service_account_jwt or client cert if Vault itself
#            authenticates to the cluster with a dedicated SA

Roles — path k8s-a/roles/<name>. Each role is a policy for minting:

service_account_name            = which SA tokens may be requested
allowed_kubernetes_namespaces   = which namespaces are allowed
token_default_ttl / token_max_ttl
# optional audience / extra audiences / token type

When a caller with a Vault token writes to k8s-a/creds/cluster-a-admin, Vault checks path policy, reads the role, builds a TokenRequest, POSTs it to the configured host with the configured CA, receives the signed JWT, and returns it under a Vault lease. The lease is how Vault tracks and can revoke the credential even though the JWT itself is not stored in Vault’s KV-style storage.

Caller (Vault token)
  →  write k8s-a/creds/<role>
  →  Vault policy check + role lookup
  →  TokenRequest → Kubernetes API
  →  JWT + lease metadata back to caller

Two mounts, two clusters

k8s-a/  →  cluster-a  (host port 6443)
k8s-b/  →  cluster-b  (host port 6444)

Separate mounts mean separate CAs and hosts, separate roles and policies, and clear audit lines (“who asked for cluster-b at 14:32”). You can grant different humans different access per cluster without cross-contamination.

Kubernetes engine vs KV v2 (Part 1)

The policy model is the same idea as Part 1. Allowing read on a KV path is the same mental model as allowing update on k8s-a/creds/cluster-a-admin: path + capability decide access.

Role and creds lifecycle (preview of Part 4)

Configuration first (real host and CA come from the kind clusters in Part 4):

vault write k8s-a/config \
  kubernetes_host="https://10.0.1.23:6443" \
  kubernetes_ca_cert=@cluster-a-ca.crt

vault write k8s-a/roles/cluster-a-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

Then a human path:

1. Authenticate via OIDC  →  Vault token + policy (e.g. k8s-a-creds)
2. vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default
3. Vault → TokenRequest on cluster-a using config from step “config”
4. Cluster returns JWT (≈15m)
5. Use JWT with kubectl (or the plugin in Part 5)
6. TTL or lease revoke ends access — no cluster cleanup step

Security properties you get by default

At the end of this post the engines are enabled but not yet configured. Part 4 extracts each kind cluster’s CA, discovers the reachable kubernetes_host from Vault’s view of the lab, writes real config and roles objects, attaches policies to the OIDC role, and tests minting from the CLI before the browser UI flow.

---

Skeleton Role and Credential Commands

The wiring (CA certificates, reachable API URL, policies) is Part 4. For shape only, the role writes look like this:

vault write k8s-a/roles/cluster-a-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

vault write k8s-b/roles/cluster-b-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

Once config and policies exist, a logged-in user mints a token with:

vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default

An illustrative success response:

Key                       Value
---                       -----
lease_id                  k8s-a/creds/cluster-a-admin/...
lease_duration            15m
lease_renewable           true
service_account_token     eyJhbG...NiIs...
service_account_name      vault-demo-sa
service_account_namespace default

These commands are not functional yet—the mounts have no cluster connection. Part 4 finishes that configuration.

---

How the Pieces Fit Together

You now have three building blocks enabled or created in the lab:

Part 1 pattern:  userpass  →  policy  →  kv
This series:     oidc      →  policy  →  kubernetes (k8s-a | k8s-b)

Part 4 connects the OIDC discovery URL and client secret, configures each engine with real cluster details, creates the policies, and walks the end-to-end path from the Vault UI to a working kubectl session.

---

What's Next?

In Part 4 you will:

Next you wire the pieces and obtain a real short-lived Kubernetes token through the browser.