[K8S] The Time My Critical Security Bug Was Just kubectl Lying To Me
Situation
My Home Lab Originally Only Used LLDAP For User Authentication, But Since The Systems I Wanted To Integrate Later Only Spoke OIDC, I Put Keycloak In Front Of It As A Bridge, With Everything Else Talking To Keycloak Instead. Wiring That Up And Testing It Turned Up Something Interesting: Got The LDAP Federation Wired Up, Got The OIDC Client Configured, Got Kubernetes Itself Trusting The Issuer. Then, Testing Whether Any Of It Actually Enforced Anything, I Ran One kubectl Command With A Made-Up Token String, And Got Back A Full Pod List. For About Thirty Seconds I Was Convinced I’d Found A Cluster-Wide Auth Bypass.
I Hadn’t. The Bug Was In My Test, Not My Cluster.
Result First:
| Setup | Keycloak bridging LDAP (LLDAP) to OIDC, Kubernetes API server configured to trust the OIDC issuer |
| First Symptom | kubectl --token="garbage" returned a full, valid pod list — looked like anonymous cluster access |
| Real Cause | sudo kubectl silently picked up root’s existing admin kubeconfig; client-cert auth (TLS layer) authenticated the request before the bogus Bearer token was ever checked |
| Also Hit | LDAP Federation schema mismatch — tutorial’s default objectClass values don’t match what LLDAP actually uses, sync silently imports zero users, no error |
| Fix | Test with raw curl + explicit Bearer header, bypassing kubectl entirely; fix schema by querying the real directory with ldapsearch instead of copying tutorial defaults |
Both traps share the same root lesson: when a config is supposed to match another system’s exact shape (a schema, an auth chain), verify against the real thing, don’t trust the tutorial default, and don’t trust a test tool that might be quietly injecting its own credentials.
Trap One: LDAP Federation’s Schema Match Is Exact, And Fails Silently
Keycloak’s LDAP User Federation builds its user-search filter from a field called userObjectClasses, roughly (&(objectClass=X)(objectClass=Y)(uid=%s)). Nearly every tutorial, including Keycloak’s own docs, defaults this to inetOrgPerson, organizationalPerson, because that’s the common AD/OpenLDAP combination.
My LDAP server is LLDAP (a lightweight LDAP implementation), and its actual user entries look like this:
ldapsearch -x -H ldap://ldap.internal:3890 \
-D "uid=svc-bind,ou=people,dc=example,dc=internal" -w '<password>' \
-b "ou=people,dc=example,dc=internal" "(uid=someuser)"
objectclass: inetOrgPerson
objectclass: posixAccount
objectclass: mailAccount
objectclass: person
No organizationalPerson. Copy the tutorial default in, and Keycloak’s search filter requires a class that literally doesn’t exist on any LLDAP user. The search always returns empty. The sync runs, reports added: 0, and throws no error at all. It just quietly does nothing. Switching to inetOrgPerson, person (both actually present) fixed it on the next sync.
The fix that should be step one, not step three: run ldapsearch against the real directory before configuring anything that has to match its schema. Don’t assume the tutorial’s defaults apply — especially once you’re off mainstream AD/OpenLDAP and onto something like LLDAP, FreeIPA, or 389-ds.
Trap Two: OIDC Login Working Doesn’t Mean Kubernetes Trusts It
Getting Headlamp to redirect to Keycloak, authenticate against LDAP, and come back with a valid ID token only proves the app recognizes you. Headlamp uses that ID token directly as a Bearer token against the Kubernetes API (it’s not proxying through its own ServiceAccount). There’s a separate, K8s-native trust relationship that has nothing to do with Keycloak or Headlamp:
# RKE2: /etc/rancher/rke2/config.yaml
kube-apiserver-arg:
- "oidc-issuer-url=https://keycloak.example.internal/realms/homelab"
- "oidc-client-id=headlamp"
- "oidc-username-claim=email"
- "oidc-groups-claim=groups"
- "oidc-ca-file=/path/to/internal-ca.pem" # required if the issuer uses a self-signed cert
Two details that are easy to miss:
oidc-ca-fileisn’t optional if your issuer runs on a self-signed cert. The API server fetches the OIDC discovery document and JWKS over its own system trust chain — it won’t inherit trust from anywhere else, and skipping this is a guaranteedx509failure.- Successful OIDC authentication is not authorization.
oidc-username-claimonly answers “who is this,” RBAC still needs an explicitClusterRoleBindingfor that identity. A user with zero bindings logs into Keycloak and Headlamp just fine, then gets403 Forbiddenon every actual API call.
Trap Three: My Security Test Was Testing The Wrong Thing
Verifying whether OIDC actually gated the Kubernetes API, I ran this first:
sudo kubectl get pods -A --token="garbage" \
--server=https://127.0.0.1:6443 --insecure-skip-tls-verify=true
# → 200 OK, full pod list
A made-up string as a token, and it returned everything. For a moment that reads as “this cluster has no auth at all.”
What actually happened: under sudo, kubectl defaults to reading root’s own kubeconfig, which already contains a valid admin client certificate. --token and --server are override flags that get merged into the same AuthInfo object. They don’t clear out a client-cert field that’s already sitting there. And TLS client-cert authentication happens at the handshake, before the request even carries an HTTP-level Bearer header. The request was authenticated as admin via mTLS the entire time. The garbage token was never actually evaluated.
The fix was to stop trusting a tool that might be silently carrying other credentials, and test the protocol directly:
# bogus token — should be 401
curl -sk -H "Authorization: Bearer garbage" \
https://127.0.0.1:6443/api/v1/namespaces/default/pods
# → 401 ✓
# real OIDC token, no ClusterRoleBinding — should be 403
curl -sk -H "Authorization: Bearer <real_id_token>" \
https://127.0.0.1:6443/api/v1/namespaces/default/pods
# → 403, with a message naming the identity and the denied verb ✓
401 and 403 that clearly trace back to the token’s content are trustworthy evidence. A 200 for an identity that shouldn’t have access is a signal to question the test setup first. It’s not a reason to immediately write up a vulnerability report.
Notes
- LDAP schema mismatches fail silently, not loudly.
added: 0with zero errors looks like “nothing to sync,” not “your filter is wrong.” Query the real directory before configuring anything schema-dependent. - “The app logged me in” and “Kubernetes trusts this identity” are two separate trust relationships, configured in two separate places, and neither implies the other.
- Authentication success is not authorization. Always confirm there’s an RBAC binding before declaring an OIDC integration “working.”
sudo+kubectlsilently inherits root’s existing kubeconfig credentials. Any override flag you pass (--token,--server) gets merged into that AuthInfo, not substituted for it — client-cert auth wins regardless of what you put in--token.- When a security test returns a result that looks like “no protection at all,” the first suspect should be the test tool, not the system. High-level CLIs (
kubectl, most cloud CLIs) auto-load credentials from multiple sources; test with the rawest possible client (curl, explicit headers) before trusting a surprising result.
How To Prevent It
| Scenario | What To Do |
|---|---|
| Any LDAP/AD integration (SSO bridges, monitoring system directory auth, CI directory lookups) | Run ldapsearch -x (or your language’s LDAP client) against the real directory before configuring schema-matching fields like userObjectClasses/memberOf — especially off mainstream AD/OpenLDAP |
| Adding an OIDC identity provider in front of Kubernetes | Remember “app trusts OIDC” and “kube-apiserver trusts OIDC” are configured separately; confirm oidc-username-claim maps to whatever field you’ll actually bind RBAC to, and don’t forget oidc-ca-file for self-signed issuers |
| Manually verifying whether a token/credential check actually works | Test with the tool closest to the raw protocol (curl + manual headers) before trusting a high-level CLI — unless you’ve confirmed it isn’t silently carrying credentials you forgot about |
| Personal projects / CTF / homelab wanting hands-on enterprise IAM experience | Keycloak’s LDAP Federation + OIDC Client + kube-apiserver --oidc-* is one of the cheapest ways to actually operate a full centralized-identity flow end to end, and every step is independently testable |
Reference:
- Keycloak Server Administration Guide
- Kubernetes — Authenticating with OpenID Connect Tokens
- Real Incident: Home RKE2 cluster, Keycloak LDAP→OIDC bridge deployment