Hardening Kubernetes RBAC: Blue Team Defense Guide
Overview & Defensive Context#
In this morning's offensive breakdown, we demonstrated how default Kubernetes configurations transform an unprivileged pod compromise into a full cluster takeover. By default, every pod running in a Kubernetes cluster mounts an ambient ServiceAccount token via projected volumes (/var/run/secrets/kubernetes.io/serviceaccount/token). When underlying roles are provisioned with overly permissive RBAC verbs—such as bind, escalate, impersonate, nodes/proxy, or create pods—an attacker who obtains arbitrary code execution can query the kube-apiserver, bind the built-in cluster-admin ClusterRole, and deploy host-mounting containers to escape onto the node.
Standard perimeter defenses, web application firewalls (WAFs), and ingress proxies remain entirely blind to this lateral movement. Because all interactions occur internally over mutual TLS (mTLS) against the Kubernetes API server (port 6443) using legitimate bearer tokens, perimeter appliances cannot inspect or disrupt the payload. Furthermore, static RBAC policies lack the dynamic context required to inspect pod specifications; an account authorized to create pods can submit a spec requesting hostPath: /andprivileged: true, effortlessly converting API privileges into node-level root execution.
[!IMPORTANT] Preventing Kubernetes privilege escalation requires a defense-in-depth posture: disabling automated token projection by default, enforcing admission control to block dangerous pod primitives, auditing RBAC policies against dangerous escalation verbs, and capturing Kubernetes API audit logs alongside kernel-level eBPF runtime telemetry.
Architecture Hardening: Defense-in-Depth Control Plane#
Eliminating pod-to-cluster escalation paths requires multiple independent verification gates along the request lifecycle. Every inbound API request and runtime action must be validated before execution.
flowchart TD
subgraph PodBoundary ["Workload Execution Boundary (Namespace)"]
Workload["Container Workload"]
TokenProj["Projected Volume (/var/run/secrets)"]
TokenDisabled["automountServiceAccountToken: false"]
end
subgraph APIServerPipeline ["Kubernetes API Server Request Lifecycle"]
AuthN{"Authentication (mTLS / JWT OIDC)"}
RBACAuthZ{"RBAC Authorization Engine"}
AdmControl{"Admission Control Gate (ValidatingAdmissionPolicy)"}
AuditLogger["Kubernetes API Audit Logger (SIEM Stream)"]
EtcdStorage["etcd Cluster State Database"]
end
subgraph NodeRuntimeLayer ["Worker Node & Kernel Runtime"]
KubeletDaemon["Kubelet Daemon (Port 10250)"]
ContainerRuntime["containerd / CRI-O"]
eBPFProbe["eBPF Telemetry Probe (Tetragon / Falco)"]
HostKernel["Host Kernel & Namespaces"]
end
Workload -.->|Blocked: No Ambient Token| TokenDisabled
Workload -->|Bearer Token Request| AuthN
AuthN -->|Valid Identity| RBACAuthZ
RBACAuthZ -->|Denied: Dangerous Verbs bind/escalate| AuditLogger
RBACAuthZ -->|Permitted Verb| AdmControl
AdmControl -->|Rejected: hostPath or Privileged Spec| AuditLogger
AdmControl -->|Approved Workload Spec| EtcdStorage
EtcdStorage -->|Schedule Pod| KubeletDaemon
KubeletDaemon --> ContainerRuntime
ContainerRuntime --> HostKernel
eBPFProbe -.->|Block Container Escape / nsenter| HostKernelHardened Configurations & Policy Implementations#
Securing Kubernetes workloads demands eliminating ambient tokens and strictly governing pod creation specs through native admission control.
1. Disabling Default ServiceAccount Token Projection#
To minimize the blast radius of any compromised pod, prevent the automatic injection of ServiceAccount tokens. This must be enforced on the default ServiceAccount within each namespace and explicitly declared on sensitive pod specifications:
## Enforce token disablement on the namespace default ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: production
automountServiceAccountToken: false
yaml
## Explicitly enforce token isolation on workload specifications
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processing-api
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: payment-api
template:
metadata:
labels:
app: payment-api
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: internal-registry.enterprise.local/payment-api:v2.4.1
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
limits:
cpu: "500m"
memory: "512Mi"
requests:
cpu: "100m"
memory: "128Mi"
2. ValidatingAdmissionPolicy: Blocking hostPath and Privileged Containers#
Kubernetes 1.30+ supports native In-Tree ValidatingAdmissionPolicy using Common Expression Language (CEL), completely eliminating the operational fragility of external admission webhooks:
## CEL Admission Policy: Reject Privileged Specs and Host Volumes
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: deny-host-mount-and-privilege
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: "!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath))"
message: "Security Policy Violation: Deploying pods with hostPath volumes is strictly forbidden."
- expression: "!has(object.spec.hostPID) || object.spec.hostPID == false"
message: "Security Policy Violation: Host PID namespace sharing is prohibited."
- expression: "!has(object.spec.hostNetwork) || object.spec.hostNetwork == false"
message: "Security Policy Violation: Host network namespace sharing is prohibited."
- expression: >
object.spec.containers.all(c, !has(c.securityContext) ||
!has(c.securityContext.privileged) || c.securityContext.privileged == false)
message: "Security Policy Violation: Privileged containers are strictly forbidden."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: bind-deny-host-mount-and-privilege
spec:
policyName: deny-host-mount-and-privilege
validationActions: [Deny]
matchResources:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system"]
Production Detection Queries & Telemetry#
Detecting RBAC escalation and pod escape attempts requires continuous analysis across two telemetry streams: the Kubernetes API server audit log and host kernel eBPF probes.
1. Validated Sigma Rule: Suspicious RoleBinding to Administrative Roles#
This Sigma rule detects unauthorized bindings to administrative ClusterRoles (cluster-admin, admin, or custom elevated roles) originated by non-exempt service accounts or anomalous users:
title: Suspicious Kubernetes Administrative RoleBinding Created
id: 7a8e91c2-3e4b-4821-99cb-1a89f2d1e034
status: test
description: Detects creation or modification of Kubernetes RoleBindings or ClusterRoleBindings targeting privileged administrative roles.
references:
- https://attack.mitre.org/techniques/T1078/002/
- https://kubernetes.io/docs/reference/access-authn-authz/rbac/
logsource:
category: application
product: kubernetes
service: audit
detection:
selection_verbs:
verb:
- 'create'
- 'patch'
- 'update'
selection_resources:
objectRef.apiGroup: 'rbac.authorization.k8s.io'
objectRef.resource:
- 'clusterrolebindings'
- 'rolebindings'
selection_target_roles:
requestObject.roleRef.name:
- 'cluster-admin'
- 'admin'
- 'edit'
- 'system:controller:*'
filter_system_users:
user.username:
- 'kubernetes-admin'
- 'system:kube-controller-manager'
- 'system:kube-scheduler'
condition: selection_verbs and selection_resources and selection_target_roles and not filter_system_users
falsepositives:
- Authorized infrastructure provisioning via CI/CD GitOps pipelines (e.g., ArgoCD, Flux) using dedicated deployment service accounts.
level: high
tags:
- attack.privilege_escalation
- attack.t1078.002
- attack.persistence
2. Comprehensive Kubernetes API Audit Policy#
Control plane nodes must be configured with a robust audit policy (/etc/kubernetes/audit-policy.yaml) logging full request and response payloads for RBAC operations and sensitive pod lifecycle events:
## Production Kubernetes Audit Policy for Control Plane
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# 1. Capture full RequestResponse for all RBAC authorization modifications
- level: RequestResponse
verbs: ["create", "update", "patch", "delete"]
resources:
- group: "rbac.authorization.k8s.io"
resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
# 2. Capture RequestResponse for privilege escalation and impersonation attempts
- level: RequestResponse
verbs: ["impersonate"]
resources:
- group: ""
resources: ["users", "groups", "serviceaccounts"]
# 3. Log interactive exec, attach, and proxy access to pods and nodes
- level: RequestResponse
verbs: ["create"]
resources:
- group: ""
resources: ["pods/exec", "pods/attach", "pods/portforward"]
- group: ""
resources: ["nodes/proxy", "pods/proxy"]
# 4. Log pod creation metadata to inspect volume mounts
- level: RequestResponse
verbs: ["create", "update"]
resources:
- group: ""
resources: ["pods"]
# 5. Drop high-volume read-only noise from controllers
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: ""
resources: ["endpoints", "services"]
3. Runtime eBPF Telemetry: Cilium Tetragon TracingPolicy#
To detect and terminate container breakout attempts using nsenterorchroot against the host root filesystem, deploy a kernel-level Tetragon TracingPolicy:
## Tetragon eBPF Policy: Detect and Terminate Host Namespace Escapes
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: block-container-host-escape
namespace: kube-system
spec:
kprobes:
- call: "sys_enter_execve"
syscall: true
args:
- index: 0
type: "string"
selectors:
- matchArgs:
- index: 0
operator: "Prefix"
values:
- "/bin/nsenter"
- "/usr/bin/nsenter"
- "/usr/sbin/chroot"
- "/sbin/chroot"
matchNamespaces:
- namespace: "production"
matchActions:
- action: Sigkill
- action: Post
[!WARNING] While API audit logs provide irrefutable records of control-plane operations, they cannot alert on in-memory processes executed inside a running container. Kernel eBPF telemetry is indispensable for detecting when an adversary escapes container namespaces and executes binaries directly against the host.
Enterprise Mitigation Matrix#
The following matrix contrasts operational workarounds, immediate hotfixes, and sustainable architectural controls:
| Mitigation Layer | Technical Control | Operational Blast Radius | Performance Overhead | Security Guarantee |
|---|---|---|---|---|
| Workaround | Disable automountServiceAccountToken across all namespace default ServiceAccounts |
Low (requires explicit volume mounts for workloads needing API access) | Zero | Prevents ambient token harvesting by unprivileged compromised pods |
| Hotfix | Deploy CEL ValidatingAdmissionPolicyblockinghostPath, hostPID, and privileged: true |
Low to Moderate (blocks legacy pods that require raw host device access) | < 1ms API latency | Completely eliminates trivial pod-to-node container breakout vectors |
| Hotfix | Revoke bind, escalate, and impersonate verbs from all non-administrative ClusterRoles |
Moderate (may break misconfigured internal operator controllers) | Zero | Prevents non-root identities from self-granting cluster-admin privileges |
| Architectural Fix | Implement GitOps-enforced RBAC pipelines with static linters (kube-rbac-audit, Krane) |
High initial setup (shifts all RBAC modifications to code review workflows) | Zero runtime overhead | Eliminates out-of-band manual role binding and enforces strict least privilege |
| Architectural Fix | Deploy eBPF runtime enforcement (Cilium Tetragon / Falco) with automated SIGKILL |
Moderate (requires modern Linux kernel 5.8+ across all cluster worker nodes) | Negligible (< 1.5% CPU) | Intercepts zero-day container breakouts and host filesystem traversal in real time |
Incident Response & Verification Playbook#
When an alert triggers for suspicious ClusterRoleBinding creation or unauthorized pod creation with host volume mounts, execute the following containment playbook:
Phase 1: Rapid Triage & Blast Radius Identification#
- Extract Authorization Events from Audit Logs: Identify the originating IP, caller identity, and target resource from the API server audit stream:
jq -r 'select(.objectRef.resource=="clusterrolebindings" and (.verb=="create" or .verb=="patch")) |
"(.requestReceivedTimestamp) | User: (.user.username) | Role: (.requestObject.roleRef.name) | CallerIP: (.sourceIPs[0])"' \
/var/log/kubernetes/audit/audit.log
- Audit Active RoleBindings for Dangerous Roles:
Identify all subjects bound to
cluster-adminacross the cluster:
kubectl get clusterrolebindings -o json | jq -r '
.items[] | select(.roleRef.name == "cluster-admin") |
"Binding: (.metadata.name) -> Subjects: ([.subjects[]? | "(.kind):(.name)"] | join(","))"'
- Locate Pods with hostPath Mounts in Non-System Namespaces: Scan all running pods across tenant namespaces for suspicious host root volume mounts:
kubectl get pods --all-namespaces -o json | jq -r '
.items[] | select(.spec.volumes[]?.hostPath != null) |
select(.metadata.namespace != "kube-system") |
"Namespace: (.metadata.namespace) | Pod: (.metadata.name) | HostPath: (.spec.volumes[].hostPath.path)"'
Phase 2: Containment & Credential Invalidation#
- Delete Rogue Bindings and Malicious Pods Immediately: Sever administrative rights by revoking the unauthorized binding and deleting rogue workloads:
kubectl delete clusterrolebinding malicious-admin-binding --ignore-not-found=true
kubectl delete pod -n production compromised-host-pod --grace-period=0 --force
- Invalidate Compromised ServiceAccount Tokens: Delete the compromised ServiceAccount or rotate its secrets to instantly invalidate all issued tokens:
# Deleting the ServiceAccount revokes all associated projected and secret tokens
kubectl delete serviceaccount -n production compromised-sa
- Cordon and Drain Affected Worker Nodes:
If a privileged container with
hostPathwas successfully scheduled and executed, treat the underlying worker node as fully compromised:
kubectl cordon node-worker-04.internal
kubectl drain node-worker-04.internal --delete-emptydir-data --force --ignore-daemonsets
[!CAUTION] Draining a node immediately evicts all workloads to other cluster nodes. Ensure cluster compute headroom is sufficient to prevent cascading resource starvation across remaining worker instances.
Phase 3: Cluster Hygiene & Verification Checklist#
- Verify In-Tree Admission Policy Enforcement:
Attempt to create a test pod mounting
/etcto ensureValidatingAdmissionPolicyactively rejects the request:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: test-hostpath-rejection
namespace: default
spec:
containers:
- name: test
image: busybox
command: ["sh", "-c", "sleep 30"]
volumeMounts:
- mountPath: /host-test
name: test-vol
volumes:
- name: test-vol
hostPath:
path: /etc
EOF
Expected Result: Error from server (Forbidden): admission webhook or ValidatingAdmissionPolicy denied the request.
- Automated RBAC Exposure Scan: Execute an RBAC assessment tool to verify zero lingering privilege escalation paths:
kubectl auth can-i --list --as=system:serviceaccount:production:payment-sa
Comments
Post a Comment