Threat Hunting Exercise: Analyzing Kubernetes Exec Privileged Pod Dataset

Extending cloud-native coverage with a simulated Kubernetes audit log (k8s_audit_exec_privileged_pod_2022-06-20T151000.json), capturing T1610: Deploy Container and T1611: Escape to Host, where an adversary with stolen or over-permissioned kubectl credentials deploys a privileged pod (or execs into an existing one) to break out of container isolation and access the underlying node's filesystem/credentials.

Step 1: Hypothesis Formation

Hypothesis: An adversary with valid but over-permissioned Kubernetes RBAC access creates a new pod with securityContext.privileged=true and a hostPath volume mount of /, or execs (kubectl exec) into an existing privileged pod, then uses the resulting host-level access to read node secrets (e.g., kubelet credentials, cloud instance metadata) or escape to the underlying host OS. Indicators:

• K8s audit log create event on pods with spec.containers[].securityContext.privileged=true and/or spec.volumes[].hostPath.path="/".
• create event on pods/exec subresource for an existing pod, especially one not typically interacted with (e.g., a kube-system component).
• Requesting user/service account with no prior history of pod creation in that namespace (baseline anomaly).
• Follow-on activity from the pod's own logs or node-level auditd showing access to /host mount paths or cloud metadata endpoint (169.254.169.254) from within the container.

Null Hypothesis: Legitimate privileged workload (e.g., a CNI plugin, storage driver, or monitoring DaemonSet that legitimately requires privileged access) or an authorized debugging session by platform engineering. Invalidate via the requesting identity's RBAC role (should be a human/CI user, not a system: service account) and absence of a change-ticket/known-workload label.

Rationale: Container escape is the Kubernetes analog to the Windows LOLBIN/service-abuse hunts earlier in this series - the audit log's pods and pods/exec resource events are the equivalent of Sysmon Event 1/10, giving us creation and access-context visibility even without host-level EDR inside every container.

Step 2: Data Sources and Scope

• Sources: Kubernetes API server audit logs (create/get/connect verbs on pods, pods/exec, pods/attach); node-level auditd or Falco for in-container syscall corroboration if available.
• Scope: ~2022-06-20T15:10:00-15:12:30 UTC; Cluster: prod-eks-01; Namespace: default; Requesting identity: ci-deploy-bot service account (token believed leaked from a public CI log).
• SIEM Queries (Falco/K8s audit via Splunk/ELK):
• Privileged create: verb=create objectRef.resource=pods requestObject.spec.containers[].securityContext.privileged=true
• Exec abuse: verb=create objectRef.resource=pods objectRef.subresource=exec | stats count by user.username, objectRef.namespace
• Baseline anomaly: user.username="system:serviceaccount:default:ci-deploy-bot" | where NOT seen creating pods in prior 30d

Step 3: Key Findings

Parsed audit log (9 events shown) shows a leaked CI service-account token used to deploy a privileged, host-mounted pod, followed by exec access retrieving node-level credentials.

Validation:

• Timeline: Recon → privileged pod creation → exec → host secret theft → metadata credential theft → persistent cluster-admin grant → pod cleanup, all within 2.5 minutes.
• False Positives: ci-deploy-bot has no legitimate need for kube-system secret access or privileged pod creation per its documented RBAC role (build-and-push only); the source IP does not match the CI provider's published runner ranges.
• Correlation: Same service-account identity throughout; the sequence (privileged pod → host file read → metadata contact → clusterrolebinding) is a well-known, repeatable K8s escape-and-persist playbook.

Step 4: Recommendations & Next Steps

• Response: Revoke/rotate the ci-deploy-bot token immediately; delete the debug-admin-binding ClusterRoleBinding; rotate the cluster CA and all node IAM credentials (assume full compromise); audit all RBAC bindings for unintended cluster-admin grants.
• Detection: Kubernetes admission control (OPA/Gatekeeper or Pod Security Admission) to block privileged: true and hostPath / mounts outside an explicit allowlist; Falco rules for read sensitive file untrusted and contact K8S API Server From Container; audit-log alert: title: Privileged Pod + Exec Combo → selection: create pods privileged=true | followed_by: create pods/exec same pod within 1m.
• Pro Tip: Treat any CI/CD service account token as a high-value target - leaked CI logs are one of the most common real-world sources of Kubernetes RBAC compromise; restrict such tokens to the absolute minimum verbs/resources and short token lifetimes.

Hypothesis confirmed - container escape via a privileged, host-mounted pod leading to cluster-wide compromise! K8s audit logs plus Falco's in-container visibility together reconstructed a chain that neither source alone would fully capture.