Threat Hunting Exercise: Analyzing Linux Loadable Kernel Module (LKM) Rootkit Dataset

This hunt uses a simulated Linux kernel/auditd dataset (linux_lkm_rootkit_2022-08-18T060000.json) capturing T1014: Rootkit, where an attacker loads a malicious Loadable Kernel Module to hide processes, files, and network connections from userspace tools (ps, ls, netstat), while maintaining a hidden backdoor.

Step 1: Hypothesis Formation

Hypothesis: An unsigned or unlisted kernel module is inserted via insmod/modprobe, after which standard userspace enumeration tools stop reporting a process/connection that is independently visible via a kernel-level or out-of-band data source (e.g., /proc raw inspection, network flow logs) - the classic signature of rootkit-based hiding. Indicators:

• insmod/modprobe/init_module() syscall for a .ko file not present in the distribution's package manifest.
• The module is absent from lsmod shortly after loading (self-hiding) or has an implausible/spoofed module name mimicking a legitimate driver.
• A process/PID visible in raw /proc/[pid] directory enumeration or in NetFlow/firewall connection logs does NOT appear in ps aux or netstat output on the host itself.
• dmesg/kernel ring buffer shows module-loading messages that are subsequently suppressed or the ring buffer is cleared.

Null Hypothesis: A legitimate, unsigned in-house or vendor-specific kernel module (e.g., a custom hardware driver, monitoring agent) was loaded through a documented change. Invalidate by checking the change-management record and confirming the module's provenance/build source.

Rationale: Kernel-level rootkits defeat nearly all userspace-only detection tooling, since the compromised kernel itself controls what ps, ls, and netstat are allowed to see; this hunt validates cross-referencing an out-of-band data source (network flow logs, hypervisor-level introspection) against userspace tool output as the only reliable way to catch a kernel-level hider.

Step 2: Data Sources and Scope

• Sources: auditd syscall monitoring (init_module, finit_module); kernel ring buffer (dmesg) snapshots; NetFlow/firewall connection logs (external to the host); hypervisor/VM introspection snapshot of process list (where available).
• Scope: ~2022-08-18T06:00:00-06:05:00 UTC; Host: lnx-web-node12 (10.230.5.9, KVM guest).
• SIEM Queries (Splunk/ELK):
• index=auditd type=SYSCALL syscall="init_module" OR syscall="finit_module" | table _time host exe module_name
• index=network dest_port=4444 src_ip="10.230.5.9" | stats count by src_port compared against index=host_netstat host="lnx-web-node12" for the same port/time - a mismatch indicates hiding.
• index=hypervisor_introspection host="lnx-web-node12" | table _time pid comm compared against index=host_ps host="lnx-web-node12".

Step 3: Key Findings

Parsed events (5 shown) confirm a kernel module was loaded and immediately hidden from lsmod, after which a backdoor process and its network connection became invisible to local ps/netstat but remained visible via NetFlow and hypervisor introspection.

Validation:

• Timeline: module load, self-removal from lsmod, and an immediately-hidden reverse-shell connection all within 90 seconds - consistent with a purpose-built LKM rootkit deployment, not a legitimate driver.
• False Positives: no change record authorizes any kernel module named nf_hook_helper; the module path (/tmp) and self-hiding behavior are inconsistent with any legitimate driver installation process.
• Correlation: NetFlow (external ground truth) and hypervisor introspection both independently confirm activity invisible to the host's own tools - conclusive, multi-source evidence of kernel-level rootkit concealment.

Step 4: Recommendations & Next Steps

• Immediate Response: Do not trust any local tool output on lnx-web-node12 going forward - take a full memory/disk forensic image via an out-of-band method (hypervisor snapshot); rebuild the host from a known-good image rather than attempting in-place remediation, since kernel-level compromise cannot be reliably cleaned; block 91.203.145.7 at the perimeter.
• Detection: Sigma-style rule (network-layer, not host-dependent): title: NetFlow Connection With No Corresponding Local Process → cross-reference active NetFlow 5-tuples against periodic authenticated netstat collection; alert on any persistent mismatch.
• Pro Tip: For any host where rootkit compromise is suspected, treat the host's own OS-level telemetry as untrustworthy - always corroborate with a source the compromised kernel cannot control, such as hypervisor introspection, network flow data external to the host, or an offline forensic image.

Hypothesis confirmed - an attacker loaded a self-hiding kernel module rootkit that concealed a reverse-shell process and its network connection from local ps/netstat, detected only via external NetFlow and hypervisor-level introspection!