CX330Blake/analyzing-linux-audit-logs-for-intrusion

Analyzing Linux Audit Logs for Intrusion

When to Use

• Investigating suspected unauthorized access or privilege escalation on Linux hosts
• Hunting for evidence of exploitation, backdoor installation, or persistence mechanisms
• Auditing compliance with security baselines (CIS, STIG, PCI-DSS) that require system call monitoring
• Reconstructing a timeline of attacker actions during incident response
• Detecting file tampering on critical system files such as /etc/passwd, /etc/shadow, or SSH keys

Do not use for network-level intrusion detection; use Suricata or Zeek for network traffic analysis. Auditd operates at the kernel level on individual hosts.

Prerequisites

• Linux system with auditd package installed and the audit daemon running (systemctl status auditd)
• Root or sudo access to configure audit rules and query logs
• Audit rules deployed via /etc/audit/rules.d/*.rules or loaded with auditctl
• Recommended: Neo23x0/auditd ruleset from GitHub for comprehensive baseline coverage
• Familiarity with Linux syscalls (execve, open, connect, ptrace, etc.)
• Log storage with sufficient retention (default location: /var/log/audit/audit.log)

Workflow

Step 1: Verify Audit Daemon Status and Configuration

Confirm the audit system is running and check the current rule set:

# Check auditd service status
systemctl status auditd

# Show current audit rules loaded in the kernel
auditctl -l

# Show audit daemon configuration
cat /etc/audit/auditd.conf | grep -E "log_file|max_log_file|num_logs|space_left_action"

# Check if the audit backlog is being exceeded (dropped events)
auditctl -s

If the backlog limit is being reached, increase it:

auditctl -b 8192

Step 2: Deploy Intrusion-Focused Audit Rules

Add rules that target common intrusion indicators. Place these in /etc/audit/rules.d/intrusion.rules:

# Monitor credential files for unauthorized reads or modifications
-w /etc/passwd -p wa -k credential_access
-w /etc/shadow -p rwa -k credential_access
-w /etc/gshadow -p rwa -k credential_access
-w /etc/sudoers -p wa -k privilege_escalation
-w /etc/sudoers.d/ -p wa -k privilege_escalation

# Monitor SSH configuration and authorized keys
-w /etc/ssh/sshd_config -p wa -k sshd_config_change
-w /root/.ssh/authorized_keys -p wa -k ssh_key_tampering

# Monitor user and group management commands
-w /usr/sbin/useradd -p x -k user_management
-w /usr/sbin/usermod -p x -k user_management
-w /usr/sbin/groupadd -p x -k user_management

# Detect process injection via ptrace
-a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k process_injection

# Monitor execution of programs from unusual directories
-a always,exit -F arch=b64 -S execve -F exe=/tmp -k exec_from_tmp
-a always,exit -F arch=b64 -S execve -F exe=/dev/shm -k exec_from_shm

# Detect kernel module loading (rootkit installation)
-a always,exit -F arch=b64 -S init_module -S finit_module -k kernel_module_load
-a always,exit -F arch=b64 -S delete_module -k kernel_module_remove
-w /sbin/insmod -p x -k kernel_module_tool
-w /sbin/modprobe -p x -k kernel_module_tool

# Monitor network socket creation for reverse shells
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_created
-a always,exit -F arch=b64 -S connect -F a0=2 -k network_connection

# Detect cron job modifications (persistence)
-w /etc/crontab -p wa -k cron_persistence
-w /etc/cron.d/ -p wa -k cron_persistence
-w /var/spool/cron/ -p wa -k cron_persistence

# Monitor log deletion or tampering
-w /var/log/ -p wa -k log_tampering

Reload rules after editing:

augenrules --load
auditctl -l | wc -l   # Confirm rule count

Step 3: Search for Intrusion Indicators with ausearch

Use ausearch to query the audit log for specific events:

# Search for all failed login attempts in the last 24 hours
ausearch -m USER_LOGIN --success no -ts recent

# Search for commands executed by a specific user
ausearch -ua 1001 -m EXECVE -ts today

# Search for all file access events on /etc/shadow
ausearch -f /etc/shadow -ts this-week

# Search for privilege escalation via sudo
ausearch -m USER_CMD -ts today

# Search for kernel module loading events
ausearch -k kernel_module_load -ts this-month

# Search for processes executed from /tmp (common attack staging)
ausearch -k exec_from_tmp -ts this-week

# Search for SSH key modifications
ausearch -k ssh_key_tampering -ts this-month

# Search events in a specific time range
ausearch -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00

# Interpret syscall numbers and format output readably
ausearch -k credential_access -i -ts today

Step 4: Generate Summary Reports with aureport

Use aureport to produce aggregate summaries for triage:

# Summary of all authentication events
aureport -au -ts this-week --summary

# Report of all failed events
aureport --failed --summary -ts today

# Report of executable runs
aureport -x --summary -ts today

# Report of all anomaly events
aureport --anomaly -ts this-week

# Report of file access events
aureport -f --summary -ts today

# Report of all events by key (maps to custom rule keys)
aureport -k --summary -ts this-month

Step 5: Reconstruct the Attack Timeline

Combine ausearch queries to build a chronological narrative:

# Identify the initial access timestamp
ausearch -m USER_LOGIN -ua 0 --success yes -ts this-week -i | head -50

# Trace what the attacker did after gaining access
ausearch -ua <UID> -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i | aureport -f -i

# Extract all commands executed during the incident window
ausearch -m EXECVE -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i

# Check for persistence mechanisms installed
ausearch -k cron_persistence -ts "03/15/2026 14:00:00" -i
ausearch -k ssh_key_tampering -ts "03/15/2026 14:00:00" -i

Step 6: Forward Audit Logs to SIEM

Configure audisp-remote or auditbeat to ship logs to a central SIEM:

# Using audisp-remote plugin (/etc/audit/plugins.d/au-remote.conf)
active = yes
direction = out
path = /sbin/audisp-remote
type = always

# Remote target (/etc/audit/audisp-remote.conf)
remote_server = siem.internal.corp
port = 6514
transport = tcp

Key Concepts

| Term | Definition | |------|------------| | auditd | Linux Audit daemon that receives kernel audit events and writes to /var/log/audit/audit.log | | auditctl | CLI to control the audit system: add/remove rules, check status, set backlog size | | ausearch | Query tool for searching audit logs by message type, user, file, key, or time range | | aureport | Reporting tool that generates aggregate summaries for triage and compliance | | audit rule key (-k) | User-defined label on audit rules for fast filtering with ausearch and aureport | | augenrules | Merges /etc/audit/rules.d/ files into audit.rules and loads into the kernel |

Verification

• [ ] auditd is running and rules are loaded (auditctl -l returns expected rule count)
• [ ] No audit backlog overflow (auditctl -s shows lost: 0)
• [ ] ausearch returns events for each custom key
• [ ] aureport generates non-empty summaries for authentication, executable, and file events
• [ ] Timeline reconstruction produces a coherent chronological sequence
• [ ] Critical file watches trigger alerts on test modifications
• [ ] Logs are forwarding to central SIEM
• [ ] Audit rules persist across reboot (rules in /etc/audit/rules.d/)