seikaikyo/triaging-security-alerts-in-splunk

Triaging Security Alerts in Splunk

When to Use

Use this skill when:

• SOC Tier 1 analysts need to process the Incident Review queue in Splunk Enterprise Security (ES)
• Notable events require rapid severity classification and initial investigation before escalation
• Alert volume exceeds capacity and analysts need a systematic triage methodology
• Management requests metrics on alert disposition (true positive, false positive, benign)

Do not use for deep forensic investigation — escalate to Tier 2/3 after initial triage confirms malicious activity.

Prerequisites

• Splunk Enterprise Security 7.x+ with Incident Review dashboard configured
• CIM-normalized data sources (Windows Event Logs, firewall, proxy, endpoint)
• Role with ess_analyst capability for notable event status updates
• Familiarity with SPL (Search Processing Language)

Workflow

Step 1: Access Incident Review and Prioritize Queue

Open the Incident Review dashboard in Splunk ES. Sort notable events by urgency (calculated from severity x priority). Apply filters to focus on unassigned events:

| `notable`
| search status="new" OR status="unassigned"
| sort - urgency
| table _time, rule_name, src, dest, user, urgency, status
| head 50

Focus on Critical and High urgency events first. Group related alerts by src or dest to identify attack chains rather than treating each alert independently.

Step 2: Investigate the Notable Event Context

For each notable event, pivot to raw events. Example for a brute force alert:

index=wineventlog sourcetype="WinEventLog:Security" EventCode=4625
src_ip="192.168.1.105"
earliest=-1h latest=now
| stats count by src_ip, dest, user, status
| where count > 10
| sort - count

Check if the source IP is internal (lateral movement) or external (perimeter attack). Cross-reference with asset and identity lookups:

| `notable`
| search rule_name="Brute Force Access Behavior Detected"
| lookup asset_lookup_by_cidr ip AS src OUTPUT category, owner, priority
| lookup identity_lookup_expanded identity AS user OUTPUT department, managedBy
| table _time, src, dest, user, category, owner, department

Step 3: Correlate Across Data Sources

Check if the same source appears in other telemetry:

index=proxy OR index=firewall src="192.168.1.105" earliest=-24h
| stats count by index, sourcetype, action, dest_port
| sort - count

Look for corroborating evidence: Did the same IP also trigger DNS anomalies, proxy blocks, or endpoint detection alerts?

index=main sourcetype="cisco:asa" src="192.168.1.105" action=blocked earliest=-24h
| timechart span=1h count by dest_port

Step 4: Check Threat Intelligence Enrichment

Query the threat intelligence framework for known IOCs:

| `notable`
| search search_name="Threat - Threat Intelligence Match - Rule"
| lookup threat_intel_by_ip ip AS src OUTPUT threat_collection, threat_description, threat_key
| table _time, src, dest, threat_collection, threat_description, weight
| where weight >= 3

For domains, check against threat lists:

| tstats count from datamodel=Web where Web.url="*evil-domain.com*" by Web.src, Web.url, Web.status
| rename Web.* AS *

Step 5: Classify and Disposition the Alert

Update the notable event status in Incident Review:

| Disposition | Criteria | Action | |-------------|----------|--------| | True Positive | Corroborating evidence confirms malicious activity | Escalate to Tier 2, create incident ticket | | Benign True Positive | Alert fired correctly but activity is authorized (e.g., pen test) | Close with comment, add suppression if recurring | | False Positive | Alert logic matched benign behavior | Close, tune correlation search, document pattern | | Undetermined | Insufficient data to classify | Assign to Tier 2 with investigation notes |

Update via Splunk ES UI or REST API:

| sendalert update_notable_event param.status="2" param.urgency="critical"
  param.comment="Confirmed brute force from compromised workstation. Escalated to IR-2024-0431."
  param.owner="analyst_jdoe"

Step 6: Document Triage Findings

Record in the notable event comment field:

• Source/destination involved
• Data sources examined
• Correlation findings (related alerts, TI matches)
• Disposition rationale
• Next steps for escalation

| `notable`
| search rule_name="Brute Force*" status="closed"
| stats count by status_label, disposition
| addtotal

Step 7: Track Triage Metrics

Monitor triage performance over time:

| `notable`
| where status_end > 0
| eval triage_time = status_end - _time
| stats avg(triage_time) AS avg_triage_sec, median(triage_time) AS med_triage_sec,
        count by rule_name, status_label
| eval avg_triage_min = round(avg_triage_sec/60, 1)
| sort - count
| table rule_name, status_label, count, avg_triage_min

Key Concepts

| Term | Definition | |------|-----------| | Notable Event | Splunk ES alert generated by a correlation search that meets defined risk or threshold criteria | | Urgency | Calculated field combining event severity with asset/identity priority (Critical/High/Medium/Low/Informational) | | Correlation Search | Scheduled SPL query that detects threat patterns and generates notable events when conditions match | | CIM | Common Information Model — Splunk's normalized field naming convention enabling cross-source queries | | Disposition | Final classification of an alert: true positive, false positive, benign true positive, or undetermined | | MTTD/MTTR | Mean Time to Detect / Mean Time to Respond — key SOC metrics measuring detection and resolution speed |

Tools & Systems

• Splunk Enterprise Security: SIEM platform providing Incident Review dashboard, correlation searches, and risk-based alerting
• Splunk SOAR (Phantom): Orchestration platform for automating triage playbooks and enrichment actions
• Asset & Identity Framework: Splunk ES lookup tables mapping IPs to asset owners and users to departments for context enrichment
• Threat Intelligence Framework: Splunk ES module ingesting STIX/TAXII feeds and matching IOCs against notable events

Common Scenarios

• Brute Force Alerts: Correlate EventCode 4625 (failed logon) with 4624 (successful logon) from same source to determine if attack succeeded
• Malware Detection: Cross-reference endpoint AV alert with proxy logs for C2 callback confirmation
• Data Exfiltration Alert: Check outbound data volume from DLP and proxy logs against user baseline
• Privilege Escalation: Correlate EventCode 4672 (special privileges assigned) with 4720 (account created) from non-admin users
• Lateral Movement: Map EventCode 4648 (explicit credential logon) across multiple destinations from single source

Output Format

TRIAGE REPORT — Notable Event #NE-2024-08921
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alert:        Brute Force Access Behavior Detected
Time:         2024-03-15 14:23:07 UTC
Source:       192.168.1.105 (WORKSTATION-042, Finance Dept)
Destination:  10.0.5.20 (DC-PRIMARY, Domain Controller)
User:         jsmith (Finance Analyst)

Investigation:
  - 847 failed logons (4625) in 12 minutes from src
  - Successful logon (4624) at 14:35:02 after brute force
  - No proxy/DNS anomalies from src in prior 24h
  - Source not on threat intel lists

Disposition:  TRUE POSITIVE — Compromised credential
Action:       Escalated to Tier 2, ticket IR-2024-0431 created
              Account jsmith disabled pending password reset