agentsecops/detection-sigma

Sigma Detection Engineering

Overview

Sigma is to log detection what Snort is to network traffic and YARA is to files - a universal signature format for describing security-relevant log events. This skill helps create, validate, and convert Sigma rules for deployment across multiple SIEM platforms, enabling detection-as-code workflows.

Core capabilities:

• Create detection rules using Sigma format
• Convert rules to 25+ SIEM/EDR backends (Splunk, Elastic, QRadar, Sentinel, etc.)
• Validate rule syntax and logic
• Map detections to MITRE ATT&CK framework
• Build threat hunting queries
• Implement compliance-based monitoring

Quick Start

Install Dependencies

pip install pysigma pysigma-backend-splunk pysigma-backend-elasticsearch pyyaml

Create a Basic Sigma Rule

title: Suspicious PowerShell Execution
id: 7d6d30b8-5b91-4b90-a71e-4f5a3f5a3c3f
status: experimental
description: Detects suspicious PowerShell execution with encoded commands
references:
    - https://attack.mitre.org/techniques/T1059/001/
author: Your Name
date: YYYY/MM/DD
modified: YYYY/MM/DD
tags:
    - attack.execution
    - attack.t1059.001
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\powershell.exe'
        CommandLine|contains:
            - '-enc'
            - '-EncodedCommand'
            - 'FromBase64String'
    condition: selection
falsepositives:
    - Legitimate administrative scripts
level: medium

Convert Rule to Target SIEM

# Convert to Splunk
python scripts/sigma_convert.py rule.yml --backend splunk

# Convert to Elasticsearch
python scripts/sigma_convert.py rule.yml --backend elasticsearch

# Convert to Microsoft Sentinel
python scripts/sigma_convert.py rule.yml --backend sentinel

Core Workflows

Workflow 1: Detection Rule Development

Progress: [ ] 1. Identify detection requirement from threat intelligence or compliance [ ] 2. Research log sources and field mappings for target environment [ ] 3. Create Sigma rule using standard template [ ] 4. Validate rule syntax: python scripts/sigma_validate.py rule.yml [ ] 5. Test rule against sample logs or historical data [ ] 6. Convert to target SIEM format [ ] 7. Deploy and tune based on false positive rate [ ] 8. Document rule metadata and MITRE ATT&CK mapping

Work through each step systematically. Check off completed items.

Workflow 2: Threat Hunting Rule Creation

For proactive threat hunting based on TTPs:

1. Select MITRE ATT&CK Technique

   • Review threat intelligence for relevant TTPs
   • Identify technique ID (e.g., T1059.001 - PowerShell)
   • See references/mitre-attack-mapping.md for common techniques

2. Identify Log Sources

   • Determine which logs capture the technique
   • Map log source categories (process_creation, network_connection, file_event)
   • Verify log source availability in your environment

3. Define Detection Logic

   • Create selection criteria matching suspicious patterns
   • Add filters to reduce false positives
   • Use field modifiers for robust matching (endswith, contains, re)

4. Validate and Test

   • Run validation: python scripts/sigma_validate.py hunting-rule.yml
   • Test against known-good and known-bad samples
   • Tune detection logic based on results

5. Document and Deploy

   • Add references to threat reports
   • Document false positive scenarios
   • Convert and deploy to production SIEM

Workflow 3: Bulk Rule Conversion

When migrating between SIEM platforms:

# Validate all rules first
python scripts/sigma_validate.py --directory rules/ --report validation-report.json

# Convert entire rule set
python scripts/sigma_convert.py --directory rules/ --backend splunk --output converted/

# Generate deployment report
python scripts/sigma_convert.py --directory rules/ --backend splunk --report conversion-report.md

Review conversion report for:

• Successfully converted rules
• Rules requiring manual adjustment
• Unsupported field mappings
• Backend-specific limitations

Workflow 4: Compliance-Based Detection

For implementing compliance monitoring (PCI-DSS, NIST, ISO 27001):

1. Map Requirements to Detections

   • Identify compliance control requirements
   • Determine required log monitoring
   • See references/compliance-mappings.md

2. Create Detection Rules

   • Use compliance rule templates from assets/compliance-rules/
   • Tag rules with compliance framework (e.g., tags: [pci-dss.10.2.5])
   • Set appropriate severity levels

3. Validate Coverage

   • Run: python scripts/compliance_coverage.py --framework pci-dss
   • Review coverage gaps
   • Create additional rules as needed

4. Generate Compliance Report

   • Document detection coverage by control
   • Include sample queries and expected alerts
   • Maintain audit trail for compliance evidence

Rule Structure Reference

Required Fields

title: Human-readable rule name
id: UUID (generate with: python -c "import uuid; print(uuid.uuid4())")
status: stable|test|experimental|deprecated
description: Detailed description of what this detects
author: Your Name
date: YYYY/MM/DD
modified: YYYY/MM/DD
logsource:
    category: process_creation|network_connection|file_event|...
    product: windows|linux|macos|azure|aws|...
detection:
    selection:
        FieldName: value
    condition: selection
level: informational|low|medium|high|critical

Optional Fields

references:
    - https://attack.mitre.org/techniques/T1059/
tags:
    - attack.execution
    - attack.t1059.001
falsepositives:
    - Legitimate use cases
fields:
    - CommandLine
    - User
    - ParentImage

Detection Conditions

# Simple selection
detection:
    selection:
        Field: value
    condition: selection

# Multiple conditions (AND)
detection:
    selection:
        Field1: value1
        Field2: value2
    condition: selection

# OR conditions
detection:
    selection1:
        Field: value1
    selection2:
        Field: value2
    condition: selection1 or selection2

# NOT conditions
detection:
    selection:
        Field: suspicious_value
    filter:
        Field: legitimate_value
    condition: selection and not filter

# Complex logic
detection:
    selection:
        EventID: 4688
    suspicious_cmd:
        CommandLine|contains:
            - 'powershell'
            - 'cmd.exe'
    filter_legitimate:
        ParentImage|endswith: '\explorer.exe'
    condition: selection and suspicious_cmd and not filter_legitimate

Field Modifiers

Common modifiers for flexible matching:

• |contains - Contains substring (case-insensitive)
• |endswith - Ends with string
• |startswith - Starts with string
• |re - Regular expression match
• |all - All values must match
• |base64 - Base64-encoded value matching
• |base64offset - Base64 with offset variations

Example:

detection:
    selection:
        CommandLine|contains|all:
            - 'powershell'
            - '-enc'
        Image|endswith: '\powershell.exe'

Security Considerations

• Sensitive Data Handling: Sigma rules may reference sensitive field names or patterns. Store rules in version control with appropriate access controls. Avoid including actual sensitive data in example values.

• Access Control: Detection rules reveal defensive capabilities to adversaries. Implement role-based access for rule repositories. Limit rule modification to authorized detection engineers.

• Audit Logging: Log all rule deployments, modifications, and deletions. Track who deployed which rules to which systems. Maintain change history for compliance auditing.

• Compliance: Sigma rules support compliance monitoring (PCI-DSS 10.2, NIST SP 800-53 AU family, ISO 27001 A.12.4). Document rule-to-control mappings for audit evidence.

• Safe Defaults: Use conservative false positive filtering in production. Start rules at "experimental" status. Test thoroughly in test environment before production deployment.

Bundled Resources

Scripts

• scripts/sigma_convert.py - Convert Sigma rules to target SIEM backend formats
• scripts/sigma_validate.py - Validate Sigma rule syntax and detect common errors
• scripts/compliance_coverage.py - Analyze detection coverage for compliance frameworks
• scripts/generate_rule_template.py - Generate Sigma rule template with MITRE ATT&CK tags

References

• references/mitre-attack-mapping.md - Common MITRE ATT&CK techniques and Sigma detection patterns
• references/log-source-guide.md - Log source categories, products, and field mappings
• references/compliance-mappings.md - Compliance framework to detection rule mappings
• references/backend-support.md - Supported SIEM backends and conversion capabilities
• references/field-modifiers.md - Comprehensive guide to Sigma field modifiers and regex patterns

Assets

• assets/rule-templates/ - Pre-built Sigma rule templates for common attack patterns

  • lateral-movement.yml - Lateral movement detection template
  • privilege-escalation.yml - Privilege escalation detection template
  • persistence.yml - Persistence mechanism detection template
  • credential-access.yml - Credential dumping detection template

• assets/compliance-rules/ - Compliance-focused rule templates

  • pci-dss-monitoring.yml - PCI-DSS monitoring requirements
  • nist-800-53-audit.yml - NIST 800-53 audit logging requirements
  • iso27001-logging.yml - ISO 27001 logging and monitoring

Common Detection Patterns

Pattern 1: Process Execution Monitoring

Detect suspicious process creation with command-line analysis:

logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith:
            - '\powershell.exe'
            - '\cmd.exe'
        CommandLine|contains:
            - 'Invoke-'
            - 'IEX'
            - 'FromBase64String'

Pattern 2: Network Connection Monitoring

Detect suspicious outbound connections:

logsource:
    category: network_connection
    product: windows
detection:
    selection:
        Initiated: 'true'
        DestinationPort:
            - 4444
            - 5555
            - 8080
    filter:
        DestinationIp|startswith:
            - '10.'
            - '172.16.'
            - '192.168.'
    condition: selection and not filter

Pattern 3: File Event Monitoring

Detect file creation in suspicious locations:

logsource:
    category: file_event
    product: windows
detection:
    selection:
        TargetFilename|contains:
            - '\Windows\Temp\'
            - '\AppData\Roaming\'
        TargetFilename|endswith:
            - '.exe'
            - '.dll'
            - '.ps1'

Integration Points

CI/CD Integration

Build detection-as-code pipelines:

# .github/workflows/sigma-validation.yml
name: Sigma Rule Validation
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Validate Sigma Rules
        run: |
          pip install pysigma
          python scripts/sigma_validate.py --directory rules/
      - name: Convert to Production Format
        run: |
          python scripts/sigma_convert.py --directory rules/ --backend splunk --output converted/

SIEM Deployment

Automated rule deployment:

• Splunk: Use Splunk REST API or splunk-sdk for savedsearches
• Elasticsearch: Convert to EQL and deploy via Kibana API
• Microsoft Sentinel: Convert to KQL and deploy via Azure API
• QRadar: Convert to AQL and deploy via QRadar API

See references/backend-support.md for deployment examples.

Threat Intelligence Integration

Enrich rules with threat intel:

• Tag rules with threat actor TTPs
• Reference threat reports and IOCs
• Map to MITRE ATT&CK techniques
• Track rule effectiveness against known threats

Troubleshooting

Issue: Conversion Fails for Specific Backend

Solution: Check backend compatibility and field mappings. Some backends have limitations:

• Review references/backend-support.md for known limitations
• Use sigma_convert.py --backend <backend> --debug for detailed error output
• Check if field names are supported in target backend
• Consider custom pipeline transformations for unsupported fields

Issue: High False Positive Rate

Solution: Refine detection logic with additional filters:

1. Review false positive patterns
2. Add exclusion filters for legitimate use cases
3. Use more specific field modifiers (e.g., |endswith vs |contains)
4. Consider time-based correlation for behavioral detection
5. Test with historical data to validate tuning

Issue: Rule Not Triggering in Target SIEM

Solution: Verify log source availability and field mappings:

1. Confirm log source is ingested: Check SIEM data sources
2. Verify field names match: Use sigma_convert.py --show-fields to see mapping
3. Test converted query directly in SIEM
4. Check for case sensitivity issues in field values
5. Validate time window and search scope in SIEM

MITRE ATT&CK Integration

Tag rules with ATT&CK tactics and techniques:

tags:
    - attack.execution           # Tactic
    - attack.t1059.001          # Technique: PowerShell
    - attack.defense_evasion    # Additional tactic
    - attack.t1027              # Technique: Obfuscated Files

Common tactic tags:

• attack.initial_access
• attack.execution
• attack.persistence
• attack.privilege_escalation
• attack.defense_evasion
• attack.credential_access
• attack.discovery
• attack.lateral_movement
• attack.collection
• attack.exfiltration
• attack.command_and_control
• attack.impact

For detailed technique mappings, see references/mitre-attack-mapping.md.

Best Practices

1. Start with Community Rules: Use SigmaHQ repository (3000+ peer-reviewed rules) as foundation
2. Version Control: Store rules in Git with meaningful commit messages
3. Test Before Deploy: Validate against historical data in test environment
4. Document Tuning: Track false positive patterns and tuning decisions
5. Map to Frameworks: Tag all rules with MITRE ATT&CK and compliance mappings
6. Automate Validation: Use CI/CD to validate rules on every change
7. Monitor Effectiveness: Track rule trigger rates and true positive rates
8. Regular Updates: Review and update rules based on new threat intelligence

References

• Sigma Specification
• SigmaHQ Rule Repository
• pySigma Documentation
• Sigma Converter Web Tool
• MITRE ATT&CK Framework