edercnj/x-handle-incident

Global Output Policy

• Language: English ONLY.
• Tone: Technical, Direct, and Concise.
• Efficiency: Remove all conversational fillers and greetings to save tokens.

Skill: Incident Response

Purpose

Provides an interactive incident response guide for {{PROJECT_NAME}} that walks the team through the complete process from detection to resolution. Classifies severity, loads severity-specific checklists, coordinates communication, triggers postmortems, and tracks action items.

Triggers

• /x-handle-incident — start interactive severity classification
• /x-handle-incident SEV1 — start SEV1 critical incident response
• /x-handle-incident SEV2 --postmortem — SEV2 incident with postmortem generation
• /x-handle-incident SEV3 --notify — SEV3 incident with communication templates
• /x-handle-incident SEV1 --postmortem --notify — full incident response workflow

Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | severity | positional | (interactive) | Severity level: SEV1, SEV2, SEV3, or SEV4 | | --postmortem | boolean | false | Generate postmortem document (auto-enabled for SEV1/SEV2) | | --notify | boolean | false | Generate communication templates for all channels |

Workflow

1. CLASSIFY     -> Analyze impact description and classify severity (SEV1-SEV4)
2. LOAD         -> Load severity-specific checklist from SRE Practices KP
3. GUIDE        -> Conduct team through Detection -> Triage -> Mitigation -> Resolution
4. COMMUNICATE  -> Generate communication templates (status page, Slack, email)
5. POSTMORTEM   -> Generate postmortem document from template (SEV1/SEV2 or --postmortem)
6. TRACK        -> Register action items with owners and deadlines

Step 1 — Classify Severity

If severity is provided as argument, validate and use directly. If omitted, ask the user about the impact and suggest classification.

| Severity | Label | Criteria | Response Time | Update Frequency | |----------|-------|----------|---------------|------------------| | SEV1 | Critical | Total service outage, significant financial loss, data breach affecting all users | 15 min | Every 30 min | | SEV2 | High | Major feature unavailable, severe performance degradation, large user impact | 30 min | Every 1 hour | | SEV3 | Medium | Minor feature impacted, workaround available, limited user impact | 4 hours | Every 4 hours | | SEV4 | Low | Cosmetic issue, minimal user impact, no business impact | Next business day | Daily |

Severity Decision Criteria:

• User Impact: How many users are affected? Is it all users or a subset?
• Financial Impact: Is there direct revenue loss or potential financial liability?
• Data Impact: Is there data loss, corruption, or unauthorized access?
• Scope: Is the issue isolated or widespread across regions/services?

Step 2 — Load Checklist

Load the severity-specific checklist from the SRE Practices knowledge pack (knowledge/sre-practices/). Each severity level has different response requirements:

SEV1 — Critical Checklist:

• [ ] Incident Commander assigned within 5 minutes
• [ ] Response team assembled (engineering, on-call, management)
• [ ] Communication channel created (war room / bridge call)
• [ ] Initial status page update published
• [ ] Executive stakeholders notified within 15 minutes
• [ ] All hands on deck — pull in additional engineers as needed

SEV2 — High Checklist:

• [ ] On-call engineer acknowledged within 10 minutes
• [ ] Secondary engineer engaged if needed
• [ ] Communication channel created
• [ ] Status page update published within 30 minutes
• [ ] Engineering manager notified

SEV3 — Medium Checklist:

• [ ] On-call engineer acknowledged within 1 hour
• [ ] Workaround documented and communicated
• [ ] Ticket created for permanent fix
• [ ] Status page updated if user-facing

SEV4 — Low Checklist:

• [ ] Ticket created with appropriate priority
• [ ] Assigned to next sprint or backlog
• [ ] No immediate response required

Step 3 — Guide Response

Conduct the team through the incident response flow. Use the sre-engineer agent via Agent tool for reliability expertise and validation.

Detection

• Identify the alert source (monitoring, user report, automated test)
• Record alert timestamp and affected service(s)
• Confirm initial severity assessment

Triage

• Determine root cause hypothesis
• Identify blast radius (affected services, users, regions)
• Assign roles: Incident Commander (SEV1/SEV2), Communication Lead, Technical Lead

Mitigation

• Apply immediate mitigation (rollback, feature flag, scaling, failover)
• Verify mitigation effectiveness
• Document mitigation steps taken

Resolution

• Implement permanent fix
• Verify resolution across all affected components
• Confirm service restoration to normal operation
• Update status page with resolution notice

Step 4 — Communicate

Generate communication templates based on severity and channels. Frequency varies by severity level.

Status Page Template

**[INVESTIGATING/IDENTIFIED/MONITORING/RESOLVED]** — {{PROJECT_NAME}}

**Impact:** [Description of user-facing impact]
**Affected Services:** [List of affected services]
**Current Status:** [What we know and what we are doing]
**Next Update:** [Time of next update based on severity]

Slack/Teams Internal Update Template

:rotating_light: **Incident Update — [SEV level]**
**Service:** [affected service]
**Impact:** [description]
**Status:** [Investigating/Mitigating/Resolved]
**IC:** [Incident Commander name]
**Timeline:**
- [HH:MM] [Event description]
**Next Steps:** [what happens next]
**Next Update:** [time]

Email Stakeholder Notification Template

Subject: [SEV level] Incident — [Brief description]

Dear Stakeholders,

We are currently experiencing [description of the issue]
affecting [scope of impact].

**Severity:** [SEV1/SEV2/SEV3/SEV4]
**Impact:** [User-facing impact description]
**Status:** [Current status]
**ETA for Resolution:** [Estimate or "Under investigation"]

We will provide updates every [frequency based on severity].

Best regards,
[Team Name]

Update Frequency by Severity:

| Severity | Update Frequency | Channels | |----------|-----------------|----------| | SEV1 | Every 30 min | Status Page, Slack, Email | | SEV2 | Every 1 hour | Status Page, Slack, Email | | SEV3 | Every 4 hours | Status Page, Slack | | SEV4 | Daily | Slack |

Step 5 — Postmortem

If --postmortem flag is provided, or severity is SEV1 or SEV2, generate a postmortem document from the _TEMPLATE-POSTMORTEM.md template.

Postmortem is triggered when:

• Severity is SEV1 (always)
• Severity is SEV2 (always)
• --postmortem flag is explicitly provided (any severity)

Postmortem document is pre-filled with:

• Incident timeline from the response flow
• Severity classification and impact assessment
• Mitigation and resolution steps taken
• Participants and their roles
• Action items identified during the response

If the postmortem template is not available, generate an inline postmortem with the basic structure:

# Postmortem — [Incident Title]

## Incident Summary
| Field | Value |
|-------|-------|
| Severity | [SEV level] |
| Date | [Date] |
| Duration | [Duration] |
| Affected Services | [Services] |

## Timeline
[Pre-filled from incident response]

## Root Cause Analysis
[To be completed]

## Action Items
[Pre-filled from tracked items]

## Lessons Learned
[To be completed]

Step 6 — Track Action Items

Register all action items identified during the incident response. Each action item includes:

| Field | Description | |-------|-------------| | ID | Sequential identifier (AI-001, AI-002, ...) | | Description | What needs to be done | | Owner | Person responsible | | Deadline | Target completion date | | Priority | HIGH / MEDIUM / LOW | | Status | OPEN / IN_PROGRESS / DONE |

Output format:

## Action Items

| ID | Description | Owner | Deadline | Priority | Status |
|----|-------------|-------|----------|----------|--------|
| AI-001 | [description] | [owner] | [date] | HIGH | OPEN |
| AI-002 | [description] | [owner] | [date] | MEDIUM | OPEN |

Severity Definitions

| Severity | Label | Description | Response Time | Update Frequency | Postmortem Required | |----------|-------|-------------|---------------|------------------|---------------------| | SEV1 | Critical | Total service outage, significant financial loss, data breach | 15 min | 30 min | Yes (always) | | SEV2 | High | Major feature unavailable, degraded experience for many users | 30 min | 1 hour | Yes (always) | | SEV3 | Medium | Minor feature impacted, workaround available | 4 hours | 4 hours | Only if --postmortem | | SEV4 | Low | Cosmetic issue, minimal impact | Next business day | Daily | Only if --postmortem |

Communication Templates

See Step 4 above for complete templates per channel (Status Page, Slack/Teams, Email) and update frequency per severity level.

Error Handling

| Scenario | Action | |----------|--------| | Severity not provided | Ask the user about the impact and suggest classification based on description | | Invalid severity (e.g., SEV5) | Reject with message: "Invalid severity. Use SEV1, SEV2, SEV3, or SEV4" | | --postmortem without template | Generate inline postmortem with basic structure | | No Incident Commander available | Assign the requesting user as IC and recommend finding a replacement | | Incomplete information | Proceed with available data, note gaps in postmortem |

Integration Notes

| Skill | Relationship | Context | |-------|-------------|---------| | x-troubleshoot-operations | called-by | Escalates to this skill when an issue becomes a production incident | | sre-engineer (agent) | calls | Delegates reliability expertise and checklist validation via Agent tool | | sre-practices (KP) | reads | References knowledge/sre-practices/ for incident management processes |

• Uses _TEMPLATE-POSTMORTEM.md for postmortem document generation. Fallback: inline postmortem with basic structure when template is absent.
• Uses _TEMPLATE-INCIDENT-RESPONSE.md for severity classification reference
• Can be used standalone or as part of on-call response workflow