cnoe-io/incident-investigation

Incident Investigation

Perform multi-agent investigation by correlating PagerDuty incidents, Jira tickets, and ArgoCD deployment history to identify root cause and impacted systems.

Instructions

Phase 1: Gather Incident Data (PagerDuty Agent)

1. Fetch active incidents - list all triggered and acknowledged incidents
2. For each incident, collect:
   • Incident ID, title, urgency, and status
   • Service affected and escalation policy
   • Triggered timestamp and duration
   • Assigned responders and acknowledgment status
   • Alert details and monitoring source

Phase 2: Correlate with Tickets (Jira Agent)

1. Search for related Jira tickets using:
   • Incident ID or service name in ticket descriptions
   • Recent tickets with labels like incident, outage, p0, p1
   • Tickets linked to the affected service or component
2. For each related ticket, collect:
   • Ticket key, summary, status, assignee
   • Priority and labels
   • Comments with recent updates

Phase 3: Check Recent Deployments (ArgoCD Agent)

1. Search for recent deployments in the last 24 hours:
   • Applications related to the affected service
   • Any applications with recent sync operations
   • Failed syncs or rollbacks
2. For each deployment, collect:
   • Application name and sync status
   • Deployment timestamp
   • Revision/commit that was deployed
   • Sync result (success, failed, pruned resources)

Phase 4: Build Incident Timeline

1. Merge all events into a chronological timeline:
   • Deployments -> Alerts triggered -> Incident created -> Responses
2. Identify correlations:
   • Did a deployment happen shortly before the incident?
   • Are multiple services affected (blast radius)?
   • Is there a pattern (recurring incident)?
3. Assess impact:
   • Which services/teams are impacted?
   • Customer-facing or internal only?
   • Estimated time to resolution

Output Format

## Incident Investigation Report

### Active Incidents
| Incident | Service | Urgency | Duration | Status |
|----------|---------|---------|----------|--------|
| INC-1234 | payment-api | High | 45m | Acknowledged |

### Timeline
| Time (UTC) | Event | Source |
|------------|-------|--------|
| 14:00 | ArgoCD sync: payment-api v2.3.1 deployed | ArgoCD |
| 14:12 | Alert: payment-api error rate >5% | PagerDuty |
| 14:15 | Incident INC-1234 created (High urgency) | PagerDuty |
| 14:18 | Acknowledged by @oncall-engineer | PagerDuty |

### Probable Root Cause
Deployment of payment-api v2.3.1 at 14:00 UTC introduced a regression.
The error rate spike began 12 minutes after deployment.

### Related Jira Tickets
- **PAY-567**: "Payment API 500 errors after v2.3.1" (In Progress)
- **PAY-550**: "Migrate payment-api to new DB schema" (Done - merged 2 days ago)

### Recommended Actions
1. **Immediate**: Rollback payment-api to v2.3.0 via ArgoCD
2. **Short-term**: Review commit diff between v2.3.0 and v2.3.1
3. **Follow-up**: Create post-mortem Jira ticket

Examples

• "Show me active PagerDuty incidents and find related Jira tickets"
• "Investigate the current outage - what changed recently?"
• "Correlate the payment-api incident with recent deployments"
• "Build a timeline for incident INC-1234"

Guidelines

• Always start with PagerDuty for the source of truth on active incidents
• Look at the 24-hour window before the incident for deployment correlation
• If no incidents are active, report that clearly and suggest checking resolved incidents
• When multiple incidents exist, group by service to identify blast radius
• Include direct links to PagerDuty incidents and Jira tickets in the output
• For recurring incidents, note the frequency and link to previous occurrences
• Never suggest changes that could make the situation worse (e.g., deploying during an active incident)