automateyournetwork/webex-incident-workflow

WebEx Incident Workflow

WebEx API Capabilities Used

| API | Purpose | |-----|---------| | Messages -- Create | Post incident updates to spaces | | Messages -- Create (Adaptive Card) | Rich incident declaration and status cards | | Rooms -- Create | Create dedicated incident spaces | | Rooms -- Get Details | Check space metadata | | Memberships -- Create | Add responders to incident spaces | | Memberships -- List | Identify who is in the incident space | | People -- List | Look up engineers by name/email for escalation | | People -- Get Details | Check engineer availability | | Webhooks | Receive Adaptive Card action submissions (IC claim, status updates) |

Incident Lifecycle in WebEx

Phase 1: Detection & Declaration

When a critical alert triggers (from webex-network-alerts skill or human report), post an Adaptive Card to the primary alerts space:

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "Container",
      "style": "attention",
      "items": [
        {
          "type": "TextBlock",
          "text": "INCIDENT DECLARED -- Network Outage",
          "weight": "Bolder",
          "size": "Large",
          "color": "Attention"
        }
      ]
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Severity:", "value": "P1 -- Service Impacting" },
        { "title": "Detected:", "value": "2024-02-21 14:32 UTC" },
        { "title": "Reporter:", "value": "NetClaw (automated)" }
      ]
    },
    {
      "type": "TextBlock",
      "text": "**Symptoms:**\n- R1 unreachable (ping 0%)\n- 47 downstream routes lost\n- 3 OSPF adjacencies down\n- BGP peer to ISP: IDLE",
      "wrap": true
    },
    {
      "type": "TextBlock",
      "text": "**Impact:**\n- Site A has no WAN connectivity\n- Estimated affected users: ~200",
      "wrap": true
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Incident Commander:", "value": "Awaiting claim" },
        { "title": "ServiceNow:", "value": "CR/INC pending" }
      ]
    },
    {
      "type": "ActionSet",
      "actions": [
        {
          "type": "Action.Submit",
          "title": "Claim IC Role",
          "data": { "action": "claim_ic", "incident_id": "INC-001" }
        }
      ]
    }
  ]
}

Optionally create a dedicated incident space for P1 events:

Room Name: INC-001 -- R1 Network Outage -- 2024-02-21

Add the NOC team, on-call engineers, and management as members via the Memberships API.

Phase 2: Triage & Assignment

When an engineer clicks "Claim IC Role" on the Adaptive Card (received via webhook):

**Incident Team Formed**
**IC:** John Smith (claimed at 14:35 UTC)
**NetClaw:** Automated investigation assistant

**Triage Checklist:**
- [x] Alert generated and posted
- [x] Incident declared (P1)
- [x] IC assigned -- John Smith
- [ ] ServiceNow incident created
- [ ] Upstream device checked
- [ ] Blast radius confirmed
- [ ] Customer communication sent

_NetClaw beginning automated investigation..._

Phase 3: Automated Investigation

NetClaw runs diagnostics and posts results as threaded replies:

**Automated Investigation -- Step 1/4**
_Checking upstream device R2 for connectivity to R1..._

Post each step result:

**Investigation Results -- Step 1/4**
**Ping from R2 to R1 (10.1.1.1):** 0% success -- R1 unreachable from upstream

**Investigation Results -- Step 2/4**
**R2 interface Gi1 (toward R1):** up/up, 0 CRC errors, last input 4 min ago
> Physical layer looks OK from R2 side

**Investigation Results -- Step 3/4**
**R2 OSPF neighbors:** R1 missing from neighbor table (was FULL)
> OSPF adjacency lost, DR election may be in progress

**Investigation Results -- Step 4/4**
**R2 logs (last 30 min):**

14:31:47: %OSPF-5-ADJCHG: Nbr 1.1.1.1 on Gi1 from FULL to DOWN 14:31:48: %LINEPROTO-5-UPDOWN: Line protocol on Gi1, changed to down 14:32:01: %LINEPROTO-5-UPDOWN: Line protocol on Gi1, changed to up 14:32:15: %OSPF-5-ADJCHG: Nbr 1.1.1.1 on Gi1 from DOWN to INIT

**Analysis:** R2 saw Gi1 flap at 14:31. Line protocol came back up but OSPF hasn't re-converged. Likely physical issue on R1 side causing interface bounce.

Phase 4: Status Updates

Post periodic status updates as Adaptive Cards:

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "TextBlock",
      "text": "Status Update -- 14:50 UTC (18 min elapsed)",
      "weight": "Bolder",
      "size": "Medium"
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Status:", "value": "Investigating" },
        { "title": "Finding:", "value": "R1 appears to have reloaded unexpectedly. R2 sees the link recover but R1 is not responding to OSPF hellos yet." },
        { "title": "Next Step:", "value": "Waiting for R1 to complete boot sequence. Checking console access." },
        { "title": "ETA:", "value": "Unknown -- dependent on R1 recovery" }
      ]
    },
    {
      "type": "TextBlock",
      "text": "_ServiceNow INC0012345 updated_",
      "isSubtle": true
    }
  ]
}

Phase 5: Resolution

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "Container",
      "style": "good",
      "items": [
        {
          "type": "TextBlock",
          "text": "INCIDENT RESOLVED",
          "weight": "Bolder",
          "size": "Large",
          "color": "Good"
        }
      ]
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Duration:", "value": "34 minutes (14:32 -- 15:06 UTC)" },
        { "title": "Resolution:", "value": "R1 experienced a software crash. Device auto-reloaded and recovered. All OSPF adjacencies re-established. Full routing restored." },
        { "title": "Root Cause:", "value": "Software crash -- Traceback found in logs indicating bug CSCxx12345. TAC case recommended." }
      ]
    },
    {
      "type": "TextBlock",
      "text": "**Post-Resolution Verification:**\n- R1 reachable: 100% ping success\n- OSPF neighbors: 3/3 FULL\n- BGP peer: Established\n- Route count: 47 routes (matches baseline)\n- Connectivity: 100% to all targets",
      "wrap": true
    },
    {
      "type": "TextBlock",
      "text": "_ServiceNow INC0012345 resolved | GAIT session abc123 closed_",
      "isSubtle": true
    }
  ]
}

Phase 6: Post-Incident Review

**Post-Incident Review -- Scheduled**
**Incident:** Network Outage -- R1 crash
**Date:** 2024-02-22 10:00 UTC
**Space:** This thread

**Review Artifacts (attached):**
1. Timeline of events
2. R1 show logging output
3. R1 show version (confirms reload reason)
4. GAIT audit trail (full session)
5. Pre/post health check comparison

**Discussion Topics:**
- Was detection fast enough?
- Was automated investigation helpful?
- What monitoring gaps exist?
- Should R1 be upgraded to patched version?
- Do we need redundant path for this link?

Escalation Matrix

| Severity | Notify | Escalate After | Space | |----------|--------|---------------|-------| | P1 | IC + Manager + NOC | 15 min | Dedicated incident space | | P2 | IC + Team | 30 min | NetClaw Alerts | | P3 | Assigned engineer | 4 hours | NetClaw Alerts | | P4 | Queue only | Next business day | NetClaw General |

Before escalating, check the engineer's WebEx presence/status:

• If engineer appears Away or in DND, escalate to next person in rotation
• Never suppress P1 escalation regardless of status

Adaptive Card Actions for Incident Management

Use Action.Submit buttons on Adaptive Cards for interactive incident management:

| Button | Action Data | Purpose | |--------|------------|---------| | Claim IC Role | {"action":"claim_ic"} | Engineer takes ownership | | Update Status | {"action":"update_status"} | Post a new status update | | Mark Resolved | {"action":"resolve"} | Close the incident | | Schedule PIR | {"action":"schedule_pir"} | Set up post-incident review |

These actions are received via WebEx Webhooks (attachmentActions resource) and processed by NetClaw to update the incident state.

ServiceNow Integration

Create ServiceNow incident at Phase 1:

python3 $MCP_CALL "python3 -u $SERVICENOW_MCP_SCRIPT" create_incident \
  '{"short_description":"P1 - R1 unreachable, WAN outage Site A","description":"R1 is unreachable. 47 routes lost, 3 OSPF adjacencies down. Impact: ~200 users at Site A without WAN connectivity.","urgency":"1","impact":"1","category":"Network"}'

Update ServiceNow as incident progresses and close on resolution.

GAIT Audit Trail

Record every phase in GAIT:

python3 $MCP_CALL "python3 -u $GAIT_MCP_SCRIPT" gait_record_turn \
  '{"input":{"role":"assistant","content":"INCIDENT P1: R1 unreachable. Phase 1 declared, Phase 2 IC assigned @engineer1, Phase 3 automated investigation shows R1 crash, Phase 4 monitoring recovery, Phase 5 resolved after 34 min. INC0012345 closed.","artifacts":[]}}'