troykelly/autonomous-orchestration

Autonomous Orchestration

Overview

Orchestrates long-running autonomous work across multiple issues, spawning parallel workers, monitoring CI, and persisting state across sessions.

Core principle: GitHub is the source of truth. Workers are disposable. State survives restarts.

Announce at start: "I'm using autonomous-orchestration to work through [SCOPE]. Starting autonomous operation now."

Prerequisites

• worker-dispatch skill for spawning workers
• worker-protocol skill for worker behavior
• ci-monitoring skill for CI/WAKE handling
• Git worktrees support (workers use isolated worktrees)
• GitHub CLI (gh) authenticated
• GitHub Project Board configured

State Management

CRITICAL: All state is stored in GitHub. NO local state files.

| State Store | Purpose | Used For | |-------------|---------|----------| | Project Board Status | THE source of truth | Ready, In Progress, In Review, Blocked, Done | | Issue Comments | Activity log | Worker assignment, progress, deviations | | Labels | Lineage only | spawned-from:#N, depth:N, epic-* | | MCP Memory | Fast cache + active marker | Read optimization, active orchestration detection |

See: reference/state-management.md for detailed state queries and updates.

Context Compaction Survival

CRITICAL: Orchestration must survive mid-loop context compaction.

On Start: Write Active Marker

# Write to MCP Memory when orchestration starts
mcp__memory__create_entities([{
  "name": "ActiveOrchestration",
  "entityType": "Orchestration",
  "observations": [
    "Status: ACTIVE",
    "Scope: [MILESTONE/EPIC/unbounded]",
    "Tracking Issue: #[NUMBER]",
    "Started: [ISO_TIMESTAMP]",
    "Repository: [owner/repo]",
    "Phase: BOOTSTRAP|MAIN_LOOP",
    "Last Loop: [ISO_TIMESTAMP]"
  ]
}])

On Each Loop Iteration: Update Marker

mcp__memory__add_observations({
  "observations": [{
    "entityName": "ActiveOrchestration",
    "contents": ["Last Loop: [ISO_TIMESTAMP]", "Phase: MAIN_LOOP"]
  }]
})

On Complete: Remove Marker

mcp__memory__delete_entities({
  "entityNames": ["ActiveOrchestration"]
})

On Session Resume (After Compaction)

Session-start skill checks for active orchestration:

# Check MCP Memory for active orchestration
ACTIVE=$(mcp__memory__open_nodes({"names": ["ActiveOrchestration"]}))

if [ -n "$ACTIVE" ]; then
  echo "⚠️ ACTIVE ORCHESTRATION DETECTED"
  echo "Scope: [from ACTIVE]"
  echo "Tracking: [from ACTIVE]"
  echo ""
  echo "Resuming orchestration loop..."
  # Invoke autonomous-orchestration skill to resume
fi

This ensures: Even if context compacts mid-loop, the next session will detect the active orchestration and resume it.

Immediate Start (User Consent Implied)

The user's request for autonomous operation IS their consent. No additional confirmation required.

When the user requests autonomous work:

1. Identify scope - Parse user request for milestone, epic, specific issues, or "all"
2. Announce intent - Briefly state what you're about to do
3. Start immediately - Begin orchestration without waiting for additional input

## Starting Autonomous Operation

**Scope:** [MILESTONE/EPIC/ISSUES or "all open issues"]
**Workers:** Up to 5 parallel
**Mode:** Continuous until complete

Beginning work now...

Do NOT ask for "PROCEED" or any confirmation. The user asked for autonomous operation - that is the confirmation.

Automatic Scope Detection

When the user requests autonomous operation without specifying a scope:

Priority Order

1. User-specified scope - If user mentions specific issues, epics, or milestones
2. Urgent/High Priority standalone issues - Issues with priority:urgent or priority:high labels not part of an epic
3. Epic-based sequential work - Work through epics in order, completing all issues within each epic
4. Remaining standalone issues - Any issues not part of an epic

detect_work_scope() {
  # 1. Check for urgent/high priority standalone issues first
  PRIORITY_ISSUES=$(gh issue list --state open \
    --label "priority:urgent,priority:high" \
    --json number,labels \
    --jq '[.[] | select(.labels | map(.name) | any(startswith("epic-")) | not)] | .[].number')

  if [ -n "$PRIORITY_ISSUES" ]; then
    echo "priority_standalone"
    echo "$PRIORITY_ISSUES"
    return
  fi

  # 2. Get epics in order (by creation date)
  EPICS=$(gh issue list --state open --label "type:epic" \
    --json number,title,createdAt \
    --jq 'sort_by(.createdAt) | .[].number')

  if [ -n "$EPICS" ]; then
    echo "epics"
    echo "$EPICS"
    return
  fi

  # 3. Fall back to all open issues
  ALL_ISSUES=$(gh issue list --state open --json number --jq '.[].number')
  echo "all_issues"
  echo "$ALL_ISSUES"
}

Continuous Operation Until Complete

Autonomous operation continues until ALL of:

• No open issues remain in scope
• No open PRs awaiting merge
• No issues in "In Progress" or "In Review" status

The operation does NOT pause for:

• Progress updates
• Confirmation between issues
• Switching between epics
• Any user input (unless blocked by a fatal error)

PR Resolution Bootstrap Phase

CRITICAL: Before spawning ANY new workers, resolve all existing open PRs first.

┌──────────────────────────────────────────────────────────┐
│                    BOOTSTRAP PHASE                        │
│             (Runs ONCE before main loop)                  │
└─────────────────────────┬────────────────────────────────┘
                          │
                          ▼
               ┌───────────────────┐
               │ GET OPEN PRs      │
               │                   │
               │ Filter out:       │
               │ - release/*       │
               │ - release-        │
               │   placeholder     │
               └─────────┬─────────┘
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
        ┌───────────┐         ┌───────────┐
        │ Has PRs?  │─── No ──│ → MAIN    │
        │           │         │   LOOP    │
        └─────┬─────┘         └───────────┘
              │ Yes
              ▼
        ┌───────────────────────────────┐
        │ FOR EACH PR:                  │
        │                               │
        │ 1. Check CI status            │
        │ 2. Verify review artifact     │
        │ 3. Merge if ready OR          │
        │ 4. Wait/fix if not            │
        └───────────────────────────────┘
                          │
                          ▼
                    MAIN LOOP

Bootstrap Implementation

resolve_existing_prs() {
  echo "=== PR RESOLUTION BOOTSTRAP ==="

  # Get all open PRs, excluding release placeholders
  OPEN_PRS=$(gh pr list --json number,headRefName,labels \
    --jq '[.[] | select(
      (.headRefName | startswith("release/") | not) and
      (.labels | map(.name) | index("release-placeholder") | not)
    )] | .[].number')

  if [ -z "$OPEN_PRS" ]; then
    echo "No actionable PRs to resolve. Proceeding to main loop."
    return 0
  fi

  echo "Found PRs to resolve: $OPEN_PRS"

  for pr in $OPEN_PRS; do
    echo "Processing PR #$pr..."

    # Get CI status
    ci_status=$(gh pr checks "$pr" --json state --jq '.[].state' 2>/dev/null | sort -u)

    # Get linked issue
    ISSUE=$(gh pr view "$pr" --json body --jq '.body' | grep -oE 'Closes #[0-9]+' | grep -oE '[0-9]+' | head -1)

    if [ -z "$ISSUE" ]; then
      echo "  ⚠ No linked issue found, skipping"
      continue
    fi

    # Check if CI passed
    if echo "$ci_status" | grep -q "FAILURE"; then
      echo "  ❌ CI failing - triggering ci-monitoring for PR #$pr"
      # Invoke ci-monitoring skill to fix
      handle_ci_failure "$pr"
      continue
    fi

    if echo "$ci_status" | grep -q "PENDING"; then
      echo "  ⏳ CI pending for PR #$pr, will check in main loop"
      continue
    fi

    if echo "$ci_status" | grep -q "SUCCESS"; then
      # Verify review artifact
      REVIEW_EXISTS=$(gh api "/repos/$OWNER/$REPO/issues/$ISSUE/comments" \
        --jq '[.[] | select(.body | contains("<!-- REVIEW:START -->"))] | length' 2>/dev/null || echo "0")

      if [ "$REVIEW_EXISTS" = "0" ]; then
        echo "  ⚠ No review artifact - requesting review for #$ISSUE"
        gh issue comment "$ISSUE" --body "## Review Required

PR #$pr has passing CI but no review artifact.

**Action needed:** Complete comprehensive-review and post artifact to this issue.

---
*Bootstrap phase - Orchestrator*"
        continue
      fi

      # All checks pass - merge
      echo "  ✅ Merging PR #$pr"
      gh pr merge "$pr" --squash --delete-branch
      mark_issue_done "$ISSUE"
    fi
  done

  echo "=== BOOTSTRAP COMPLETE ==="
}

Release Placeholder Detection

PRs are excluded from bootstrap resolution if:

| Condition | Example | |-----------|---------| | Branch starts with release/ | release/v2.0.0, release/2025-01 | | Has release-placeholder label | Manual exclusion | | Has do-not-merge label | Explicit hold |

Orchestration Loop

┌──────────────────────────────────────────────────────────┐
│                       MAIN LOOP                          │
└─────────────────────────┬────────────────────────────────┘
                          │
      ┌───────────────────┼───────────────────┐
      ▼                   ▼                   ▼
┌───────────┐      ┌───────────┐      ┌───────────┐
│ CHECK     │      │ CHECK     │      │ SPAWN     │
│ WORKERS   │      │ CI/PRs    │      │ WORKERS   │
└─────┬─────┘      └─────┬─────┘      └─────┬─────┘
      │                  │                  │
      └──────────────────┼──────────────────┘
                         │
                         ▼
               ┌───────────────────┐
               │ EVALUATE STATE    │
               │                   │
               │ All done? → Exit  │
               │ Waiting? → SLEEP  │
               │ Work? → Continue  │
               └───────────────────┘

See: reference/loop-implementation.md for full loop code.

Loop Steps

1. Check Deviation Resolution - Resume issues whose children are all closed
2. Check CI/PRs - Monitor for merge readiness, verify review artifacts
3. MERGE GREEN PRs - Any PR with passing CI is merged IMMEDIATELY
4. Spawn Workers - Up to 5 parallel workers from Ready queue
5. Evaluate State - Determine next action (continue, sleep, complete)
6. Brief Pause - 30 second interval between iterations

CRITICAL: Merge Green PRs Immediately

Every loop iteration must check for and merge passing PRs:

# In each loop iteration
for pr in $(gh pr list --json number,statusCheckRollup --jq '.[] | select(.statusCheckRollup | all(.conclusion == "SUCCESS")) | .number'); do
  # Check for do-not-merge label
  if ! gh pr view "$pr" --json labels --jq '.labels[].name' | grep -q "do-not-merge"; then
    echo "Merging PR #$pr (CI passed)"
    gh pr merge "$pr" --squash --delete-branch

    # Get linked issue and mark done
    ISSUE=$(gh pr view "$pr" --json body --jq '.body' | grep -oE 'Closes #[0-9]+' | grep -oE '[0-9]+' | head -1)
    if [ -n "$ISSUE" ]; then
      # Update project board status to Done
      mark_issue_done "$ISSUE"
    fi
  fi
done

Do NOT:

• Report "PRs are ready for merge" and stop
• Wait for user to request merge
• Summarize completed work and ask for next steps

The loop continues until scope is complete. Green PR = immediate merge.

Scope Types

Milestone

gh issue list --milestone "v1.0.0" --state open --json number --jq '.[].number'

Epic

gh issue list --label "epic-dark-mode" --state open --json number --jq '.[].number'

Unbounded (All Open Issues)

gh issue list --state open --json number --jq '.[].number'

Do NOT ask for "UNBOUNDED" confirmation. The user's request is their consent.

Failure Handling

Workers that fail do NOT immediately become blocked:

Attempt 1 → Research → Attempt 2 → Research → Attempt 3 → Research → Attempt 4 → BLOCKED

Only after 3+ research cycles is an issue marked as blocked.

See: reference/failure-recovery.md for research cycle implementation.

Blocked Determination

An issue is only marked blocked when:

• Multiple research cycles completed (3+)
• Research concludes "impossible without external input"
• Examples: missing credentials, requires human decision, external service down

SLEEP/WAKE

Entering SLEEP

Orchestration sleeps when:

• All issues are either blocked or in review
• No work can proceed without external event

State is posted to GitHub tracking issue (survives crashes).

WAKE Mechanisms

• Session start - Checks CI status on new Codex session
• Manual - codex resume [SESSION_ID]

Checklist

Before starting orchestration:

• [ ] Scope identified (explicit or auto-detected)
• [ ] Git worktrees available (git worktree list)
• [ ] GitHub CLI authenticated (gh auth status)
• [ ] No uncommitted changes in main worktree
• [ ] Tracking issue exists with orchestration-tracking label
• [ ] Project board configured with Status field

Bootstrap phase:

• [ ] Existing open PRs detected
• [ ] Release placeholders excluded (release/*, release-placeholder, do-not-merge labels)
• [ ] CI status checked for each PR
• [ ] Review artifacts verified before merge
• [ ] PRs merged or flagged for attention
• [ ] Bootstrap complete before spawning workers

During orchestration:

• [ ] Workers spawned with worktree isolation
• [ ] Worker status tracked via Project Board (NOT labels)
• [ ] CI status monitored
• [ ] Review artifacts verified before PR merge
• [ ] Failed workers trigger research cycles
• [ ] Handovers happen at turn limit
• [ ] SLEEP entered when only waiting on CI
• [ ] Deviation resolution checked each loop
• [ ] Status posted to tracking issue

Review Enforcement

CRITICAL: The orchestrator verifies review compliance:

1. Before PR merge:

   • Review artifact exists in issue comments
   • Review status is COMPLETE
   • Unaddressed findings = 0

2. Child issues (from deferred findings):

   • Follow full issue-driven-development process
   • Have their own code reviews
   • Track via spawned-from:#N label

3. Deviation handling:

   • Parent status set to Blocked on project board
   • Resumes only when all children closed

Integration

This skill coordinates:

| Skill | Purpose | |-------|---------| | worker-dispatch | Spawning workers | | worker-protocol | Worker behavior | | worker-handover | Context passing | | ci-monitoring | CI and WAKE handling | | research-after-failure | Research cycles | | issue-driven-development | Worker follows this | | comprehensive-review | Workers must complete before PR | | project-board-enforcement | ALL state queries and updates |

Reference Files

• reference/state-management.md - State queries, updates, deviation handling
• reference/loop-implementation.md - Full loop code and helpers
• reference/failure-recovery.md - Research cycles, blocked handling, SLEEP/WAKE