charon-fan/self-improving-agent

Self-Improving Agent

"An AI agent that learns from every interaction, accumulating patterns and insights to continuously improve its own capabilities." — Based on 2025 lifelong learning research

Overview

This is a universal self-improvement system that learns from ALL skill experiences, not just PRDs. It implements a complete feedback loop with:

• Multi-Memory Architecture: Semantic + Episodic + Working memory
• Self-Correction: Detects and fixes skill guidance errors
• Self-Validation: Periodically verifies skill accuracy
• Hooks Integration: Auto-triggers on skill events (before_start, after_complete, on_error)
• Evolution Markers: Traceable changes with source attribution

Research-Based Design

Based on 2025 research:

| Research | Key Insight | Application | |----------|-------------|-------------| | SimpleMem | Efficient lifelong memory | Pattern accumulation system | | Multi-Memory Survey | Semantic + Episodic memory | World knowledge + experiences | | Lifelong Learning | Continuous task stream learning | Learn from every skill use | | Evo-Memory | Test-time lifelong learning | Real-time adaptation |

The Self-Improvement Loop

┌─────────────────────────────────────────────────────────────────┐
│                    UNIVERSAL SELF-IMPROVEMENT                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Skill Event → Extract Experience → Abstract Pattern → Update  │
│        │                  │                │         │          │
│        ▼                  ▼                ▼         ▼          │
│   ┌─────────────────────────────────────────────────────┐       │
│   │              MULTI-MEMORY SYSTEM                      │       │
│   ├─────────────────────────────────────────────────────┤       │
│   │  Semantic Memory   │  Episodic Memory  │ Working Memory │  │
│   │  (Patterns/Rules)  │  (Experiences)    │  (Current)     │  │
│   │  memory/semantic/  │  memory/episodic/ │  memory/working/│  │
│   └─────────────────────────────────────────────────────┘       │
│                                                                 │
│   ┌─────────────────────────────────────────────────────┐       │
│   │              FEEDBACK LOOP                            │       │
│   │  User Feedback → Confidence Update → Pattern Adapt   │       │
│   └─────────────────────────────────────────────────────┘       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

When This Activates

Automatic Triggers (via hooks)

| Event | Trigger | Action | |-------|---------|--------| | before_start | Any skill starts | Log session start | | after_complete | Any skill completes | Extract patterns, update skills | | on_error | Bash returns non-zero exit | Capture error context, trigger self-correction |

Manual Triggers

• User says "自我进化", "self-improve", "从经验中学习"
• User says "分析今天的经验", "总结教训"
• User asks to improve a specific skill

Evolution Priority Matrix

Trigger evolution when new reusable knowledge appears:

| Trigger | Target Skill | Priority | Action | |---------|--------------|----------|--------| | New PRD pattern discovered | prd-planner | High | Add to quality checklist | | Architecture tradeoff clarified | architecting-solutions | High | Add to decision patterns | | API design rule learned | api-designer | High | Update template | | Debugging fix discovered | debugger | High | Add to anti-patterns | | Review checklist gap | code-reviewer | High | Add checklist item | | Perf/security insight | performance-engineer, security-auditor | High | Add to patterns | | UI/UX spec issue | prd-planner, architecting-solutions | High | Add visual spec requirements | | React/state pattern | debugger, refactoring-specialist | Medium | Add to patterns | | Test strategy improvement | test-automator, qa-expert | Medium | Update approach | | CI/deploy fix | deployment-engineer | Medium | Add to troubleshooting |

Multi-Memory Architecture

1. Semantic Memory (memory/semantic-patterns.json)

Stores abstract patterns and rules reusable across contexts:

{
  "patterns": {
    "pattern_id": {
      "id": "pat-2025-01-11-001",
      "name": "Pattern Name",
      "source": "user_feedback|implementation_review|retrospective",
      "confidence": 0.95,
      "applications": 5,
      "created": "2025-01-11",
      "category": "prd_structure|react_patterns|async_patterns|...",
      "pattern": "One-line summary",
      "problem": "What problem does this solve?",
      "solution": { ... },
      "quality_rules": [ ... ],
      "target_skills": [ ... ]
    }
  }
}

2. Episodic Memory (memory/episodic/)

Stores specific experiences and what happened:

memory/episodic/
├── 2025/
│   ├── 2025-01-11-prd-creation.json
│   ├── 2025-01-11-debug-session.json
│   └── 2025-01-12-refactoring.json

{
  "id": "ep-2025-01-11-001",
  "timestamp": "2025-01-11T10:30:00Z",
  "skill": "debugger",
  "situation": "User reported data not refreshing after form submission",
  "root_cause": "Empty callback in onRefresh prop",
  "solution": "Implement actual refresh logic in callback",
  "lesson": "Always verify callbacks are not empty functions",
  "related_pattern": "callback_verification",
  "user_feedback": {
    "rating": 8,
    "comments": "This was exactly the issue"
  }
}

3. Working Memory (memory/working/)

Stores current session context:

memory/working/
├── current_session.json   # Active session data
├── last_error.json        # Error context for self-correction
└── session_end.json       # Session end marker

Self-Improvement Process

Phase 1: Experience Extraction

After any skill completes, extract:

What happened:
  skill_used: {which skill}
  task: {what was being done}
  outcome: {success|partial|failure}

Key Insights:
  what_went_well: [what worked]
  what_went_wrong: [what didn't work]
  root_cause: {underlying issue if applicable}

User Feedback:
  rating: {1-10 if provided}
  comments: {specific feedback}

Phase 2: Pattern Abstraction

Convert experiences to reusable patterns:

| Concrete Experience | Abstract Pattern | Target Skill | |--------------------|------------------|--------------| | "User forgot to save PRD notes" | "Always persist thinking to files" | prd-planner | | "Code review missed SQL injection" | "Add security checklist item" | code-reviewer | | "Callback was empty, didn't work" | "Verify callback implementations" | debugger | | "Net APY position ambiguous" | "UI specs need exact relative positions" | prd-planner |

Abstraction Rules:

If experience_repeats 3+ times:
  pattern_level: critical
  action: Add to skill's "Critical Mistakes" section

If solution_was_effective:
  pattern_level: best_practice
  action: Add to skill's "Best Practices" section

If user_rating >= 7:
  pattern_level: strength
  action: Reinforce this approach

If user_rating <= 4:
  pattern_level: weakness
  action: Add to "What to Avoid" section

Phase 3: Skill Updates

Update the appropriate skill files with evolution markers:

<!-- Evolution: 2025-01-12 | source: ep-2025-01-12-001 | skill: debugger -->

## Pattern Added (2025-01-12)

**Pattern**: Always verify callbacks are not empty functions

**Source**: Episode ep-2025-01-12-001

**Confidence**: 0.95

### Updated Checklist
- [ ] Verify all callbacks have implementations
- [ ] Test callback execution paths

Correction Markers (when fixing wrong guidance):

<!-- Correction: 2025-01-12 | was: "Use callback chain" | reason: caused stale refresh -->

## Corrected Guidance

Use direct state monitoring instead of callback chains:
```typescript
// ✅ Do: Direct state monitoring
const prevPendingCount = usePrevious(pendingCount);

### Phase 4: Memory Consolidation

1. **Update semantic memory** (`memory/semantic-patterns.json`)
2. **Store episodic memory** (`memory/episodic/YYYY-MM-DD-{skill}.json`)
3. **Update pattern confidence** based on applications/feedback
4. **Prune outdated patterns** (low confidence, no recent applications)

## Promotion Policy

Self-improvement has two separate jobs:

1. **Capture** facts, corrections, failed assumptions, and reusable patterns as memory or proposal artifacts.
2. **Promote** only validated patterns into `SKILL.md`, `AGENTS.md`, docs, or CLI behavior.

Default to capture-first. Promote a change only when one of these is true:

- The user explicitly asks to update a skill or repository instruction.
- The same pattern recurs across multiple episodes.
- A focused test or review proves the current guidance is wrong or incomplete.
- The change is low-risk documentation that preserves existing behavior and is clearly traceable.

Promotion targets:

| Artifact | Use For | Approval Level |
|----------|---------|----------------|
| `memory/episodic/*.json` | Raw episode facts and signals | Auto |
| `memory/semantic-patterns.json` | Candidate reusable patterns with confidence | Auto |
| `memory/proposals/*.md` | Proposed skill/doc/code changes with evidence | Auto |
| `SKILL.md` / `references/` | Validated workflow guidance | Ask first unless user requested editing |
| `AGENTS.md` / repo rules | Cross-repo behavior or hard constraints | Ask first |
| CLI/runtime code | Automation semantics | Require tests |

## Self-Correction (on_error hook)

Triggered when:
- Bash command returns non-zero exit code
- Tests fail after following skill guidance
- User reports the guidance produced incorrect results

**Process:**

```markdown
## Self-Correction Workflow

1. Detect Error
   - Capture error context from working/last_error.json
   - Identify which skill guidance was followed

2. Verify Root Cause
   - Was the skill guidance incorrect?
   - Was the guidance misinterpreted?
   - Was the guidance incomplete?

3. Create Proposal
   - Write a proposal with evidence, affected skill names, and expected behavior
   - Add correction marker text in the proposal, not directly in the skill yet
   - Update related patterns in semantic memory with low initial confidence

4. Validate Fix
   - Test the corrected guidance
   - Ask user to verify

5. Promote
   - Apply the skill/doc/code change after validation or explicit approval
   - Keep the source episode/proposal id in the change note

Example:

<!-- Correction: 2025-01-12 | was: "useMemo for claimable ids" | reason: stale data at click time -->

## Self-Correction: Click-Time Computation

**Issue**: Using useMemo for claimable IDs caused stale data
**Fix**: Compute at click time for always-fresh data
**Pattern**: click_time_vs_open_time_computation

Self-Validation

Use the validation template in references/appendix.md when reviewing updates.

Hooks Integration

Runtime Trigger Source

agent-playbook self-improve reads skill chaining from each skill's SKILL.md frontmatter:

metadata:
  hooks:
    after_complete:
      - trigger: self-improving-agent
        mode: background
        reason: "Extract patterns"

Treat metadata.hooks as the source of truth. Do not maintain a second hardcoded hook map in runtime code. This keeps skill behavior auditable and lets Skill Creator style reviews inspect the same file that the agent executes.

Wiring Hooks in Claude Code Settings

Add to Claude Code settings (~/.claude/settings.json):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash|Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/pre-tool.sh \"$TOOL_NAME\" \"$TOOL_INPUT\""
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/post-bash.sh \"$TOOL_OUTPUT\" \"$EXIT_CODE\""
          }
        ]
      }
    ],
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/session-end.sh"
          }
        ]
      }
    ]
  }
}

Replace ${SKILLS_DIR} with your actual skills path.

Additional References

See references/appendix.md for memory structure, workflow diagrams, metrics, feedback templates, and research links.

Best Practices

DO

• ✅ Learn from EVERY skill interaction
• ✅ Extract patterns at the right abstraction level
• ✅ Update multiple related skills
• ✅ Track confidence and apply counts
• ✅ Ask for user feedback on improvements
• ✅ Use evolution/correction markers for traceability
• ✅ Validate guidance before applying broadly
• ✅ Write proposals before mutating durable skill guidance
• ✅ Keep hook routing in metadata.hooks

DON'T

• ❌ Over-generalize from single experiences
• ❌ Update skills without confidence tracking
• ❌ Ignore negative feedback
• ❌ Make changes that break existing functionality
• ❌ Create contradictory patterns
• ❌ Update skills without understanding context
• ❌ Silently promote self-improvement findings into repo rules
• ❌ Duplicate hook definitions in CLI code and skill frontmatter

Quick Start

After any skill completes, this agent automatically:

1. Analyzes what happened
2. Extracts patterns and insights
3. Writes memory and proposal artifacts
4. Promotes validated improvements when approval or evidence is sufficient
5. Reports summary to user

References

• SimpleMem: Efficient Lifelong Memory for LLM Agents
• A Survey on the Memory Mechanism of Large Language Model Agents
• Lifelong Learning of LLM based Agents
• Evo-Memory: DeepMind's Benchmark
• Let's Build a Self-Improving AI Agent
• OpenClaw self-improving-agent skill
• OpenCrabs local self-improving agent
• ELL-StuLife experience-driven lifelong learning