abanoub-ashraf/debugging

Systematic Debugging

Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches.

When to Use This Skill

• Tracking down elusive bugs
• Investigating performance issues
• Understanding unfamiliar codebases
• Debugging production issues
• Analyzing crash dumps and stack traces
• Profiling application performance
• Investigating memory leaks
• Test failures and flaky tests
• Build failures
• Integration issues

Use this ESPECIALLY when:

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes. Symptom fixes are failure.

Core Principles

1. The Scientific Method

1. Observe: What's the actual behavior?
2. Hypothesize: What could be causing it?
3. Experiment: Test your hypothesis
4. Analyze: Did it prove/disprove your theory?
5. Repeat: Until you find the root cause

2. Debugging Mindset

Don't Assume:

• "It can't be X" - Yes it can
• "I didn't change Y" - Check anyway
• "It works on my machine" - Find out why

Do:

• Reproduce consistently
• Isolate the problem
• Keep detailed notes
• Question everything
• Take breaks when stuck

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully

   • Don't skip past errors or warnings
   • They often contain the exact solution
   • Read stack traces completely
   • Note line numbers, file paths, error codes

2. Reproduce Consistently

   • Can you trigger it reliably?
   • What are the exact steps?
   • Does it happen every time?
   • If not reproducible, gather more data, don't guess

3. Check Recent Changes

   • What changed that could cause this?
   • Git diff, recent commits
   • New dependencies, config changes
   • Environmental differences

4. Gather Evidence in Multi-Component Systems

   WHEN system has multiple components (CI → build → signing, API → service → database):

   For EACH component boundary:
     - Log what data enters component
     - Log what data exits component
     - Verify environment/config propagation
     - Check state at each layer
   
   Run once to gather evidence showing WHERE it breaks
   THEN analyze evidence to identify failing component
   THEN investigate that specific component
   

5. Trace Data Flow

   • Where does bad value originate?
   • What called this with bad value?
   • Keep tracing up until you find the source
   • Fix at source, not at symptom

   See root-cause-tracing.md for the complete backward tracing technique.

Phase 2: Pattern Analysis

Find the pattern before fixing:

1. Find Working Examples

   • Locate similar working code in same codebase
   • What works that's similar to what's broken?

2. Compare Against References

   • If implementing pattern, read reference implementation COMPLETELY
   • Don't skim - read every line
   • Understand the pattern fully before applying

3. Identify Differences

   • What's different between working and broken?
   • List every difference, however small
   • Don't assume "that can't matter"

Phase 3: Hypothesis and Testing

Scientific method:

1. Form Single Hypothesis

   • State clearly: "I think X is the root cause because Y"
   • Write it down
   • Be specific, not vague

2. Test Minimally

   • Make the SMALLEST possible change to test hypothesis
   • One variable at a time
   • Don't fix multiple things at once

3. Verify Before Continuing

   • Did it work? Yes → Phase 4
   • Didn't work? Form NEW hypothesis
   • DON'T add more fixes on top

Phase 4: Implementation

Fix the root cause, not the symptom:

1. Create Failing Test Case

   • Simplest possible reproduction
   • Automated test if possible
   • MUST have before fixing

2. Implement Single Fix

   • Address the root cause identified
   • ONE change at a time
   • No "while I'm here" improvements

3. Verify Fix

   • Test passes now?
   • No other tests broken?
   • Issue actually resolved?

4. If 3+ Fixes Failed: Question Architecture

   • Is this pattern fundamentally sound?
   • Should we refactor architecture vs. continue fixing symptoms?
   • Discuss with your human partner before attempting more fixes

Debugging Tools by Language

JavaScript/TypeScript Debugging

// Chrome DevTools Debugger
function processOrder(order: Order) {
    debugger;  // Execution pauses here

    const total = calculateTotal(order);
    console.log('Total:', total);

    // Conditional breakpoint
    if (order.items.length > 10) {
        debugger;  // Only breaks if condition true
    }

    return total;
}

// Console debugging techniques
console.log('Value:', value);                    // Basic
console.table(arrayOfObjects);                   // Table format
console.time('operation'); /* code */ console.timeEnd('operation');  // Timing
console.trace();                                 // Stack trace
console.assert(value > 0, 'Value must be positive');  // Assertion

// Performance profiling
performance.mark('start-operation');
// ... operation code
performance.mark('end-operation');
performance.measure('operation', 'start-operation', 'end-operation');
console.log(performance.getEntriesByType('measure'));

VS Code Debugger Configuration:

// .vscode/launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "Debug Program",
            "program": "${workspaceFolder}/src/index.ts",
            "preLaunchTask": "tsc: build - tsconfig.json",
            "outFiles": ["${workspaceFolder}/dist/**/*.js"],
            "skipFiles": ["<node_internals>/**"]
        },
        {
            "type": "node",
            "request": "launch",
            "name": "Debug Tests",
            "program": "${workspaceFolder}/node_modules/jest/bin/jest",
            "args": ["--runInBand", "--no-cache"],
            "console": "integratedTerminal"
        }
    ]
}

Python Debugging

# Built-in debugger (pdb)
import pdb

def calculate_total(items):
    total = 0
    pdb.set_trace()  # Debugger starts here

    for item in items:
        total += item.price * item.quantity

    return total

# Breakpoint (Python 3.7+)
def process_order(order):
    breakpoint()  # More convenient than pdb.set_trace()
    # ... code

# Post-mortem debugging
try:
    risky_operation()
except Exception:
    import pdb
    pdb.post_mortem()  # Debug at exception point

# Logging for debugging
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

def fetch_user(user_id):
    logger.debug(f'Fetching user: {user_id}')
    user = db.query(User).get(user_id)
    logger.debug(f'Found user: {user}')
    return user

# Profile performance
import cProfile
import pstats

cProfile.run('slow_function()', 'profile_stats')
stats = pstats.Stats('profile_stats')
stats.sort_stats('cumulative')
stats.print_stats(10)  # Top 10 slowest

Go Debugging

// Delve debugger
// Install: go install github.com/go-delve/delve/cmd/dlv@latest
// Run: dlv debug main.go

import (
    "fmt"
    "runtime"
    "runtime/debug"
)

// Print stack trace
func debugStack() {
    debug.PrintStack()
}

// Panic recovery with debugging
func processRequest() {
    defer func() {
        if r := recover(); r != nil {
            fmt.Println("Panic:", r)
            debug.PrintStack()
        }
    }()

    // ... code that might panic
}

// Memory profiling
import _ "net/http/pprof"
// Visit http://localhost:6060/debug/pprof/

// CPU profiling
import (
    "os"
    "runtime/pprof"
)

f, _ := os.Create("cpu.prof")
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
// ... code to profile

Advanced Debugging Techniques

Technique 1: Binary Search Debugging (Git Bisect)

# Git bisect for finding regression
git bisect start
git bisect bad                    # Current commit is bad
git bisect good v1.0.0            # v1.0.0 was good

# Git checks out middle commit
# Test it, then:
git bisect good   # if it works
git bisect bad    # if it's broken

# Continue until bug found
git bisect reset  # when done

Technique 2: Differential Debugging

Compare working vs broken:

| Aspect | Working | Broken | |--------------|-----------------|-----------------| | Environment | Development | Production | | Node version | 18.16.0 | 18.15.0 | | Data | Empty DB | 1M records | | User | Admin | Regular user | | Browser | Chrome | Safari | | Time | During day | After midnight |

Technique 3: Trace Debugging

// Function call tracing
function trace(target: any, propertyKey: string, descriptor: PropertyDescriptor) {
    const originalMethod = descriptor.value;

    descriptor.value = function(...args: any[]) {
        console.log(`Calling ${propertyKey} with args:`, args);
        const result = originalMethod.apply(this, args);
        console.log(`${propertyKey} returned:`, result);
        return result;
    };

    return descriptor;
}

class OrderService {
    @trace
    calculateTotal(items: Item[]): number {
        return items.reduce((sum, item) => sum + item.price, 0);
    }
}

Technique 4: Memory Leak Detection

// Chrome DevTools Memory Profiler
// 1. Take heap snapshot
// 2. Perform action
// 3. Take another snapshot
// 4. Compare snapshots

// Node.js memory debugging
if (process.memoryUsage().heapUsed > 500 * 1024 * 1024) {
    console.warn('High memory usage:', process.memoryUsage());

    // Generate heap dump
    require('v8').writeHeapSnapshot();
}

// Find memory leaks in tests
let beforeMemory: number;

beforeEach(() => {
    beforeMemory = process.memoryUsage().heapUsed;
});

afterEach(() => {
    const afterMemory = process.memoryUsage().heapUsed;
    const diff = afterMemory - beforeMemory;

    if (diff > 10 * 1024 * 1024) {  // 10MB threshold
        console.warn(`Possible memory leak: ${diff / 1024 / 1024}MB`);
    }
});

Debugging Patterns by Issue Type

Pattern 1: Intermittent/Flaky Bugs

1. Add extensive logging

   • Log timing information
   • Log all state transitions
   • Log external interactions

2. Look for race conditions

   • Concurrent access to shared state
   • Async operations completing out of order
   • Missing synchronization

3. Check timing dependencies

   • setTimeout/setInterval
   • Promise resolution order
   • Animation frame timing

4. Stress test

   • Run many times
   • Vary timing
   • Simulate load

See condition-based-waiting.md for fixing timing-dependent tests.

Pattern 2: Performance Issues

1. Profile first

   • Don't optimize blindly
   • Measure before and after
   • Find bottlenecks

2. Common culprits

   • N+1 queries
   • Unnecessary re-renders
   • Large data processing
   • Synchronous I/O

3. Tools

   • Browser DevTools Performance tab
   • Lighthouse
   • Python: cProfile, line_profiler
   • Node: clinic.js, 0x

Pattern 3: Production Bugs

1. Gather evidence

   • Error tracking (Sentry, Bugsnag)
   • Application logs
   • User reports
   • Metrics/monitoring

2. Reproduce locally

   • Use production data (anonymized)
   • Match environment
   • Follow exact steps

3. Safe investigation

   • Don't change production
   • Use feature flags
   • Add monitoring/logging
   • Test fixes in staging

Red Flags - STOP and Follow Process

If you catch yourself thinking:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "Add multiple changes, run tests"
• "Skip the test, I'll manually verify"
• "It's probably X, let me fix that"
• "I don't fully understand but this might work"
• "One more fix attempt" (when already tried 2+)

ALL of these mean: STOP. Return to Phase 1.

Common Rationalizations

| Excuse | Reality | |--------|---------| | "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. | | "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. | | "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. | | "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. | | "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. | | "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |

Quick Reference

| Phase | Key Activities | Success Criteria | |-------|---------------|------------------| | 1. Root Cause | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY | | 2. Pattern | Find working examples, compare | Identify differences | | 3. Hypothesis | Form theory, test minimally | Confirmed or new hypothesis | | 4. Implementation | Create test, fix, verify | Bug resolved, tests pass |

Quick Debugging Checklist

When Stuck, Check:

• [ ] Spelling errors (typos in variable names)
• [ ] Case sensitivity (fileName vs filename)
• [ ] Null/undefined values
• [ ] Array index off-by-one
• [ ] Async timing (race conditions)
• [ ] Scope issues (closure, hoisting)
• [ ] Type mismatches
• [ ] Missing dependencies
• [ ] Environment variables
• [ ] File paths (absolute vs relative)
• [ ] Cache issues (clear cache)
• [ ] Stale data (refresh database)

Supporting Techniques

These techniques are part of systematic debugging and available in this directory:

• root-cause-tracing.md - Trace bugs backward through call stack to find original trigger
• defense-in-depth.md - Add validation at multiple layers after finding root cause
• condition-based-waiting.md - Replace arbitrary timeouts with condition polling
• find-polluter.sh - Script to find which test causes pollution

Real-World Impact

From debugging sessions:

• Systematic approach: 15-30 minutes to fix
• Random fixes approach: 2-3 hours of thrashing
• First-time fix rate: 95% vs 40%
• New bugs introduced: Near zero vs common