abcnuts/serverless-debugging

Serverless Debugging

Systematic debugging methodology for serverless/edge functions where console.log may not appear in logs and traditional debugging tools are unavailable.

When to Use This Skill

• Feature deployed but not triggering (code exists but never executes)
• Console.log statements don't appear in logs
• No visible errors but behavior doesn't match expectations
• Database state doesn't update as expected
• Type coercion or default value bugs
• Async/timing issues in distributed systems

Core Methodology: 4-Phase Approach

Phase 1: Evidence Gathering

Collect evidence from ALL layers before forming hypotheses. Avoid jumping to conclusions.

Database Layer:

• Query actual data state (not assumptions)
• Check default values vs. expected values
• Verify row counts, timestamps, update patterns
• Look for NULL vs. 0 vs. empty string differences

Code Layer:

• Verify deployed code matches local code
• Check function version numbers
• Review conditional logic and type checks
• Identify all code paths that could prevent execution

Runtime Layer:

• Check deployment status (ACTIVE, version number)
• Review available logs (even if incomplete)
• Note what SHOULD appear in logs but doesn't
• Check for silent failures (no error, no success)

Environment Layer:

• Verify environment variables are set
• Check permissions and RLS policies
• Confirm API keys and authentication
• Review timeout settings

Evidence Checklist: See references/evidence-checklist.md for comprehensive list

Phase 2: Pattern Analysis

Compare working vs. broken behavior to identify differences.

Key Questions:

• What works? What doesn't?
• When did it last work? What changed?
• Are there similar features that DO work? How do they differ?
• What's the simplest test case that reproduces the issue?

Common Patterns:

• Logging doesn't work in this environment (find alternatives)
• Code path never reached (condition always false)
• Type mismatch (0 vs null, string vs number)
• Async race condition (timing-dependent)
• Environment-specific behavior (works locally, fails in production)

Phase 3: Hypothesis Formation

Form specific, testable hypotheses based on evidence.

Good Hypothesis Characteristics:

• Explains ALL observed symptoms
• Testable with available tools
• Specific about root cause (not vague)
• Based on evidence, not assumptions

Example Hypotheses:

• "Database default is 0, but code checks for null"
• "Function never reaches line X because condition Y is always false"
• "Environment variable Z is not set in production"
• "RLS policy blocks access for this user role"

Testing Without Logs:

• Use database writes as "print statements"
• Add error responses with diagnostic info
• Create minimal reproduction cases
• Test conditions in isolation

Phase 4: Implementation & Verification

Fix the root cause and verify across all layers.

Fix Implementation:

• Make minimal, targeted changes
• Document the bug and fix
• Update tests to prevent regression
• Deploy new version

Verification:

• Database state updates as expected
• Feature triggers in expected scenarios
• No side effects on other features
• End-to-end test passes

Common Serverless Bugs

See references/common-serverless-bugs.md for detailed catalog. Quick reference:

Type Coercion Issues:

• 0 === null → false (check for both)
• undefined === null → false
• Empty string vs null vs undefined
• Number strings vs numbers ("0" vs 0)

Async/Timing Issues:

• Fire-and-forget operations that fail silently
• Race conditions in parallel operations
• Timeout before completion
• Promise not awaited

Environment Differences:

• Console.log works locally but not in production
• Environment variables not set
• Different runtime versions
• Permission differences

Database Issues:

• RLS policies blocking access
• Default values vs. expected values
• Transaction isolation issues
• Connection pool exhaustion

Alternative Debugging Strategies

When console.log doesn't work, use these alternatives:

Database-Driven Debugging:

-- Create debug log table
CREATE TABLE debug_logs (
  id SERIAL PRIMARY KEY,
  timestamp TIMESTAMPTZ DEFAULT NOW(),
  function_name TEXT,
  checkpoint TEXT,
  data JSONB
);

-- Write debug info from code
INSERT INTO debug_logs (function_name, checkpoint, data)
VALUES ('my-function', 'before-extraction', jsonb_build_object('count', message_count));

Error Response Debugging:

// Return diagnostic info in error response
return new Response(JSON.stringify({
  debug: {
    messageCount: messageCount,
    lastExtraction: lastExtraction,
    shouldExtract: shouldExtractMemories(messageCount, lastExtraction)
  }
}), { status: 200 });

Code Analysis:

• Read deployed code directly (if accessible)
• Trace execution paths on paper
• Check type of every variable in conditions
• Verify all branches can be reached

See references/logging-alternatives.md for more strategies.

Practical Tools

Deployment Verification:

# Use scripts/check_deployment.py
python scripts/check_deployment.py <function-name> <local-path>

Database State Analysis:

# Use scripts/database_debugger.py
python scripts/database_debugger.py <table-name> <condition>

Type Analysis:

# Use scripts/type_analyzer.py
python scripts/type_analyzer.py <file-path> <function-name>

Real-World Example: Wade Memory Extraction Bug

Symptom: Proactive memory extraction never triggered despite code being deployed.

Phase 1 Evidence:

• ✅ Code deployed (v48)
• ✅ Database schema correct
• ✅ Messages stored (8-10 messages)
• ❌ Console.log never appeared
• ❌ last_extraction_at stayed at 0

Phase 2 Pattern:

• Console.log doesn't work in Supabase Edge Functions logs API
• Needed alternative debugging approach

Phase 3 Hypothesis:

• Database default for last_extraction_at is 0, not null
• Code checks if (lastExtraction === null) → always false when 0
• Takes wrong branch: messageCount - 0 >= 10 instead of messageCount >= 4

Phase 4 Fix:

// Before: Only checked null
if (lastExtractionAt === null) {
  return messageCount >= 4;
}

// After: Check both null AND 0
if (lastExtractionAt === null || lastExtractionAt === 0) {
  return messageCount >= 4;
}

Result: Feature now works correctly, triggers after 4 messages.

Best Practices

1. Gather evidence before hypothesizing - Resist the urge to guess
2. Use database as debugging tool - When logs fail, database writes work
3. Check type coercion carefully - 0, null, undefined are all different
4. Verify deployed code - Don't assume local matches production
5. Test in isolation - Minimal reproduction cases reveal root cause
6. Document the process - Future debugging benefits from past patterns

Quick Start

1. Read references/evidence-checklist.md and gather evidence from all layers
2. Identify patterns using Phase 2 questions
3. Form testable hypothesis
4. Use scripts/ tools to verify hypothesis
5. Implement fix and verify across all layers
6. Document bug and fix for future reference