abcnuts/serverless-debugging

Serverless Debugging

Systematic debugging methodology for serverless/edge functions where console.log may not appear in logs and traditional debugging tools are unavailable.

When to Use This Skill

• Feature deployed but not triggering (code exists but never executes)
• Console.log statements don't appear in logs
• No visible errors but behavior doesn't match expectations
• Database state doesn't update as expected
• Type coercion or default value bugs
• Async/timing issues in distributed systems

Core Methodology: 4-Phase Approach

Phase 1: Evidence Gathering

Collect evidence from ALL layers before forming hypotheses. Avoid jumping to conclusions.

Database Layer:

• Query actual data state (not assumptions)
• Check default values vs. expected values
• Verify row counts, timestamps, update patterns
• Look for NULL vs. 0 vs. empty string differences

Code Layer:

• Verify deployed code matches local code
• Check function version numbers
• Review conditional logic and type checks
• Identify all code paths that could prevent execution

Runtime Layer:

• Check deployment status (ACTIVE, version number)
• Review available logs (even if incomplete)
• Note what SHOULD appear in logs but doesn't
• Check for silent failures (no error, no success)

Environment Layer:

• Verify environment variables are set
• Check permissions and RLS policies
• Confirm API keys and authentication
• Review timeout settings

Evidence Checklist: See references/evidence-checklist.md for comprehensive list

Phase 2: Pattern Analysis

Compare working vs. broken behavior to identify differences.

Key Questions:

• What works? What doesn't?
• When did it last work? What changed?
• Are there similar features that DO work? How do they differ?
• What's the simplest test case that reproduces the issue?

Common Patterns:

• Logging doesn't work in this environment (find alternatives)
• Code path never reached (condition always false)
• Type mismatch (0 vs null, string vs number)
• Async race condition (timing-dependent)
• Environment-specific behavior (works locally, fails in production)

Phase 3: Hypothesis Formation

Form specific, testable hypotheses based on evidence.

Good Hypothesis Characteristics:

• Explains ALL observed symptoms
• Testable with available tools
• Specific about root cause (not vague)
• Based on evidence, not assumptions

Example Hypotheses:

• "Database default is 0, but code checks for null"
• "Function never reaches line X because condition Y is always false"
• "Environment variable Z is not set in production"
• "RLS policy blocks access for this user role"

Testing Without Logs:

• Use database writes as "print statements"
• Add error responses with diagnostic info
• Create minimal reproduction cases
• Test conditions in isolation

Phase 4: Implementation & Verification

Fix the root cause and verify across all layers.

Fix Implementation:

• Make minimal, targeted changes
• Document the bug and fix
• Update tests to prevent regression
• Deploy new version

Verification:

• Database state updates as expected
• Feature triggers in expected scenarios
• No side effects on other features
• End-to-end test passes

Common Serverless Bugs

See references/common-serverless-bugs.md for detailed catalog. Quick reference:

Type Coercion Issues:

• 0 === null → false (check for both)
• undefined === null → false
• Empty string vs null vs undefined
• Number strings vs numbers ("0" vs 0)

Async/Timing Issues:

• Fire-and-forget operations that fail silently
• Race conditions in parallel operations
• Timeout before completion
• Promise not awaited

Environment Differences:

• Console.log works locally but not in production
• Environment variables not set
• Different runtime versions
• Permission differences

Database Issues:

• RLS policies blocking access
• Default values vs. expected values
• Transaction isolation issues
• Connection pool exhaustion

Alternative Debugging Strategies

When console.log doesn't work, use these alternatives:

Database-Driven Debugging:

-- Create debug log table
CREATE TABLE debug_logs (
  id SERIAL PRIMARY KEY,
  timestamp TIMESTAMPTZ DEFAULT NOW(),
  function_name TEXT,
  checkpoint TEXT,
  data JSONB
);

-- Write debug info from code
INSERT INTO debug_logs (function_name, checkpoint, data)
VALUES ('my-function', 'before-extraction', jsonb_build_object('count', message_count));

Error Response Debugging:

// Return diagnostic info in error response
return new Response(JSON.stringify({
  debug: {
    messageCount: messageCount,
    lastExtraction: lastExtraction,
    shouldExtract: shouldExtractMemories(messageCount, lastExtraction)
  }
}), { status: 200 });

Code Analysis:

• Read deployed code directly (if accessible)
• Trace execution paths on paper
• Check type of every variable in conditions
• Verify all branches can be reached

See references/logging-alternatives.md for more strategies.

Practical Tools

Deployment Verification:

# Use scripts/check_deployment.py
python scripts/check_deployment.py <function-name> <local-path>

Database State Analysis:

# Use scripts/database_debugger.py
python scripts/database_debugger.py <table-name> <condition>

Type Analysis:

# Use scripts/type_analyzer.py
python scripts/type_analyzer.py <file-path> <function-name>

Real-World Example: Wade Memory Extraction Bug

Symptom: Proactive memory extraction never triggered despite code being deployed.

Phase 1 Evidence:

• ✅ Code deployed (v48)
• ✅ Database schema correct
• ✅ Messages stored (8-10 messages)
• ❌ Console.log never appeared
• ❌ last_extraction_at stayed at 0

Phase 2 Pattern:

• Console.log doesn't work in Supabase Edge Functions logs API
• Needed alternative debugging approach

Phase 3 Hypothesis:

• Database default for last_extraction_at is 0, not null
• Code checks if (lastExtraction === null) → always false when 0
• Takes wrong branch: messageCount - 0 >= 10 instead of messageCount >= 4

Phase 4 Fix:

// Before: Only checked null
if (lastExtractionAt === null) {
  return messageCount >= 4;
}

// After: Check both null AND 0
if (lastExtractionAt === null || lastExtractionAt === 0) {
  return messageCount >= 4;
}

Result: Feature now works correctly, triggers after 4 messages.

Best Practices

1. Gather evidence before hypothesizing - Resist the urge to guess
2. Use database as debugging tool - When logs fail, database writes work
3. Check type coercion carefully - 0, null, undefined are all different
4. Verify deployed code - Don't assume local matches production
5. Test in isolation - Minimal reproduction cases reveal root cause
6. Document the process - Future debugging benefits from past patterns

Quick Start

1. Read references/evidence-checklist.md and gather evidence from all layers
2. Identify patterns using Phase 2 questions
3. Form testable hypothesis
4. Use scripts/ tools to verify hypothesis
5. Implement fix and verify across all layers
6. Document bug and fix for future reference

When 3+ Fixes Fail: Question the Architecture

Red flag pattern: Multiple fix attempts, each revealing new problems in different layers.

Symptoms

• ✅ Fix #1: Added environment variables → Still failing
• ✅ Fix #2: Force redeployed → Still failing
• ✅ Fix #3: Added to all environments → Still failing
• ✅ Fix #4: Created lazy initialization → Deployment failed
• ✅ Fix #5: Tried different deployment method → New error

Pattern: Not converging on a solution. Each fix reveals a NEW problem.

What This Means

Per systematic-debugging Phase 4.5:

If 3+ fixes failed: STOP and question the architecture

This is NOT a bug. This is an ARCHITECTURAL PROBLEM.

Root Cause Analysis

The real issues:

1. Generic error messages - Can't see actual problem
2. No direct code access - Can't iterate quickly
3. Complex deployment pipeline - Adds failure points
4. No local testing - Can't reproduce issue
5. Limited observability - Can't see runtime state

What to Do

STOP attempting more fixes. Instead:

1. Get better error visibility

   • Add comprehensive error logging (see references/enhanced-error-logging.md)
   • Access full logs (not truncated)
   • Add diagnostic endpoints

2. Simplify the workflow

   • Test locally first
   • Use proper version control (GitHub)
   • Set up CI/CD pipeline
   • Remove manual deployment steps

3. Question the fundamentals

   • Is this deployment method sound?
   • Are we fighting the platform?
   • Should we refactor the approach?

Real-World Example: Stripe Integration

Attempts:

1. Added Stripe keys to Vercel → Still failing
2. Force redeployed → Still failing
3. Added keys to all environments → Still failing
4. Created lazy initialization fix → Deployment failed
5. Tried Google Drive transfer → Downloaded 0 bytes

Root cause: Generic error message hid the actual Stripe error

Solution: Add comprehensive error logging to see real error, then fix that

Lesson: After 3 failed fixes, we should have stopped and added better logging FIRST

Key Takeaways

• 3+ failed fixes = architectural problem
• Stop fixing, start observing
• Add logging before adding fixes
• Question the deployment workflow
• Test locally before deploying

See also: systematic-debugging skill Phase 4.5

Lazy Initialization for External Services

Problem: Module-level initialization fails in serverless environments.

// ❌ DON'T: Initialize at module load
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

Why it fails:

• Environment variables may not be ready at cold start
• No validation that keys exist
• Initialization errors hidden
• Cached instances retain old values

Solution: Lazy initialization pattern

// ✅ DO: Initialize on first use
let stripeInstance = null;

function getStripe() {
  if (!stripeInstance) {
    const key = process.env.STRIPE_SECRET_KEY;
    if (!key) throw new Error('STRIPE_SECRET_KEY not set');
    stripeInstance = require('stripe')(key);
  }
  return stripeInstance;
}

Benefits:

• ✅ Environment variables validated
• ✅ Clear error if missing
• ✅ Initialization after env vars loaded
• ✅ Proper error logging

See references/lazy-initialization-patterns.md for complete examples.

Enhanced Error Logging

Problem: Generic error messages hide root cause.

// ❌ DON'T: Hide error details
catch (error) {
  res.status(500).json({ error: 'An error occurred' });
}

Solution: Log full error details

// ✅ DO: Log comprehensive error info
catch (error) {
  console.error('Operation failed:', {
    message: error.message,
    type: error.type,
    code: error.code,
    stack: error.stack,
    context: { priceId, userId }
  });
  res.status(500).json({ 
    error: error.message,
    type: error.type,
    code: error.code
  });
}

See references/enhanced-error-logging.md for patterns.

Diagnostic Health Endpoints

Add health check endpoints to verify configuration:

app.get('/api/health', (req, res) => {
  const config = {
    stripe: !!process.env.STRIPE_SECRET_KEY,
    supabase: !!process.env.SUPABASE_URL
  };
  
  const keyPrefixes = {
    stripe: config.stripe ? process.env.STRIPE_SECRET_KEY.substring(0, 7) : 'none'
  };
  
  res.json({ 
    status: 'ok',
    configured: config,
    keyPrefixes: keyPrefixes
  });
});

Usage:

curl https://your-api.vercel.app/api/health

This immediately shows:

• Which services are configured
• Key prefixes (safe to log)
• Whether environment variables are set