harsh040506/performance-profiling

Performance Profiling

Systematically identify and eliminate performance bottlenecks using measurement-first principles.

---

The Performance Investigation Process

1. Measure → establish baseline
2. Identify → which layer is slow?
3. Profile → what is causing it within that layer?
4. Optimize → targeted fix
5. Verify → measure again, compare to baseline
6. Don't guess → return to step 1 if uncertain

The rule: measure before and after every change. Intuition about performance is usually wrong.

---

Baseline Measurement

Establish quantitative baselines before touching anything:

HTTP Services

# wrk (full featured)
wrk -t4 -c50 -d30s \
  -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/endpoint

# hey (simpler, good output)
hey -n 10000 -c 100 -m GET http://localhost:3000/api/endpoint

# k6 (scriptable, integrates with CI/CD)
k6 run script.js

# Apache Bench
ab -n 10000 -c 100 http://localhost:3000/api/endpoint

Record all of: P50, P95, P99, RPS, error rate.

Database Queries

-- PostgreSQL: measure a specific query
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) 
SELECT * FROM orders WHERE customer_id = 12345 ORDER BY created_at DESC LIMIT 20;

-- Look for: total execution time, number of rows, Seq Scan vs Index Scan, buffer cache hits
-- Target: execution time < 100ms for single queries, < 10ms for hot paths

-- Find slowest queries (requires pg_stat_statements extension)
SELECT 
  round(mean_exec_time::numeric, 2) AS avg_ms,
  calls,
  round(total_exec_time::numeric, 0) AS total_ms,
  left(query, 100) AS query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

CLI Programs / Scripts

# Unix time
time python script.py

# Detailed system resource usage
/usr/bin/time -v python script.py  # Linux
/usr/bin/time -l python script.py  # macOS

# Wall time vs CPU time — if wall >> CPU, you're waiting on I/O

---

Identifying the Bottleneck Layer

Before profiling application code, determine which layer is responsible.

The Performance Stack

User browser / client
    ↓ DNS + TLS + network
Load balancer
    ↓
Application server (CPU / memory / event loop)
    ↓
Database (disk I/O / CPU / indexes / locks)
    ↓
External APIs (network + their performance)
    ↓
Cache layer (Redis / Memcached)

Diagnostic Approach

# Is it the network?
# Compare: curl timing from close server vs. far
curl -o /dev/null -s -w "DNS: %{time_namelookup}s | Connect: %{time_connect}s | TLS: %{time_appconnect}s | First byte: %{time_starttransfer}s | Total: %{time_total}s\n" https://api.example.com/endpoint

# Is it CPU?
# Check CPU during load test
top -b -n 3 | grep "Cpu\|%CPU"
# >80% CPU → application CPU bound, profile the code
# <30% CPU but slow → I/O bound (database, network, disk)

# Is it memory / GC?
# Node.js
node --max-old-space-size=4096 --expose-gc server.js
# Trigger GC manually, watch pause duration

# Is it database?
# Add timing around DB calls in application code
# Look for: individual queries > 100ms, many small queries (N+1)

# Is it an external API?
# Add per-call timing with a span or log statement
const start = Date.now();
const result = await externalApi.call(params);
logger.info({ duration_ms: Date.now() - start, api: 'payment-provider' }, 'API call completed');

---

Language-Specific Profiling Techniques

JavaScript / Node.js

CPU profiling:

# Built-in profiler (good for getting started)
node --prof --prof-process server.js
# Load test → Ctrl+C → converts .log to readable text

# 0x (flamegraph — best visual tool)
npx 0x --output-dir ./profile -- node server.js
# Load test → Ctrl+C → opens flamegraph in browser

# Clinic.js (comprehensive diagnostics)
npx clinic flame -- node server.js
npx clinic doctor -- node server.js   # Identifies event loop lag, GC issues, I/O

Event loop lag (the Node.js killer):

// Detect event loop lag
const { monitorEventLoopDelay } = require('perf_hooks');
const histogram = monitorEventLoopDelay({ resolution: 10 });
histogram.enable();

setInterval(() => {
  const p99Lag = histogram.percentile(99) / 1e6; // Convert ns to ms
  if (p99Lag > 10) {
    logger.warn({ event_loop_lag_p99_ms: p99Lag }, 'Event loop lag detected');
  }
  histogram.reset();
}, 5000);

CommonNode.js performance issues:

• Synchronous I/O (fs.readFileSync, fs.writeFileSync) in request handlers
• Large JSON.parse/stringify in the hot path
• Crypto operations without worker threads
• Blocking DNS lookups (dns.lookup instead of dns.resolve)

Python

# cProfile — find which functions are slow
import cProfile, pstats, io

def profile_function(func, *args, **kwargs):
    pr = cProfile.Profile()
    pr.enable()
    result = func(*args, **kwargs)
    pr.disable()
    
    s = io.StringIO()
    stats = pstats.Stats(pr, stream=s)
    stats.sort_stats('cumulative')
    stats.print_stats(30)
    print(s.getvalue())
    return result

# line_profiler — find which LINES are slow
# pip install line_profiler
@profile  # Decorator added by line_profiler
def slow_function(data):
    results = []
    for item in data:
        results.append(process(item))  # Is this the slow line?
    return results

# py-spy — attach to running process without restarts
# (Production-safe profiler)
py-spy top --pid <PID>                  # Like 'top' but for Python functions
py-spy record --output flame.svg --pid <PID>  # Flamegraph of running process

Common Python performance issues:

• Using CPython for CPU-intensive loops (use NumPy/Cython/Rust extension instead)
• Creating lists/dicts in tight loops instead of using generators
• Not using __slots__ for classes with many instances
• String concatenation with + in loops (use join())
• Missing asyncio for I/O-bound code

Go

// Add pprof to your server
import (
    "net/http"
    _ "net/http/pprof"  // This is all you need
)

// In main:
go http.ListenAndServe("localhost:6060", nil)

# Capture 30-second CPU profile
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Check heap (memory usage)
go tool pprof http://localhost:6060/debug/pprof/heap

# Goroutine leaks (goroutines that never stop)
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Interactive flamegraph
go tool pprof -http=:8080 cpu.prof

Common Go performance issues:

• Escape analysis causing unexpected heap allocation — use go build -gcflags="-m" to see
• Goroutine leaks — channels that are never read from
• Mutex contention — use sync/atomic for simple counters
• fmt.Sprintf in hot paths — use strings.Builder instead

---

Database Performance

Reading EXPLAIN ANALYZE Output

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE customer_id = 5;

Key things to look for:

| Term | What it means | Action if bad | |------|-------------|--------------| | Seq Scan on large table | Reading entire table (no index) | Add index | | Index Scan | Using index — good | Fine | | Rows Removed by Filter: 50000 | Index exists but not selective enough | Better index or composite index | | Buffers: shared hit=0 read=5000 | Most blocks read from disk | Increase shared_buffers, add index | | Actual time ≫ estimated time | Stale statistics | ANALYZE table_name |

Index Design Principles

-- Equality first, range second, highest cardinality first
-- Query: WHERE user_id = ? AND status = 'active' AND created_at > ?
CREATE INDEX idx_orders_user_status_date
  ON orders(user_id, status, created_at)
  WHERE deleted_at IS NULL;  -- Partial index — smaller, faster for common filter

-- Use CONCURRENTLY to avoid table lock in production
CREATE INDEX CONCURRENTLY idx_name ON table(column);

Connection Pool Tuning

Too few connections: requests queue up, latency spikes under load
Too many connections: database thrashes, memory exhaustion

PostgreSQL max_connections × 0.8 = usable connections
Distribute across all app instances

Example: 100 max_connections × 0.8 = 80 usable
With 4 app instances: pool_size = 20 per instance

// node-postgres connection pool
const pool = new Pool({
  max: 20,                  // Max connections per pool
  idleTimeoutMillis: 30000, // Close idle connections after 30s
  connectionTimeoutMillis: 2000, // Fail fast if pool is full
});

---

Caching Strategy

When to Cache

Cache when:

• Data is read far more than it's written
• Computation or query is expensive
• Slight staleness is acceptable
• Data can be stored in a key-value structure

Don't cache when:

• Data must be real-time accurate
• Dataset is huge and hit rate would be very low
• Cache invalidation logic is complex enough to introduce bugs

Cache Levels

Browser cache → CDN → Application cache (in-process) → Redis → Database

Use the closest cache that can serve the request with acceptable freshness.

Redis Caching Pattern

const CACHE_TTL = 300; // 5 minutes

async function getCachedData(key: string): Promise<Data> {
  // Check cache
  const cached = await redis.get(key);
  if (cached !== null) {
    return JSON.parse(cached);
  }
  
  // Cache miss — fetch from source
  const data = await fetchFromDatabase(key);
  
  // Store with TTL
  await redis.setex(key, CACHE_TTL, JSON.stringify(data));
  
  return data;
}

// Cache invalidation on write
async function updateData(id: string, updates: Partial<Data>): Promise<void> {
  await db.update(id, updates);
  await redis.del(`data:${id}`);   // Invalidate specific key
  await redis.del(`data:list:*`);  // Invalidate list keys (use SCAN for production)
}

---

Performance Optimization Reference

| Issue | Symptom | Fix | Expected gain | |-------|---------|-----|--------------| | N+1 queries | Many small DB queries per request | Eager loading, batch queries | 10–100× | | Missing index | Seq scan on large table | Add targeted index | 10–1000× | | Synchronous I/O in async code | Event loop lag | Move to worker thread | 5–50× | | No caching of expensive data | Same query repeatedly | Add Redis cache | 10–100× | | Serialized I/O | Sequential awaits that could be parallel | Promise.all, goroutines | Proportional to I/O count | | CPU-bound in interpreted language | High CPU, slow loops | Compiled extension, SIMD, batch | 10–100× | | GC pressure | Periodic latency spikes | Reduce allocations, tune GC | 2–10× | | Connection pool exhaustion | Requests queuing | Increase pool size (within DB limits) | Eliminates queuing | | Large payload | Slow network transfer | Pagination, compression, CDN | Proportional to payload size | | DNS lookups per request | Consistent added latency | Connection pooling / keep-alive | 10–200ms |

Deeper Reference

For ready-to-run profiling commands across languages and runtimes, see:

• references/profiling-commands.md — copy-paste profiling commands for Python (cProfile, py-spy), Go (pprof), Node.js (clinic.js, 0x), PostgreSQL slow query analysis, and Linux system profiling (perf, flamegraph)