lidge-jun/dev-backend

Dev-Backend — Production-Grade Backend Engineering

Build reliable, secure, and maintainable server-side applications. This skill has modular references for specialized guidance — read the relevant ones before coding. It activates by change surface whenever work primarily touches APIs, servers, services, jobs, data access, schemas, migrations, or operational backend behavior.

C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev is canonical: dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.

Modular References

| File | When to Read | What It Covers | | -------------------------------------- | ------------------------------ | ---------------------------------------------------------------------- | | references/core/crud-api.md | C2 ordinary CRUD/resource endpoints | Route/schema/service/query basics, five operations, error+permission mapping | | references/core/api-design.md | New/changed API style, or C3+ API work (C2 ordinary slice: crud-api.md alone suffices) | REST conventions, response envelopes, HTTP status, pagination, GraphQL, gRPC, tRPC | | references/core/api-lifecycle.md | API versioning, deprecation, migration | Versioning strategy, RFC 9745/8594 lifecycle, oasdiff CI gate | | references/core/architecture.md | New features at C3+ (C2 ordinary slice: crud-api.md alone suffices) | Layered architecture, DDD, SOLID, when to split, monolith vs micro | | references/core/anti-slop-backend.md | New endpoints, classes, or modules | Banned patterns: god classes, raw SQL in services, magic numbers, etc. | | references/core/observability.md | Production deployments | OpenTelemetry, structured logging, distributed tracing, alerting | | references/core/health-checks.md | Production/long-lived services | Liveness, readiness, startup probes, dependency checks | | references/core/process-isolation.md | CPU-bound or untrusted work | worker_threads vs child_process vs separate service, communication, resource limits | | references/core/caching.md | Performance optimization | Redis-compatible (Valkey/Redis) patterns, CDN, connection pooling, cache invalidation | | references/stacks/node.md | Node.js/TypeScript projects | Express/Fastify, middleware, Zod validation, ESM, error handling | | references/stacks/python.md | Python projects | FastAPI/Django, Pydantic, async patterns, testing | | references/stacks/database.md | Database design/optimization | PostgreSQL, MongoDB, indexing, N+1, migrations, ORM comparison | | references/core/ml-serving.md | ML model deployment, GPU inference | vLLM/SGLang runtime selection, FastAPI+GPU patterns, dynamic batching, quantization | | references/core/llm-integration.md | RAG, LLM API integration, prompt engineering | Chunking, hybrid search, vector DB, structured output, LangChain/LlamaIndex 2026 | | references/core/mobile-api.md | Mobile app API patterns | BFF, push notifications, offline sync, mobile auth, API optimization |

Read api-design.md + anti-slop-backend.md first, then the relevant stack file. For C2 ordinary slices, crud-api.md alone suffices; read api-design.md/architecture.md for new API styles or C3+ work.

When backend decisions depend on current external API docs, API lifecycle changes, LLM/RAG provider behavior, dependency freshness, or package/source evidence, read the active search skill and follow its query-rewrite, source-fetch, and evidence-status rules.

---

0. Stack Detection & Architecture Clarification

Auto-detect (existing projects)

| File Found | Project Type | | ----------------------------------- | ----------------- | | tsconfig.json | TypeScript (Node) | | package.json (no ts) | JavaScript (Node) | | pyproject.toml/requirements.txt | Python | | go.mod | Go | | Cargo.toml | Rust |

If config files exist → detect silently and proceed.

Architecture Clarification (new or ambiguous projects)

When the request has unspecified technology or unclear scope, clarify before coding:

1. Identify what's ambiguous from this list:

| Dimension | Options to present | | ------------ | ----------------------------------------------------------------------------------- | | API style | REST (default) · GraphQL (BFF/mobile) · gRPC (internal microservices) · tRPC (TS monorepo) | | Database | PostgreSQL (default, ACID) · MongoDB (flexible schema) · SQLite (embedded) | | Auth method | JWT + refresh (stateless) · Session-based (simple) · OAuth 2.1 (3rd party) | | Realtime | Not needed (default) · WebSocket · SSE · Polling | | Architecture | Monolith (default) · Modular monolith · Microservices |

2. Recommend one with reasoning: cite project context. e.g., "Small team → monolith + PostgreSQL + JWT is the simplest starting point."
3. Over-engineering guard: A CRUD API probably doesn't need GraphQL + microservices + event sourcing. Simple → complex, not the reverse.
4. One round limit: 2-3 options → recommend → confirm → proceed.

If the user already specifies clear tech (e.g. "FastAPI로 REST API 만들어줘"), skip this entirely.

Node/framework defaults (verified 2026-07-02): production uses Active/Maintenance LTS — Node 24 (Active LTS) for new services, Node 22 (Maintenance). Framework: Fastify for greenfield Node APIs (schema/Pino/plugin structure); Express 5 for legacy/ecosystem compatibility; Hono for edge/serverless/multi-runtime Web-Standards APIs. New TS validation baseline is Zod v4 (read the migration guide before upgrading v3 projects). Sources: references/stacks/node.md § Sources.

For new Node backend source files, prefer .ts when the repo supports TypeScript or is greenfield. Inherit dev TypeScript strict-compatibility rules. If backend boundaries are unclear, read existing source-of-truth docs/logs first, then document routes, services, repositories, data stores, and runtime commands in the repo's existing SOT before broad implementation.

---

1. Architecture Decision

Before coding, identify the right pattern:

| Team Size | Default Starting Point | | --------- | ----------------------- | | 1-3 devs | Modular monolith | | 4-10 devs | Modular monolith or SOA | | 10+ devs | Consider microservices |

Default to monolith. Extract only when you have a proven need (different scaling, independent deployment, technology mismatch).

See references/core/architecture.md for full decision matrices.

API Protocol Decision

| Protocol | Choose When | Avoid When | |----------|-------------|------------| | REST | Public/partner APIs, simple CRUD, caching matters | Clients need flexible data shapes | | GraphQL | Mobile/BFF client aggregation, multiple resources per request | Simple CRUD, server-to-server, file uploads | | gRPC | Internal microservices, high-perf binary, bidirectional streaming | Browser clients (without gRPC-Web), public APIs | | tRPC (v11) | TypeScript shared end-to-end: monorepo, internal tools | Polyglot environments, public APIs |

Hybrid pattern (verified 2026-07-02):

Public/Partner → REST (OpenAPI 3.1+; prefer 3.2 where tooling supports it)
Mobile/Web BFF → GraphQL (client aggregation; Apollo Federation ONLY when multiple
                 independently-owned subgraphs must compose into a supergraph)
Internal services → gRPC (Protobuf contracts)
TS internal tools → tRPC v11 (zero-codegen type safety)

Deprecations: emit Deprecation (RFC 9745) and optional Sunset (RFC 8594) headers, but client adoption is uneven — always pair with OpenAPI/changelog dates and deprecated- endpoint traffic dashboards.

See references/core/api-design.md for protocol-specific patterns.

Long-Lived Connection Operation

Rules for SSE, WebSocket, and any connection held open beyond a single request-response cycle.

Lifecycle Rules:

| Parameter | Default | Rationale | |-----------|---------|-----------| | Heartbeat interval | 15-30s | Detect dead connections before TCP timeout (varies by proxy) | | Reconnection backoff | Exponential 1s-30s with jitter | Prevent thundering herd on server restart | | Max connection duration | 1h (SSE), 24h (WebSocket) | Force reconnect to rebalance and prevent memory leaks | | Connections per client | Cap at 6 (SSE) or 1-2 (WebSocket) | Browser limits + server memory budget |

Server-Side Requirements:

• Connection registry: Track all active connections in-memory (Map by client/session ID). Required for graceful drain and debugging.
• Graceful drain on deploy: Stop accepting new connections → send "reconnect" frame to existing → wait drain timeout → close.
• Memory budget: Allocate max memory per connection (e.g., 2KB buffer). Monitor total; reject new connections when approaching limit.
• Backpressure: If client stops consuming, buffer up to N messages then drop oldest or disconnect.

Pattern — "202 + Job ID" for Long Operations:

Instead of holding a connection open for a slow operation:

POST /generate → 202 { jobId: "j_abc123" }
GET /jobs/j_abc123 → { status: "processing", progress: 0.6 }
                   → { status: "complete", result: {...} }

Use SSE/WebSocket only for push notifications about job status — not for the operation itself.

Banned:

| Banned | Fix | |--------|-----| | Unbounded connections (no cap, no registry) | Connection registry + cap per client + global max | | No heartbeat (rely on TCP keepalive only) | Application-level heartbeat every 15-30s | | Blocking event loop per connection (sync work in message handler) | Offload to worker thread or queue; handler stays async | | Holding connection open for >5s synchronous work | Return 202 + job ID; notify via push when done | | No reconnection logic on client side | Implement exponential backoff with jitter |

---

2. Layered Architecture (Default; Allow Serverless Handlers, Vertical Slices, and Small Scripts When Appropriate)

Routes → Controllers → Services → Repositories → Database
  │          │             │            │
  │          │             │            └── Data access only
  │          │             └── Business logic (validation at controller boundary — service trusts caller per dev-architecture §4)
  │          └── Parse HTTP, format response
  └── URL mapping, middleware

Rules:

• Routes: URL patterns + middleware only. No logic.
• Controllers: parse input, call services, format output. No business rules.
• Services: receive/return plain data (not req/res). All logic here.
• Repositories: abstract DB access. Services access data through repositories only.

Repository Pattern (Interface Abstraction)

Use repository interfaces so services depend on abstractions, enabling mocking and swapping implementations.

Async Task Queue Patterns

When work exceeds what an HTTP response cycle should hold open, use a queue.

Decision: Queue vs Direct:

| Condition | Use Queue | Use Direct | |-----------|-----------|------------| | Execution time >5s | Yes | No | | Must be retryable on failure | Yes | No | | Fire-and-forget (caller doesn't wait) | Yes | No | | <1s, idempotent, caller needs immediate result | No | Yes | | Real-time user-facing validation | No | Yes |

Pattern — Accept, Queue, Notify:

1. Client  → POST /tasks       → Server validates, enqueues
2. Server  → 202 { jobId }     → Client receives immediately
3. Worker  → picks from queue  → executes task
4. Client  → GET /tasks/{id}   → polls status (or receives webhook/SSE push)
5. Worker  → completes         → writes result, triggers notification

Queue Selection Guide:

| Queue | When | Notes | |-------|------|-------| | BullMQ (Redis-compatible) | Node.js, need retries + priorities + rate limiting | Most mature Node queue; needs Redis/Valkey | | Celery (Redis/RabbitMQ) | Python, distributed workers, periodic tasks | De facto Python standard | | pg-boss (PostgreSQL) | Node.js, Postgres is the durable system of record, moderate scale | No extra infra; SKIP_LOCKED-based | | Simple DB queue (polling) | Small scale (<100 jobs/min), any language | status column + SELECT FOR UPDATE SKIP LOCKED | | SQS / Cloud Tasks | Serverless, managed, very high scale | No infra to manage; at-least-once delivery | | Temporal | Durable multi-step workflows: sagas, human-in-loop, long-running AI/business processes | Workflow engine, NOT a default queue replacement |

Required Safeguards:

| Safeguard | Rule | |-----------|------| | Idempotency key | Every enqueue call must include a unique idempotency key; dedup on insert | | Dead letter queue (DLQ) | Failed 3x (configurable) → move to DLQ → alert → manual review | | Max retries | Set explicit limit (default: 3); exponential backoff between attempts | | Timeout per job | Every job has a max execution time; kill and retry on exceed | | Visibility timeout | Lock duration > expected execution time; prevent duplicate processing | | Observability | Emit metrics: queue depth, processing time p95, DLQ size, failure rate |

Banned:

| Banned | Fix | |--------|-----| | Synchronous long operation blocking HTTP response (>5s) | Enqueue + return 202 + job ID | | Queue without DLQ | Always configure DLQ; alert on DLQ depth > 0 | | Infinite retries (no max) | Set maxRetries=3 with exponential backoff | | No idempotency (duplicate jobs on retry) | Idempotency key on every enqueue; dedup in worker | | No timeout on job execution | Set per-job timeout; kill + mark failed on exceed | | Polling without backoff (tight loop) | Poll with interval (1-5s) or use blocking pop / push notification |

---

3. Error Handling

| Type | HTTP | Log Level | | -------------- | ---- | ------------- | | Validation | 400 | warn | | Authentication | 401 | warn | | Authorization | 403 | warn | | Not found | 404 | info | | Conflict | 409 | warn | | Rate limit | 429 | info | | Internal error | 500 | error + stack |

Use a centralized AppError class (DEFAULT — when the repo already has an error convention, follow it instead). Distinguish operational vs programmer errors.

Error Taxonomy (AppError Hierarchy)

Create an AppError base class with statusCode, code, and isOperational properties. Extend for each error type (ValidationError, NotFoundError, etc.).

Result Pattern (conditional)

Consider the Result/Either pattern (e.g. neverthrow) for recoverable domain errors where explicit error handling improves clarity — adopt it only when the project scope justifies it and the repo doesn't already settle error style (HEURISTIC, not a universal requirement).

| Library | When to Use | |---------|-------------| | neverthrow | Default choice — small explicit Result<T, E> for recoverable domain errors | | Effect | Only when the app benefits from a full effect runtime: typed errors, retries, resources, concurrency, tracing, service composition |

Rule: Use Result where recoverable/domain errors are first-class. Reserve try/catch for error boundaries (middleware, top-level handlers) only.

---

4. Middleware Execution Order

Apply in this sequence (order matters):

1. Request ID generation
2. Request logging
3. Security headers (CORS, CSP, HSTS)
4. Rate limiting
5. Authentication
6. Authorization
7. Body parsing
8. Input validation (schema)
9. Route handler
10. Error handler
11. Response logging

---

5. API Response Contract

API endpoints should use a stable response envelope (DEFAULT) unless the protocol (GraphQL, gRPC, SSE) defines its own or the repo already has a different established contract — follow the existing contract first. Envelope, OTel, health checks, and deployment-readiness checks are production-surface concerns (dev §0.4 shared definition), conditional by project scope, not universal blockers.

Rules:

• success boolean at top level — never infer from HTTP status alone
• error.code is machine-readable (UPPER_SNAKE), error.message is human-readable
• meta.requestId on every response — enables cross-service tracing
• Pagination uses cursor-based (after/before) for large datasets, offset-based (page/pageSize) for admin UIs
• Nullability: prefer consistent key presence; omit when sparse payloads are intentional
• Timestamps: ISO 8601 UTC (2024-01-15T09:30:00Z), never Unix epoch in JSON
• Money: integer cents + currency code, never floating point

See references/core/api-design.md for protocol-specific patterns (REST, GraphQL, gRPC, tRPC).

---

6. Caching Strategy

Decision rules:

• Say Redis-compatible, not Redis-only (verified 2026-07-02): prefer Valkey (Linux Foundation, BSD) for permissive OSS/self-hosted defaults; choose Redis when managed-service, module, or license posture justifies it (Redis relicensed 2024; Redis 8 added AGPLv3).
• Cache only after correctness is proven on the uncached path.
• Prefer cache-aside by default; use write-through only when strong consistency matters.
• Every key has a namespace, version, stable identifier, TTL, and invalidation trigger.
• Never cache error responses or personalized CDN responses; protect cached PII with encryption and access controls.
• Add stampede protection for hot keys and monitor hit rate, pool exhaustion, and stale-read incidents.

See references/core/caching.md for TTL guidance, Redis-compatible cache patterns, CDN rules, invalidation triggers, connection pooling, and code examples.

---

7. Observability (OpenTelemetry)

Decision rules:

• Production services emit traces, metrics, and structured JSON logs with requestId, traceId, and spanId.
• OTel maturity (verified 2026-07-02): traces and metrics are Stable in JS/Python; logs are still Development — the baseline is trace/span-correlated structured logs + OTel traces/metrics; adopt OTel Logs export only where that maturity is acceptable.
• Start with OTel auto-instrumentation, then add custom spans only for business-critical or non-instrumented work.
• Never log PII, secrets, full request/response bodies, or noisy stack traces outside error boundaries.
• Page only on customer-impacting signals tied to SLOs; use warning alerts for capacity trends.

See references/core/observability.md for OTel setup, structured logging conventions, trace propagation, dashboards, RUM correlation, and alerting guidance.

---

8. Skeleton Project Evaluation

When starting from a template or boilerplate, verify before building on top:

| Check | What to Verify | |-------|---------------| | Dependencies | Up-to-date? CVEs? Unnecessary packages? | | Architecture fit | Does the template's structure match your actual needs? | | Auth/security | Is the auth pattern appropriate for your use case? | | Database | Is the ORM/query builder suitable for your data model? | | Dead code | Remove unused example routes, models, and middleware | | Config management | Environment-based config, no hardcoded values |

Treat templates as starting points, not gospel. Strip to essentials, then add what you need.

---

9. API Performance Targets (HEURISTIC defaults, not universal SLOs)

These are aggressive healthy-service starting budgets. Each product MUST define its own SLOs from user journeys and alert on error-budget burn, not raw percentile thresholds alone (Google SRE SLO practice; see observability.md Sources).

| Metric | Default budget | Escalation | |--------|--------|-----------| | p50 response time (reads) | ≤ 50ms | Profile with tracing | | p95 response time (reads) | ≤ 200ms, alert at >500ms (see observability.md) | Optimization target | | p95 response time (writes) | ≤ 500ms | Acceptable for complex writes | | p99 response time | ≤ 1000ms | Investigate outliers | | Error rate | < 0.1% target, alert at >1% (see observability.md) | Optimization target |

• Measure at the handler level, not including network
• Use Server-Timing header to expose backend timing to frontend
• Log slow queries (> 100ms) with EXPLAIN output
• Connection pool (long-running servers): min = CPU cores, max = CPU cores × 4; for serverless/Lambda use min = 0–2

API responses that drive UI must include descriptive error messages (not just codes) for screen reader announcement, pagination metadata (total count) for assistive technology, and Content-Language header matching response body language.

10. SEO Support Endpoints

When the app serves web pages (SSR/SSG):

• GET /sitemap.xml — dynamic sitemap generation with <lastmod>
• GET /robots.txt — configurable per-environment (disallow staging/preview)
• Structured data: provide JSON-LD data in API responses when frontend needs it
• Redirect chains: max 1 hop (301 for permanent, 308 for POST-preserving)

11. Deployment Patterns

• Blue-green: deploy to inactive environment, verify health + smoke tests, swap traffic
• Canary: route 5% → 25% → 100% with automated rollback on error rate spike
• Rollback: always keep previous version deployed and ready to swap back
• Database migrations: separate from code deploy, backward-compatible (expand-then-contract)
• Feature flags: use for gradual rollout of risky changes

---

12. Pre-Flight Checklist

• Analytical data / ETL / pipeline quality: load dev-data.
• API consumer context / frontend contract alignment: load dev-frontend.
• Test strategy / verification harnesses / QA execution: load dev-testing.
• Backend failure RCA methodology: load dev-debugging.
• New project setup / file placement conventions: load dev-scaffolding.

Before delivering:

• [ ] Consistent response envelope on every endpoint
• [ ] Input validation with schema (Zod, Pydantic, etc.)
• [ ] Authentication middleware on protected routes
• [ ] Rate limiting on public endpoints
• [ ] Structured JSON logging with requestId and traceId
• [ ] Error handler returns proper HTTP codes via AppError hierarchy
• [ ] No raw SQL in service layer
• [ ] No hardcoded secrets
• [ ] Migrations have rollback
• [ ] Observability: traces and structured logs wired (see references/core/observability.md)
• [ ] Health endpoints: /health (liveness) and /ready (readiness) — see references/core/health-checks.md
• [ ] API performance: p95 reads ≤ 200ms, slow queries logged with EXPLAIN (§9)
• [ ] SEO endpoints: sitemap.xml + robots.txt if serving web pages (§10)
• [ ] Security review: delegate to dev-security/SKILL.md for production readiness
• [ ] Stack-specific rules followed (see references/stacks/)