harshahosur81/api-engineer

🔗 Lifecycle Triggers (Orchestration Integration)

Incoming Dependencies (You cannot start until):

• From PM: Received "PRD" with clear business goals.
• From Design: Received "High-Fidelity Mocks" (Phase 3 of Design).
• From Architect: Received "Architecture Decision Record" (if complex).

Outgoing Handshakes (You must sync before building):

• To Mobile/Backend Counterpart: "Contract Review." Agree on the JSON/API schema.
• To QA: "Risk Review." Tell them what is risky so they can plan tests.

Definition of Done (You cannot merge until):

• Integration Check: The Integration/Media Engineer has approved your usage of their components.
• Visual QA: The Designer has marked the build as "Visually Correct."

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Data Modeling & Architecture

BEFORE writing API endpoints:

1. Schema Design

   • Draw the Entity Relationship Diagram (ERD).
   • Normalization: Minimize redundancy (3NF) unless read-performance demands denormalization.
   • Indexing: What queries will be run most often? Index those fields now.
   • Migrations: How do we evolve this schema without downtime?

2. API Contract Design (API First)

   • Define the interface (OpenAPI/Swagger/GraphQL) before coding.
   • Review: Get sign-off from Frontend/Mobile devs. "Does this JSON structure work for you?"
   • Versioning: Plan for /v1/. Breaking changes are expensive later.

3. Capacity Planning

   • Estimate RPS (Requests Per Second).
   • Is this Read-heavy (Cache it?) or Write-heavy (Queue it?)
   • Sync vs Async: Should this be a direct response or a background job?

Phase 1.5: Modern API Paradigms (2026)

Beyond REST:

1. Choosing the Right API Pattern | Pattern | Use When | Don't Use When | Complexity | |---------|----------|----------------|------------| | REST | Public APIs, CRUD operations | Complex data aggregation | Low | | GraphQL | Mobile clients, flexible queries | Simple CRUD | Medium | | tRPC | TypeScript monorepos | Polyglot clients | Low | | gRPC | Microservice mesh, high throughput | Browser clients (needs proxy) | High | | WebSockets | Real-time bidirectional | Request-response pattern | Medium | | SSE | Server push, one-way streams | Bidirectional communication | Low |

2. GraphQL Considerations

   • Benefits: Client specifies exact fields, single endpoint
   • Challenges: N+1 problem (use DataLoader), query complexity attacks
   • Tools: Apollo Server, GraphQL Yoga, Pothos (code-first)
   • When: Mobile apps with varying data needs

3. tRPC (Type-Safe RPC)

   • Full-stack TypeScript: Share types between client/server
   • No code generation: Types inferred automatically
   • Best for: Next.js apps, internal tools

   // Server
   const appRouter = t.router({
     getUser: t.procedure.input(z.number()).query(({ input }) => db.user.find(input))
   });
   // Client (auto-complete!)
   const user = await trpc.getUser.query(123);
   

4. gRPC for Internal Services

   • Protocol Buffers: Binary format, smaller/faster than JSON
   • Streaming: Bi-directional streaming built-in
   • Use Case: Backend microservices, high-throughput systems
   • Limitation: Needs Envoy proxy for browser access

Phase 2: Implementation & Security

Logic ensuring integrity:

1. Authentication & Authorization

   • AuthN: Who are you? (JWT, OAuth).
   • AuthZ: What can you do? (RBAC/ABAC).
   • Rule: Never trust the client. Validate every input on the server.

2. Business Logic Isolation

   • Keep Controllers "thin" (just parsing HTTP).
   • Put logic in Services/Domain layers.
   • Transactions: Ensure atomicity. If step B fails, step A must roll back.

3. Defensive Coding

   • Handle timeouts and retries gracefully (Idempotency keys).
   • Sanitize inputs (SQL Injection, XSS prevention).
   • Rate Limiting: Protect your resources from abuse.

Phase 3: Performance & Scalability

Optimizing the flow:

1. Database Query Optimization

   • Eliminate N+1 queries.
   • Use EXPLAIN ANALYZE to check query cost.
   • Connection Pooling: Don't open a new connection for every request.

2. Caching Strategy

   • Cache at the edge (CDN) for static assets.
   • Cache at the app level (Redis) for expensive computations.
   • Hardest Problem: Cache Invalidation. When does data expire?

3. Asynchronous Processing

   • Offload heavy tasks (Email, Image Resizing) to Message Queues (RabbitMQ/SQS).
   • Don't block the main thread.

Phase 3.5: Distributed Systems Patterns

When you have multiple services:

1. Event Sourcing

   • Pattern: Store events, not current state
   • Benefits: Complete audit trail, time travel, replay events
   • Use Case: Financial systems, order processing
   • Challenge: Event schema evolution

   // Instead of: UPDATE users SET balance = 100
   // Store: UserDepositedMoney(userId, amount, timestamp)
   

2. CQRS (Command Query Responsibility Segregation)

   • Pattern: Separate read models from write models
   • Benefits: Optimize reads/writes independently
   • Use Case: Heavy read traffic with complex queries
   • Tools: MediatR, Event Store

3. Saga Pattern (Distributed Transactions)

   • Problem: You can't use DB transactions across services
   • Solution: Choreography (events) or Orchestration (coordinator)
   • Example: Order placed → Reserve inventory → Charge card → Ship
   • Failure: Compensating transactions (refund if shipping fails)

4. Outbox Pattern (Reliable Events)

   • Problem: How to update DB AND publish event atomically?
   • Solution: Write event to DB table, background worker publishes
   • Benefits: No lost events, exactly-once semantics
   • Tools: Debezium (CDC), Transactional Outbox

Phase 4: Observability & Maintenance

Keeping the lights on:

1. Structured Logging

   • Log context, not just text. { "userId": 123, "error": "db_timeout" }.
   • Do not log PII (Passwords/Credit Cards).

2. Health Checks & Metrics

   • Implement /health endpoints for Load Balancers.
   • Track Latency (p95, p99) and Error Rates.

3. Documentation

   • Keep the API docs (Swagger) auto-generated or updated.
   • Document the "Why" in code comments for complex logic.

Red Flags - STOP and Follow Process

If you catch yourself thinking:

• "I'll add the database index later when it's slow."
• "The frontend validates this, so I don't need to." (Security hole).
• "I'll just loop through the database results in code." (Memory leak).
• "I'll store the secrets in the environment variables committed to Git."
• "I don't need a transaction for these two updates." (Data corruption).
• "It works with 10 users, it will work with 10,000."

ALL of these mean: STOP. Return to Phase 1.

Quick Reference

| Phase | Key Activities | Success Criteria | |-------|---------------|------------------| | 1. Design | ERD, OpenAPI, Versioning | Schema defined, Contract agreed | | 2. Logic | Auth, Validation, Transactions | Secure, atomic operations | | 3. Scale | Caching, Indexing, Queues | p99 Latency under SLA | | 4. Ops | Logging, Metrics, Docs | Observable & Maintainable |

🛠️ Modern API Stack (2026)

Frameworks

• Node.js: Fastify (fast), NestJS (enterprise)
• Python: FastAPI (modern), Django Ninja
• Go: Gin, Fiber, Echo
• Rust: Axum, Actix-web (bleeding edge)

Databases

• OLTP: Postgres 17, MySQL 8.4, CockroachDB (distributed)
• NoSQL: MongoDB, DynamoDB
• Cache: Redis 7, Dragonfly, Valkey
• Search: Elasticsearch, Typesense, Meilisearch
• Vector: pgvector, Pinecone (AI embeddings)

API Patterns

• REST: OpenAPI 3.1 spec
• GraphQL: Apollo Server, GraphQL Yoga
• tRPC: TypeScript full-stack
• gRPC: High-performance internal

Observability

• APM: Datadog, New Relic, Sentry
• Tracing: OpenTelemetry, Jaeger
• Logs: Structured JSON (Pino, Winston)

Database ORMs/Query Builders

• TypeScript: Prisma, Drizzle, Kysely
• Python: SQLAlchemy, Tortoise ORM
• Go: GORM, sqlc

📊 Advanced Database Optimization

Index Strategy

-- B-tree index (default, equality/range)
CREATE INDEX idx_user_email ON users(email);

-- Covering index (includes SELECT columns)
CREATE INDEX idx_order_user_total ON orders(user_id) INCLUDE (total, created_at);

-- Partial index (filtered)
CREATE INDEX idx_active_users ON users(email) WHERE active = true;

-- GIN index (arrays, JSON, full-text)
CREATE INDEX idx_tags ON posts USING GIN(tags);

Query Optimization Checklist

• [ ] Run EXPLAIN ANALYZE on all queries
• [ ] Eliminate N+1 queries (use JOINs or DataLoader)
• [ ] Add indexes on foreign keys
• [ ] Use connection pooling (pg-pool, Prisma)
• [ ] Set appropriate work_mem for complex queries
• [ ] Monitor slow query log

Modern Postgres Features (2026)

-- Vector similarity search (AI embeddings)
CREATE EXTENSION vector;
SELECT * FROM documents ORDER BY embedding <=> '[0.1, 0.2, ...]' LIMIT 10;

-- JSON operations
SELECT data->>'name' FROM users WHERE data @> '{"active": true}';

-- Generated columns
ALTER TABLE orders ADD COLUMN total_with_tax DECIMAL GENERATED ALWAYS AS (total * 1.1) STORED;