curiositech/service-mesh-microservices-expert

Service Mesh & Microservices Expert

Design and operate service meshes for secure, observable, and resilient microservice communication using Istio, Envoy, and Linkerd.

Activation Triggers

Activate on: "service mesh", "Istio", "Envoy", "sidecar proxy", "circuit breaker", "service discovery", "mTLS", "traffic management", "canary deployment", "Linkerd"

NOT for: Edge/API gateway → api-gateway-reverse-proxy-expert | Application instrumentation → observability-apm-expert | Container orchestration basics → relevant DevOps skill

Quick Start

1. Evaluate need — service meshes add complexity; justified at 10+ services with cross-cutting concerns
2. Choose mesh — Istio (full-featured), Linkerd (lightweight), Cilium (eBPF, no sidecar)
3. Enable mTLS — zero-trust between all services, mesh handles certificate rotation
4. Configure traffic policies — retries, timeouts, circuit breakers per service pair
5. Deploy with canary — use mesh traffic splitting for safe rollouts (90/10, 80/20, etc.)

Core Capabilities

| Domain | Technologies | |--------|-------------| | Meshes | Istio 1.24+, Linkerd 2.16+, Cilium Service Mesh | | Data Plane | Envoy Proxy, Linkerd2-proxy, eBPF (Cilium) | | Security | mTLS, SPIFFE/SPIRE, AuthorizationPolicy | | Traffic | VirtualService, DestinationRule, traffic splitting | | Observability | Kiali, automatic Prometheus metrics, distributed tracing |

Architecture Patterns

Sidecar Proxy Architecture (Istio)

┌────────────────── Pod ──────────────────┐
│  ┌──────────┐     ┌──────────────────┐  │
│  │  App     │────→│  Envoy Sidecar   │──┼──→ Other services
│  │ Container│←────│  (injected auto) │←─┼──  (via their sidecars)
│  └──────────┘     └──────────────────┘  │
└─────────────────────────────────────────┘

Envoy intercepts all inbound/outbound traffic:
- mTLS encryption/decryption
- Retry, timeout, circuit breaking
- Metrics collection (RED)
- Access logging
- Traffic routing rules

Circuit Breaker + Retry Configuration

# Istio DestinationRule: circuit breaker for payment-service
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5        # trip after 5 errors
      interval: 10s                   # check every 10s
      baseEjectionTime: 30s           # eject for 30s minimum
      maxEjectionPercent: 50          # never eject >50% of hosts
---
# Istio VirtualService: retry policy
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts: [payment-service]
  http:
    - route:
        - destination:
            host: payment-service
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure
      timeout: 10s

Canary Deployment with Traffic Splitting

# Route 90% to v1, 10% to v2 (canary)
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts: [order-service]
  http:
    - route:
        - destination:
            host: order-service
            subset: v1
          weight: 90
        - destination:
            host: order-service
            subset: v2
          weight: 10

# Progressive: 90/10 → 70/30 → 50/50 → 0/100
# Roll back instantly by setting v1 weight to 100

Anti-Patterns

1. Mesh for 3 services — service mesh overhead is not justified for small deployments; use direct HTTP with retry libraries
2. Ignoring resource overhead — each Envoy sidecar uses 50-100MB RAM and adds ~1ms latency; budget for it
3. Application-level retries + mesh retries — double retries cause amplification; choose one layer for retry logic
4. Permissive mTLS forever — start with PERMISSIVE for migration, but move to STRICT mode to enforce zero-trust
5. No traffic policies — deploying a mesh without configuring timeouts, retries, and circuit breakers wastes the mesh

Quality Checklist

• [ ] mTLS mode set to STRICT (not PERMISSIVE) in production
• [ ] Circuit breakers configured per upstream service
• [ ] Retry budgets set to prevent amplification (retries < 20% of total traffic)
• [ ] Timeouts explicit on every VirtualService (no infinite waits)
• [ ] Canary deployment tested with traffic splitting
• [ ] Kiali or equivalent mesh topology dashboard available
• [ ] Sidecar resource limits set (CPU/memory requests and limits)
• [ ] AuthorizationPolicy restricts which services can call which
• [ ] Mesh telemetry feeding into observability stack (Prometheus, Jaeger)
• [ ] Sidecar injection verified (no pods running without proxy)