georgeguimaraes/otp-thinking

OTP Thinking

Paradigm shifts for OTP design. These insights challenge typical concurrency and state management patterns.

The Iron Law

GENSERVER IS A BOTTLENECK BY DESIGN

A GenServer processes ONE message at a time. Before creating one, ask:

1. Do I actually need serialized access?
2. Will this become a throughput bottleneck?
3. Can reads bypass the GenServer via ETS?

The ETS pattern: GenServer owns ETS table, writes serialize through GenServer, reads bypass it entirely with :read_concurrency.

No exceptions: Don't wrap stateless functions in GenServer. Don't create GenServer "for organization".

GenServer Patterns

| Function | Use For | |----------|---------| | call/3 | Synchronous requests expecting replies | | cast/2 | Fire-and-forget messages |

When in doubt, use call to ensure back-pressure. Set appropriate timeouts for call/3.

Use handle_continue/2 for post-init work—keeps init/1 fast and non-blocking.

Task.Supervisor, Not Task.async

Task.async spawns a linked process—if task crashes, caller crashes too.

| Pattern | On task crash | |---------|---------------| | Task.async/1 | Caller crashes (linked, unsupervised) | | Task.Supervisor.async/2 | Caller crashes (linked, supervised) | | Task.Supervisor.async_nolink/2 | Caller survives, can handle error |

Use Task.Supervisor for: Production code, graceful shutdown, observability, async_nolink. Use Task.async for: Quick experiments, scripts, when crash-together is acceptable.

DynamicSupervisor + Registry = Named Dynamic Processes

DynamicSupervisor only supports :one_for_one (dynamic children have no ordering). Use Registry for names—never create atoms dynamically:

defp via_tuple(id), do: {:via, Registry, {MyApp.Registry, id}}

PartitionSupervisor scales DynamicSupervisor for millions of children.

:pg for Distributed, Registry for Local

| Tool | Scope | Use Case | |------|-------|----------| | Registry | Single node | Named dynamic processes | | :pg | Cluster-wide | Process groups, pub/sub |

:pg replaced deprecated :pg2. Horde provides distributed supervisor/registry with CRDTs.

Broadway vs Oban: Different Problems

| Tool | Use For | |------|---------| | Broadway | External queues (SQS, Kafka, RabbitMQ) — data ingestion with batching | | Oban | Background jobs with database persistence |

Broadway is NOT a job queue.

Broadway Gotchas

Processors are for runtime, not code organization. Dispatch to modules in handle_message, don't add processors for different message types.

one_for_all is for Broadway bugs, not your code. Your handle_message errors are caught and result in failed messages, not supervisor restarts.

Handle expected failures in the producer (connection loss, rate limits). Reserve max_restarts for unexpected bugs.

Supervision Strategies Encode Dependencies

| Strategy | Children Relationship | |----------|----------------------| | :one_for_one | Independent | | :one_for_all | Interdependent (all restart) | | :rest_for_one | Sequential dependency |

Use :max_restarts and :max_seconds to prevent restart loops.

Think about failure cascades BEFORE coding.

Abstraction Decision Tree

Need state?
├── No → Plain function
└── Yes → Complex behavior?
    ├── No → Agent
    └── Yes → Supervision?
        ├── No → spawn_link
        └── Yes → Request/response?
            ├── No → Task.Supervisor
            └── Yes → Explicit states?
                ├── No → GenServer
                └── Yes → GenStateMachine

Storage Options

| Need | Use | |------|-----| | Memory cache | ETS (:read_concurrency for reads) | | Static config | :persistent_term (faster than ETS) | | Disk persistence | DETS (2GB limit) | | Transactions/Distribution | Mnesia |

:sys Debugs ANY OTP Process

:sys.get_state(pid)        # Current state
:sys.trace(pid, true)      # Trace events (TURN OFF when done!)

Telemetry Is Built Into Everything

Phoenix, Ecto, and most libraries emit telemetry events. Attach handlers:

:telemetry.attach("my-handler", [:phoenix, :endpoint, :stop], &handle/4, nil)

Use Telemetry.Metrics + reporters (StatsD, Prometheus, LiveDashboard).

Red Flags - STOP and Reconsider

• GenServer wrapping stateless computation
• Task.async in production when you need error handling
• Creating atoms dynamically for process names
• Single GenServer becoming throughput bottleneck
• Using Broadway for background jobs (use Oban)
• Using Oban for external queue consumption (use Broadway)
• No supervision strategy reasoning

Any of these? Re-read The Iron Law and use the Abstraction Decision Tree.