Book 2 β€” OTP Core: GenServer & Supervisors

In Book 1 you hand-rolled a stateful process: a recursive receive loop. OTP (the Open Telecom Platform β€” Erlang's standard library of battle-tested patterns) packages that loop, plus every edge case ever hit in 40 years of production telecom, into reusable behaviours. The two you'll use daily: GenServer and Supervisor.

πŸ““ Companion notebook: book2-otp-core.livemd β€” you'll build a real GenServer, crash it under a supervisor, and visualize the supervision tree live with Kino.

1. From hand-rolled loop to behaviour

Your Book 1 Counter.loop/1 had problems you haven't hit yet: no reply timeouts, no handling of unexpected messages (mailbox leak!), no clean shutdown, no debug tracing, no code upgrades, and callers had to know the raw message protocol. A behaviour is a contract: OTP provides the generic machinery (the loop, the edge cases), and you fill in callbacks with your business logic.

Mental model for JS devs: a behaviour is like a framework base class or an interface with lifecycle hooks β€” think React's component lifecycle. You never write the event loop; you write init, handle_call, handle_cast and OTP calls them at the right moments, in a real process it manages for you.

2. GenServer: the stateful server process

Here's the counter as a GenServer. Note the split: client API (plain functions, run in the caller's process) vs server callbacks (run inside the GenServer process):

defmodule Counter do
  use GenServer

  ## Client API β€” runs in the caller's process
  def start_link(initial \\ 0) do
    GenServer.start_link(__MODULE__, initial, name: __MODULE__)
  end

  def increment,  do: GenServer.cast(__MODULE__, :increment)
  def value,      do: GenServer.call(__MODULE__, :value)

  ## Server callbacks β€” run inside the GenServer process
  @impl true
  def init(initial), do: {:ok, initial}          # return value = initial state

  @impl true
  def handle_cast(:increment, state) do
    {:noreply, state + 1}                         # new state, no reply
  end

  @impl true
  def handle_call(:value, _from, state) do
    {:reply, state, state}                        # reply value, new state
  end
end

Map this onto what you already know: init/1 sets initial state (like a constructor). handle_call/3 answers synchronous requests. handle_cast/2 handles fire-and-forget. There's also handle_info/2 for raw messages (timers, monitors, PubSub) that arrive outside the call/cast protocol. State is whatever value you return β€” the "new state" threading from Book 1's recursion, now managed for you.

Naming: name: __MODULE__ registers the process under a name so clients call Counter.value() without holding a PID. One process per name β€” fine for singletons (a cache, a rate limiter); for many instances of the same server (one per user, per game…) you'll use a Registry in Book 3.

3. call vs cast β€” the decision that shapes your system

call (sync)cast (async)
Callerblocks until reply (default 5s timeout)returns :ok immediately
Deliveryconfirmed β€” you know it was processedfire-and-forget, no confirmation
Back-pressureβœ… built in β€” callers wait, mailbox can't flood❌ none β€” a fast producer can flood the mailbox
Use forreads, anything needing a result or confirmation, writes you must know succeedednotifications, metrics, logs β€” things you can afford to lose
Rule of thumb: when unsure, use call β€” even for writes with no interesting result. The blocking is a feature: it's natural back-pressure. The classic production incident is a cast-flooded GenServer whose mailbox grows until the VM runs out of memory. (Remember from Book 1: send never blocks and never fails.)

QUIZ 2.1

A GenServer takes 100ms to process each message. 1,000 clients cast to it simultaneously, once per 50ms each. What happens?
A GenServer is ONE process = one message at a time = max ~10 msg/s here, while ~20,000/s arrive. Nothing is dropped, nothing is parallelized β€” the mailbox just grows. Fixes: use call (back-pressure), shard across many processes, or batch.

4. Supervisors: crash recovery as architecture

A supervisor is a process whose only job is to start children, watch them (links + trapped exits β€” exactly Book 1's NaiveSupervisor, hardened), and restart them per policy when they die.

defmodule MyApp.Supervisor do
  use Supervisor

  def start_link(opts), do: Supervisor.start_link(__MODULE__, :ok, opts)

  @impl true
  def init(:ok) do
    children = [
      {Counter, 0},                    # {module, init_arg}
      {Cache, []},
      {RateLimiter, max: 100}
    ]
    Supervisor.init(children, strategy: :one_for_one)
  end
end

Each child spec tells the supervisor how to start (start_link) and restart the child. Restart policies per child: :permanent (always restart β€” the default), :transient (restart only on abnormal exit), :temporary (never restart).

Try it: crash some processes

πŸ”§ Supervision tree simulator β€” click a worker to crash it

Supervisor
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
Worker A
alive
Worker B
alive
Worker C
alive
// crash a worker to see the strategy in action

5. Restart strategies

StrategyWhen a child dies…Use when
:one_for_oneonly that child restartschildren are independent (the default choice)
:one_for_allALL children are killed and restartedchildren share state / are useless without each other
:rest_for_onethat child and every child started after it restartlater children depend on earlier ones (start order = dependency order)

Supervisors also have a restart intensity: by default, more than 3 restarts in 5 seconds means the supervisor itself gives up and dies β€” escalating the failure to its supervisor. This is intentional: persistent failures bubble up the tree, restarting ever-larger subsystems, until either something with a clean slate fixes it, or the whole node dies (and your orchestrator restarts that). Failure handling becomes a hierarchy, not a scattering of try/catches.

QUIZ 2.2

Children in order: [Database, Cache, Web]. Cache is useless without Database; Web needs both. Cache crashes. Which strategy restarts Cache and Web but leaves Database alone?
:rest_for_one restarts the crashed child plus everything started after it. Database (started before Cache) keeps running; Cache and Web restart in order. This is why child order matters: it encodes your dependency graph.

6. Designing supervision trees

Real systems nest supervisors under supervisors. A typical shape:

MyApp.Sup Repo (DB pool) Workers.Sup Endpoint Worker 1 ... Worker N Start order left→right = dependency order. A crashing Worker never touches the Repo. If Workers.Sup exceeds restart intensity, MyApp.Sup restarts the whole worker subsystem. Purple = supervisor, green-bordered = worker (GenServer etc.)

Design guidelines: put things that fail together under the same supervisor; put independent things under separate ones. Order children by dependency. Keep state you can't afford to lose (or can rebuild) as low/isolated as possible so restarts are cheap. And remember the payoff β€” this is the "error kernel" pattern: the deeper you go in the tree, the more disposable the process; the closer to the root, the more sacred.

Deep dive: where does lost state go?

A restarted GenServer calls init/1 fresh β€” its state is gone. That's the point (state was possibly corrupted), but sometimes you need recovery: re-read from a database or ETS table (Book 3), rebuild from an event log, or fetch from a peer. A useful discipline: separate "precious" state (owned by a stable process or external store) from "working" state (rebuilt on restart). Supervisors give you the restart; you design what restart means.

Recap

GenServer = your Book 1 receive-loop with production armor: sync call (with back-pressure), async cast, raw handle_info. Supervisor = links + trap_exit with policy: strategies (:one_for_one / :one_for_all / :rest_for_one), per-child restart types, intensity limits that escalate failure up the tree. Now go crash things in the notebook β€” it's the best part.