Design the Unhappy Path First

Most of us learn to program by making the happy path work. The request comes in, the data is there, the downstream service answers, and we ship it. That instinct is fine for a demo. It is a liability for anything that has to stay up.

After a year of building systems that sit on top of networks I do not control, I have come to believe the opposite of how I was taught: the unhappy path is the design. The happy path mostly takes care of itself. Here are three failure-mode habits that have paid for themselves over and over.

Timeouts are a design decision, not a default

Every call that leaves your process — a database query, an HTTP request, a socket read — can hang. Not might in theory hang. It will hang, on some unlucky day, when a peer goes away without sending a FIN and your call sits there holding a connection, a thread, and a slice of your capacity.

It usually responds in 20ms is not a guarantee. It is an average that says nothing about the tail. The fix is boring and absolute: nothing waits forever.

// Generic, illustrative: pick the timeout per call, not globally
result = await with_timeout(800ms, fetch(req));
match result {
    Ok(value) => handle(value),
    Err(Timeout) => fallback_or_fail_fast(),
}

The hard part is not the syntax — it is deciding the number. A timeout is a statement about how long this operation is allowed to matter. Treat it as a product decision, not a magic constant copied from a tutorial.

Backpressure beats buffering

When work arrives faster than you can process it, the tempting move is to add a bigger queue. Resist it. An unbounded buffer does not absorb overload — it hides it, lets latency climb invisibly, and converts a small slowdown into an out-of-memory crash later, when it is hardest to debug.

A bounded queue forces an honest question at the boundary: when we are full, what do we do? Reject early with a clear signal? Drop the lowest-priority work? Block the producer so the pressure propagates back to the source? All three are valid. Pretending the situation cannot happen is not. Backpressure is simply letting that we are full signal travel upstream, so the whole system slows down together instead of one component falling over alone.

Retries need jitter and a budget

Retries are the most dangerous reliability tool because they feel free. They are not, and a naive retry loop has two failure modes baked in.

• Synchronized retries. If a downstream service blips and a thousand clients all retry at exactly 100ms, 200ms, 400ms, you have built a thundering herd that re-DDoSes the service right as it tries to recover. Add randomized jitter so the herd spreads out.
• Retry storms. Retries multiply load precisely when the system is least able to take it. Cap them with a per-request budget, a circuit breaker, or both, so a struggling dependency gets room to breathe instead of a second wave.

// Exponential backoff WITH jitter and a hard cap
delay = min(base * 2 ** attempt, max_delay)
sleep(delay * random_between(0.5, 1.5))

The unglamorous part is the point

None of this shows up in a demo. Nobody screenshots a graceful degradation. But the gap between a system that wobbles under stress and one that topples is almost entirely in these unglamorous decisions: the timeout you set, the queue you bounded, the retry you capped.

So when I start something new now, I sketch the failure paths first — what times out, what fills up, what gets retried, and what the user sees when each of those happens. Build that scaffold, and the happy path slots in almost as an afterthought, which is exactly where it belongs.