Backpressure & Circuit Breaker
On this page
The recommendations service starts answering in 10 seconds instead of 50 ms. The product page calls it on every request and waits. Within a minute, every product-page thread is stuck waiting, the connection pool is empty, and product pages time out, even for users who never scroll to recommendations. Meanwhile, retries from those timeouts hit the recommendations service even harder. One slow dependency has taken down a healthy service, and the healthy service is making the sick one sicker.
Two ideas stop this kind of cascade. A circuit breaker stops calling a dependency that is failing. Backpressure makes an overloaded component push back on its callers instead of accepting work it cannot finish.
How it works
Timeouts first
Without a timeout, a slow dependency holds your threads, connections and memory for as long as it likes. Set a timeout on every remote call, based on the dependency's normal latency (for example, p99 is 80 ms, so time out at 300 ms), and on the total request (a deadline that is passed downstream so deeper services stop work the caller has already given up on).
Circuit breaker

A circuit breaker wraps calls to one dependency and has three states:
- Closed: calls go through. The breaker tracks the recent failure rate (errors and timeouts), for example over the last 20 calls or 10 seconds.
- Open: once the failure rate passes a threshold (say 50%), the breaker opens. Calls fail immediately without touching the network. Your threads are free, and the dependency gets room to recover.
- Half-open: after a cool-down (say 30 seconds), a few test calls are allowed. If they succeed, the breaker closes; if not, it opens again.
When the breaker is open, the caller uses a fallback: cached recommendations, a default "popular items" list, hiding the widget, or a clear error for features that can't degrade.
Breakers are per dependency (and often per endpoint), and live in each caller instance. Service meshes and client libraries provide them.
Backpressure

If work arrives faster than it can be done, it has to wait somewhere. An unbounded queue just converts overload into growing latency (users wait minutes for answers they've given up on) and then an out-of-memory crash. Backpressure makes the limit explicit:
- Bounded queues and pools: cap the queue length, thread pool and connection pool. When full, reject new work right away.
- Reject early with a clear signal:
429 Too Many Requestsor503 Service UnavailablewithRetry-After, so clients back off instead of hammering. - Shed load by priority: keep checkout, drop analytics; keep logged-in users, drop crawlers.
- Slow the producer: in pull-based systems (Kafka consumers, streaming with credits), consumers only take what they can handle and the backlog stays in durable storage.
- Autoscale on queue depth or concurrency, knowing it takes minutes, so the other mechanisms must cover the gap.
Retries done safely
Retries help with brief blips and hurt during real outages. Rules:
- Retry only idempotent operations (or ones with an idempotency key).
- Use exponential backoff with jitter: 100 ms, 200 ms, 400 ms, each randomized, so clients don't retry in sync.
- Cap retries (2–3) and use a retry budget: for example, retries may add at most 10% extra traffic.
- Retry at one layer. If the browser, gateway and service each retry 3 times, one failure becomes 27 calls.
Trade-offs and costs
- Rejecting work makes some users see errors on purpose. That is the point: a few clean errors instead of everyone timing out.
- Thresholds need tuning: too sensitive and the breaker flaps on normal blips; too loose and it opens too late. Base them on real latency and error data.
- Fallbacks must be designed per feature and tested; a fallback that is never exercised tends to be broken when you need it.
- Per-instance breakers each learn on their own, so with 100 instances it takes a moment for all to open.
- Bulkheads (separate pools per dependency) cost extra threads or connections but stop one dependency from starving the others.
When to reach for it
- Any service that calls other services, especially optional ones (recommendations, reviews, personalization).
- Calls to third parties you don't control: payment providers, email, maps.
- Queue-based pipelines where producers can outrun consumers.
- Anywhere the interviewer asks "what happens if X is slow?" Slow is often worse than down.
Example
A product page that calls Price, Inventory, Reviews and Recommendations. Assumptions: 5,000 product-page requests per second, 50 instances, each with 200 worker threads.
- Timeouts: Price and Inventory 200 ms (required), Reviews and Recommendations 150 ms (optional). Overall page deadline 800 ms.
- Bulkheads: each dependency gets its own connection pool (50 connections per instance), so a slow Recommendations service can use at most 50 of the 200 threads.
- Circuit breakers: open at 50% failures over the last 10 seconds (minimum 20 calls), cool down 30 seconds, 5 test calls in half-open.
- Fallbacks: Recommendations → cached "popular in this category" list (refreshed every 10 minutes). Reviews → hide the section. Price → no fallback; return a clear error, since showing a stale price is worse than none.
- Backpressure: each instance accepts at most 300 in-flight requests; beyond that the gateway gets 503 and routes to other instances, or returns 503 with Retry-After to the client.
- Retries: only the gateway retries, once, for idempotent GETs, with jitter, within a 10% budget.
Common mistakes
- No timeouts, or default timeouts of 30–60 seconds.
- Unbounded queues and thread pools "so nothing is dropped".
- Retry storms: retries at every layer, without backoff or a budget.
- Breakers without fallbacks, which just turn slow errors into fast errors for features that could have degraded.
- Treating "slow" as healthy. Count timeouts as failures.
- Sharing one connection pool across all dependencies.
Say this in the interview
"Every downstream call has a timeout and its own connection pool, so a slow Recommendations service can't take all our threads. Each dependency sits behind a circuit breaker that opens at 50% failures and fails fast, and while it's open we serve cached popular items. Each instance caps in-flight requests and returns 503 with Retry-After when full. Only the gateway retries, once, with jittered backoff, so a failure can't multiply into a retry storm."
Used in
- Rate Limiting Algorithms, Message Queue, API Gateway, Idempotency & Deduplication.
- Design a Flash Sale System, Design a Food Delivery Platform like DoorDash / Uber Eats, Design a Logging System.