API Gateway
On this page
Your shop has grown from one app into a User Service, an Order Service and a Search Service. The mobile app now needs three hostnames. Each service checks login tokens with its own copy of the auth code, and two of them have different bugs. Rate limits exist on one service and not the others, so a scraper simply hits Search. Every change to TLS certificates or logging has to be made three times.
The common work in front of every service, accept the connection, check who is calling, check they are not calling too often, and send them to the right place, belongs in one component. That component is the API gateway.
How it works

Clients know one hostname, api.systemcraft.app. The gateway holds a routing table:
/users/* → user-service (load balancer → 6 instances)
/orders/* → order-service (load balancer → 10 instances)
/search/* → search-service (load balancer → 4 instances)
Each service sits behind its own load balancer, which spreads requests across healthy instances. The gateway decides which service; the load balancer decides which instance.
One request inside the gateway

- Terminate TLS: decrypt HTTPS once at the edge of your system.
- Authenticate: verify the JWT or session token. A bad or expired token gets
401right here. - Rate limit: check the caller's quota, often against a shared counter in Redis. Over the limit gets
429 Too Many Requestswith aRetry-Afterheader. - Route: match the path and method, pick the upstream, and add headers such as
X-User-Id: u_123and a request ID for tracing. - Forward and log: send the request on, record status and latency, and return the response.
Rejected requests stop early and never cost a service any work. Services can trust the verified user ID header because only the gateway can reach them (they are on a private network).
Other jobs it often takes
- Request shaping: small transformations, such as API versioning (
/v1→ old service,/v2→ new one). - Canary releases: send 5% of traffic to a new version.
- Timeouts and retries toward services, and circuit breaking when one is failing.
- Response caching for public, read-mostly endpoints.
- CORS, request size limits, IP allow and block lists.
Trade-offs and costs
- An extra hop: a few milliseconds of latency per request. Keep gateway work light (no database lookups per request beyond the rate-limit counter; cache token-verification keys).
- A shared dependency: if every gateway instance fails, everything fails. Run many stateless instances across zones behind DNS or a cloud load balancer, and roll out config changes gradually.
- Config sprawl: routes and policies for many teams live in one place. Keep them in version control with review.
- Temptation to add logic: business rules in the gateway make it slow and couple teams. Aggregation of several services' responses for one screen belongs in a separate backend-for-frontend service, not in the gateway.
When to reach for it
On our system design boards, the API Gateway is the single entry point when several services sit behind one public API: Client → API Gateway → services. It also spreads calls across each service's servers, so we don't draw a separate load balancer behind it. For a single service or internal infrastructure (a cache, a queue, a blob store), a load balancer is the entry point instead. It is the place you name when the interviewer asks:
- "How do you authenticate requests?" → the gateway verifies tokens.
- "How do you stop abuse?" → rate limiting at the gateway.
- "How do clients find services?" → one hostname; the gateway routes.
- "How do you roll out a new version safely?" → weighted routing at the gateway.
For purely internal infrastructure that no client calls directly (a storage engine's replication, a batch pipeline), there may be nothing for a gateway to front, and you can leave it off the board.
Example
Public API for the course site. Assumptions: 20,000 requests per second at peak, free users limited to 100 requests per minute, paid users to 1,000.
- Deployment: 8 gateway instances in 2 zones behind a cloud load balancer. Each handles about 2,500 requests per second, with room to lose one zone.
- Auth: JWTs signed by the auth service. The gateway caches the public signing keys and verifies tokens locally, about 0.1 ms each, with no call per request.
- Rate limits: a token bucket per user ID in a Redis cluster, one round trip of about 1 ms. If Redis is unreachable, the gateway fails open with a local in-memory limit so a Redis outage does not take down the API.
- Routing:
/v1/lessons/\to the Lesson Service,/v1/progress/\to the Progress Service. A new Progress Service version gets 5% of traffic for an hour before full rollout. - Observability: every request gets an
X-Request-Id; latency and status are logged per route, so "429s are up on /v1/progress" is one dashboard away.
Common mistakes
- Leaving the gateway off the board, then having no place to put auth and rate limiting.
- Putting business logic in it, like pricing rules or order validation.
- Making it a single instance, a single point of failure.
- Confusing it with the load balancer. The gateway routes by path and applies policies; the load balancer spreads load across instances of one service.
- Letting services be reachable around it, so the
X-User-Idheader could be forged. - Calling the auth service on every request instead of verifying tokens locally.
Say this in the interview
"Clients hit the API gateway first. It terminates TLS, verifies the JWT locally using cached keys, and applies a per-user token-bucket rate limit backed by Redis, returning 429 when exceeded. Then it routes by path to each service's load balancer and adds the verified user ID and a request ID. It's stateless, so we run several instances across zones. Business logic stays in the services."
Used in
- Rate Limiting Algorithms, Backpressure & Circuit Breaker, 50-Minute Interview Framework.
- Design a Rate Limiter, Design a URL Shortener, Design Quota Service.