Token Rate Limits and Quotas
On this page
A company runs an internal LLM gateway with a limit of 100 requests per minute per team. One team starts a batch job that sends 10 requests a minute, each with a 100,000-token document. They are far below the request limit, yet they consume the whole provider quota, and every chat user in the company starts getting "429 Too Many Requests". For LLMs, the expensive unit is the token, not the request, so limits, quotas and fairness have to be measured in tokens.
How it works

What to limit
- Tokens per minute (TPM), input and output, per tenant or API key: protects shared capacity.
- Requests per minute (RPM): still useful against floods of tiny requests.
- Concurrent requests: long generations hold GPU slots; capping in-flight requests protects latency.
- Monthly token or spend quota: protects the budget and maps to pricing plans.
Counting tokens before you know the answer
The output length is unknown when a request arrives. A common approach uses a token bucket measured in tokens:

- Reserve at admission: count the input tokens (with the right tokenizer) and add the request's
max_tokens. If the bucket holds that many, take them; otherwise queue or reject. - Reconcile when the response finishes: subtract the actual output tokens and return the unused reservation to the bucket.
- Refill continuously at the tenant's rate (for example 100,000 tokens per minute), up to a burst cap.
This makes clients that set a sensible max_tokens get more throughput, which is the right incentive.
Layers of limits

- Per user inside a tenant, so one script can't use a whole team's quota.
- Per tenant or API key: plan quotas and fairness between customers or teams.
- Per model deployment: the real capacity of your GPUs or your provider account. Provider limits are themselves usually TPM and RPM per key and region.
When the bucket is empty
- Return 429 with a
Retry-Afterheader, so well-behaved clients back off. - Queue short bursts for a few seconds rather than failing immediately; give interactive traffic priority over batch.
- Degrade: route to a smaller or self-hosted model, or a second provider (see Model Routing and Fallbacks).
- Clients retry with exponential backoff and jitter, never in tight loops.
Trade-offs and costs
- Reserving max_tokens is pessimistic: clients that set a huge max_tokens lock up capacity they won't use. Cap max_tokens per plan, or reserve an estimate and allow small overdrafts.
- Distributed counting: with many gateway instances, buckets live in a shared store (for example Redis with atomic scripts); a local approximation is faster but less exact.
- Strict vs fair: hard per-tenant caps waste idle capacity; "borrowing" unused capacity improves utilisation but needs priority rules to pull it back.
In production
- Metrics: tokens per tenant per minute, 429 rate, queue time, reservation vs actual usage, provider-side 429s.
- Separate batch from interactive (different keys, queues or deployments) so overnight jobs can't hurt daytime chat.
- Monthly quotas need alerts at 80% and 100%, and a clear behaviour (block, slow down or bill overage).
- Failure modes: a retry storm after a provider outage multiplying load; a tokenizer mismatch undercounting tokens; one region's provider quota exhausted while others sit idle.
When to reach for it
Any shared LLM access: an internal gateway for many teams, a multi-tenant SaaS product, a public API, or simply protecting your provider quota from your own batch jobs.
Example
An internal gateway gives each team 200,000 tokens per minute and caps max_tokens at 4,000, with batch traffic on a separate key limited to 30% of provider capacity. The team from the story now gets its 10 big requests queued in its own lane, while chat traffic keeps p95 time to first token under 1 s. Reservations average 2.1× actual usage, so the team lowered its default max_tokens and gained about 40% more throughput within the same limit (example numbers).
Common mistakes
- Limiting only requests per minute.
- Counting with the wrong tokenizer, especially after switching models.
- No reconciliation, so reserved-but-unused tokens are lost until refill.
- Retrying immediately on 429, making overload worse.
- One shared pool for batch and interactive traffic.
Say this in the interview
"For LLMs I limit tokens, not just requests, because one request can be a hundred thousand tokens. I'd use a token bucket per tenant: reserve input tokens plus max_tokens at admission, reconcile with actual output after, refill at the plan's rate. Limits stack per user, per tenant and per deployment, checked at the gateway. When empty: 429 with Retry-After, short queues with priority for interactive traffic, or fallback to another model, and clients back off with jitter."
Interview questions on this
- Why are request limits not enough for LLMs? Request sizes vary by orders of magnitude; tokens track real cost and capacity.
- How do you limit tokens you haven't generated yet? Reserve max_tokens up front and reconcile after the response.
- How do you implement this across many gateway nodes? A shared counter store with atomic operations, or per-node buckets with periodic sync.
- How do you keep batch jobs from hurting chat? Separate keys, queues or deployments, with priority for interactive traffic.
- What should a client do on 429? Respect Retry-After, back off exponentially with jitter, and cap retries.
Used in
- AI Engineering: Tokens and Tokenization; Model Routing and Fallbacks; Semantic Caching; Serving LLMs: Metrics and Autoscaling.
- AI System Design: Token rate limits and caching; Design an LLM Gateway.
- System Design: Design a Rate Limiter.