Choosing a Model: Size, Open vs Closed, Cost
On this page
A team builds its feature on the top model of a public leaderboard. It works, but each answer takes eight seconds and the monthly bill passes the feature's revenue. Another team picks the cheapest small model, and its answers to tricky questions are wrong just often enough to generate complaints. Both skipped the real question: which model is good enough for this task, at what latency, cost and data constraints? Interviewers ask this constantly, because it's a decision AI engineers make every month.
How it works
The axes that matter

- Quality on your task. Not a public benchmark score, but your own eval set: real inputs, expected outputs, scored automatically or by a judge.
- Latency. Time to first token and tokens per second. Larger models and reasoning modes are slower.
- Cost. Price per input and output token for APIs; GPU hours, engineering time and idle capacity for self-hosting.
- Context window and features: long inputs, tool calling, structured output, vision, audio.
- Data and compliance: can prompts leave your network, where is data processed, is it retained, which regions?
- Control: fine-tuning, pinned versions, no surprise behaviour changes, offline or on-device use.
Model families in practice
- Large frontier models (via API): the best reasoning and broad knowledge; highest price and latency.
- Small or "mini" API models: much cheaper and faster, often good enough for classification, extraction, routing and simple chat.
- Open-weight models: download and run yourself or through a host. Full control and data stays in your environment; you take on serving, scaling and upgrades.
- Reasoning modes: spend extra "thinking" tokens before answering; better on maths, code and multi-step problems, but slower and pricier per request.
A quick way to narrow the choice

The pattern that comes up most in real systems is routing: send each request to the cheapest model that can handle it, and escalate to a larger one only when needed (detected by a classifier, by task type, or by the small model's own low confidence).
API or self-host?

Trade-offs and costs
| Choice | Wins | Costs |
|---|---|---|
| Large API model | Quality, fastest to ship | Price, latency, data leaves your network |
| Small API model | Price, speed | Weaker on hard cases |
| Self-hosted open model | Data control, fine-tuning, steady cost at high volume | GPU and ops work, capacity planning |
| Routing across models | Most of the savings with little quality loss | More moving parts, router errors |
In production
- Evals before switching. Keep a fixed test set and run every candidate against it; compare quality, p50/p95 latency and cost per request side by side.
- Pin versions. "Latest" aliases change behaviour under you; pin and upgrade deliberately.
- Fallbacks. Have a second provider or model ready for outages and rate limits (see Model Routing and Fallbacks).
- Revisit often. Prices and models change every few months; a choice that was right last year may be expensive now.
- Failure modes: quality drop after a silent upgrade; a router sending hard questions to the small model; self-hosted GPUs idle at night but still billed.
When to reach for it
At the start of every feature, at each major model release, and when cost or latency becomes a problem. In interviews it appears as "which model would you use and why?" inside almost every AI system design question.
Example
Assumptions (illustrative): an API model at a blended $1 per million tokens, versus self-hosting an open model on two GPUs at $2 per GPU-hour, which can process about 1,500 tokens per second together.
- Self-hosting cost: 2 × $2 × 730 hours ≈ $2,920 per month, before engineering time.
- Self-hosted capacity at full load: 1,500 × 3,600 × 730 ≈ 3.9 billion tokens per month.
- Break-even: $2,920 / $1 per million ≈ 2.9 billion tokens per month, which needs the GPUs about 74% busy around the clock.
What this means: at these prices the API is cheaper unless traffic is both large and steady, because real traffic has peaks and quiet nights. Self-hosting still wins when data can't leave, when you need a fine-tuned model, or when volume is far above break-even.
Common mistakes
- Choosing by leaderboard instead of your own eval set.
- Using the biggest model for everything, including tasks a small model handles.
- Comparing price per token only, ignoring that a cheaper model may need longer prompts or retries.
- Forgetting people cost and idle GPUs in self-hosting estimates.
- No fallback when the single provider has an outage.
Say this in the interview
"I'd start from constraints: can data go to a vendor, what latency and cost per request we can afford. Then I'd build a small eval set from real inputs and test a few candidates, picking the cheapest that meets the quality bar. If only part of the traffic is hard, I'd route: small model by default, large model on escalation. API vs self-hosting is a volume and data-control question; I'd do the break-even math including utilisation and people."
Interview questions on this
- How would you decide between a large and a small model? Run both on a task-specific eval set; pick the cheapest that passes; consider routing.
- When does self-hosting make sense? Data residency or privacy needs, fine-tuned models, or high steady volume above break-even.
- How do you protect against a provider changing the model? Pin versions, keep evals in CI, and have a fallback provider.
- What's a reasoning model good for, and what does it cost? Multi-step maths, code and planning; more tokens, latency and money per request.
- How would a router decide where to send a request? By task type, a lightweight classifier, input length, or escalation when the small model's answer fails a check.
Used in
- AI Engineering: Model Routing and Fallbacks; How Would You Cut LLM Cost by 10x?; Fine-tune, RAG or Prompt?; GPU Utilization and Cost; How Many GPUs Do You Need to Serve a 70B Model?
- AI System Design: Design an LLM Gateway; Design an LLM Inference Platform.