How Would You Cut LLM Cost by 10x?
On this page
Problem statement
"Our AI support assistant costs about $120,000 a month in model API fees and usage keeps growing. Finance wants it cut by 10x without users noticing. How would you approach it?"
A practical question that tests whether you know where LLM cost comes from and can stack savings without breaking quality.
Why it happens

- Every request pays for the whole prompt: a long system prompt with tool definitions, plus many retrieved chunks, plus chat history (see Context Windows and Token Cost).
- One model for everything: greetings, simple FAQs and hard troubleshooting all go to the largest model.
- Repeated work: the same questions answered from scratch thousands of times.
- Verbose outputs: long answers cost more, and output tokens are usually priced higher.
- Hidden calls: query rewriting, guardrails, judges and retries add calls per user message.
Options
Weaknegotiate a discount or switch to the cheapest model
A discount helps a little; switching everything to the cheapest model may cut cost but quietly breaks hard cases, and nobody measures the damage until complaints arrive.
Goodtrim tokens and cache
Measure tokens per request by part, then shorten the system prompt, retrieve 5 strong chunks instead of 20, summarise older history, and cap answer length. Turn on provider prompt caching for the stable prefix and add an exact-match cache for repeated requests. These are low-risk and often cut cost by half.
Strongstack levers, measured one by one

- Trim: shorter prompts and context (example: input tokens −50%).
- Route: a small, fast model handles the easy majority; escalate to the large model on hard intents or low confidence (see Model Routing and Fallbacks).
- Cache: semantic cache for frequent, non-personal questions (see Semantic Caching); prefix caching for the static part of the prompt.
- Batch: move offline work (summaries, tagging, evals) to discounted batch processing.
- Remove hidden calls: merge query rewriting into the main call, run cheap classifiers instead of LLM guardrails where possible.
- Consider fine-tuning or distillation so a small model handles a narrow task as well as a large one (see Distillation), and self-hosting only above the break-even volume (see Choosing a Model).
With the example numbers, trimming (×0.59), routing 70% of traffic to a model ten times cheaper (×0.37), a 20% semantic cache hit rate (×0.8) and batching 30% offline work at half price (×0.85) take the bill from 100 to about 15. Shorter answers and prefix-cache discounts usually close the gap to 10x.
How to measure it
- Cost per successful outcome (per resolved conversation, per completed task), not only per call: a cheaper model that needs more turns can cost more.
- Tokens per request by part (system, context, history, output) and by feature.
- Quality: eval-set correctness and faithfulness, escalation rate, user feedback, for each change.
- Routing mix and cache hit rates, and wrong-hit rate for the semantic cache.
Example answer flow
"I'd first get a per-feature, per-prompt-part breakdown. If 80% is input tokens and everything goes to the biggest model, I'd trim the prompt and context, add routing so simple intents use a small model, cache frequent questions, and push offline jobs to batch. After each step I'd re-run the eval set and watch cost per resolved conversation. If we're still short of 10x, I'd look at distilling the most common task into a small fine-tuned model."
Follow-up questions
- How do you decide which requests are "easy" enough for the small model?
- A cheaper model resolves fewer conversations. How do you compare total cost?
- When does self-hosting beat API pricing? Walk through the math.
- How does prompt caching work, and what breaks it?
- How would you prevent costs from creeping back up?
- What's the risk of a semantic cache in customer support?
Related
- Concepts: Context Windows and Token Cost; Choosing a Model; Model Routing and Fallbacks; Semantic Caching; Prefix Caching; Distillation.
- Practice: How Would You Cut LLM Latency in Half?
- AI System Design: Design an LLM Gateway; Design ChatGPT.