LLM API Costs Add Up Fast
A single GPT-4o call costs about $0.01 to $0.05 depending on prompt and response length. That sounds small until you multiply it by 50,000 queries per day. Suddenly you are spending $500 to $2,500 daily on API calls alone. Most of that spend is avoidable with the right optimizations.
Here are the techniques that actually move the needle, ordered by impact.
1. Cache Repeated Queries
This is the single biggest cost saver. In most applications, 20% to 40% of queries are duplicates or near-duplicates. A user asking "What is your return policy?" on Monday will ask the same thing on Tuesday. Why pay for the LLM call twice?
Exact-match caching is simple: hash the prompt, store the response in Redis, and return the cached response if the hash matches. Semantic caching goes further by embedding the query and returning cached responses for queries that are similar enough (cosine similarity above 0.95).
Expected savings: 20% to 40% of total API costs, depending on how repetitive your query patterns are.
2. Route to Cheaper Models
Not every query needs GPT-4o. Simple questions ("What are your office hours?"), classification tasks, and structured extraction can run on GPT-4o-mini at roughly 1/10th the cost, or on Claude 3 Haiku at even less.
Build a router that classifies incoming queries by difficulty and sends them to the appropriate model. A simple keyword or embedding-based classifier can route 60% to 70% of queries to a cheaper model without noticeable quality loss.
Expected savings: 40% to 60% of remaining API costs after caching.
3. Shorten Your Prompts
Tokens cost money. Every unnecessary instruction, every verbose example, every redundant system message adds to your bill. Review your prompts and cut anything that does not improve output quality.
Common prompt bloat:
- System prompts over 500 tokens that repeat the same instruction in three different ways
- Few-shot examples that could be replaced with a clear instruction
- RAG context windows that include entire documents when a 200-token excerpt would work
Measure your average prompt length. If it is over 2,000 tokens, there is almost certainly room to cut.
Expected savings: 10% to 30%, proportional to how much you trim.
4. Reduce Max Tokens on Responses
If your application only needs a 100-word answer, set max_tokens accordingly. Without a limit, the model might generate 500-word responses that your application truncates anyway, but you still pay for every token generated.
5. Batch API Calls
OpenAI's Batch API offers a 50% discount on API calls processed within a 24-hour window. If your workload is not latency-sensitive (report generation, data enrichment, content moderation on uploaded files), batch it.
Expected savings: 50% on batched workloads.
6. Use Open-Source Models for Internal Tasks
Tasks that run internally (data cleaning, summarization of internal documents, classification for routing) do not need GPT-4. Self-hosted Llama 3 8B or Mistral 7B can handle these at near-zero marginal cost after the initial infrastructure setup.
The tradeoff: you pay for GPU compute instead of API calls. For high-volume internal tasks, self-hosting is almost always cheaper.
7. Implement Request Deduplication
In real-time applications, the same user might fire multiple identical requests (double-clicks, page refreshes, retry logic). Deduplicate at the application layer so you do not send the same prompt to the API multiple times within a short window.
Putting It Together: A Real Example
A customer support chatbot handling 30,000 queries per day on GPT-4o:
| Before optimization | After optimization |
|---|---|
| 30,000 GPT-4o calls/day | 8,000 GPT-4o calls + 12,000 GPT-4o-mini + 10,000 cache hits |
| ~$1,200/day | ~$240/day |
| $36,000/month | $7,200/month |
That is an 80% reduction. The quality metrics (CSAT score, escalation rate) stayed flat because the router correctly identified which queries needed the more capable model.
Frequently Asked Questions
Does caching cause stale answers?
It can if your underlying data changes frequently. Set TTL (time to live) on cache entries. For a FAQ bot, 24-hour TTL works. For real-time data, cache for 5 to 15 minutes or skip caching for those query types.
How do I know which model to route to?
Start simple: route by query length and keyword matching. Short, simple queries go to the cheap model. Long, complex queries go to the expensive model. Refine with an embedding-based classifier as you collect data.