To reduce LLM API costs without losing quality, measure cost per successful task, then remove waste through shorter context, prompt caching, response limits, model routing, batching, and controlled retries. An 80 percent reduction is possible for some workloads, but it must be demonstrated against the application's own evaluation set.
Measure LLM API costs at the task level
Token spend is a useful accounting input, not the final success metric. A cheaper call that produces more retries, escalations, or wrong answers can cost more overall. Track each request from user intent through every model call, tool call, and retry, then connect the total cost to a successful or failed task.
Record model, input tokens, output tokens, cached tokens, latency, route, prompt version, and response status. For agents, include the number of steps and repeated tool calls. Group results by task type because extraction, support, search, and long-form generation have different quality and cost profiles.
Establish a baseline before changing anything. Use a representative evaluation set and production traffic distribution. Report median cost, p95 cost, success rate, latency, and the share of requests that trigger retries or fallbacks. That baseline prevents a saving in one component from hiding damage elsewhere.
Reduce prompt and retrieval tokens before changing models
Long prompts often contain repeated policy text, unused examples, excessive conversation history, and more retrieved passages than the task needs. Remove instructions that do not change evaluation results. Keep stable policy and schema text in a predictable prefix so provider caching can recognize it.
Retrieval can create more token waste than the system prompt. Tune chunking, top-k, filters, and reranking so the model receives evidence that can affect the answer. Do not send ten documents when two answer the question. Preserve citations and enough surrounding context to avoid cutting away qualifications.
Set output limits from the product requirement. If a field accepts 100 words, do not pay for 800 and truncate the result later. Prefer structured fields when the application needs a known shape. Test shorter prompts and outputs against the same evaluation set, including difficult and ambiguous inputs.
Use prompt caching for stable repeated prefixes
Prompt caching lowers the cost of input that a provider has already processed. OpenAI's prompt caching documentation explains that supported requests can reuse matching prompt prefixes and report cached token use in the API response. The useful design principle is broader than one provider: place stable instructions first and variable user data later.
Provider prompt caching is not the same as returning a saved answer. It reuses computation for repeated input prefixes while the model still produces a response. Response caching skips the model call entirely for a validated repeated request. The latter needs careful cache keys, permission boundaries, freshness rules, and invalidation.
Semantic response caching can reuse an answer for similar questions, but similarity alone is not permission to share data or assume the answer remains current. Partition caches by tenant and policy context. Exclude requests that depend on live balances, private records, or rapidly changing facts unless freshness is explicit.
Route each request to the least costly model that passes
Model routing sends easier tasks to a smaller or lower-cost model and reserves a more capable model for harder cases. Start with task classes, not a vague complexity score. Classification, entity extraction, formatting, and common support questions may have clear acceptance tests. Legal analysis, difficult reasoning, or high-risk refusals may require a stronger route.
Build the router from evaluation evidence. Run candidate models on each task class, choose the least costly one that meets the quality threshold, and define a fallback for low confidence or validation failure. Monitor route changes because the traffic mix and provider models can change.
A production AI chatbot development project should test routing against actual conversations, handoff rules, retrieval quality, and safety cases. A cheaper answer is useful only when users receive the same required outcome.
Batch work that does not need an immediate answer
Offline enrichment, document classification, evaluation runs, summaries, and scheduled reports often tolerate delayed completion. The OpenAI Batch API currently processes asynchronous batches within a 24-hour window at a discount. Other providers offer their own batch or provisioned options. Compare current terms before estimating savings.
Separate interactive and offline queues. Give every batch item an idempotency key, store its input version, and handle partial failures. Batching should not turn one invalid record into a repeated job that consumes the expected saving.
Control retries, loops, and duplicate requests
Automatic retries can quietly multiply spend. Retry only transient failures, cap attempts, and use exponential backoff with jitter. Do not retry validation failures with the same prompt and parameters unless the retry changes something that could correct the result.
Agent loops need budgets for steps, tokens, tool calls, and elapsed time. Stop when the agent repeats the same action or has no new evidence. Deduplicate requests at the application boundary so double clicks, network retries, and repeated queue delivery do not create separate model calls.
Teams that need shared tracing, routing, and budget controls can connect these measures to LLMOps cost governance rather than maintaining isolated counters inside each feature.
Calculate a defensible cost reduction
- Run the baseline. Measure total model cost and successful-task rate for a fixed evaluation set and a representative traffic sample.
- Apply one change at a time. Test context reduction, caching, routing, batching, and retry controls separately before combining them.
- Recheck quality by task class. Keep safety, groundedness, format validity, and escalation thresholds fixed.
- Model cache misses and traffic shifts. A saving that relies on an unusually high cache hit rate may disappear after launch.
- Report net cost. Include vector search, reranking, gateways, storage, GPUs, and engineering overhead, instead of stopping at the primary API invoice.
Do not promise an 80 percent cut before measurement. Use it as a hypothesis. The reliable strategy is to eliminate repeated work, send only necessary context, choose models from evaluation results, and keep a rollback path when quality moves.