LLM Cost Optimization: Token Budgets, Caching and Models

An AI feature can look cheap in a demo and become expensive in normal use. Real customers paste long documents, reopen conversations and retry answers that do not solve their problem. Background jobs, tool results and failed requests add work that a simple price-per-message estimate misses.

Cost control starts with a measurable product outcome. For a documentation assistant, that might be a supported answer that avoids a support ticket. For a content tool, it might be an accepted draft. Optimise spending against that outcome rather than celebrating a lower unit price while users need twice as many attempts.

Build a budget from actual workloads

Measure cost per successful task, not just average request cost. Capture input tokens, output tokens, cache-related usage, retries and any separately billed tools. Provider accounting differs, so reconcile your estimates against billing records rather than assuming every token category has the same rate.

Create a representative workload with short, typical and unusually large inputs. Include conversations with several turns and requests that fail. Multiply the measured distribution by expected traffic, then model bursts and growth. Averages alone hide the long tail that often dominates spend.

Separate interactive and background work. An answer that blocks a customer has different latency needs from a nightly classification job. Background work may tolerate batching or a slower route if the provider supports it, but deadlines and quality requirements still apply.

Use the JSON formatter to inspect a redacted usage event and verify that operation, token counts and billing categories are consistently represented. Formatting makes records easier to review; it does not validate your accounting model. Keep personal content and credentials out of shared debugging tools.

Bound tokens before optimising prices

Set an explicit input budget for each operation. Limit document size, retrieve only relevant passages and remove duplicated instructions. Count tokens with the provider's supported tooling where available; character counts are rough estimates and vary across languages, code and other content types.

Choose an output ceiling that fits the task. A classification result does not need the same allowance as an implementation plan. A maximum is a guardrail, not a request for a fixed response length. Test truncated outputs and distinguish a complete short response from one cut off before a required field.

Long conversations need deliberate context management. Preserve facts needed for the current decision, summarise older material where appropriate and retain source references for verification. Summaries can lose exceptions or introduce errors, so do not use them as the sole record for sensitive decisions.

A budget policy might begin like this:

{
"operation": "support_draft",
"maxInputTokens": 6000,
"maxOutputTokens": 700,
"maxAttempts": 2,
"maxToolRounds": 2,
"fallback": "human_review"
}

These numbers are illustrative, not universal defaults. Calibrate them against your corpus and service limits. The guide to production prompt templates explains how tighter task contracts make these limits easier to apply without damaging usefulness.

Distinguish prompt caching from answer caching

Prompt caching generally reuses computation for eligible repeated input, subject to provider rules. It does not mean a previously generated answer is returned. Stable instruction prefixes and reusable reference material may help, but eligibility, minimum sizes, retention and pricing differ across services.

Read the Anthropic documentation and the OpenAI API documentation for the provider you actually use. Do not assume a cache discount applies to the whole request or that a cached prefix makes generation free. Record observed cache usage and measure the effect under realistic traffic.

Answer caching is your application's reuse of a prior result. It can avoid an inference call entirely, but only when the result remains valid. Include prompt version, model configuration, source revision, locale and relevant permissions in the cache design. Two identical questions can require different answers for different accounts.

Cache only within an appropriate security scope. A tenant identifier is not sufficient if permissions differ within the tenant. Avoid caching personalised answers publicly, and define invalidation when source policies or access rights change. A stale answer that triggers incorrect advice can cost more than the saved request.

Start with exact-match caching for well-understood operations. Semantic caching adds the risk that similar wording conceals a different intention. Evaluate false matches carefully before reusing answers across questions, especially where dates, amounts or eligibility conditions matter.

Route models by demonstrated task quality

Use task-based routing instead of sending every request to the largest available model. A smaller model may handle label selection reliably while a more capable one is needed for conflicting evidence or complex synthesis. The distinction must come from evaluation, not assumptions based on marketing categories.

Build a labelled test set and compare acceptance rates, latency and total cost. Include escalations and repeated attempts in the calculation. A cheap route that fails often can be more expensive after fallback, especially if the initial failure damages the context used by the second model.

Keep escalation rules observable. Missing evidence, failed schema validation or explicit task complexity can be useful signals. A model's unsupported claim that it is confident is not a reliable routing policy. For high-impact operations, human review may be the appropriate fallback rather than another model.

The Google AI documentation offers another provider reference for supported capabilities. Verify context limits and tool behaviour before substituting models. The AI coding assistants directory can support implementation research, but your own task results should decide which route ships.

Enforce budgets and monitor regressions

Implement budget enforcement on the server. Apply per-user, tenant and application limits, reserve capacity before starting work and reconcile actual usage afterwards. Concurrency can defeat a simple read-then-check balance. Use atomic reservations or another design that prevents several requests spending the same remaining allowance.

Cap retries and tool loops. A response that requests another search indefinitely should hit a known stopping condition and return a useful failure state. Network errors do not guarantee the provider performed no billable work, so account for uncertain outcomes conservatively.

Follow the API security guide for proxy boundaries and abuse controls. Cost optimisation cannot compensate for an exposed credential or an anonymous endpoint with unrestricted generation. Spending limits and access controls solve related but different problems.

Monitor cost distributions by operation and version, not only the monthly total. Alert when input length, retries, cache misses or escalation rates change. Release a cheaper route gradually and inspect acceptance rates alongside savings. Keep a rollback path when an apparent optimisation merely moves expense into support work.

Key takeaways

• Measure total cost against successful tasks, including retries and tools.
• Set input, output and iteration budgets for each operation.
• Treat prompt caching and application answer caching as different mechanisms.
• Choose models with representative evaluations and explicit escalation rules.
• Enforce spending limits server-side and monitor quality alongside savings.