LLM bills creep up quietly. A feature ships against a frontier model because that was the fastest way to get it working, traffic grows, and a few months later inference is one of the larger lines on the cloud invoice. The common first instinct is to rewrite prompts to be shorter. That helps a little, but it is rarely the lever that moves the bill. The cost of an LLM request is set by which model you call, how many tokens go in and come back, and how often you avoid the call entirely. Those are the things worth attacking, and they respond best when you treat inference as a FinOps problem rather than a one-off tuning exercise.
Why the bill grows and where prompts fit
Per-request cost is a product of input tokens, output tokens, and the price of the model tier you chose. Prompt edits touch only one of those terms, and usually the smaller one, because the system prompt and retrieved context tend to dwarf the user's message. Trimming a system prompt by a few sentences feels productive, but if every request still hits the most expensive model and still pulls in the same large context block, you have changed almost nothing.
The levers that actually reduce spend are structural. They decide which model handles a request, how much context that model has to read, and whether the request needs to be made at all. We have built this discipline into LLM work for clients such as the British Council AiBC assistant, and the same FinOps habits carry over from the serverless AWS cost work we do on cloud infrastructure and DevOps.
Model routing: match the model to the request
Most production traffic is not hard. A large share of requests are short questions, classifications, extractions, or rewrites that a small model answers correctly. Routing sends those to the cheaper model and reserves the frontier model for the requests that genuinely need it. Because the price gap between tiers is large, moving the easy majority down a tier is often a large reduction on its own, without touching the quality of the hard requests at all.
The work is in deciding what counts as easy. You can route on a cheap signal such as input length or request type, on a fast classifier that labels difficulty, or on confidence: try the small model first and escalate to the larger one only when the answer fails a check. The escalation pattern costs a second call on the hard cases but keeps the common path cheap. Routing decisions belong in the orchestration layer, which is where the request already has the context it needs to make the call. If that layer is new to you, our piece on what LLM orchestration is covers where routing sits.
Caching: stop paying for the same answer twice
The cheapest request is the one you never make. Caching works at three layers, and they stack.
Exact-match caching
Identical prompts return a stored result. Trivial to add, and it pays off whenever the same query repeats, which is more common than teams expect for FAQ-style and templated traffic.
Semantic caching
Near-duplicate queries reuse a prior answer by comparing embeddings rather than exact text. It catches paraphrases, but it needs a similarity threshold you have tested, or it will serve answers to questions that only look alike.
Provider prompt caching
Providers can cache a shared prefix, such as a long system prompt or retrieved document, and bill the repeated portion at a reduced rate. You pay once to warm it, then reuse it across requests in the window.
Exact-match and semantic caching remove whole calls. Provider prompt caching keeps the call but cuts the cost of the part that repeats, which is why it pairs well with a large, stable system prompt. Order the shared content first so the cacheable prefix is as long as possible, and measure your hit rate so you know the cache is earning its place rather than just adding a layer.
Token budgets and context size
Context is usually the largest cost term, because retrieved passages, conversation history, and tool definitions all ride along on every request. Set a token budget per feature and treat it as a constraint, not a guideline. If a feature is allowed a fixed number of input tokens, you are forced to ask whether you really need the top twenty retrieved chunks or whether five reranked ones do the job, and whether the full conversation history matters or a summary would do.
A few habits keep context honest: retrieve fewer, better passages rather than many mediocre ones; summarise or truncate long histories instead of replaying them in full; cap output length so the model cannot ramble at your expense; and strip tool definitions the model will not need for the current step. None of this is a prompt rewrite. It is deciding what the model is allowed to read and say.
Output tokens are not free
Output is often priced higher than input, so a verbose response style costs more than it looks. Ask for structured, short output where the downstream consumer is code rather than a person, and set a maximum length so a runaway generation cannot quietly inflate the bill.
Batching and concurrency
When work is not interactive, batching changes the economics. Asynchronous batch endpoints process large volumes at a reduced rate in exchange for relaxed latency, which suits overnight enrichment, back-catalogue classification, and evaluation runs. If a job does not need an answer in the next second, it probably should not be paying interactive prices.
Concurrency is the other side of throughput. Rate limits and connection handling decide whether a batch finishes in an hour or a day, and getting them right is closer to infrastructure work than to prompt work. This is the same ground we cover building production LLM systems under AI agents and LLM products.
Choosing a tier, and when to self-host
The right tier is the cheapest model that passes your evaluation on a given task, not the most capable model overall. That means you need an eval set per task so you can drop a tier with evidence rather than a hunch. Many teams discover that a mid-tier or small model clears the bar for most of their traffic once routing is in place.
Self-hosting an open model becomes worth considering at sustained high volume, where the per-request saving outweighs the cost of running and keeping GPUs busy. Below that threshold it usually loses, because idle accelerators and the engineering to operate them cost more than a hosted API. The honest comparison is not the headline per-token price; it is fully loaded cost including utilisation, on-call, and the people who keep it running.
The levers that move the bill
- Model routing: small model for easy requests, frontier model for the hard ones, with escalation on failure.
- Caching at three layers: exact-match, semantic, and provider prompt caching for shared prefixes.
- Token budgets per feature, covering retrieved context, history, and capped output length.
- Batching for non-interactive work, plus concurrency tuned to provider rate limits.
- Tier selection backed by an eval set, with self-hosting reserved for sustained high volume.
Measuring cost: the FinOps loop
You cannot optimise what you cannot attribute. Tag every request with the feature and the tenant that triggered it, and record tokens and cost alongside the response. Without per-feature and per-tenant attribution, the bill is one undifferentiated number and every optimisation is a guess. With it, you can see that one feature or one customer drives most of the spend, and aim your effort there.
The FinOps loop is the process that keeps cost from drifting back up. Visibility comes first: dashboards that break inference cost down by feature, tenant, and model tier. Optimisation follows: apply the levers above where the data points, and verify the saving rather than assuming it. Accountability closes the loop: give the teams that own each feature a view of its cost so the trade-off between quality and spend sits with the people making product decisions. Inference behaves like any other variable cloud cost, so the same loop that controls compute and storage works here.
How to prioritise
Do not start with prompt edits. Start by making the bill legible, then go after the structural levers in the order that pays back fastest for your traffic shape.
Order of attack
- Attribution first. Tag requests by feature and tenant and put cost on a dashboard, so every later change can be measured.
- Model routing next. It is usually the single largest reduction, because it moves the easy majority of traffic to a cheaper tier.
- Caching after that. Exact-match is quick to ship; add semantic and provider prompt caching where hit rates justify them.
- Token budgets and capped output. Cut context and response size feature by feature, guided by what the dashboard shows.
- Batching and tier review last. Move non-interactive work to batch rates, and re-test whether each task can drop a tier.
- Keep the FinOps loop running. Costs drift as traffic and features change, so visibility and accountability are ongoing, not a one-time pass.
Treated this way, cutting inference cost is a repeatable discipline that does not trade away quality. Route the request, avoid the call when you can, budget the tokens, and keep the numbers in front of the people who own the features. The savings hold because the process keeps running.