How to Cut Your AI Product's Model Costs Without Making It Worse

To reduce LLM costs without making your AI product worse, measure before you cut. Log the input, output and cached tokens of every model call with a feature name and a user id, add up cost per feature and per active user, and find the few calls that drive most of the bill. Then cut in order of least quality risk: trim prompts and context, cap output length, turn on prompt caching, move non-urgent work to a batch API, route easy tasks to a smaller model only after it passes your eval set, and stop paying for retries and runaway loops. Run the same eval set before and after every change, so a cheaper bill never means a worse product.

The startup and prices in the worked example below are made up. Use your provider's current pricing page for real numbers.

Why does an AI bill grow faster than revenue?

Most early AI products are built for speed, not cost. You pick the strongest model, paste the whole document into the prompt and keep the full chat history because it is one line of code. With ten users, nobody notices. With a few hundred, the API bill grows every week while revenue grows every month.

Usage is also lumpy. A few heavy users, one chatty feature or one background job can make up most of the spend. Cut blindly and you will shave a feature that costs nothing while missing the one that costs everything.

Pricing is covered in our guide to pricing an AI product when every user costs you different money. This post is about getting the same quality for less money.

How do you measure cost per feature and per user?

Your provider dashboard shows what you spent, not which feature or user spent it. You need one line of your own logging per model call.

Log every model call

OpenAI and Claude API responses include a usage block with token counts. Store it with your own context. A log line can be this simple:

{"ts":"2026-10-30T09:12:04Z","user_id":"u_123","feature":"call_summary","model":"big-model","input_tokens":9120,"cached_tokens":0,"output_tokens":884,"retries":0,"cost_usd":0.0253}

That cost uses made-up prices of $2 per million input tokens and $8 per million output tokens. Compute cost_usd from a small price table in your code and update it when prices change. Log retries and tool-call steps as their own rows, or you will never see them.

Build a cost-per-feature table

After a week, group the log by feature: calls per month, average input and output tokens, monthly cost and share of total. Divide total cost by monthly active users to get cost per user.

Sort by monthly cost. Usually one or two rows hold most of the money. That is your list.

Also sort users by cost. If your top five users cost more than the next fifty, the fix may be a limit or a plan change rather than an engineering project. The DEV write-up on metering an AI feature with per-user token quotas covers that guardrail.

What should you cut first?

Start with cuts that cannot change what the user sees, then move toward the ones that can. Savings are described loosely on purpose, because they depend on your prompts and traffic.

How do you trim prompts and context without losing quality?

Input tokens are usually the biggest line because you resend the same material on every call. Three places hide most of the waste:

• System prompt bloat. Every bug report adds a rule and nobody removes the old ones. Delete rules the model follows anyway, merge duplicates, and keep one or two examples instead of eight.

• Whole documents. If a user asks one question about a 40-page contract, sending all 40 pages on every message is the most expensive way to answer it. Retrieve the relevant sections instead, or summarise the document once and reuse the summary.

• Chat history. Resending every past turn makes each message cost more than the last. Keep the last few turns in full and replace older turns with a short running summary.

Here is what a trim looks like for a made-up sales-call assistant:

Trim one thing at a time and rerun your evals after each change. Some of those 31 rules exist for a reason.

Should you cap output length?

Yes, for most features. Output tokens usually cost more than input tokens (on Anthropic's current price table, five times more per token), and models are happy to write three paragraphs when one sentence would do. Set a maximum output token limit per feature, and say the length you want in the prompt, such as "three bullet points" or "under 80 words". The prompt instruction does most of the work. The hard limit is a safety net.

Test what happens when the limit is hit. A summary cut off mid-sentence looks broken, so leave headroom above the length you asked for.

How does prompt caching cut OpenAI and Claude API costs?

Prompt caching lets the provider reuse work on the identical start of a prompt, so repeated input tokens are billed at a lower rate. The rule that matters on both providers is the same: put stable content first and changing content last. System prompt, tool definitions and shared reference material go at the top. Timestamps, user names and the current question go at the end. One changed character early in the prompt breaks the match for everything after it.

The details differ. OpenAI's prompt caching guide says caching is on by default for supported models, that a prefix must meet a minimum length before it can be cached (1,024 tokens on its newer models, varying on earlier ones), and that cached input is billed at a reduced rate, discounted by up to 95 percent depending on the model. On its newer models, cache writes cost a bit more than normal input, so caching pays off when the prefix is actually reused.

Anthropic's prompt caching docs say you opt in with cache_control. The default cache lives for 5 minutes and is refreshed each time it is used, with a 1-hour option at extra cost. Writing to the 5-minute cache costs 1.25 times the base input price, and reading from it costs 0.1 times the base input price on most models. Prompts below a model's minimum length are simply not cached, with no error, so check the usage fields to confirm you are getting hits.

Track your cache hit rate per feature: cached input tokens divided by total input tokens. Near zero on a long fixed prompt means something early in the prompt changes on every call.

When does a batch API make sense?

Any job where nobody is waiting on the answer should not pay real-time prices: nightly digests, bulk tagging, backfilling old summaries and running your eval set.

OpenAI and Anthropic both sell this at half price. OpenAI's Batch API offers a 50 percent discount compared with its synchronous APIs, a separate pool of higher rate limits, and completion within 24 hours. Anthropic's Message Batches API charges 50 percent of standard prices, says most batches finish in under an hour, and expires requests that are not processed within 24 hours.

The cost is plumbing: submit requests, poll for completion, and match results back by your own id, because they can return in any order. Build it once and every background job can use it.

How do you route easy tasks to a cheaper model safely?

Model routing means sending each task to the cheapest model that does it well enough. Many products use their strongest model for everything, including classifying a message or writing a two-line email, which smaller models often handle fine.

Do it feature by feature, never as a global switch:

1. Pick one high-volume feature with a narrow, checkable job.

2. Run your eval set for that feature on the current model and on the cheaper one.

3. Switch only if the cheaper model passes the cases that matter. Read the failures, not only the score.

4. Ship to a slice of traffic first and watch complaints, retries and thumbs-down for a week.

If you do not have an eval set yet, build one before routing anything. Our guide to building an eval set for your AI product shows how to get 30 to 50 real cases in a few days.

What about response caching, retries and runaway agents?

Response caching is different from prompt caching: you store the full answer and skip the model call when an identical request comes in, such as the same document summarised twice. Key the cache on a hash of the model, prompt and settings, give answers an expiry, and skip it for anything personal or time-sensitive.

Retries and loops are the hidden cost. Check for these:

• Retry wrappers that retry bad requests that will never succeed.

• Double requests from a double click or a re-render.

• Agents with no step limit, looping on one confused task.

• Validation loops that ask the model to fix its own output again and again.

Cap retries, retry only on errors that can succeed, set a hard step limit per agent run, and alert when one user's daily cost jumps far above their normal. The SideHunt guide to keeping a side project's monthly bills under $20 has a simple habit for spend alerts and monthly bill checks.

How do you keep embeddings and RAG costs down?

Retrieval cuts cost when it replaces whole documents, but it adds its own waste. First, retrieve fewer chunks. If you pass ten and answers only use the top three, you pay for the rest on every question. Test fewer chunks against your eval set and keep the smallest number that does not hurt answers.

Second, do not re-embed content that has not changed. Store a hash of each chunk's text with its embedding and embed again only when the hash changes.

Is self-hosting an open-weight model worth it at small scale?

Usually not yet. You pay for GPUs whether or not anyone is using them, plus your time on deployment, scaling and monitoring. At early-stage volumes, API prices plus the cuts above tend to win. It starts to make sense with steady, high volume on a narrow task, a data rule that forbids APIs, or someone on the team who already runs this kind of infrastructure.

What is the quality guard for every cost cut?

No cost change ships without an eval run before and after: same cases, same grading, side by side. Log the date, the change, the cost difference per call and the eval difference.

Then watch real signals for a week: thumbs-down rate, regenerate clicks and support tickets about wrong answers. If any moves the wrong way, roll back first and investigate second.

A worked example: a made-up startup halves its model bill

Saltreed is invented for this example, and so are all of its numbers. It turns sales call transcripts into summaries and follow-up emails for small B2B teams, and it has 400 monthly active users. The arithmetic uses made-up example prices: a big model at $2 per million input tokens and $8 per million output tokens, and a small model at $0.40 and $1.60.

Before: where the money went

That is $3.54 per active user per month. Chat is about two thirds of the bill because every message resent the full transcript and conversation: 30,000 calls times 14,000 input tokens is 420 million tokens at $2 per million, or $840, plus 12 million output tokens at $8 per million, or $96.

After: four changes, one at a time

The founders changed one feature per week and ran their eval set before and after each change.

The new bill is $684.96, about $1.71 per active user and roughly half where it started. Prompt caching is left out because its saving depends on hit rates they had not measured yet. One change failed: sending only 2 transcript sections to chat made answers miss details, the eval set caught it, and they settled on more sections.

A 14-day plan to reduce LLM costs

AI model cost FAQ

What is a normal LLM cost per user for an early AI product?

There is no honest single number. It depends on your model, prompt length, usage and how heavy your top users are. Compute your own: monthly model spend divided by monthly active users, then the same per feature. What matters is that number against what each user pays you, and which way it is moving.

Does prompt caching reduce quality?

No. Both OpenAI and Anthropic say in their docs that prompt caching does not change how output is generated. It only changes how the repeated start of your prompt is processed and billed. What can hurt quality is reorganising a prompt badly to make it cache-friendly, so run your evals after reordering.

Should I switch to an open-source model to save money?

Not as a first step. Open-weight models can be cheaper for some tasks, but the cuts above are faster and lower risk. If you test one, treat it like any model switch: one feature at a time, against your eval set, with a way to roll back.

When should I raise prices instead of cutting costs?

When you have cut the obvious waste and heavy users still cost more than they pay, or users get far more value than the price reflects. Cost cuts fix waste, not a price set too low. Our post on AI pricing models when costs vary per user walks through that decision.

Once your AI product's costs make sense, you can launch it on EarlyHunt and put it in front of early adopters.