LLM Cost Optimization: Cut Your AI Bill Without Hurting Quality

The bill grows because new features, bigger models and hidden loops quietly add tokens, while the invoice shows only one total per model. LLM cost optimization starts by making that spending visible, feature by feature and customer by customer.

Picture a common setup. A meeting-notes app joins online meetings and writes a transcript: the full text of who said what. A large model then turns it into a summary and a list of action items. The setup is made up, but the problem is real.

Customers pay Rs 299 a month. Three months ago the app had 5,000 paying customers and an AI bill of about Rs 2 lakh (200,000) a month. This month, with the same 5,000 customers, the bill is Rs 9 lakh.

Revenue is 5,000 times Rs 299, about Rs 15 lakh. The AI bill now eats 60 percent of it. Each customer costs Rs 180 a month in AI and pays Rs 299.

The invoice from the model service shows one line per model and a total. It does not say which feature or which customer spent the money. Meanwhile the team added a chat feature called Ask your meetings, a follow-up email agent and a weekly digest email. It also moved every feature to the largest model, to be safe.

The first idea in the room is to switch everything to the cheapest model and hope. That is the naive fix, and a later section shows it breaking a feature. The better first move is one week of measuring.

You need five words to follow this guide. Each one gets a plain meaning and an example from the meeting-notes app.

Token. A token is a small piece of text that the model reads or writes, often part of a word. The word meeting might be one token, while a long or rare word may be split into several. Example: a one-hour transcript is about 12,000 tokens.

Input tokens and output tokens. Input tokens are everything you send to the model: your instructions plus the data. Output tokens are everything the model writes back. Example: the transcript is input; the summary is output.

Rate. The rate is the price per token, usually quoted per million tokens, with one price for input and a higher one for output. Example: Rs 250 per million input tokens and Rs 1,000 per million output tokens on the large model. These are illustrative numbers, not any real price list.

Unit cost. Unit cost is the AI cost of one useful thing your product does, such as one summary, one answered question or one customer for a month. Example: Rs 180 per customer per month is a unit cost, and it is the number the business cares about.

Test set. A test set is a fixed group of real examples with known good answers, used to check quality after every change. Example: the team keeps 200 past meetings where a person checked the correct action items.

LLM cost optimization is cutting the money spent per useful result, such as per summary or per customer, while quality stays where it was. It is a loop you keep running, not a one-time clean-up.

The loop has four parts. Measure cost per feature and per user. Fix the biggest line first. Check quality on the test set. Then add limits in code, so the saving does not drift back.

The FinOps Foundation, a group that works on cloud cost practice, puts the same step first in its paper on AI token costs. Its maturity path starts with visibility: tagging each call with the app and team that made it. It then adds unit costs, such as cost per query and per user.

Most model services charge per token: input tokens times the input rate, plus output tokens times the output rate. Output usually costs more per token. The FinOps Foundation paper, updated 3 June 2026, says input-to-output price ratios commonly fall between 1 to 3 and 1 to 5, varying by model and provider.

Think of the electricity meter at home. Units are tokens, and the bill is units times the tariff. Nobody asks whether a unit ran a useful fan or a forgotten light. A wasted token costs exactly as much as a useful one.

Where the picture breaks: your meter has one tariff, but a model service has two, and output costs more. You also cannot cap a fan's units per hour, but you can cap output tokens per call.

Here is one summary call in the meeting-notes app, before any fixes. Look at the two arrows into the model, then at the two bill boxes.

Rs 5.25 sounds tiny. But the app handles 40,000 meetings a month, so this one call costs Rs 2.1 lakh a month.

Here is the same maths in Python, so you can reuse it. Keep the prices in one place in your code, so a price change means editing one line.

PRICE_PER_MILLION = {                    # illustrative rupee prices, not a real price list
    "large": {"in": 250, "out": 1000},
    "small": {"in": 25, "out": 100},
}

def call_cost(model, input_tokens, output_tokens):
    price = PRICE_PER_MILLION[model]
    rupees = input_tokens * price["in"] + output_tokens * price["out"]
    return rupees / 1_000_000

per_meeting = call_cost("large", 15_000, 1_500)
print(per_meeting)            # 5.25
print(per_meeting * 40_000)   # 210000.0 rupees a month

PRICE_PER_MILLION stores the input and output rate for each model. call_cost multiplies each token count by its rate and adds the two. Then it divides by one million, because the rates are per million tokens.

What just happened? Run it with Python 3 and you should see 5.25 and 210000.0, both in rupees. If this maths and your invoice disagree, a token count or a price in your table is wrong.

Some tokens are easy to forget. They often explain a bill that looks too high.

• Chat history: the model itself does not remember earlier messages, so each new question usually carries the whole conversation as input again. The tenth question costs more than the first.
• Reasoning text: some models write step-by-step working before the answer. That is generated output too. In vLLM 0.32.0, an open-source model server released on 18 October 2026, it counts in the output total even when the request hides it.
• Thrown-away replies: a reply you reject, such as a cut-off one, was still generated, so its tokens are on the bill. A retry is a new call, billed again.
• Agent rounds: an agent re-sends its growing notes every round, so each round costs more than the one before.
• Language: the same meaning can take many more tokens in some languages than in English. One 2023 study found gaps of up to 15 times between languages, depending on the tokenizer (the part that splits text into tokens).

You send every AI call through one small function that saves the feature name, the user, the model and the token counts to a cost log. Then you add up the log by feature and by user.

You have to build this yourself. The FinOps paper notes that a provider's bill typically breaks usage down at most by API key or project. It has no idea of your features, teams or customers.

Here is a small cost log you can run. It uses SQLite, a database that ships with Python, and call_cost from the first code block. The three logged calls are illustrative.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE cost_log (feature TEXT, user_id TEXT, model TEXT, "
           "input_tokens INT, output_tokens INT, cost_rs REAL)")

def log_call(feature, user_id, model, input_tokens, output_tokens):
    cost = call_cost(model, input_tokens, output_tokens)
    db.execute("INSERT INTO cost_log VALUES (?, ?, ?, ?, ?, ?)",
               (feature, user_id, model, input_tokens, output_tokens, cost))

log_call("summary", "user_1", "large", 15_000, 1_500)
log_call("chat", "user_1", "large", 60_000, 300)
log_call("chat", "user_2", "large", 60_000, 300)

for row in db.execute("SELECT feature, COUNT(*), ROUND(SUM(cost_rs), 2) "
                      "FROM cost_log GROUP BY feature ORDER BY 3 DESC"):
    print(row)
# ('chat', 2, 30.6)
# ('summary', 1, 5.25)

Run both blocks together and you should see two rows. Chat comes first at Rs 30.6 for two calls, then summary at Rs 5.25 for one. The GROUP BY line adds up the rupees per feature, most expensive first.

In the real app, one function wraps every model call. It calls the model, reads the token counts from the reply and calls log_call. A function like this, which wraps another one and adds a job to it, is called a wrapper.

The team's rule: no code may call the model directly, so no spending can hide. Follow the calls from the features down to the model, and the cost rows sideways into the log.

The team follows four steps to get its first useful numbers.

1. 1

   Route every call through the wrapper

   Search the code for direct model calls and replace each one with the wrapper and a feature name.

2. 2

   Collect a few days of rows

   Let real traffic fill the cost log. Three normal weekdays are enough to see the pattern.

3. 3

   Check the total against the invoice

   Scale the log's daily total up to a month. If it is far from the invoice, some call is still skipping the wrapper or its counts.

4. 4

   Group by feature, then by user

   Add up rupees per feature first, then per customer inside the most expensive feature.

Here is what the cost log showed for a month, using made-up numbers. Notice that three of the six lines did not exist three months ago.

• Ask your meetings chat: Rs 2.6 lakh. Every question sent five full transcripts, about 60,000 tokens, so each question cost about Rs 15.30.
• Meeting summary: Rs 2.1 lakh, at Rs 5.25 per meeting, as worked out above.
• Action items: Rs 1.6 lakh. A separate call sent the whole transcript again, at Rs 4 per meeting.
• Follow-up email agent: Rs 1.0 lakh. Most runs were cheap, but some got stuck for 30 or more rounds.
• Action-item retries: Rs 0.8 lakh. Failed calls were being sent again up to 10 times.
• Weekly digest: Rs 0.9 lakh, sent to every customer every Monday.

The per-customer view had its own surprise. One customer had connected a script to the chat and asked 4,000 questions in a month. At Rs 15.30 a question, that one Rs 299 customer cost Rs 61,200.

You remove tokens the model does not need to read, and ask for only the output you will actually use. Every token you cut is cut from every single call, so small trims add up fast.

The team starts with the summary and action items, because together they cost Rs 3.7 lakh. It makes four changes.

• Shorter instructions: the 3,000-token prompt had 10 long examples. The team keeps 2 short ones and cuts repeated rules, bringing it to 1,000 tokens, including the action-item instructions.
• A cleaner transcript: the raw text had a timestamp on every line and many um and haan haan fillers. Removing them takes a one-hour transcript from 12,000 to 9,500 tokens.
• One call instead of two: summary and action items now come from the same call, so the transcript is sent once, not twice.
• A shorter answer: instead of summarise this meeting, the prompt asks for 5 bullet points and an action-item list. The answer drops from 2,000 tokens across both calls to 900.

Now redo the maths with the illustrative prices. Input is 1,000 plus 9,500, which is 10,500 tokens, costing about Rs 2.63. Output is 900 tokens, costing Rs 0.90. One meeting now costs about Rs 3.53 instead of Rs 9.25 for the two old calls.

Across 40,000 meetings, summary plus action items drop from Rs 3.7 lakh to about Rs 1.41 lakh a month, because the model simply reads and writes less.

The team also caps output at 1,200 tokens on this call. Model APIs let you set a maximum output length, often a field called max_completion_tokens or the older max_tokens. Generation stops there. It is only a safety net, because it can cut an answer off mid-sentence.

The chat gets the same treatment. Instead of five full transcripts, it now finds the parts of past meetings that match the question and sends only those. This is called retrieval: searching your own data first and sending only the matching pieces.

That is about 9,000 input tokens, instructions included. Each question drops from about Rs 15.30 to about Rs 2.55, and the chat bill falls from Rs 2.6 lakh to about Rs 43,000.

Research points the same way. A 2023 study compressed prompts by cutting them down token by token while trying to keep their meaning. It reported up to 20 times compression with little performance loss on its four test datasets. Your own test set still decides what is safe for your product.

Caching reuses work the model has already done, cheaper models handle easy jobs, and batching runs non-urgent jobs later at a lower price. Each one has its own guide, so here is just enough to know when to reach for it.

Caching, in brief

Prompt caching means the service reuses a repeated start of a prompt at a lower price. The FinOps paper reports 80 to 90 percent off input cost for the cached part, where a service offers it. The chat's instructions are the same for every question, so they fit well.

The Prompt Caching Explained guide covers how it works. This guide leaves caching out of the maths to keep it simple.

Cheaper models, in brief

Model tiering means using a smaller, cheaper model for easy jobs and the large model only where it is needed. Remember the naive fix of moving everything to the small model. The team tests it on the summary call first, and action items drop from 92 percent right to 81 percent.

That change is rejected. The weekly digest is different. It can be built from the summaries that already exist, so the small model does it just as well on the test set.

Research backs trying cheap models first. A 2023 study of model cascades reported matching the best single model at up to 98 percent lower cost on its test tasks. A cascade asks models one after another, cheap first, and stops when a scoring step judges an answer reliable. The LLM Model Selection per Task guide covers tiering and routing properly.

Batching for jobs that can wait

Batch mode means sending many requests together as one job and collecting the results later. The FinOps paper says batch pricing typically gives 50 percent off real-time prices. In return you wait: results come back within a window your provider states, not in seconds.

The weekly digest fits well. Nobody needs it until Monday morning, so every Saturday night the app sends that week's 5,000 digest requests as one batch. The digest now reads those short summaries instead of full transcripts.

Shorter input, the small model and batch pricing together bring the digest from Rs 90,000 to about Rs 2,000 a month, using illustrative prices. A summary someone is waiting for on screen must stay live; batch is only for work no person is waiting on.

Every retry and every agent round is a full, billed call. A loop with no limit can multiply the cost of one task by ten or more. You stop this by capping retries and rounds in your code.

A retry means sending the same request again after it fails. In the meeting-notes app, the old action-items code retried up to 10 times whenever the reply was not valid JSON (a simple text format for data).

The problem was long meetings. For about 2,000 very long meetings a month, the answer was longer than the output cap, so the JSON was cut off. Each retry had the same input and the same cap, so it was cut off again.

At an illustrative Rs 4 per try, each of those meetings cost Rs 40 for nothing: Rs 80,000 a month.

How do you know a reply was cut off? The reply says why generation stopped. In vLLM 0.32.0, the finish reason is length when the output cap or the model's context limit was reached. The context limit is the most tokens one call can hold, input and output together. Read that field before you retry.

The fixed flow is in the diagram below. Look at the question box in the middle: it asks whether trying again could possibly help, before it asks how many tries are left.

For very long meetings, the team also splits the transcript into one-hour parts before the call, so the answer fits. Retry costs fall from Rs 80,000 to about Rs 5,000 a month.

The follow-up email agent had the same disease in a different shape. It could look up contacts and calendars, then draft an email. When it could not find someone's email address, it kept searching, sometimes for 40 rounds.

The team adds two limits in code: at most 6 rounds per run, and at most Rs 5 of spending per run. When either limit is hit, the agent stops and asks the user for the missing address. The agent's bill falls from Rs 1 lakh to about Rs 15,000 a month.

Research makes the same point from the other side. A 2024 paper on how AI agents are tested argues that judging them on accuracy alone has made many needlessly costly. It suggests optimising cost and accuracy together.

You check a budget in your own code before each call. You also send an alert when spending crosses set lines, such as 80 and 100 percent of a daily budget. The invoice arrives too late to save you.

Remember the customer whose script asked 4,000 questions. The OWASP Top 10 for LLM Applications 2025, a list of top security risks for apps built on large models, calls this problem unbounded consumption. It includes denial of wallet: so many requests that the cost hurts the business.

OWASP suggests rate limits and user quotas that cap how many requests one source can make in a time window. It also suggests timeouts, throttling (slowing requests down rather than refusing them) and monitoring for unusual use.

The diagram shows the checks a chat question now passes before it reaches the model. Notice that each check can stop the call, and the cheapest checks come first.

Here are the budget rules the team writes into its code, all with made-up numbers.

• Per customer: 500 chat questions a month on the Rs 299 plan (a quota), and at most 20 a minute (a rate limit).
• Per feature: a daily rupee budget for each feature, read from the cost log. Chat gets Rs 2,000 a day.
• Per agent run: at most 6 rounds and Rs 5, as in the last section.
• Alerts: a message to the team at 80 and 100 percent of any daily budget, and another if one customer costs more than Rs 150 in a month.
• A last safety net: if the model service offers a monthly spending limit, set it above the plan so it only trips if everything else fails.

Alerts at 80 and 100 percent match the FinOps paper. It suggests measuring for 30 to 60 days first, then setting budgets at 110 to 120 percent of that baseline, your normal spend. It also warns that a provider's own rate limits count tokens per minute, not money per month.

What happens when a budget runs out is a product choice. The team keeps summaries running, because customers paid for them, and pauses only new chat questions with a clear message. Decide this before the alert fires at 2 a.m.

It depends on how much waste you start with, so nobody can promise a number before you measure. In the meeting-notes app, using made-up numbers, the bill fell from Rs 9 lakh to about Rs 2.06 lakh with quality held steady on the test set.

None of the big fixes needed a new model or a clever trick. They came from measuring, reading less, writing less and putting limits on loops. Summaries still run on the large model, because the test set showed the small one was worse.

The bill is back near where it started three months ago, with three more features than before. The chat line does not even count the new 500-question quota, which trims it a little more.

Each round re-sends everything that came before it. When each round adds a similar amount of notes, total input grows roughly with the square of the number of rounds. In long runs, twice the rounds approaches four times the input.

Take the email agent with illustrative sizes. Its base prompt is 1,000 tokens, each round adds 400 tokens of notes and tool results, and each round writes 200 output tokens.

def agent_run_cost(rounds, base=1_000, added=400, out_per_round=200):
    # round k re-sends the base prompt plus the notes from every earlier round
    total_in = sum(base + added * k for k in range(rounds))
    return call_cost("large", total_in, out_per_round * rounds)

for rounds in (6, 12, 40):
    print(rounds, round(agent_run_cost(rounds), 2))
# 6 4.2
# 12 12.0
# 40 96.0

Run it after the first code block. A 40-round run costs Rs 96 against Rs 4.20 for 6 rounds: about 23 times the cost for under 7 times the rounds. The last rounds are the most expensive ones.

The FinOps paper gives the extreme case: an agent that calls itself, or tools that recurse, can use millions of tokens in minutes. Chat has the same shape. Summarising old turns or keeping only recent ones, as the paper suggests, flattens the curve.

Usually some calls reach the log without their real token counts, or some tokens are priced at the wrong rate. Four causes cover most gaps. The server details below come from the source of vLLM 0.32.0, an open-source model server. Hosted APIs have their own switches, so check your provider's docs.

• Streamed replies without counts. A streamed reply arrives in pieces as it is written. In vLLM 0.32.0, the final piece carries token counts only if the request asks for them, or the server was started with a flag that forces them.
• Hidden reasoning. In the same server, completion_tokens counts every generated token. That includes reasoning text the request hid by setting include_reasoning to false.
• Rejected replies. Code often drops a badly formed reply before logging it, but its tokens were generated. Log every reply first, then validate it.
• Different rates. Cached input, batch jobs and price changes are billed at different rates. Store the rate used on each row, with the date it applied.

In vLLM 0.32.0, the streaming fix is one request option: set stream_options with include_usage to true. Server owners can instead start vLLM with the flag --enable-force-include-usage. Both were read in the 0.32.0 source as of 19 October 2026.

A check you can run: add up one day of rows and compare the total with your provider's usage page for the same day. Agree a tolerance first, such as 2 percent. A bigger gap means a call path is missing from the log.

Reserve the worst-case cost before each call and settle the real cost after it. A plain check of spending so far lets every call in flight through at once.

The real cost of a call is known only when it ends. If 20 questions arrive together, each one checks the budget, sees money left and runs. In our test run, 20 parallel calls passed a Rs 10 check-then-call budget and spent Rs 51 (illustrative prices, a simulated 50-millisecond call).

The worst case, though, is known before the call: the input tokens plus the output cap, at your rates. That is a second reason to set an output cap. Count the input with the model's tokenizer before sending, or round it up generously. Use a cap that covers every generated token, hidden reasoning included.

import threading

class Budget:
    def __init__(self, limit_rs):
        self.limit = limit_rs
        self.spent = 0.0
        self.lock = threading.Lock()

    def reserve(self, worst_case_rs):
        with self.lock:
            new_total = self.spent + worst_case_rs
            if max(new_total, self.limit) != self.limit:   # over the limit
                return False
            self.spent = new_total
            return True

    def settle(self, worst_case_rs, actual_rs):
        with self.lock:
            self.spent += actual_rs - worst_case_rs

budget = Budget(limit_rs=10)
worst = call_cost("large", 9_000, 1_200)   # input plus max output tokens
print(worst)                                # 3.45
print([budget.reserve(worst) for _ in range(3)])   # [True, True, False]
budget.settle(worst, 2.55)
budget.settle(worst, 2.55)
print(round(budget.spent, 2))               # 5.1
print(budget.reserve(worst))                # True

Run it after the first code block. The worst case is Rs 3.45, so two reservations fit in Rs 10 and the third is refused. After both calls settle at Rs 2.55, Rs 5.10 is spent and the next reservation succeeds. Call settle in a finally block, which runs even when the call fails. Use the reply's usage if there was one, so a failed call does not hold its worst case for good.

The lock makes check-and-add one step inside one process. With several servers, keep the counter in a shared database. Reserve with one UPDATE that adds the cost only where the new total stays within the limit, then check that one row changed.

Show that you would measure first, cut waste second and protect quality throughout. Interviewers want a calm plan with numbers, not a list of tricks.

1. 1

   State the unit cost

   Turn the bill into cost per request and per user, and compare it with what a user pays.

2. 2

   Show how you would measure

   Describe one wrapper around every model call that logs feature, user, model and tokens.

3. 3

   Cut tokens on the biggest line

   Shorter instructions, cleaner input, fewer repeated calls, and asking for a shorter answer.

4. 4

   Match the job to the price

   Mention caching for repeated prompt starts, smaller models for easy jobs, and batch mode for work that can wait.

5. 5

   Put limits in code

   Cap retries and agent rounds, reserve worst-case cost against budgets, add per-user quotas and alert before the invoice.

6. 6

   Guard quality

   Run a fixed test set before and after each change, and roll back anything that drops quality past an agreed limit.