The AI Bill Is Growing. Do We Really Need All Those Tokens?

Pradeep Kumar Vasudevan

Sep 16

4 min read

Views expressed here are my own, shaped by my experience and discussions with tech leads and practictioners

How organizations can reduce the cost of Generative and Agentic AI, now and as they scale?

AI experimentation is relatively inexpensive.

AI at enterprise scale may not be.

Once thousands of users, applications and agents start calling foundation models every day, small inefficiencies multiply quickly. Long prompts are repeatedly processed, entire documents are passed when only a few paragraphs are relevant, powerful models answer simple questions, and agents can make several LLM calls to complete a single task.

So perhaps the question should not simply be:

“How much does our model cost per million tokens?”

It should be:

“Why are we consuming those tokens in the first place?”

That distinction matters.

Where does the money go?

Most commercial LLM usage involves some combination of:

Input tokens + Cached tokens + Output tokens + Tool or Agent calls

With Agentic AI, one user request could result in several model interactions:

Plan → Search → Retrieve → Reason → Tool Call → Re-evaluate → Respond

A five-step agent is not necessarily one AI call. It could involve several.

Cost optimisation therefore needs to become part of AI architecture, not something Finance discovers after deployment.

What can organizations do quickly?

1. Stop sending the model information it doesn't need

Large context windows allow us to send huge amounts of information to an LLM.

That does not mean we should.

Instead of:

Question + 100-page document → LLM

retrieve only the information relevant to the question:

Question → Relevant evidence → LLM

Good RAG architecture is not only an accuracy technique. It can also be a cost-control mechanism.

The aim should be simple:

Give the model the minimum context required to make a good decision.

2. Cache what the model has already seen

Enterprise applications repeatedly send the same content:

System instructions, policies, tool definitions, schemas, documents and examples.

Reprocessing the same information on every request costs money.

Prompt caching allows reusable context to be processed once and reused where supported.

This means there is also an architectural question:

Are our prompts designed for reuse?

Stable instructions, policies and tool definitions should be separated from frequently changing user content wherever possible.

3. Don't use your smartest model for everything

A frontier model answering:

“Classify this email as Complaint, Query or Other” may work perfectly well.

But it may also be unnecessary.

A better approach is:

Request → Complexity Check → Appropriate Model

Simple request → smaller modelComplex request → stronger modelHigh-risk reasoning → advanced model

The future may therefore be less about selecting one enterprise LLM and more about building an intelligence routing layer.

4. Control the output too

Organisations often focus on reducing input tokens.

Output tokens cost money as well.

If an application only needs:

{ "risk": "LOW" }

there is little value in asking the model for three paragraphs explaining why.

Structured outputs, response limits and well-designed tool calls can reduce unnecessary generation.

The principle is simple:

Generate only what the person or system consuming the result actually needs.

5. Ask whether you need an LLM at all

Perhaps the cheapest token is the one you never generate.

Not every task requires AI reasoning.

If something can reliably be handled by:

then use the appropriate tool.

A well-designed agent should not first ask:

“Which model should solve this?”

It should ask:

“Does this task need a model at all?”

6. Batch workloads that can wait

Not every AI task needs a real-time response.

Document classification, embeddings, evaluations, summarisation and other bulk workloads may be processed asynchronously.

Separating:

Real-time AI

from

Can-wait AI

can create meaningful savings at scale.

What about the longer term?

Quick optimization helps.

But sustainable AI economics needs a broader architecture.

Build AI FinOps

Organizations already measure cloud infrastructure costs.

AI requires another level of visibility.

Instead of only measuring:

How much did we spend on LLMs this month?

measure:

Cost per customer interactionCost per investigationCost per document processedCost per agent taskCost per successful resolution

That means capturing information such as:

Model → Tokens → Cache Usage → Tool Calls → Latency → Cost → Outcome

This lets an organisation understand whether additional AI spend is creating additional business value.

Make model selection dynamic

Today a development team might configure:

Use Model X.

A more mature architecture could decide:

Use the least expensive model capable of meeting the quality, latency and risk requirements of this request.

The routing decision could consider:

Complexity + Risk + Cost + Latency + Confidence

This becomes particularly useful for agents.

A planning step may need a powerful reasoning model.

A classification step may need a smaller model.

A database lookup may need no LLM at all.

My Verdict

I don't think organisations should aim for the lowest possible token consumption.

They should aim for the lowest cost required to achieve the right outcome safely.

Sometimes spending more tokens on a difficult problem will be entirely justified.

The waste happens when we use the same level of intelligence, context and reasoning for every problem.

I see enterprise AI evolving towards:

Understand the task → Retrieve what matters → Check cache → Decide if an LLM is required → Select the appropriate model → Set a token budget → Execute → Measure the outcome

For me, the principle is simple:

Don't make AI cheaper by making it less intelligent. Make it cheaper by deciding where intelligence is actually required.