In the very recent past when the frontier model providers were building their user bases, tokens were subsidized and provided below cost. The commentariat kept discussing how the cost of intelligence was heading towards zero. Early adopters gorged on tokens because they could. Companies had contests to see which department could use the most tokens, the idea being that token usage was a proxy for the level of adoption. Then, in the leadup to their IPOs, Anthropic and OpenAI needed to show stronger revenue, and their costs to enterprise users went up so significantly that the likes of Uber, Amazon and Microsoft have either canceled or greatly curtailed their usage. The days of tokenmaxxing are over.
There are lots of takeaways from these events. One might be that the frontier models don’t have as much pricing power as they thought they did. Another is that the apparent true cost of frontier model tokens makes that tier of AI technology uncompetitive with the cost of human employees. Yet another is that while this technology is unquestionably useful and valuable, it’s best to optimize costs.
There are several approaches to saving tokens, but you need to make sure that a savings approach isn’t just moving costs around instead of reducing them. To build a genuinely cost-negative optimization strategy, you have to audit the Incremental Cost Tax of each solution.
Dynamic Routing
Routers are excellent on paper, but they introduce a "double-inference" or orchestration tax.
- The Tax: To route a prompt dynamically, you must first evaluate it. If you use a lightweight model (like GPT-4o-mini or a local 7B model) as the router, you are paying a token or compute tax on every single request just to decide where it goes.
- The Break-Even Calculus: If your router costs $0.15 per million tokens to run, and it routes a prompt to an expensive model anyway, you just paid $1.05 for a $1.00 task. If using a locally hosted model as your router, the cost will be less.
- When it's worth it: Routing only saves money if there is a significant delta between your high-tier and low-tier models, and if the router successfully diverts enough requests to the cheaper tier to offset its own inference cost.
- Mitigation: Use deterministic, algorithmic routers instead of LLM-based routers wherever possible. Simple regex pattern matching, keyword intent detection, or rapid string-length checks cost fractions of a cent in standard code execution and incur zero token overhead.
Output Length Controls
LLM billing is asymmetric, with output tokens typically costing significantly more than input tokens. There are tools at your disposal to control the length and format of that output. Let's consider 3 techniques that control output length.
Structured outputs Conversational output is thus much more costly than receiving a succinct response in JSON format. Native JSON schemas like OpenAI's response_format or Gemini's response_schema allow you to specify JSON output. You get the answer without the expensive chatter.
The Tax: To enforce a rigid schema, providers append hidden instructions or complex grammar definitions to your system prompt behind the scenes. This increases your input token count slightly on the very first request.
Mitigation: The input tax is completely neutralized if the provider supports Prompt Caching. Because the schema rules remain static across all calls, you only pay the full prefill price once; subsequent requests hit the cache at a massive discount.
Explicit Reasoning Instruction You can include an explicit reasoning or thought_process key inside your JSON schema along with how you want the model to 'think', for example using target keys for sentiment or category. This tightly steers how the model reasons about the answer, with the result bounded by a JSON payload.
Something NOT to try: Setting a hard limit via the API configuration parameter (max_tokens or max_completion_tokens) is an important safety guardrail, but it's sometimes misused as a formatting tool. Don't try to use max_tokens for this purpose. If a model is only partway through a complex thought or JSON output and it hits the max, the result will be a partial/broken output that's unusable, requiring a prompt re-run. It's counter-productive.
The better way is to use explicit constraints within the prompt, for example, 'respond with a maximum of 4 sentences' or '4 bullet points'. Other than the few tokens required for these instructions, there is no tax.
Batch Processing
If your application can tolerate a delay in processing and use their batch APIs, major providers will give you a significant discount on both input and output tokens across virtually their entire model lineups.
Batching is a great cost-savings lever because it requires zero modifications to your prompt length, zero infrastructure overhead (like hosting vector databases), and zero drop-off in model intelligence. You get the exact same flagship model performance for less cost.
That said, the "cost tax" of batch processing is paid entirely in latency, orchestration complexity, and architectural constraints. When you submit requests via a standard API endpoint, you are competing for real-time compute cycles, and the provider charges a premium for instantaneous turnaround. With a Batch API, you submit a .jsonl file containing hundreds or millions of requests. The provider queues this file and runs it on their idle hardware capacity whenever global demand dips.
The latency tax stems from the fact the the provider generally guarantees that your batch will be completed within a 24-hour window. While many batches finish in a matter of minutes or hours depending on queue traffic, you can't design a synchronous, user-facing application around this framework. If a user is waiting on a screen for an answer, batch processing is not a fit.
The asynchronous nature of batch processing burdens your backend team. They'll need to set up processes to submit the batch, store the batch_id in a database, and establish a cron job or webhook listener to continuously poll the provider's API ("Is it done yet?").
And because you're handling data in bulk, your backend system must be engineered to compile, validate, and write massive .jsonl payloads to local disk or cloud object storage before uploading them.
Error recovery is similarly more complex. If a real-time API call fails, your standard code can immediately execute a retry loop. If a batch containing 10,000 tasks finishes 6 hours later, and 45 of those tasks failed due to internal provider errors or rate limits, your application must parse the output file, isolate the specific failures, compile a new mini-batch of just the failures, and submit it back into the 24-hour queue.
Prompt Compression
Algorithmic token compression promises to strip 30% of your prompt out before it hits the provider, but it doesn't happen by magic.
- The Tax: To compress a prompt using information entropy, you have to pass that prompt through a language model (like an inexpensive cloud model, Llama-3-8B or a specialized small BERT architecture) to calculate token perplexity.
- The Break-Even Calculus: You are trading Local/Cloud Compute Cycles + Latency for API Token Savings. If you are running the compression model on a paid cloud instance (like an AWS EC2 or an internal Kubernetes cluster), you need to calculate the cost per second of that GPU/CPU time. If compressing 1,000 tokens costs more in raw cloud compute than the 300 saved API tokens would have cost, it’s obviously not worth doing.
- When it's worth it: It shines when dealing with massive, multi-megabyte context windows (e.g., passing entire codebases or lengthy legal contracts) where a 40% reduction equals millions of frontier tokens saved per API call. For short conversational turns, the compute tax completely wipes out the token savings.
Provider-Level Prefix Caching
When an LLM processes a prompt, also known as prefill, it creates a Key-Value (KV) cache in GPU memory. Prefix caching allows the provider to save that KV cache. If your next prompt starts with the exact same text, the model skips the prefill phase entirely.
The amount of savings depends on the cloud provider, and the numbers frequently change, but cached tokens are typically priced substantially below uncached tokens.
The Incremental Cost Tax
- Financial Cost: Effectively Zero or Negative.
- Latency Impact: Highly positive. Skipping the prefill stage reduces the Time-to-First-Token (TTFT) by up to 80% on large prompts.
The caveat: Prefix caching is highly sensitive to prompt structure. If you write your prompt carelessly, you will invalidate the cache on every single call, completely wiping out the savings.
Semantic Caching
Instead of making an LLM call cheaper by optimizing provider GPU memory, semantic caching aims to bypass the LLM entirely for prompts with similar meaning.
This sits inside your application infrastructure and utilizes a fast vector database like Redis or pgvector. When a user submits a query, you generate an embedding of that query and check your local cache. If a previous user asked a question that is substantially semantically identical (you decide how close is close enough), your application returns the exact answer the LLM generated last time.
The True Incremental Cost Tax of Local Semantic Caching
Unlike provider prefix caching (which reduces your bill automatically), semantic caching requires financial and computational investment:
- The Local Embedding Overhead: To evaluate if an incoming query matches something in your cache, you must turn that query into a vector. This means you have to call an embedding model (like OpenAI's
text-embedding-3-smallor a local BERT/HuggingFace model) on every single user request, regardless of whether it results in a cache hit or miss. - Database Infrastructure Costs: You need to host, manage, and pay for an active vector index instance. If you run a high-throughput, managed vector service, the flat monthly hosting fees or capital cost might outpace the token savings of low-to-medium volume applications.
- The "Cache Miss" Penalty: When a query is unique, you pay the time and money to embed it, query your vector database, realize it's a miss, and then send the full prompt to the cloud LLM anyway. For highly dynamic applications, you've added an execution tax to your worst-case cost scenarios.
On-Prem / Local Hosting Note: Local hosting means on-prem or in a virtual private cloud that you own and control.
Moving to local hardware (like Apple Silicon clusters or dedicated enterprise workstation GPUs) changes your billing from OpEx (Variable API costs) to CapEx (Fixed hardware and maintenance costs).
The Tax: The incremental costs here are heavy and front-loaded:
Hardware Amortization: A high-end local inference machine (e.g., an M3/M4 Max or Ultra with 128GB+ RAM) represents thousands of dollars in upfront capital.
Electricity and Cooling: Running inference continuous-duty consumes measurable power.
Engineering/Maintenance Overhead: The time spent configuring drivers, updating llama.cpp or MLX, managing context windows, and handling concurrency queues is time not spent building core business features.
The Break-Even Calculus: Local hosting only breaks even when your token volume passes a specific threshold.
Break-Even Volume = (Cost of Hardware + Engineering Maintenance Hours) / Equivalent Cloud API Cost per Million Tokens
When it's worth it: High-throughput, predictable workloads. If you have an internal automation agent running 24/7 processing data extractions, local hardware pays for itself in months. If your workload is spiky, idle hardware represents underutilized capital.
Fine-Tuning
For both cloud and local models, instead of putting ten examples in every prompt (few-shotting), you can fine-tune a smaller model on those examples so it natively understands the output style.
The Tax: Training Costs: Compute costs to run the fine-tuning job (often requiring specialized cloud GPUs or paid provider fine-tuning endpoints (if not running locally).
Data Preparation Overhead: The labor cost of cleaning, structuring, and formatting hundreds of high-quality training pairs.
Hosting Surcharge if using Cloud: Many cloud providers charge a premium to host a custom fine-tuned model checkpoint compared to their standard, shared public endpoints.
When it's worth it: High-frequency, standardized tasks (like customer support categorization, code generation in a specific proprietary framework, or medical scribing). If you make millions of calls a month, stripping the few-shot examples out of the runtime prompt saves real money over time.
Small Models + Retrieval + Tooling
Smaller models almost always cost less to run than larger models, whether you're hosting them yourself or using a cloud API:
- Fewer parameters → fewer computations per token.
- Less memory usage → cheaper hardware requirements.
- Higher throughput → one GPU can serve more requests.
- Lower latency → responses are often faster.
If you have a small model, good retrieval, and a few examples, a small model may perform like a much more specialized model on a constrained task. There is a tradeoff, though:
- Smaller models benefit more from good prompts, retrieval, and fine-tuning.
- Larger models have stronger reasoning and generalization, so they need less task-specific help.
That's why the trend in industry has been toward "small model + retrieval + tooling" for many business workflows rather than using the largest model for every request.
Optimization Best Use Cases
| Technique | Primary Savings Mechanism | Incremental Cost Tax | Best Fit |
|---|---|---|---|
| Dynamic Routing | Send only hard requests to expensive models | Router inference/compute on every request; added latency | Mixed workloads with a large price gap between model tiers |
| Output Length Controls | Reduce completion tokens via concise prompts and structured outputs (JSON/schema) | Minimal engineering effort; risk of truncation/retries if hard token caps are misused | High-volume conversational or extraction workloads |
| Prompt Compression | Shrink large input contexts before API submission | Extra local/cloud compute plus latency for the compressor model | Very large prompts (codebases, contracts, long reports) |
| Provider-Level Prefix Caching | Reuse KV cache for identical prompt prefixes | Usually negligible; vendor-specific cache write semantics | Repeated system prompts, templates, and long static context |
| Local Semantic Caching | Bypass the LLM entirely for semantically equivalent queries | Embedding generation, vector DB infrastructure, cache-miss overhead | FAQ-style apps, support bots, repetitive enterprise queries |
| On-Prem / Local Hosting | Replace per-token API spend with owned infrastructure | Hardware amortization, power/cooling, engineering operations | Predictable, high-throughput workloads and data-residency requirements |
| Fine-Tuning (Replacing Few-Shot Examples) | Remove repeated examples from runtime prompts; improve task adherence | Training compute, data preparation, possible hosting surcharge | Millions of calls/month on standardized tasks |
| Small Models + Retrieval + Tooling | Use cheaper models for most requests; escalate only when needed | Retrieval infrastructure, prompt/tool engineering, occasional fallback to larger models | Classification, extraction, KB Q&A, routing, and constrained workflows |
This seems like a lot to think about, and it’s true that there are several options, each with its own set of nuances, but you only need one of these approaches to work to start saving money and lowering the threshold for when an AI implementation starts yielding ROI.
You can also take an incremental approach. The cheapest token is the one you never send, so first remove unnecessary output and context, then maximize cache hits and small-model usage, and only after that consider heavier-weight optimizations such as routing, compression, fine-tuning or small models, or local hosting.