llm-cost-optimizer

---
name: llm-cost-optimizer
description: Use when LLM API costs come up in architecture, feature design, or model choice: high spend, single-model traffic, uncapped max_tokens, or no per-feature cost logging.
category: Engineering
version: 1.0.0
tools: []
---

# LLM Cost Optimizer

Cut LLM API spend by 40-80% without degrading user-facing quality, using model routing, caching, prompt compression, and observability to make every token count. Treat AI API costs as engineering costs -- measure first, optimize second, monitor always.

## Step 0: Classify Before You Ask

Classify which mode applies from what's already been said before asking follow-up questions:

| Mode | When to use |
|---|---|
| **Cost Audit** | Spend exists but no clear picture of where it goes |
| **Optimize Existing System** | Cost drivers are known; apply targeted fixes |
| **Design Cost-Efficient Architecture** | Building new AI features; wire in cost controls before launch |

If ambiguous, ask once for whatever context below isn't already known.

## Context Needed

Current LLM providers/models in use, monthly spend and which features drive it, whether token logging exists; target cost reduction, latency constraints, acceptable quality floor; request volume and token-count distribution (p50/p95/p99), repeated/similar prompts (caching potential), mix of task types.

## Mode 1: Cost Audit

Instrument first, optimize second. Log per-request: model, input tokens, output tokens, latency, endpoint/feature, user segment, calculated cost. Sort by feature x model x token count -- usually 2-3 endpoints drive the majority of spend; target those first. Classify each request by complexity:

| Complexity | Characteristics | Right Model Tier |
|---|---|---|
| Simple | Classification, extraction, yes/no, short output | Small (Haiku, GPT-4o-mini, Gemini Flash) |
| Medium | Summarization, structured output, moderate reasoning | Mid (Sonnet, GPT-4o) |
| Complex | Multi-step reasoning, code gen, long context | Large (Opus, o3) |

If token logging doesn't exist yet, that's the first deliverable -- not prompt compression, not routing. You cannot optimize what you cannot see.

## Mode 2: Optimize Existing System

Apply in ROI order; measure impact at each step before moving to the next.

1. **Model routing (60-80% cost reduction on routed traffic)** -- route by task complexity, not by default. Small models: classification, extraction, simple Q&A, formatting, short summaries. Mid models: structured output, moderate summarization, code completion. Large models: complex reasoning, long-context analysis, agentic tasks, code generation. Even routing 20% of traffic to a cheaper model produces meaningful savings.
2. **Prompt caching (40-90% reduction on cacheable traffic)** -- supported by Anthropic (`cache_control`), OpenAI (automatic on some models), Google (context caching). Cache system prompts, static context, document chunks, few-shot examples. Target hit rates: >60% for document Q&A, >40% for chatbots with static system prompts. Flag a system prompt over ~2,000 tokens sent on every request as a high-value target.
3. **Output length control (20-40% reduction)** -- force conciseness: explicit length instructions ("3 sentences or fewer"), schema-constrained JSON output, per-endpoint `max_tokens` hard caps (never global), stop sequences for list/structured output. An uncapped `max_tokens` endpoint is always a cost leak.
4. **Prompt compression (15-30% input token reduction)** -- remove filler without losing meaning ("Please carefully analyze the following text and provide..." becomes "Analyze:"); drop context already in the system prompt; strip HTML/markdown when plain text works. Over-compression causes hallucination and retries that erase the savings -- compress filler, preserve task-critical instructions.
5. **Semantic caching (30-60% hit rate on repeated queries)** -- cache responses keyed by embedding similarity rather than exact match (tools: GPTCache, LangChain cache, Redis + embedding lookup). Cosine similarity >0.95 is a safe threshold for serving a cached response.
6. **Request batching (10-25% reduction via amortized overhead)** -- batch non-latency-sensitive requests and process async queues off-peak.

## Mode 3: Design Cost-Efficient Architecture

Wire these in before launch -- retrofitting costs more.

- **Budget envelopes** per feature/user tier/day -- hard limits plus soft alerts at 80% of limit.
- **Routing layer**: classify -> route -> call. Never call the large model by default.
- **Tiered model access**: free users don't need the most expensive model; assign tiers by user tier at design time.
- **Cost observability**: spend by feature/model, cost per active user, week-over-week trend, anomaly alerts -- the monitoring foundation, not optional.
- **Graceful degradation** when budget is exceeded: smaller model -> cached response -> queue for async.

## Proactive Flags (surface unprompted)

| Signal | Action |
|---|---|
| No per-feature cost breakdown | Instrument logging before any other change |
| All requests hitting one model | Model monoculture is the #1 overspend pattern; propose routing |
| System prompt >2,000 tokens every request | Flag as a high-value caching target |
| `max_tokens` not set per endpoint | Flag as an active cost leak |
| No cost alerts configured | Spend spikes go undetected for days; set p95 cost-per-request alerts |
| Free-tier users on the same model as paid | Tier model access by user tier |

## Failure Modes and Recovery

| Situation | Response |
|---|---|
| No token logs exist | Stop -- a logging schema is deliverable #1; return once baseline data exists |
| Can't identify which feature drives spend | Provide an instrumentation plan; schedule a review after 2 weeks of data |
| Routing classifier adds too much latency | Fall back to rule-based routing (token-count thresholds, endpoint tags) |
| Cache hit rate below 20% | Diagnose prompt variability/dynamic context; recommend semantic caching or reconsider what's cached |
| Prompt compression degrades quality | Restore the section; flag it as compression-resistant |

## Handoff

If the conversation shifts to prompt quality/effectiveness, retrieval pipeline design, broader observability beyond cost, or latency profiling as the primary concern, @mention the teammate who owns that area instead of continuing inline.

## Output Artifacts

| Request | Deliverable |
|---|---|
| Cost audit | Per-feature spend breakdown, top 3 optimization targets, projected savings |
| Model routing design | Routing decision tree with model recommendations per task type and cost delta |
| Caching strategy | What to cache, cache key design, expected hit rate, implementation pattern |
| Prompt optimization | Token-by-token audit with before/after token counts |
| Architecture review | Cost-efficiency scorecard (0-100) with prioritized fixes and projected savings |

## Communication Standard

Bottom line first -- cost impact before explanation. Every finding states what, why, and how. Actions get owners and deadlines. Tag confidence: verified / medium / assumed.

## Anti-Patterns

| Anti-Pattern | Better Approach |
|---|---|
| Largest model for every request (wastes 5-10x on the 80%+ of requests that are simple) | Route by classified complexity to the cheapest adequate model |
| Optimizing prompts before measuring | Instrument token logging and cost-per-request first |
| Caching by exact string match only | Embedding-based semantic caching with a cosine similarity threshold |
| Single global max_tokens (wastes or truncates) | Set per endpoint from measured p95 output length |
| Ignoring system prompt size (3,000 tokens/request is a hidden multiplier) | Cache static system prompts; strip unnecessary instructions |
| Treating cost optimization as one-time | Continuous monitoring, weekly spend reports, anomaly alerts |
| Compressing to the point of ambiguity | Compress filler and redundant context; preserve task-critical instructions |