Rate-Limiting, Costs, and Optimization

The Bill That Compounds

A single engineer using Claude Code heavily can generate hundreds of millions of tokens a month. Across a team, the numbers become genuinely expensive. Most overspend comes from habits, not essential usage — which means most of it is recoverable.

The four cost drivers, ranked

1. Context size per call — long-context calls dominate cost
2. Model tier — flagship models are 5-10x the price of mid-tier siblings
3. Call frequency — agentic loops can burn through tokens in minutes
4. Output length — long outputs cost 3-5x more per token than inputs

Cost per million tokens, roughly (April 2026)

The levers that actually work

• Prompt caching: 90% input discount after first call — always on for long context
• Right-size the model: use mid/small tier for 80% of work, flagship only when needed
• Limit context: paste only what's relevant, not the whole repo
• Cap max_tokens on output: 2048 is enough for most coding tasks
• Batch requests: one call with 5 questions beats 5 calls
• Model routing: a gateway picks the cheapest model that meets quality

A simple routing pattern

// Route simple tasks to cheap models, hard tasks to flagships. async function routeTask(task: Task) { const complexity = await classifyComplexity(task); // classifyComplexity uses a small/cheap model if (complexity === 'trivial') { return callModel('haiku', task); // $0.25/M } if (complexity === 'moderate') { return callModel('sonnet', task); // $3/M } return callModel('opus', task); // $15/M } // Rule of thumb: at scale, 70-80% of tasks route to cheap tiers. // Average cost drops 3-5x with no measurable quality loss.A router in front of your agent is the single highest-leverage optimization. Gateways like Vercel AI Gateway and LiteLLM do this with config.

Rate limits to know

• Provider-level: requests/minute and tokens/minute caps by tier
• Tool-level: Claude Code, Cursor, and Windsurf all enforce daily/weekly quotas
• Org-level: your company may set per-user caps; check before a big day
• Burst-level: retry with exponential backoff on 429s, never just hammer

Team FinOps for AI

1. Set per-user monthly budgets with alerts at 50% / 80% / 100%
2. Tag every API call with user, project, and feature for accurate attribution
3. Review spend weekly in the first month of new tool adoption, monthly after
4. Publish a leaderboard of efficient users — peer pressure beats policy
5. Negotiate enterprise volume discounts once spend is predictable

When self-hosting makes sense

At sustained high volume, self-hosting an open-weights model like Llama 4 or Qwen 3 can undercut API pricing. The crossover happens around $10-20k/month of API spend, higher if you need flagship quality. Below that, running GPUs is a distraction.

The first rule of AI cost optimization: the bill you can't see is the bill you can't control.

— A FinOps lead

The big idea: AI coding bills scale with habits, not just headcount. Caching, routing, context hygiene, and budget visibility get you 80% of the savings. Skip those and you're burning money the market is teaching other teams to keep.