2026-01-14 03:09:18 +00:00
|
|
|
|
# AI Cost Management and Optimization\n\n**Status**: ✅ Production-Ready (cost tracking, budgets, caching benefits)\n\nComprehensive guide to managing LLM API costs, optimizing usage through caching and rate limiting, and tracking spending. The provisioning platform\nincludes built-in cost controls to prevent runaway spending while maximizing value.\n\n## Cost Overview\n\n### API Provider Pricing\n\n| | Provider | Model | Input | Output | Per MTok | |\n| | ---------- | ------- | ------- | -------- | ---------- | |\n| | **Anthropic** | Claude Sonnet 4 | $3 | $15 | $0.003 input / $0.015 output | |\n| | | Claude Opus 4 | $15 | $45 | Higher accuracy, longer context | |\n| | | Claude Haiku 4 | $0.80 | $4 | Fast, for simple queries | |\n| | **OpenAI** | GPT-4 Turbo | $0.01 | $0.03 | Per 1K tokens | |\n| | | GPT-4 | $0.03 | $0.06 | Legacy, avoid | |\n| | | GPT-4o | $5 | $15 | Per MTok | |\n| | **Local** | Llama 2, Mistral | Free | Free | Hardware cost only | |\n\n### Cost Examples\n\n```\nScenario 1: Generate simple database configuration\n - Input: 500 tokens (description + schema)\n - Output: 200 tokens (generated config)\n - Cost: (500 × $3 + 200 × $15) / 1,000,000 = $0.0045\n - With caching (hit rate 50%): $0.0023\n\nScenario 2: Deep troubleshooting analysis\n - Input: 5000 tokens (logs + context)\n - Output: 2000 tokens (analysis + recommendations)\n - Cost: (5000 × $3 + 2000 × $15) / 1,000,000 = $0.045\n - With caching (hit rate 70%): $0.0135\n\nScenario 3: Monthly usage (typical organization)\n - ~1000 config generations @ $0.005 = $5\n - ~500 troubleshooting calls @ $0.045 = $22.50\n - ~2000 form assists @ $0.002 = $4\n - ~200 agent executions @ $0.10 = $20\n - **Total: ~$50-100/month for small org**\n - **Total: ~$500-1000/month for large org**\n```\n\n## Cost Control Mechanisms\n\n### Request Caching\n\nCaching is the primary cost reduction strategy, cutting costs by 50-80%:\n\n```\nWithout Caching:\n User 1: "Generate PostgreSQL config" → API call → $0.005\n User 2: "Generate PostgreSQL config" → API call → $0.005\n Total: $0.010 (2 identical requests)\n\nWith LRU Cache:\n User 1: "Generate PostgreSQL config" → API call → $0.005\n User 2: "Generate PostgreSQL config" → Cache hit → $0.00001\n Total: $0.00501 (500x cost reduction for identical)\n\nWith Semantic Cache:\n User 1: "Generate PostgreSQL database config" → API call → $0.005\n User 2: "Create a PostgreSQL database" → Semantic hit → $0.00001\n (Slightly different wording, but same intent)\n Total: $0.00501 (near 500x reduction for similar)\n```\n\n### Cache Configuration\n\n```\n[ai.cache]\nenabled = true\ncache_type = "redis" # Distributed cache across instances\nttl_seconds = 3600 # 1-hour cache lifetime\n\n# Cache size limits\nmax_size_mb = 500\neviction_policy = "lru" # Least Recently Used\n\n# Semantic caching - cache similar queries\n[ai.cache.semantic]\nenabled = true\nsimilarity_threshold = 0.95 # Cache if 95%+ similar to previous query\ncache_embeddings = true # Cache embedding vectors themselves\n\n# Cache metrics\n[ai.cache.metrics]\ntrack_hit_rate = true\ntrack_space_usage = true\nalert_on_low_hit_rate = true\n```\n\n### Rate Limiting\n\nPrevent usage spikes from unexpected costs:\n\n```\n[ai.limits]\n# Per-request limits\nmax_tokens = 4096\nmax_input_tokens = 8192\nmax_output_tokens = 4096\n\n# Throughput limits\nrpm_limit = 60 # 60 requests per minute\nrpm_burst = 100 # Allow burst to 100\ndaily_request_limit = 5000 # Max 5000 requests/day\n\n# Cost limits\ndaily_cost_limit_usd = 100 # Stop at $100/day\nmonthly_cost_limit_usd = 2000 # Stop at $2000/month\n\n# Budget alerts\nwarn_at_percent = 80 # Warn when at 80% of daily budget\nstop_at_percent = 95 # Stop when at 95% of budget\n```\n\n### Workspace-Level Budgets\n\n```\n[ai.workspace_budgets]\n# Per-workspace cost limits\ndev.daily_limit_usd = 10\nstaging.daily_limit_usd = 50\nprod.daily_limit_usd = 100\n\n# Can override globally
|
