12 KiB
12 KiB
AI Cost Management and Optimization
Status: ✅ Production-Ready (cost tracking, budgets, caching benefits)
Comprehensive guide to managing LLM API costs, optimizing usage through caching and rate limiting, and tracking spending. The provisioning platform includes built-in cost controls to prevent runaway spending while maximizing value.
Cost Overview
API Provider Pricing
| Provider | Model | Input | Output | Per MTok | | | ---------- | ------- | ------- | -------- | ---------- | | | Anthropic | Claude Sonnet 4 | $3 | $15 | $0.003 input / $0.015 output | | | | Claude Opus 4 | $15 | $45 | Higher accuracy, longer context | | | | Claude Haiku 4 | $0.80 | $4 | Fast, for simple queries | | | OpenAI | GPT-4 Turbo | $0.01 | $0.03 | Per 1K tokens | | | | GPT-4 | $0.03 | $0.06 | Legacy, avoid | | | | GPT-4o | $5 | $15 | Per MTok | | | Local | Llama 2, Mistral | Free | Free | Hardware cost only | |
Cost Examples
Scenario 1: Generate simple database configuration
- Input: 500 tokens (description + schema)
- Output: 200 tokens (generated config)
- Cost: (500 × $3 + 200 × $15) / 1,000,000 = $0.0045
- With caching (hit rate 50%): $0.0023
Scenario 2: Deep troubleshooting analysis
- Input: 5000 tokens (logs + context)
- Output: 2000 tokens (analysis + recommendations)
- Cost: (5000 × $3 + 2000 × $15) / 1,000,000 = $0.045
- With caching (hit rate 70%): $0.0135
Scenario 3: Monthly usage (typical organization)
- ~1000 config generations @ $0.005 = $5
- ~500 troubleshooting calls @ $0.045 = $22.50
- ~2000 form assists @ $0.002 = $4
- ~200 agent executions @ $0.10 = $20
- **Total: ~$50-100/month for small org**
- **Total: ~$500-1000/month for large org**
Cost Control Mechanisms
Request Caching
Caching is the primary cost reduction strategy, cutting costs by 50-80%:
Without Caching:
User 1: "Generate PostgreSQL config" → API call → $0.005
User 2: "Generate PostgreSQL config" → API call → $0.005
Total: $0.010 (2 identical requests)
With LRU Cache:
User 1: "Generate PostgreSQL config" → API call → $0.005
User 2: "Generate PostgreSQL config" → Cache hit → $0.00001
Total: $0.00501 (500x cost reduction for identical)
With Semantic Cache:
User 1: "Generate PostgreSQL database config" → API call → $0.005
User 2: "Create a PostgreSQL database" → Semantic hit → $0.00001
(Slightly different wording, but same intent)
Total: $0.00501 (near 500x reduction for similar)
Cache Configuration
[ai.cache]
enabled = true
cache_type = "redis" # Distributed cache across instances
ttl_seconds = 3600 # 1-hour cache lifetime
# Cache size limits
max_size_mb = 500
eviction_policy = "lru" # Least Recently Used
# Semantic caching - cache similar queries
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.95 # Cache if 95%+ similar to previous query
cache_embeddings = true # Cache embedding vectors themselves
# Cache metrics
[ai.cache.metrics]
track_hit_rate = true
track_space_usage = true
alert_on_low_hit_rate = true
Rate Limiting
Prevent usage spikes from unexpected costs:
[ai.limits]
# Per-request limits
max_tokens = 4096
max_input_tokens = 8192
max_output_tokens = 4096
# Throughput limits
rpm_limit = 60 # 60 requests per minute
rpm_burst = 100 # Allow burst to 100
daily_request_limit = 5000 # Max 5000 requests/day
# Cost limits
daily_cost_limit_usd = 100 # Stop at $100/day
monthly_cost_limit_usd = 2000 # Stop at $2000/month
# Budget alerts
warn_at_percent = 80 # Warn when at 80% of daily budget
stop_at_percent = 95 # Stop when at 95% of budget
Workspace-Level Budgets
[ai.workspace_budgets]
# Per-workspace cost limits
dev.daily_limit_usd = 10
staging.daily_limit_usd = 50
prod.daily_limit_usd = 100
# Can override globally for specific workspaces
teams.team-a.monthly_limit = 500
teams.team-b.monthly_limit = 300
Cost Tracking
Track Spending
# View current month spending
provisioning admin costs show ai
# Forecast monthly spend
provisioning admin costs forecast ai --days-remaining 15
# Analyze by feature
provisioning admin costs analyze ai --by feature
# Analyze by user
provisioning admin costs analyze ai --by user
# Export for billing
provisioning admin costs export ai --format csv --output costs.csv
Cost Breakdown
Month: January 2025
Total Spending: $285.42
By Feature:
Config Generation: $150.00 (52%) [300 requests × avg $0.50]
Troubleshooting: $95.00 (33%) [80 requests × avg $1.19]
Form Assistance: $30.00 (11%) [5000 requests × avg $0.006]
Agents: $10.42 (4%) [20 runs × avg $0.52]
By Provider:
Anthropic (Claude): $200.00 (70%)
OpenAI (GPT-4): $85.42 (30%)
Local: $0 (0%)
By User:
alice@company.com: $50.00 (18%)
bob@company.com: $45.00 (16%)
...
other (20 users): $190.42 (67%)
By Workspace:
production: $150.00 (53%)
staging: $85.00 (30%)
development: $50.42 (18%)
Cache Performance:
Requests: 50,000
Cache hits: 35,000 (70%)
Cache misses: 15,000 (30%)
Cost savings from cache: ~$175 (38% reduction)
Optimization Strategies
Strategy 1: Increase Cache Hit Rate
# Longer TTL = more cache hits
[ai.cache]
ttl_seconds = 7200 # 2 hours instead of 1 hour
# Semantic caching helps with slight variations
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.90 # Lower threshold = more hits
# Result: Increase hit rate from 65% → 80%
# Cost reduction: 15% → 23%
Strategy 2: Use Local Models
[ai]
provider = "local"
model = "mistral-7b" # Free, runs on GPU
# Cost: Hardware ($5-20/month) instead of API calls
# Savings: 50-100 config generations/month × $0.005 = $0.25-0.50
# Hardware amortized cost: <$0.50/month on existing GPU
# Tradeoff: Slightly lower quality, 2x slower
Strategy 3: Use Haiku for Simple Tasks
Task Complexity vs Model:
Simple (form assist): Claude Haiku 4 ($0.80/$4)
Medium (config gen): Claude Sonnet 4 ($3/$15)
Complex (agents): Claude Opus 4 ($15/$45)
Example optimization:
Before: All tasks use Sonnet 4
- 5000 form assists/month: 5000 × $0.006 = $30
After: Route by complexity
- 5000 form assists → Haiku: 5000 × $0.001 = $5 (83% savings)
- 200 config gen → Sonnet: 200 × $0.005 = $1
- 10 agent runs → Opus: 10 × $0.10 = $1
Strategy 4: Batch Operations
# Instead of individual requests, batch similar operations:
# Before: 100 configs, 100 separate API calls
provisioning ai generate "PostgreSQL config" --output db1.ncl
provisioning ai generate "PostgreSQL config" --output db2.ncl
# ... 100 calls = $0.50
# After: Batch similar requests
provisioning ai batch --input configs-list.yaml
# Groups similar requests, reuses cache
# ... 3-5 API calls = $0.02 (90% savings)
Strategy 5: Smart Feature Enablement
[ai.features]
# Enable high-ROI features
config_generation = true # High value, moderate cost
troubleshooting = true # High value, higher cost
rag_search = true # Low cost, high value
# Disable low-ROI features if cost-constrained
form_assistance = false # Low value, non-zero cost (if budget tight)
agents = false # Complex, requires multiple calls
Budget Management Workflow
1. Set Budget
# Set monthly budget
provisioning config set ai.budget.monthly_limit_usd 500
# Set daily limit
provisioning config set ai.limits.daily_cost_limit_usd 50
# Set workspace limits
provisioning config set ai.workspace_budgets.prod.monthly_limit 300
provisioning config set ai.workspace_budgets.dev.monthly_limit 100
2. Monitor Spending
# Daily check
provisioning admin costs show ai
# Weekly analysis
provisioning admin costs analyze ai --period week
# Monthly review
provisioning admin costs analyze ai --period month
3. Adjust If Needed
# If overspending:
# - Increase cache TTL
# - Enable local models for simple tasks
# - Reduce form assistance (high volume, low cost but adds up)
# - Route complex tasks to Haiku instead of Opus
# If underspending:
# - Enable new features (agents, form assistance)
# - Increase rate limits
# - Lower cache hit requirements (broader semantic matching)
4. Forecast and Plan
# Current monthly run rate
provisioning admin costs forecast ai
# If trending over budget, recommend actions:
# - Reduce daily limit
# - Switch to local model for 50% of tasks
# - Increase batch processing
# If trending under budget:
# - Enable agents for automation workflows
# - Enable form assistance across all workspaces
Cost Allocation
Chargeback Models
Per-Workspace Model:
Development workspace: $50/month
Staging workspace: $100/month
Production workspace: $300/month
------
Total: $450/month
Per-User Model:
Each user charged based on their usage
Encourages efficiency
Difficult to track/allocate
Shared Pool Model:
All teams share $1000/month budget
Budget splits by consumption rate
Encourages optimization
Most flexible
Cost Reporting
Generate Reports
# Monthly cost report
provisioning admin costs report ai
--format pdf
--period month
--output cost-report-2025-01.pdf
# Detailed analysis for finance
provisioning admin costs report ai
--format xlsx
--include-forecasts
--include-optimization-suggestions
# Executive summary
provisioning admin costs report ai
--format markdown
--summary-only
Cost-Benefit Analysis
ROI Examples
Scenario 1: Developer Time Savings
Problem: Manual config creation takes 2 hours
Solution: AI config generation, 10 minutes (12x faster)
Time saved: 1.83 hours/config
Hourly rate: $100
Value: $183/config
AI cost: $0.005/config
ROI: 36,600x (far exceeds cost)
Scenario 2: Troubleshooting Efficiency
Problem: Manual debugging takes 4 hours
Solution: AI troubleshooting analysis, 2 minutes
Time saved: 3.97 hours
Value: $397/incident
AI cost: $0.045/incident
ROI: 8,822x
Scenario 3: Reduction in Failed Deployments
Before: 5% of 1000 deployments fail (50 failures)
Failure cost: $500 each (lost time, data cleanup)
Total: $25,000/month
After: With AI analysis, 2% fail (20 failures)
Total: $10,000/month
Savings: $15,000/month
AI cost: $200/month
Net savings: $14,800/month
ROI: 74:1
Advanced Cost Optimization
Hybrid Strategy (Recommended)
✓ Local models for:
- Form assistance (high volume, low complexity)
- Simple validation checks
- Document retrieval (RAG)
Cost: Hardware only (~$500 setup)
✓ Cloud API for:
- Complex generation (requires latest model capability)
- Troubleshooting (needs high accuracy)
- Agents (complex reasoning)
Cost: $50-200/month per organization
Result:
- 70% of requests → Local (free after hardware amortization)
- 30% of requests → Cloud ($50/month)
- 80% overall cost reduction vs cloud-only
Monitoring and Alerts
Cost Anomaly Detection
# Enable anomaly detection
provisioning config set ai.monitoring.anomaly_detection true
# Set thresholds
provisioning config set ai.monitoring.cost_spike_percent 150
# Alert if daily cost is 150% of average
# System alerts:
# - Daily cost exceeded by 10x normal
# - New expensive operation (agent run)
# - Cache hit rate dropped below 40%
# - Rate limit nearly exhausted
Alert Configuration
[ai.monitoring.alerts]
enabled = true
spike_threshold_percent = 150
check_interval_minutes = 5
[ai.monitoring.alerts.channels]
email = "ops@company.com"
slack = "[https://hooks.slack.com/..."](https://hooks.slack.com/...")
pagerduty = "integration-key"
# Alert thresholds
[ai.monitoring.alerts.thresholds]
daily_budget_warning_percent = 80
daily_budget_critical_percent = 95
monthly_budget_warning_percent = 70
Related Documentation
- Architecture - AI system overview
- Configuration - Cost control settings
- Security Policies - Cost-aware policies
- RAG System - Caching details
- ADR-015 - Design decisions
Last Updated: 2025-01-13 Status: ✅ Production-Ready Average Savings: 50-80% through caching Typical Cost: $50-500/month per organization ROI: 100:1 to 10,000:1 depending on use case