provisioning/docs/src/ai/cost-management.md

# AI Cost Management and Optimization

**Status**: ✅ Production-Ready (cost tracking, budgets, caching benefits)

Comprehensive guide to managing LLM API costs, optimizing usage through caching and rate limiting, and tracking spending. The provisioning platform
includes built-in cost controls to prevent runaway spending while maximizing value.

## Cost Overview

### API Provider Pricing

| Provider | Model | Input | Output | Per MTok |  |
| ---------- | ------- | ------- | -------- | ---------- |  |
| **Anthropic** | Claude Sonnet 4 | $3 | $15 | $0.003 input / $0.015 output |  |
|  | Claude Opus 4 | $15 | $45 | Higher accuracy, longer context |  |
|  | Claude Haiku 4 | $0.80 | $4 | Fast, for simple queries |  |
| **OpenAI** | GPT-4 Turbo | $0.01 | $0.03 | Per 1K tokens |  |
|  | GPT-4 | $0.03 | $0.06 | Legacy, avoid |  |
|  | GPT-4o | $5 | $15 | Per MTok |  |
| **Local** | Llama 2, Mistral | Free | Free | Hardware cost only |  |

### Cost Examples

```bash
Scenario 1: Generate simple database configuration
  - Input: 500 tokens (description + schema)
  - Output: 200 tokens (generated config)
  - Cost: (500 × $3 + 200 × $15) / 1,000,000 = $0.0045
  - With caching (hit rate 50%): $0.0023

Scenario 2: Deep troubleshooting analysis
  - Input: 5000 tokens (logs + context)
  - Output: 2000 tokens (analysis + recommendations)
  - Cost: (5000 × $3 + 2000 × $15) / 1,000,000 = $0.045
  - With caching (hit rate 70%): $0.0135

Scenario 3: Monthly usage (typical organization)
  - ~1000 config generations @ $0.005 = $5
  - ~500 troubleshooting calls @ $0.045 = $22.50
  - ~2000 form assists @ $0.002 = $4
  - ~200 agent executions @ $0.10 = $20
  - **Total: ~$50-100/month for small org**
  - **Total: ~$500-1000/month for large org**
```

## Cost Control Mechanisms

### Request Caching

Caching is the primary cost reduction strategy, cutting costs by 50-80%:

```bash
Without Caching:
  User 1: "Generate PostgreSQL config" → API call → $0.005
  User 2: "Generate PostgreSQL config" → API call → $0.005
  Total: $0.010 (2 identical requests)

With LRU Cache:
  User 1: "Generate PostgreSQL config" → API call → $0.005
  User 2: "Generate PostgreSQL config" → Cache hit → $0.00001
  Total: $0.00501 (500x cost reduction for identical)

With Semantic Cache:
  User 1: "Generate PostgreSQL database config" → API call → $0.005
  User 2: "Create a PostgreSQL database" → Semantic hit → $0.00001
  (Slightly different wording, but same intent)
  Total: $0.00501 (near 500x reduction for similar)
```

### Cache Configuration

```toml
[ai.cache]
enabled = true
cache_type = "redis"  # Distributed cache across instances
ttl_seconds = 3600    # 1-hour cache lifetime

# Cache size limits
max_size_mb = 500
eviction_policy = "lru"  # Least Recently Used

# Semantic caching - cache similar queries
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.95  # Cache if 95%+ similar to previous query
cache_embeddings = true      # Cache embedding vectors themselves

# Cache metrics
[ai.cache.metrics]
track_hit_rate = true
track_space_usage = true
alert_on_low_hit_rate = true
```

### Rate Limiting

Prevent usage spikes from unexpected costs:

```toml
[ai.limits]
# Per-request limits
max_tokens = 4096
max_input_tokens = 8192
max_output_tokens = 4096

# Throughput limits
rpm_limit = 60                    # 60 requests per minute
rpm_burst = 100                   # Allow burst to 100
daily_request_limit = 5000        # Max 5000 requests/day

# Cost limits
daily_cost_limit_usd = 100        # Stop at $100/day
monthly_cost_limit_usd = 2000     # Stop at $2000/month

# Budget alerts
warn_at_percent = 80              # Warn when at 80% of daily budget
stop_at_percent = 95              # Stop when at 95% of budget
```

### Workspace-Level Budgets

```toml
[ai.workspace_budgets]
# Per-workspace cost limits
dev.daily_limit_usd = 10
staging.daily_limit_usd = 50
prod.daily_limit_usd = 100

# Can override globally for specific workspaces
teams.team-a.monthly_limit = 500
teams.team-b.monthly_limit = 300
```

## Cost Tracking

### Track Spending

```bash
# View current month spending
provisioning admin costs show ai

# Forecast monthly spend
provisioning admin costs forecast ai --days-remaining 15

# Analyze by feature
provisioning admin costs analyze ai --by feature

# Analyze by user
provisioning admin costs analyze ai --by user

# Export for billing
provisioning admin costs export ai --format csv --output costs.csv
```

### Cost Breakdown

```bash
Month: January 2025

Total Spending: $285.42

By Feature:
  Config Generation:    $150.00 (52%) [300 requests × avg $0.50]
  Troubleshooting:      $95.00  (33%) [80 requests × avg $1.19]
  Form Assistance:      $30.00  (11%) [5000 requests × avg $0.006]
  Agents:               $10.42  (4%)  [20 runs × avg $0.52]

By Provider:
  Anthropic (Claude):   $200.00 (70%)
  OpenAI (GPT-4):       $85.42  (30%)
  Local:                $0      (0%)

By User:
  alice@company.com:    $50.00  (18%)
  bob@company.com:      $45.00  (16%)
  ...
  other (20 users):     $190.42 (67%)

By Workspace:
  production:           $150.00 (53%)
  staging:              $85.00  (30%)
  development:          $50.42  (18%)

Cache Performance:
  Requests: 50,000
  Cache hits: 35,000 (70%)
  Cache misses: 15,000 (30%)
  Cost savings from cache: ~$175 (38% reduction)
```

## Optimization Strategies

### Strategy 1: Increase Cache Hit Rate

```bash
# Longer TTL = more cache hits
[ai.cache]
ttl_seconds = 7200  # 2 hours instead of 1 hour

# Semantic caching helps with slight variations
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.90  # Lower threshold = more hits

# Result: Increase hit rate from 65% → 80%
# Cost reduction: 15% → 23%
```

### Strategy 2: Use Local Models

```toml
[ai]
provider = "local"
model = "mistral-7b"  # Free, runs on GPU

# Cost: Hardware ($5-20/month) instead of API calls
# Savings: 50-100 config generations/month × $0.005 = $0.25-0.50
# Hardware amortized cost: <$0.50/month on existing GPU

# Tradeoff: Slightly lower quality, 2x slower
```

### Strategy 3: Use Haiku for Simple Tasks

```bash
Task Complexity vs Model:

Simple (form assist): Claude Haiku 4 ($0.80/$4)
Medium (config gen): Claude Sonnet 4 ($3/$15)
Complex (agents): Claude Opus 4 ($15/$45)

Example optimization:
  Before: All tasks use Sonnet 4
  - 5000 form assists/month: 5000 × $0.006 = $30
  
  After: Route by complexity
  - 5000 form assists → Haiku: 5000 × $0.001 = $5 (83% savings)
  - 200 config gen → Sonnet: 200 × $0.005 = $1
  - 10 agent runs → Opus: 10 × $0.10 = $1
```

### Strategy 4: Batch Operations

```bash
# Instead of individual requests, batch similar operations:

# Before: 100 configs, 100 separate API calls
provisioning ai generate "PostgreSQL config" --output db1.ncl
provisioning ai generate "PostgreSQL config" --output db2.ncl
# ... 100 calls = $0.50

# After: Batch similar requests
provisioning ai batch --input configs-list.yaml
# Groups similar requests, reuses cache
# ... 3-5 API calls = $0.02 (90% savings)
```

### Strategy 5: Smart Feature Enablement

```toml
[ai.features]
# Enable high-ROI features
config_generation = true    # High value, moderate cost
troubleshooting = true      # High value, higher cost
rag_search = true           # Low cost, high value

# Disable low-ROI features if cost-constrained
form_assistance = false     # Low value, non-zero cost (if budget tight)
agents = false              # Complex, requires multiple calls
```

## Budget Management Workflow

### 1. Set Budget

```bash
# Set monthly budget
provisioning config set ai.budget.monthly_limit_usd 500

# Set daily limit
provisioning config set ai.limits.daily_cost_limit_usd 50

# Set workspace limits
provisioning config set ai.workspace_budgets.prod.monthly_limit 300
provisioning config set ai.workspace_budgets.dev.monthly_limit 100
```

### 2. Monitor Spending

```bash
# Daily check
provisioning admin costs show ai

# Weekly analysis
provisioning admin costs analyze ai --period week

# Monthly review
provisioning admin costs analyze ai --period month
```

### 3. Adjust If Needed

```bash
# If overspending:
# - Increase cache TTL
# - Enable local models for simple tasks
# - Reduce form assistance (high volume, low cost but adds up)
# - Route complex tasks to Haiku instead of Opus

# If underspending:
# - Enable new features (agents, form assistance)
# - Increase rate limits
# - Lower cache hit requirements (broader semantic matching)
```

### 4. Forecast and Plan

```bash
# Current monthly run rate
provisioning admin costs forecast ai

# If trending over budget, recommend actions:
# - Reduce daily limit
# - Switch to local model for 50% of tasks
# - Increase batch processing

# If trending under budget:
# - Enable agents for automation workflows
# - Enable form assistance across all workspaces
```

## Cost Allocation

### Chargeback Models

**Per-Workspace Model**:
```bash
Development workspace: $50/month
Staging workspace:     $100/month
Production workspace:  $300/month
------
Total:                 $450/month
```

**Per-User Model**:
```bash
Each user charged based on their usage
Encourages efficiency
Difficult to track/allocate
```

**Shared Pool Model**:
```bash
All teams share $1000/month budget
Budget splits by consumption rate
Encourages optimization
Most flexible
```

## Cost Reporting

### Generate Reports

```bash
# Monthly cost report
provisioning admin costs report ai 
  --format pdf 
  --period month 
  --output cost-report-2025-01.pdf

# Detailed analysis for finance
provisioning admin costs report ai 
  --format xlsx 
  --include-forecasts 
  --include-optimization-suggestions

# Executive summary
provisioning admin costs report ai 
  --format markdown 
  --summary-only
```

## Cost-Benefit Analysis

### ROI Examples

```bash
Scenario 1: Developer Time Savings
  Problem: Manual config creation takes 2 hours
  Solution: AI config generation, 10 minutes (12x faster)
  Time saved: 1.83 hours/config
  Hourly rate: $100
  Value: $183/config
  
  AI cost: $0.005/config
  ROI: 36,600x (far exceeds cost)

Scenario 2: Troubleshooting Efficiency
  Problem: Manual debugging takes 4 hours
  Solution: AI troubleshooting analysis, 2 minutes
  Time saved: 3.97 hours
  Value: $397/incident
  
  AI cost: $0.045/incident
  ROI: 8,822x

Scenario 3: Reduction in Failed Deployments
  Before: 5% of 1000 deployments fail (50 failures)
  Failure cost: $500 each (lost time, data cleanup)
  Total: $25,000/month
  
  After: With AI analysis, 2% fail (20 failures)
  Total: $10,000/month
  Savings: $15,000/month
  
  AI cost: $200/month
  Net savings: $14,800/month
  ROI: 74:1
```

## Advanced Cost Optimization

### Hybrid Strategy (Recommended)

```bash
✓ Local models for:
  - Form assistance (high volume, low complexity)
  - Simple validation checks
  - Document retrieval (RAG)
  Cost: Hardware only (~$500 setup)

✓ Cloud API for:
  - Complex generation (requires latest model capability)
  - Troubleshooting (needs high accuracy)
  - Agents (complex reasoning)
  Cost: $50-200/month per organization

Result:
  - 70% of requests → Local (free after hardware amortization)
  - 30% of requests → Cloud ($50/month)
  - 80% overall cost reduction vs cloud-only
```

## Monitoring and Alerts

### Cost Anomaly Detection

```bash
# Enable anomaly detection
provisioning config set ai.monitoring.anomaly_detection true

# Set thresholds
provisioning config set ai.monitoring.cost_spike_percent 150
# Alert if daily cost is 150% of average

# System alerts:
# - Daily cost exceeded by 10x normal
# - New expensive operation (agent run)
# - Cache hit rate dropped below 40%
# - Rate limit nearly exhausted
```

### Alert Configuration

```toml
[ai.monitoring.alerts]
enabled = true
spike_threshold_percent = 150
check_interval_minutes = 5

[ai.monitoring.alerts.channels]
email = "ops@company.com"
slack = "[https://hooks.slack.com/..."](https://hooks.slack.com/...")
pagerduty = "integration-key"

# Alert thresholds
[ai.monitoring.alerts.thresholds]
daily_budget_warning_percent = 80
daily_budget_critical_percent = 95
monthly_budget_warning_percent = 70
```

## Related Documentation

- [Architecture](architecture.md) - AI system overview
- [Configuration](configuration.md) - Cost control settings
- [Security Policies](security-policies.md) - Cost-aware policies
- [RAG System](rag-system.md) - Caching details
- [ADR-015](../architecture/adr/adr-015-ai-integration-architecture.md) - Design decisions

---

**Last Updated**: 2025-01-13
**Status**: ✅ Production-Ready
**Average Savings**: 50-80% through caching
**Typical Cost**: $50-500/month per organization
**ROI**: 100:1 to 10,000:1 depending on use case