## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# ADR-012: Three-Tier LLM Routing (Rules + Dynamic + Override)

Status: Accepted | Implemented Date: 2024-11-01 Deciders: LLM Architecture Team Technical Story: Balancing predictability (static rules) with flexibility (dynamic selection) in provider routing

## Decision

Implementar three-tier routing system para seleción de LLM providers: Rules → Dynamic → Override.

## Rationale

1. Rules-Based: Predictable routing para tareas conocidas (Architecture → Claude Opus)
2. Dynamic: Runtime selection basado en availability, latency, budget
3. Override: Manual selection con audit logging para troubleshooting/testing
4. Balance: Combinación de determinismo y flexibilidad

## Alternatives Considered

### ❌ Static Rules Only

- Pros: Predictable, simple
- Cons: No adaptación a provider failures, no dynamic cost optimization

### ❌ Dynamic Only

- Pros: Flexible, adapts to runtime conditions
- Cons: Unpredictable routing, harder to debug, cold-start problem

### ✅ Three-Tier Hybrid (CHOSEN)

- Predictable baseline + flexible adaptation + manual override

## Trade-offs

Pros:

- ✅ Predictable baseline (rules)
- ✅ Automatic adaptation (dynamic)
- ✅ Manual control when needed (override)
- ✅ Audit trail of decisions
- ✅ Graceful degradation

Cons:

- ⚠️ Added complexity (3 selection layers)
- ⚠️ Rule configuration maintenance
- ⚠️ Override can introduce inconsistency if overused

## Implementation

Tier 1: Rules-Based Routing:

    #![allow(unused)]
    fn main() {
    // crates/vapora-llm-router/src/router.rs
    pub struct RoutingRules {
        rules: Vec<(Pattern, ProviderId)>,
    }

    impl RoutingRules {
        pub fn apply(&self, task: &Task) -> Option<ProviderId> {
            for (pattern, provider) in &self.rules {
                if pattern.matches(&task.description) {
                    return Some(provider.clone());
                }
            }
            None
        }
    }

    // Example rules
    let rules = vec![
        (Pattern::contains("architecture"), "claude-opus"),
        (Pattern::contains("code generation"), "gpt-4"),
        (Pattern::contains("quick query"), "gemini-flash"),
        (Pattern::contains("test"), "ollama"),
    ];
    }

Tier 2: Dynamic Selection:

    #![allow(unused)]
    fn main() {
    pub async fn select_dynamic(
        task: &Task,
        providers: &[LLMClient],
    ) -> Result<&LLMClient> {
        // Score providers by: availability, latency, cost
        let scores: Vec<(ProviderId, f64)> = providers
            .iter()
            .map(|p| {
                let availability = check_availability(p).await;
                let latency = estimate_latency(p).await;
                let cost = get_cost_per_token(p);

                let score = availability * 0.5
                          - latency_penalty(latency) * 0.3
                          - cost_penalty(cost) * 0.2;
                (p.id.clone(), score)
            })
            .collect();

        // Select highest scoring provider
        scores
            .into_iter()
            .max_by(|a, b| a.1.partial_cmp(&b.1).unwrap())
            .ok_or(Error::NoProvidersAvailable)
    }
    }

Tier 3: Manual Override:

    #![allow(unused)]
    fn main() {
    pub async fn route_task(
        task: &Task,
        override_provider: Option<ProviderId>,
    ) -> Result<String> {
        let provider_id = if let Some(override_id) = override_provider {
            // Tier 3: Manual override (log for audit)
            audit_log::log_override(&task.id, &override_id, &current_user())?;
            override_id
        } else if let Some(rule_provider) = apply_routing_rules(task) {
            // Tier 1: Rules-based
            rule_provider
        } else {
            // Tier 2: Dynamic selection
            select_dynamic(task, &self.providers).await?.id.clone()
        };

        self.clients
            .get(&provider_id)
            .complete(&task.prompt)
            .await
    }
    }

Configuration:

    # config/llm-routing.toml

    # Tier 1: Rules
    [[routing_rules]]
    pattern = "architecture"
    provider = "claude"
    model = "claude-opus"

    [[routing_rules]]
    pattern = "code_generation"
    provider = "openai"
    model = "gpt-4"

    [[routing_rules]]
    pattern = "quick_query"
    provider = "gemini"
    model = "gemini-flash"

    [[routing_rules]]
    pattern = "test"
    provider = "ollama"
    model = "llama2"

    # Tier 2: Dynamic scoring weights
    [dynamic_scoring]
    availability_weight = 0.5
    latency_weight = 0.3
    cost_weight = 0.2

    # Tier 3: Override audit settings
    [override_audit]
    log_all_overrides = true
    require_reason = true

Key Files:

- `/crates/vapora-llm-router/src/router.rs` (routing logic)
- `/crates/vapora-llm-router/src/config.rs` (rule definitions)
- `/crates/vapora-backend/src/audit.rs` (override logging)

## Verification

    # Test rules-based routing
    cargo test -p vapora-llm-router test_rules_routing

    # Test dynamic scoring
    cargo test -p vapora-llm-router test_dynamic_scoring

    # Test override with audit logging
    cargo test -p vapora-llm-router test_override_audit

    # Integration test: task routing through all tiers
    cargo test -p vapora-llm-router test_full_routing_pipeline

    # Verify audit trail
    cargo run -p vapora-backend -- audit query --type llm_override --limit 50

Expected Output:

- Rules correctly match task patterns
- Dynamic scoring selects best available provider
- Overrides logged with user and reason
- Fallback to next tier if previous fails
- All three tiers functional and audited

## Consequences

### Operational

- Routing rules maintained in Git (versioned)
- Dynamic scoring requires provider health checks
- Overrides tracked in audit trail for compliance

### Performance

- Rule matching: O(n) patterns (pre-compiled for speed)
- Dynamic scoring: Concurrent provider checks (~50ms)
- Override bypasses both: immediate execution

### Monitoring

- Track which tier was used per request
- Alert if dynamic tier used frequently (rules insufficient)
- Report override usage patterns (identify gaps in rules)

### Debugging

- Audit trail shows exact routing decision
- Reason recorded for overrides
- Helps identify rule gaps or misconfiguration

## References

- `/crates/vapora-llm-router/src/router.rs` (routing implementation)
- `/crates/vapora-llm-router/src/config.rs` (rule configuration)
- `/crates/vapora-backend/src/audit.rs` (audit logging)
- ADR-007 (Multi-Provider LLM)
- ADR-015 (Budget Enforcement)

Related ADRs: ADR-007 (Multi-Provider), ADR-015 (Budget), ADR-016 (Cost Efficiency)