provisioning/docs/src/infrastructure/batch-workflow-multi-provider.md

# Multi-Provider Batch Workflow Examples

This document provides practical examples of orchestrating complex deployments and operations across multiple cloud providers using the batch workflow
system.

## Table of Contents

- [Overview](#overview)
- [Workflow 1: Coordinated Multi-Provider Deployment](#workflow-1-coordinated-multi-provider-deployment)
- [Workflow 2: Multi-Provider Disaster Recovery Failover](#workflow-2-multi-provider-disaster-recovery-failover)
- [Workflow 3: Cost Optimization Workload Migration](#workflow-3-cost-optimization-workload-migration)
- [Workflow 4: Multi-Region Database Replication](#workflow-4-multi-region-database-replication)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)

## Overview

The batch workflow system enables declarative orchestration of operations across multiple providers with:

- **Dependency Tracking**: Define what must complete before what
- **Error Handling**: Automatic rollback on failure
- **Idempotency**: Safe to re-run workflows
- **Status Tracking**: Real-time progress monitoring
- **Recovery Checkpoints**: Resume from failure points

## Workflow 1: Coordinated Multi-Provider Deployment

**Use Case**: Deploy web application across DigitalOcean, AWS, and Hetzner with proper sequencing and dependencies.

**Workflow Characteristics**:
- Database created first (dependencies)
- Backup storage ready before compute
- Web servers scale once database ready
- Health checks before considering complete

### Workflow Definition

```
# file: workflows/multi-provider-deployment.yml

name: multi-provider-app-deployment
version: "1.0"
description: "Deploy web app across three cloud providers"

parameters:
  do_region: "nyc3"
  aws_region: "us-east-1"
  hetzner_location: "nbg1"
  web_server_count: 3

phases:
  # Phase 1: Create backup storage first (independent)
  - name: "provision-backup-storage"
    provider: "hetzner"
    description: "Create backup storage volume in Hetzner"

    operations:
      - id: "create-backup-volume"
        action: "create-volume"
        config:
          name: "webapp-backups"
          size: 500
          location: "{{ hetzner_location }}"
          format: "ext4"

        tags: ["storage", "backup"]

    on_failure: "alert"
    on_success: "proceed"

  # Phase 2: Create database (independent, but must complete before app)
  - name: "provision-database"
    provider: "aws"
    description: "Create managed PostgreSQL database"
    depends_on: []  # Can run in parallel with Phase 1

    operations:
      - id: "create-rds-instance"
        action: "create-db-instance"
        config:
          identifier: "webapp-db"
          engine: "postgres"
          engine_version: "14.6"
          instance_class: "db.t3.medium"
          allocated_storage: 100
          multi_az: true
          backup_retention_days: 30

        tags: ["database", "primary"]

      - id: "create-security-group"
        action: "create-security-group"
        config:
          name: "webapp-db-sg"
          description: "Security group for RDS"

        depends_on: ["create-rds-instance"]

      - id: "configure-db-access"
        action: "authorize-security-group"
        config:
          group_id: "{{ create-security-group.id }}"
          protocol: "tcp"
          port: 5432
          cidr: "10.0.0.0/8"

        depends_on: ["create-security-group"]

        timeout: 60

  # Phase 3: Create web tier (depends on database being ready)
  - name: "provision-web-tier"
    provider: "digitalocean"
    description: "Create web servers and load balancer"
    depends_on: ["provision-database"]  # Wait for database

    operations:
      - id: "create-droplets"
        action: "create-droplet"
        config:
          name: "web-server"
          size: "s-2vcpu-4gb"
          region: "{{ do_region }}"
          image: "ubuntu-22-04-x64"
          count: "{{ web_server_count }}"
          backups: true
          monitoring: true

        tags: ["web", "production"]

        timeout: 300
        retry:
          max_attempts: 3
          backoff: exponential

      - id: "create-firewall"
        action: "create-firewall"
        config:
          name: "web-firewall"
          inbound_rules:
            - protocol: "tcp"
              ports: "22"
              sources: ["0.0.0.0/0"]
            - protocol: "tcp"
              ports: "80"
              sources: ["0.0.0.0/0"]
            - protocol: "tcp"
              ports: "443"
              sources: ["0.0.0.0/0"]

        depends_on: ["create-droplets"]

      - id: "create-load-balancer"
        action: "create-load-balancer"
        config:
          name: "web-lb"
          algorithm: "round_robin"
          region: "{{ do_region }}"
          forwarding_rules:
            - entry_protocol: "http"
              entry_port: 80
              target_protocol: "http"
              target_port: 80
            - entry_protocol: "https"
              entry_port: 443
              target_protocol: "http"
              target_port: 80
          health_check:
            protocol: "http"
            port: 80
            path: "/health"
            interval: 10

        depends_on: ["create-droplets"]

  # Phase 4: Network configuration (depends on all resources)
  - name: "configure-networking"
    description: "Setup VPN tunnels and security between providers"
    depends_on: ["provision-web-tier"]

    operations:
      - id: "setup-vpn-tunnel-do-aws"
        action: "create-vpn-tunnel"
        config:
          source_provider: "digitalocean"
          destination_provider: "aws"
          protocol: "ipsec"
          encryption: "aes-256"

        timeout: 120

      - id: "setup-vpn-tunnel-aws-hetzner"
        action: "create-vpn-tunnel"
        config:
          source_provider: "aws"
          destination_provider: "hetzner"
          protocol: "ipsec"
          encryption: "aes-256"

  # Phase 5: Validation and verification
  - name: "verify-deployment"
    description: "Verify all resources are operational"
    depends_on: ["configure-networking"]

    operations:
      - id: "health-check-droplets"
        action: "run-health-check"
        config:
          targets: "{{ create-droplets.ips }}"
          endpoint: "/health"
          expected_status: 200
          timeout: 30

        timeout: 300

      - id: "health-check-database"
        action: "verify-database"
        config:
          host: "{{ create-rds-instance.endpoint }}"
          port: 5432
          database: "postgres"
          timeout: 30

      - id: "health-check-backup"
        action: "verify-volume"
        config:
          volume_id: "{{ create-backup-volume.id }}"
          status: "available"

# Rollback strategy: if any phase fails
rollback:
  strategy: "automatic"
  on_phase_failure: "rollback-previous-phases"
  preserve_data: true

# Notifications
notifications:
  on_start: "slack:#deployments"
  on_phase_complete: "slack:#deployments"
  on_failure: "slack:#alerts"
  on_success: "slack:#deployments"

# Validation checks
pre_flight:
  - check: "credentials"
    description: "Verify all provider credentials"
  - check: "quotas"
    description: "Verify sufficient quotas in each provider"
  - check: "dependencies"
    description: "Verify all dependencies are available"
```

### Execution Flow

```
┌─────────────────────────────────────────────────────────┐
│ Start Deployment                                        │
└──────────────────┬──────────────────────────────────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
        ▼                     ▼
   ┌─────────────┐    ┌──────────────────┐
   │  Hetzner    │    │      AWS         │
   │  Backup     │    │   Database       │
   │ (Phase 1)   │    │   (Phase 2)      │
   └──────┬──────┘    └────────┬─────────┘
          │                    │
          │ Ready              │ Ready
          └────────┬───────────┘
                   │
                   ▼
            ┌──────────────────┐
            │  DigitalOcean    │
            │   Web Tier       │
            │  (Phase 3)       │
            │ - Droplets       │
            │ - Firewall       │
            │ - Load Balancer  │
            └────────┬─────────┘
                     │
                     ▼
            ┌──────────────────┐
            │  Network Setup   │
            │  (Phase 4)       │
            │ - VPN Tunnels    │
            └────────┬─────────┘
                     │
                     ▼
            ┌──────────────────┐
            │  Verification    │
            │  (Phase 5)       │
            │ - Health Checks  │
            └────────┬─────────┘
                     │
                     ▼
            ┌──────────────────┐
            │  Deployment OK   │
            │  (Ready to use)  │
            └──────────────────┘
```

## Workflow 2: Multi-Provider Disaster Recovery Failover

**Use Case**: Automated failover from primary provider (DigitalOcean) to backup provider (Hetzner) on detection of failure.

**Workflow Characteristics**:
- Continuous health monitoring
- Automatic failover trigger
- Database promotion
- DNS update
- Verification before considering complete

### Workflow Definition

```
# file: workflows/multi-provider-dr-failover.yml

name: multi-provider-dr-failover
version: "1.0"
description: "Automated failover from DigitalOcean to Hetzner"

parameters:
  primary_provider: "digitalocean"
  backup_provider: "hetzner"
  dns_provider: "aws"
  health_check_threshold: 3

phases:
  # Phase 1: Monitor primary provider
  - name: "monitor-primary"
    description: "Continuous health monitoring of primary"

    operations:
      - id: "health-check-primary"
        action: "run-health-check"
        config:
          provider: "{{ primary_provider }}"
          resources: ["web-servers", "load-balancer"]
          checks:
            - type: "http"
              endpoint: "/health"
              expected_status: 200
            - type: "database"
              host: "db.primary.example.com"
              query: "SELECT 1"
            - type: "connectivity"
              test: "ping"
          interval: 30  # Check every 30 seconds

        timeout: 300

      - id: "aggregate-health"
        action: "aggregate-metrics"
        config:
          source: "{{ health-check-primary.results }}"
          failure_threshold: 3  # 3 consecutive failures trigger failover

  # Phase 2: Trigger failover (conditional on failure)
  - name: "trigger-failover"
    description: "Activate disaster recovery if primary fails"
    depends_on: ["monitor-primary"]
    condition: "{{ aggregate-health.status }} == 'FAILED'"

    operations:
      - id: "alert-on-failure"
        action: "send-notification"
        config:
          type: "critical"
          message: "Primary provider ({{ primary_provider }}) has failed. Initiating failover..."
          recipients: ["ops-team@example.com", "slack:#alerts"]

      - id: "enable-backup-infrastructure"
        action: "scale-up"
        config:
          provider: "{{ backup_provider }}"
          target: "warm-standby-servers"
          desired_count: 3
          instance_type: "cx31"

        timeout: 300
        retry:
          max_attempts: 3

      - id: "promote-database-replica"
        action: "promote-read-replica"
        config:
          provider: "aws"
          replica_identifier: "backup-db-replica"
          to_master: true

        timeout: 600  # Allow time for promotion

  # Phase 3: Network failover
  - name: "network-failover"
    description: "Switch traffic to backup provider"
    depends_on: ["trigger-failover"]

    operations:
      - id: "update-load-balancer"
        action: "reconfigure-load-balancer"
        config:
          provider: "{{ dns_provider }}"
          record: "api.example.com"
          old_backend: "do-lb-{{ primary_provider }}"
          new_backend: "hz-lb-{{ backup_provider }}"

      - id: "update-dns"
        action: "update-dns-record"
        config:
          provider: "route53"
          record: "example.com"
          old_value: "do-lb-ip"
          new_value: "hz-lb-ip"
          ttl: 60

      - id: "update-cdn"
        action: "update-cdn-origin"
        config:
          cdn_provider: "cloudfront"
          distribution_id: "E123456789ABCDEF"
          new_origin: "backup-lb.hetzner.com"

  # Phase 4: Verify failover
  - name: "verify-failover"
    description: "Verify backup provider is operational"
    depends_on: ["network-failover"]

    operations:
      - id: "health-check-backup"
        action: "run-health-check"
        config:
          provider: "{{ backup_provider }}"
          resources: ["backup-servers"]
          endpoint: "/health"
          expected_status: 200
          timeout: 30

        timeout: 300

      - id: "verify-database"
        action: "verify-database"
        config:
          provider: "aws"
          database: "backup-db-promoted"
          query: "SELECT COUNT(*) FROM users"
          expected_rows: "> 0"

      - id: "verify-traffic"
        action: "verify-traffic-flow"
        config:
          endpoint: "https://example.com"
          expected_response_time: "< 500 ms"
          expected_status: 200

  # Phase 5: Activate backup fully
  - name: "activate-backup"
    description: "Run at full capacity on backup provider"
    depends_on: ["verify-failover"]

    operations:
      - id: "scale-to-production"
        action: "scale-up"
        config:
          provider: "{{ backup_provider }}"
          target: "all-backup-servers"
          desired_count: 6

        timeout: 600

      - id: "configure-persistence"
        action: "enable-persistence"
        config:
          provider: "{{ backup_provider }}"
          resources: ["backup-servers"]
          persistence_type: "volume"

# Recovery strategy for primary restoration
recovery:
  description: "Restore primary provider when recovered"
  phases:
    - name: "detect-primary-recovery"
      operation: "health-check"
      target: "primary-provider"
      success_criteria: "3 consecutive successful checks"

    - name: "resync-data"
      operation: "database-resync"
      direction: "backup-to-primary"
      timeout: 3600

    - name: "failback"
      operation: "switch-traffic"
      target: "primary-provider"
      verification: "100% traffic restored"

# Notifications
notifications:
  on_failover_start: "pagerduty:critical"
  on_failover_complete: "slack:#ops"
  on_failover_failed: ["pagerduty:critical", "email:cto@example.com"]
  on_recovery_start: "slack:#ops"
  on_recovery_complete: "slack:#ops"
```

### Failover Timeline

```
Time    Event
────────────────────────────────────────────────────
00:00   Health check detects failure (3 consecutive failures)
00:01   Alert sent to ops team
00:02   Backup infrastructure scaled to 3 servers
00:05   Database replica promoted to master
00:10   DNS updated (TTL=60s, propagation ~2 minutes)
00:12   Load balancer reconfigured
00:15   Traffic verified flowing through backup
00:20   Backup scaled to full production capacity (6 servers)
00:25   Fully operational on backup provider

Total RTO: 25 minutes (including DNS propagation)
Data loss (RPO): < 5 minutes (database replication lag)
```

## Workflow 3: Cost Optimization Workload Migration

**Use Case**: Migrate running workloads to cheaper provider (DigitalOcean to Hetzner) for cost reduction.

**Workflow Characteristics**:
- Parallel deployment on target provider
- Gradual traffic migration
- Rollback capability
- Cost tracking

### Workflow Definition

```
# file: workflows/cost-optimization-migration.yml

name: cost-optimization-migration
version: "1.0"
description: "Migrate workload from DigitalOcean to Hetzner for cost savings"

parameters:
  source_provider: "digitalocean"
  target_provider: "hetzner"
  migration_speed: "gradual"  # or "aggressive"
  traffic_split: [10, 25, 50, 75, 100]  # Gradual percentages

phases:
  # Phase 1: Create target infrastructure
  - name: "create-target-infrastructure"
    description: "Deploy identical workload on Hetzner"

    operations:
      - id: "provision-servers"
        action: "create-server"
        config:
          provider: "{{ target_provider }}"
          name: "migration-app"
          server_type: "cpx21"  # Better price/performance than DO
          count: 3

        timeout: 300

  # Phase 2: Verify target is ready
  - name: "verify-target"
    description: "Health checks on target infrastructure"
    depends_on: ["create-target-infrastructure"]

    operations:
      - id: "health-check"
        action: "run-health-check"
        config:
          provider: "{{ target_provider }}"
          endpoint: "/health"

        timeout: 300

  # Phase 3: Gradual traffic migration
  - name: "migrate-traffic"
    description: "Gradually shift traffic to target provider"
    depends_on: ["verify-target"]

    operations:
      - id: "set-traffic-10"
        action: "set-traffic-split"
        config:
          source: "{{ source_provider }}"
          target: "{{ target_provider }}"
          percentage: 10
          duration: 300

      - id: "verify-10"
        action: "verify-traffic-flow"
        config:
          target_percentage: 10
          error_rate_threshold: 0.1

      - id: "set-traffic-25"
        action: "set-traffic-split"
        config:
          percentage: 25
          duration: 600

      - id: "set-traffic-50"
        action: "set-traffic-split"
        config:
          percentage: 50
          duration: 900

      - id: "set-traffic-75"
        action: "set-traffic-split"
        config:
          percentage: 75
          duration: 900

      - id: "set-traffic-100"
        action: "set-traffic-split"
        config:
          percentage: 100
          duration: 600

  # Phase 4: Cleanup source
  - name: "cleanup-source"
    description: "Remove old infrastructure from source provider"
    depends_on: ["migrate-traffic"]

    operations:
      - id: "verify-final"
        action: "run-health-check"
        config:
          provider: "{{ target_provider }}"
          duration: 3600  # Monitor for 1 hour

      - id: "decommission-source"
        action: "delete-resources"
        config:
          provider: "{{ source_provider }}"
          resources: ["droplets", "load-balancer"]
          preserve_backups: true

# Cost tracking
cost_tracking:
  before:
    provider: "{{ source_provider }}"
    estimated_monthly: "$72"

  after:
    provider: "{{ target_provider }}"
    estimated_monthly: "$42"

  savings:
    monthly: "$30"
    annual: "$360"
    percentage: "42%"
```

## Workflow 4: Multi-Region Database Replication

**Use Case**: Setup database replication across multiple providers and regions for disaster recovery.

**Workflow Characteristics**:
- Create primary database
- Setup read replicas in other providers
- Configure replication
- Monitor lag

### Workflow Definition

```
# file: workflows/multi-region-replication.yml

name: multi-region-replication
version: "1.0"
description: "Setup database replication across providers"

phases:
  # Primary database
  - name: "create-primary"
    provider: "aws"
    operations:
      - id: "create-rds"
        action: "create-db-instance"
        config:
          identifier: "app-db-primary"
          engine: "postgres"
          instance_class: "db.t3.medium"
          region: "us-east-1"

  # Secondary replica
  - name: "create-secondary-replica"
    depends_on: ["create-primary"]
    provider: "aws"
    operations:
      - id: "create-replica"
        action: "create-read-replica"
        config:
          source: "app-db-primary"
          region: "eu-west-1"
          identifier: "app-db-secondary"

  # Tertiary replica in different provider
  - name: "create-tertiary-replica"
    depends_on: ["create-primary"]
    operations:
      - id: "setup-replication"
        action: "setup-external-replication"
        config:
          source_provider: "aws"
          source_db: "app-db-primary"
          target_provider: "hetzner"
          replication_slot: "hetzner_replica"
          replication_type: "logical"

  # Monitor replication
  - name: "monitor-replication"
    depends_on: ["create-tertiary-replica"]
    operations:
      - id: "check-lag"
        action: "monitor-replication-lag"
        config:
          replicas:
            - name: "secondary"
              warning_threshold: 300
              critical_threshold: 600
            - name: "tertiary"
              warning_threshold: 1000
              critical_threshold: 2000
          interval: 60
```

## Best Practices

### 1. Workflow Design

- **Define Clear Dependencies**: Explicitly state what must happen before what
- **Use Idempotent Operations**: Workflows should be safe to re-run
- **Set Realistic Timeouts**: Account for cloud provider delays
- **Plan for Failures**: Define rollback strategies
- **Test Workflows**: Run in staging before production

### 2. Orchestration

- **Parallel Execution**: Run independent phases in parallel for speed
- **Checkpoints**: Add verification at each phase
- **Progressive Deployment**: Use gradual traffic shifting
- **Monitoring Integration**: Track metrics during workflow
- **Notifications**: Alert team at key points

### 3. Cost Management

- **Calculate ROI**: Track cost savings from optimizations
- **Monitor Resource Usage**: Watch for over-provisioning
- **Implement Cleanup**: Remove old resources after migration
- **Review Regularly**: Reassess provider choices

## Troubleshooting

### Issue: Workflow Stuck in Phase

**Diagnosis**:
```
provisioning workflow status workflow-id --verbose
```

**Solution**:
- Increase timeout if legitimate long operation
- Check provider logs for actual status
- Manually intervene if necessary
- Use `--skip-phase` to skip problematic phase

### Issue: Rollback Failed

**Diagnosis**:
```
provisioning workflow rollback workflow-id --dry-run
```

**Solution**:
- Review what resources were created
- Manually delete resources if needed
- Fix root cause of failure
- Re-run workflow

### Issue: Data Inconsistency After Failover

**Diagnosis**:
```
provisioning database verify-consistency
```

**Solution**:
- Check replication lag before failover
- Manually resync if necessary
- Use backup to restore consistency
- Run validation queries

## Summary

Batch workflows enable complex multi-provider orchestration with:

- Coordinated deployment across providers
- Automated failover and recovery
- Gradual workload migration
- Cost optimization
- Disaster recovery

Start with simple workflows and gradually add complexity as you gain confidence.