## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Disaster Recovery Runbook

Step-by-step procedures for recovering VAPORA from various disaster scenarios.

## Disaster Severity Levels

### Level 1: Critical 🔴

Complete Service Loss - Entire VAPORA unavailable

Examples:

- Complete cluster failure
- Complete data center outage
- Database completely corrupted
- All backups inaccessible

RTO: 2-4 hours RPA: Up to 1 hour of data loss possible

### Level 2: Major 🟠

Partial Service Loss - Some services unavailable

Examples:

- Single region down
- Database corrupted but backups available
- One service completely failed
- Primary storage unavailable

RTO: 30 minutes - 2 hours RPA: Minimal data loss

### Level 3: Minor 🟡

Degraded Service - Service running but with issues

Examples:

- Performance issues
- One pod crashed
- Database connection issues
- High error rate

RTO: 5-15 minutes RPA: No data loss

## Disaster Assessment (First 5 Minutes)

### Step 1: Declare Disaster State

When any of these occur, declare a disaster:

    # Q1: Is the service accessible?
    curl -v https://api.vapora.com/health

    # Q2: How many pods are running?
    kubectl get pods -n vapora

    # Q3: Can we access the database?
    kubectl exec -n vapora pod/<name> -- \
      surreal query "SELECT * FROM projects LIMIT 1"

    # Q4: Are backups available?
    aws s3 ls s3://vapora-backups/

Decision Tree:

    Can access service normally?
      YES → No disaster, escalate to incident response
      NO → Continue

    Can reach any pods?
      YES → Partial disaster (Level 2-3)
      NO → Likely total disaster (Level 1)

    Can reach database?
      YES → Application issue, not data issue
      NO → Database issue, need restoration

    Are backups accessible?
      YES → Recovery likely possible
      NO → Critical situation, activate backup locations

### Step 2: Severity Assignment

Based on assessment:

    # Level 1 Criteria (Critical)
    - 0 pods running in vapora namespace
    - Database completely unreachable
    - All backup locations inaccessible
    - Service down >30 minutes

    # Level 2 Criteria (Major)
    - <50% pods running
    - Database reachable but degraded
    - Primary backups inaccessible but secondary available
    - Service down 5-30 minutes

    # Level 3 Criteria (Minor)
    - >75% pods running
    - Database responsive but with errors
    - Backups accessible
    - Service down <5 minutes

    Assignment: Level ___

    If Level 1: Activate full DR plan
    If Level 2: Activate partial DR plan
    If Level 3: Use normal incident response

### Step 3: Notify Key Personnel

    # For Level 1 (Critical) DR
    send_message_to = [
      "@cto",
      "@ops-manager",
      "@database-team",
      "@infrastructure-team",
      "@product-manager"
    ]

    message = """
    🔴 DISASTER DECLARED - LEVEL 1 CRITICAL

    Service: VAPORA (Complete Outage)
    Severity: Critical
    Time Declared: [UTC]
    Status: Assessing

    Actions underway:
    1. Activating disaster recovery procedures
    2. Notifying stakeholders
    3. Engaging full team

    Next update: [+5 min]

    /cc @all-involved
    """

    post_to_slack("#incident-critical")
    page_on_call_manager(urgent=true)

## Disaster Scenario Procedures

### Scenario 1: Complete Cluster Failure

Symptoms:

- kubectl commands time out or fail
- No pods running in any namespace
- Nodes unreachable
- All services down

Recovery Steps:

#### Step 1: Assess Infrastructure (5 min)

    # Try basic cluster operations
    kubectl cluster-info
    # If: "Unable to connect to the server"

    # Check cloud provider status
    # AWS: Check AWS status page, check EC2 instances
    # GKE: Check Google Cloud console
    # On-prem: Check infrastructure team

    # Determine: Is infrastructure failed or just connectivity?

#### Step 2: If Infrastructure Failed

Activate Secondary Infrastructure (if available):

    # 1. Access backup/secondary infrastructure
    export KUBECONFIG=/path/to/backup/kubeconfig

    # 2. Verify it's operational
    kubectl cluster-info
    kubectl get nodes

    # 3. Prepare for database restore
    # (See: Scenario 2 - Database Recovery)

If No Secondary: Activate failover to alternate region

    # 1. Contact cloud provider
    # AWS: Open support case - request emergency instance launch
    # GKE: Request cluster creation in different region

    # 2. While infrastructure rebuilds:
    # - Retrieve backups
    # - Prepare restore scripts
    # - Brief team on ETA

#### Step 3: Restore Database (See Scenario 2)

#### Step 4: Deploy Services

    # Once infrastructure ready and database restored

    # 1. Apply ConfigMaps
    kubectl apply -f vapora-configmap.yaml

    # 2. Apply Secrets
    kubectl apply -f vapora-secrets.yaml

    # 3. Deploy services
    kubectl apply -f vapora-deployments.yaml

    # 4. Wait for pods to start
    kubectl rollout status deployment/vapora-backend -n vapora --timeout=10m

    # 5. Verify health
    curl http://localhost:8001/health

#### Step 5: Verification

    # 1. Check all pods running
    kubectl get pods -n vapora
    # All should show: Running, 1/1 Ready

    # 2. Verify database connectivity
    kubectl logs deployment/vapora-backend -n vapora | tail -20
    # Should show: "Successfully connected to database"

    # 3. Test API
    curl http://localhost:8001/api/projects
    # Should return project list

    # 4. Check data integrity
    # Run validation queries:
    SELECT COUNT(*) FROM projects;          # Should > 0
    SELECT COUNT(*) FROM users;             # Should > 0
    SELECT COUNT(*) FROM tasks;             # Should > 0

### Scenario 2: Database Corruption/Loss

Symptoms:

- Database queries return errors
- Data integrity issues
- Corruption detected in logs

Recovery Steps:

#### Step 1: Assess Database State (10 min)

    # 1. Try to connect
    kubectl exec -n vapora pod/surrealdb-0 -- \
      surreal sql --conn ws://localhost:8000 \
      --user root --pass "$DB_PASSWORD" \
      "SELECT COUNT(*) FROM projects"

    # 2. Check for error messages
    kubectl logs -n vapora pod/surrealdb-0 | tail -50 | grep -i error

    # 3. Assess damage
    # Is it:
    # - Connection issue (might recover)
    # - Data corruption (need restore)
    # - Complete loss (restore from backup)

#### Step 2: Backup Current State (for forensics)

    # Before attempting recovery, save current state

    # Export what's remaining
    kubectl exec -n vapora pod/surrealdb-0 -- \
      surreal export --conn ws://localhost:8000 \
      --user root --pass "$DB_PASSWORD" \
      --output /tmp/corrupted-export.sql

    # Download for analysis
    kubectl cp vapora/surrealdb-0:/tmp/corrupted-export.sql \
      ./corrupted-export-$(date +%Y%m%d-%H%M%S).sql

#### Step 3: Identify Latest Good Backup

    # Find most recent backup before corruption
    aws s3 ls s3://vapora-backups/database/ --recursive | sort

    # Latest backup timestamp
    # Should be within last hour

    # Download backup
    aws s3 cp s3://vapora-backups/database/2026-01-12/vapora-db-010000.sql.gz \
      ./vapora-db-restore.sql.gz

    gunzip vapora-db-restore.sql.gz

#### Step 4: Restore Database

    # Option A: Restore to same database (destructive)
    # WARNING: This will overwrite current database

    kubectl exec -n vapora pod/surrealdb-0 -- \
      rm -rf /var/lib/surrealdb/data.db

    # Restart pod to reinitialize
    kubectl delete pod -n vapora surrealdb-0
    # Pod will restart with clean database

    # Import backup
    kubectl exec -n vapora pod/surrealdb-0 -- \
      surreal import --conn ws://localhost:8000 \
      --user root --pass "$DB_PASSWORD" \
      --input /tmp/vapora-db-restore.sql

    # Wait for import to complete (5-15 minutes)

Option B: Restore to temporary database (safer)

    # 1. Create temporary database pod
    kubectl run -n vapora restore-test --image=surrealdb/surrealdb:latest \
      -- start file:///tmp/restore-test

    # 2. Restore to temporary
    kubectl cp ./vapora-db-restore.sql vapora/restore-test:/tmp/
    kubectl exec -n vapora restore-test -- \
      surreal import --conn ws://localhost:8000 \
      --user root --pass "$DB_PASSWORD" \
      --input /tmp/vapora-db-restore.sql

    # 3. Verify restored data
    kubectl exec -n vapora restore-test -- \
      surreal sql "SELECT COUNT(*) FROM projects"

    # 4. If good: Restore production
    kubectl delete pod -n vapora surrealdb-0
    # Wait for pod restart
    kubectl cp ./vapora-db-restore.sql vapora/surrealdb-0:/tmp/
    kubectl exec -n vapora surrealdb-0 -- \
      surreal import --conn ws://localhost:8000 \
      --user root --pass "$DB_PASSWORD" \
      --input /tmp/vapora-db-restore.sql

    # 5. Cleanup test pod
    kubectl delete pod -n vapora restore-test

#### Step 5: Verify Recovery

    # 1. Database responsive
    kubectl exec -n vapora pod/surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects"

    # 2. Application can connect
    kubectl logs deployment/vapora-backend -n vapora | tail -5
    # Should show successful connection

    # 3. API working
    curl http://localhost:8001/api/projects

    # 4. Data valid
    # Check record counts match pre-backup
    # Check no corruption in key records

### Scenario 3: Configuration Corruption

Symptoms:

- Application misconfigured
- Pods failing to start
- Wrong values in environment

Recovery Steps:

#### Step 1: Identify Bad Configuration

    # 1. Get current ConfigMap
    kubectl get configmap -n vapora vapora-config -o yaml > current-config.yaml

    # 2. Compare with known-good backup
    aws s3 cp s3://vapora-backups/configs/2026-01-12/configmaps.yaml .

    # 3. Diff to find issues
    diff configmaps.yaml current-config.yaml

#### Step 2: Restore Previous Configuration

    # 1. Get previous ConfigMap from backup
    aws s3 cp s3://vapora-backups/configs/2026-01-11/configmaps.yaml ./good-config.yaml

    # 2. Apply previous configuration
    kubectl apply -f good-config.yaml

    # 3. Restart pods to pick up new config
    kubectl rollout restart deployment/vapora-backend -n vapora
    kubectl rollout restart deployment/vapora-agents -n vapora

    # 4. Monitor restart
    kubectl get pods -n vapora -w

#### Step 3: Verify Configuration

    # 1. Pods should restart and become Running
    kubectl get pods -n vapora
    # All should show: Running, 1/1 Ready

    # 2. Check pod logs
    kubectl logs deployment/vapora-backend -n vapora | tail -10
    # Should show successful startup

    # 3. API operational
    curl http://localhost:8001/health

### Scenario 4: Data Center/Region Outage

Symptoms:

- Entire region unreachable
- Multiple infrastructure components down
- Network connectivity issues

Recovery Steps:

#### Step 1: Declare Regional Failover

    # 1. Confirm region is down
    ping production.vapora.com
    # Should fail

    # Check status page
    # Cloud provider should report outage

    # 2. Declare failover
    declare_failover_to_region("us-west-2")

#### Step 2: Activate Alternate Region

    # 1. Switch kubeconfig to alternate region
    export KUBECONFIG=/path/to/backup-region/kubeconfig

    # 2. Verify alternate region up
    kubectl cluster-info

    # 3. Download and restore database
    aws s3 cp s3://vapora-backups/database/latest/ . --recursive

    # 4. Restore services (as in Scenario 1, Step 4)

#### Step 3: Update DNS/Routing

    # Update DNS to point to alternate region
    aws route53 change-resource-record-sets \
      --hosted-zone-id Z123456 \
      --change-batch '{
        "Changes": [{
          "Action": "UPSERT",
          "ResourceRecordSet": {
            "Name": "api.vapora.com",
            "Type": "A",
            "AliasTarget": {
              "HostedZoneId": "Z987654",
              "DNSName": "backup-region-lb.elb.amazonaws.com",
              "EvaluateTargetHealth": false
            }
          }
        }]
      }'

    # Wait for DNS propagation (5-10 minutes)

#### Step 4: Verify Failover

    # 1. DNS resolves to new region
    nslookup api.vapora.com

    # 2. Services accessible
    curl https://api.vapora.com/health

    # 3. Data intact
    curl https://api.vapora.com/api/projects

#### Step 5: Communicate Failover

    Post to #incident-critical:

    ✅ FAILOVER TO ALTERNATE REGION COMPLETE

    Primary Region: us-east-1 (Down)
    Active Region: us-west-2 (Restored)

    Status:
    - All services running: ✓
    - Database restored: ✓
    - Data integrity verified: ✓
    - Partial data loss: ~30 minutes of transactions

    Estimated Data Loss: 30 minutes (11:30-12:00 UTC)
    Current Time: 12:05 UTC

    Next steps:
    - Monitor alternate region closely
    - Begin investigation of primary region
    - Plan failback when primary recovered

    Questions? /cc @ops-team

## Post-Disaster Recovery

### Phase 1: Stabilization (Ongoing)

    # Continue monitoring for 4 hours minimum

    # Checks every 15 minutes:
    ✓ All pods Running
    ✓ API responding
    ✓ Database queries working
    ✓ Error rates normal
    ✓ Performance baseline

### Phase 2: Root Cause Analysis

Start within 1 hour of service recovery:

    Questions to answer:

    1. What caused the disaster?
       - Hardware failure
       - Software bug
       - Configuration error
       - External attack
       - Human error

    2. Why wasn't it detected earlier?
       - Monitoring gap
       - Alert misconfiguration
       - Alert fatigue

    3. How did backups perform?
       - Were they accessible?
       - Restore time as expected?
       - Data loss acceptable?

    4. What took longest in recovery?
       - Finding backups
       - Restoring database
       - Redeploying services
       - Verifying integrity

    5. What can be improved?
       - Faster detection
       - Faster recovery
       - Better documentation
       - More automated recovery

### Phase 3: Recovery Documentation

    Create post-disaster report:

    Timeline:
    - 11:30 UTC: Disaster detected
    - 11:35 UTC: Database restore started
    - 11:50 UTC: Services redeployed
    - 12:00 UTC: All systems operational
    - Duration: 30 minutes

    Impact:
    - Users affected: [X]
    - Data lost: [X] transactions
    - Revenue impact: $[X]

    Root cause: [Description]

    Contributing factors:
    1. [Factor 1]
    2. [Factor 2]

    Preventive measures:
    1. [Action] by [Owner] by [Date]
    2. [Action] by [Owner] by [Date]

    Lessons learned:
    1. [Lesson 1]
    2. [Lesson 2]

### Phase 4: Improvements Implementation

Due date: Within 2 weeks

    Checklist for improvements:

    □ Update backup strategy (if needed)
    □ Improve monitoring/alerting
    □ Automate more recovery steps
    □ Update runbooks with learnings
    □ Train team on new procedures
    □ Test improved procedures
    □ Document for future reference
    □ Incident retrospective meeting

## Disaster Recovery Drill

### Quarterly DR Drill

Purpose: Test DR procedures before real disaster

Schedule: Last Friday of each quarter at 02:00 UTC

    def quarterly_dr_drill [] {
      print "=== QUARTERLY DISASTER RECOVERY DRILL ==="
      print $"Date: (date now | format date %Y-%m-%d %H:%M:%S UTC)"
      print ""

      # 1. Simulate database corruption
      print "1. Simulating database corruption..."
      # Create test database, introduce corruption

      # 2. Test restore procedure
      print "2. Testing restore from backup..."
      # Download backup, restore to test database

      # 3. Measure restore time
      let start_time = (date now)
      # ... restore process ...
      let end_time = (date now)
      let duration = $end_time - $start_time
      print $"Restore time: ($duration)"

      # 4. Verify data integrity
      print "3. Verifying data integrity..."
      # Check restored data matches pre-backup

      # 5. Document results
      print "4. Documenting results..."
      # Record in DR drill log

      print ""
      print "Drill complete"
    }

### Drill Checklist

    Pre-Drill (1 week before):
    □ Notify team of scheduled drill
    □ Plan specific scenario to test
    □ Prepare test environment
    □ Have runbooks available

    During Drill:
    □ Execute scenario as planned
    □ Record actual timings
    □ Document any issues
    □ Note what went well
    □ Note what could improve

    Post-Drill (within 1 day):
    □ Debrief meeting
    □ Review recorded times vs. targets
    □ Discuss improvements
    □ Update runbooks if needed
    □ Thank team for participation
    □ Document lessons learned

    Post-Drill (within 1 week):
    □ Implement identified improvements
    □ Test improvements
    □ Verify procedures updated
    □ Archive drill documentation

## Disaster Recovery Readiness

### Recovery Readiness Checklist

    Infrastructure:
    □ Primary region configured
    □ Backup region prepared
    □ Load balancing configured
    □ DNS failover configured

    Data:
    □ Hourly database backups
    □ Backups encrypted
    □ Backups tested (monthly)
    □ Multiple backup locations

    Configuration:
    □ ConfigMaps backed up (daily)
    □ Secrets encrypted and backed up
    □ Infrastructure code in Git
    □ Deployment manifests versioned

    Documentation:
    □ Disaster procedures documented
    □ Runbooks current and tested
    □ Team trained on procedures
    □ Escalation paths clear

    Testing:
    □ Monthly restore test passes
    □ Quarterly DR drill scheduled
    □ Recovery times meet RTO/RPA

    Monitoring:
    □ Alerts for backup failures
    □ Backup health checks running
    □ Recovery procedures monitored

### RTO/RPA Targets

Scenario | RTO | RPA
--- | --- | ---
Single pod failure | 5 min | 0 min
Database corruption | 1 hour | 1 hour
Node failure | 15 min | 0 min
Region outage | 2 hours | 15 min
Complete cluster loss | 4 hours | 1 hour

## Disaster Recovery Contacts

    Role: Contact: Phone: Slack:
    Primary DBA: [Name] [Phone] @[slack]
    Backup DBA: [Name] [Phone] @[slack]
    Infra Lead: [Name] [Phone] @[slack]
    Backup Infra: [Name] [Phone] @[slack]
    CTO: [Name] [Phone] @[slack]
    Ops Manager: [Name] [Phone] @[slack]

    Escalation:
    Level 1: [Name] - notify immediately
    Level 2: [Name] - notify within 5 min
    Level 3: [Name] - notify within 15 min

## Quick Reference: Disaster Steps

    1. ASSESS (First 5 min)
       - Determine disaster severity
       - Assess damage scope
       - Get backup location access

    2. COMMUNICATE (Immediately)
       - Declare disaster
       - Notify key personnel
       - Start status updates (every 5 min)

    3. RECOVER (Next 30-120 min)
       - Activate backup infrastructure if needed
       - Restore database from latest backup
       - Redeploy applications
       - Verify all systems operational

    4. VERIFY (Continuous)
       - Check pod health
       - Verify database connectivity
       - Test API endpoints
       - Monitor error rates

    5. STABILIZE (Next 4 hours)
       - Monitor closely
       - Watch for anomalies
       - Verify performance normal
       - Check data integrity

    6. INVESTIGATE (Within 1 hour)
       - Root cause analysis
       - Document what happened
       - Plan improvements
       - Update procedures

    7. IMPROVE (Within 2 weeks)
       - Implement improvements
       - Test improvements
       - Update documentation
       - Train team