## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Database Recovery Procedures

Detailed procedures for recovering SurrealDB in various failure scenarios.

## Quick Reference: Recovery Methods

Scenario | Method | Time | Data Loss
--- | --- | --- | ---
Pod restart | Automatic pod recovery | 2 min | 0
Pod crash | Persistent volume intact | 3 min | 0
Corrupted pod | Restart from snapshot | 5 min | 0
Corrupted database | Restore from backup | 15 min | 0-60 min
Complete loss | Restore from backup | 30 min | 0-60 min

## SurrealDB Architecture

    VAPORA Database Layer

    SurrealDB Pod (Kubernetes)
    ├── PersistentVolume: /var/lib/surrealdb/
    ├── Data file: data.db (RocksDB)
    ├── Index files: *.idx
    └── Wal (Write-ahead log): *.wal

    Backed up to:
    ├── Hourly exports: S3 backups/database/
    ├── CloudSQL snapshots: AWS/GCP snapshots
    └── Archive backups: Glacier (monthly)

## Scenario 1: Pod Restart (Most Common)

Cause: Node maintenance, resource limits, health check failure

Duration: 2-3 minutes Data Loss: None

### Recovery Procedure

    # Most of the time, just restart the pod

    # 1. Delete the pod
    kubectl delete pod -n vapora surrealdb-0

    # 2. Pod automatically restarts (via StatefulSet)
    kubectl get pods -n vapora -w

    # 3. Verify it's Ready
    kubectl get pod surrealdb-0 -n vapora
    # Should show: 1/1 Running

    # 4. Verify database is accessible
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT 1"

    # 5. Check data integrity
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects"
    # Should return non-zero count

## Scenario 2: Pod CrashLoop (Container Issue)

Cause: Application crash, memory issues, corrupt index

Duration: 5-10 minutes Data Loss: None (usually)

### Recovery Procedure

    # 1. Examine pod logs to identify issue
    kubectl logs surrealdb-0 -n vapora --previous
    # Look for: "panic", "fatal", "out of memory"

    # 2. Increase resource limits if memory issue
    kubectl patch statefulset surrealdb -n vapora --type='json' \
      -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value":"2Gi"}]'

    # 3. If corrupt index, rebuild
    kubectl exec -n vapora surrealdb-0 -- \
      surreal query "REBUILD INDEX"

    # 4. If persistent issue, try volume snapshot
    kubectl delete pod -n vapora surrealdb-0
    # Use previous snapshot (if available)

    # 5. Monitor restart
    kubectl get pods -n vapora -w

## Scenario 3: Corrupted Database (Detected via Queries)

Cause: Unclean shutdown, disk issue, data corruption

Duration: 15-30 minutes Data Loss: Minimal (last hour of transactions)

### Detection

    # Symptoms to watch for
    ✗ Queries return error: "corrupted database"
    ✗ Disk check shows corruption
    ✗ Checksums fail
    ✗ Integrity check fails

    # Verify corruption
    kubectl exec -n vapora surrealdb-0 -- \
      surreal query "INFO FOR DB"
    # Look for any error messages

    # Try repair
    kubectl exec -n vapora surrealdb-0 -- \
      surreal query "REBUILD INDEX"

### Recovery: Option A - Restart and Repair (Try First)

    # 1. Delete pod to force restart
    kubectl delete pod -n vapora surrealdb-0

    # 2. Watch restart
    kubectl get pods -n vapora -w
    # Should restart within 30 seconds

    # 3. Verify database accessible
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects"

    # 4. If successful, done
    # If still errors, proceed to Option B

### Recovery: Option B - Restore from Recent Backup

    # 1. Stop database pod
    kubectl scale statefulset surrealdb --replicas=0 -n vapora

    # 2. Download latest backup
    aws s3 cp s3://vapora-backups/database/ ./ --recursive
    # Get most recent .sql.gz file

    # 3. Clear corrupted data
    kubectl delete pvc -n vapora surrealdb-data-surrealdb-0

    # 4. Recreate pod (will create new PVC)
    kubectl scale statefulset surrealdb --replicas=1 -n vapora

    # 5. Wait for pod to be ready
    kubectl wait --for=condition=Ready pod/surrealdb-0 \
      -n vapora --timeout=300s

    # 6. Restore backup
    # Extract and import
    gunzip vapora-db-*.sql.gz
    kubectl cp vapora-db-*.sql vapora/surrealdb-0:/tmp/

    kubectl exec -n vapora surrealdb-0 -- \
      surreal import \
        --conn ws://localhost:8000 \
        --user root \
        --pass $DB_PASSWORD \
        --input /tmp/vapora-db-*.sql

    # 7. Verify restored data
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects"
    # Should match pre-corruption count

## Scenario 4: Storage Failure (PVC Issue)

Cause: Storage volume corruption, node storage failure

Duration: 20-30 minutes Data Loss: None with backup

### Recovery Procedure

    # 1. Detect storage issue
    kubectl describe pvc -n vapora surrealdb-data-surrealdb-0
    # Look for: "Pod pending", "volume binding failure"

    # 2. Check if snapshot available (cloud)
    aws ec2 describe-snapshots \
      --filters "Name=tag:database,Values=vapora" \
      --query 'Snapshots[].{SnapshotId:SnapshotId,StartTime:StartTime}' \
      --sort-by StartTime | tail -10

    # 3. Create new PVC from snapshot
    kubectl apply -f - << EOF
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: surrealdb-data-surrealdb-0-restore
      namespace: vapora
    spec:
      accessModes:
        - ReadWriteOnce
      dataSource:
        kind: VolumeSnapshot
        apiGroup: snapshot.storage.k8s.io
        name: surrealdb-snapshot-latest
      resources:
        requests:
          storage: 100Gi
    EOF

    # 4. Update StatefulSet to use new PVC
    kubectl patch statefulset surrealdb -n vapora --type='json' \
      -p='[{"op": "replace", "path": "/spec/volumeClaimTemplates/0/metadata/name", "value":"surrealdb-data-surrealdb-0-restore"}]'

    # 5. Delete old pod to force remount
    kubectl delete pod -n vapora surrealdb-0

    # 6. Verify new pod runs
    kubectl get pods -n vapora -w

    # 7. Test database
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects"

## Scenario 5: Complete Data Loss (Restore from Backup)

Cause: User delete, accidental truncate, security incident

Duration: 30-60 minutes Data Loss: Up to 1 hour

### Pre-Recovery Checklist

    Before restoring, verify:
    □ What data was lost? (specific tables or entire DB?)
    □ When was it lost? (exact time if possible)
    □ Is it just one table or entire database?
    □ Do we have valid backups from before loss?
    □ Has the backup been tested before?

### Recovery Procedure

    # 1. Stop the database
    kubectl scale statefulset surrealdb --replicas=0 -n vapora
    sleep 10

    # 2. Identify backup to restore
    # Look for backup from time BEFORE data loss
    aws s3 ls s3://vapora-backups/database/ --recursive | sort
    # Example: surrealdb-2026-01-12-230000.sql.gz
    # (from 11 PM, before 12 AM loss)

    # 3. Download backup
    aws s3 cp s3://vapora-backups/database/2026-01-12-surrealdb-230000.sql.gz ./

    gunzip surrealdb-230000.sql

    # 4. Verify backup integrity before restoring
    # Extract first 100 lines to check format
    head -100 surrealdb-230000.sql

    # 5. Delete corrupted PVC
    kubectl delete pvc -n vapora surrealdb-data-surrealdb-0

    # 6. Restart database pod (will create new PVC)
    kubectl scale statefulset surrealdb --replicas=1 -n vapora

    # 7. Wait for pod to be ready and listening
    kubectl wait --for=condition=Ready pod/surrealdb-0 \
      -n vapora --timeout=300s
    sleep 10

    # 8. Copy backup to pod
    kubectl cp surrealdb-230000.sql vapora/surrealdb-0:/tmp/

    # 9. Restore backup
    kubectl exec -n vapora surrealdb-0 -- \
      surreal import \
        --conn ws://localhost:8000 \
        --user root \
        --pass $DB_PASSWORD \
        --input /tmp/surrealdb-230000.sql

    # Expected output:
    # Imported 1500+ records...
    # This should take 5-15 minutes depending on backup size

    # 10. Verify data restored
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql \
        --conn ws://localhost:8000 \
        --user root \
        --pass $DB_PASSWORD \
        "SELECT COUNT(*) as project_count FROM projects"

    # Should match pre-loss count

### Data Loss Assessment

    # After restore, compare with lost version

    # 1. Get current record count
    RESTORED_COUNT=$(kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT COUNT(*) FROM projects")

    # 2. Get pre-loss count (from logs or ticket)
    PRE_LOSS_COUNT=1500

    # 3. Calculate data loss
    if [ "$RESTORED_COUNT" -lt "$PRE_LOSS_COUNT" ]; then
      LOSS=$(( PRE_LOSS_COUNT - RESTORED_COUNT ))
      echo "Data loss: $LOSS records"
      echo "Data loss duration: ~1 hour"
      echo "Restore successful but incomplete"
    else
      echo "Data loss: 0 records"
      echo "Full recovery complete"
    fi

## Scenario 6: Backup Verification Failed

Cause: Corrupt backup file, incompatible format

Duration: 30-120 minutes (fallback to older backup) Data Loss: 2+ hours possible

### Recovery Procedure

    # 1. Identify backup corruption
    # During restore, if backup fails import:

    kubectl exec -n vapora surrealdb-0 -- \
      surreal import \
        --conn ws://localhost:8000 \
        --user root \
        --pass $DB_PASSWORD \
        --input /tmp/backup.sql

    # Error: "invalid SQL format" or similar

    # 2. Check backup file integrity
    file vapora-db-backup.sql
    # Should show: ASCII text

    head -5 vapora-db-backup.sql
    # Should show: SQL statements or surreal export format

    # 3. If corrupt, try next-oldest backup
    aws s3 ls s3://vapora-backups/database/ --recursive | sort | tail -5
    # Get second-newest backup

    # 4. Retry restore with older backup
    aws s3 cp s3://vapora-backups/database/2026-01-12-210000/ ./
    gunzip backup.sql.gz

    # 5. Repeat restore procedure with older backup
    # (As in Scenario 5, steps 8-10)

## Scenario 7: Database Size Growing Unexpectedly

Cause: Accumulation of data, logs not rotated, storage leak

Duration: Varies (prevention focus) Data Loss: None

### Detection

    # Monitor database size
    kubectl exec -n vapora surrealdb-0 -- du -sh /var/lib/surrealdb/

    # Check disk usage trend
    # (Should be ~1-2% growth per week)

    # If sudden spike:
    kubectl exec -n vapora surrealdb-0 -- \
      find /var/lib/surrealdb/ -type f -exec ls -lh {} + | sort -k5 -h | tail -20

### Cleanup Procedure

    # 1. Identify large tables
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT table, count(*) FROM meta::tb GROUP BY table ORDER BY count DESC"

    # 2. If logs table too large
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "DELETE FROM audit_logs WHERE created_at < now() - 90d"

    # 3. Rebuild indexes to reclaim space
    kubectl exec -n vapora surrealdb-0 -- \
      surreal query "REBUILD INDEX"

    # 4. If still large, delete old records from other tables
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "DELETE FROM tasks WHERE status = 'archived' AND updated_at < now() - 1y"

    # 5. Monitor size after cleanup
    kubectl exec -n vapora surrealdb-0 -- du -sh /var/lib/surrealdb/

## Scenario 8: Replication Lag (If Using Replicas)

Cause: Replica behind primary, network latency

Duration: Usually self-healing (seconds to minutes) Data Loss: None

### Detection

    # Check replica lag
    kubectl exec -n vapora surrealdb-replica -- \
      surreal sql "SHOW REPLICATION STATUS"

    # Look for: "Seconds_Behind_Master" > 5 seconds

### Recovery

    # Usually self-healing, but if stuck:

    # 1. Check network connectivity
    kubectl exec -n vapora surrealdb-replica -- ping surrealdb-primary -c 5

    # 2. Restart replica
    kubectl delete pod -n vapora surrealdb-replica

    # 3. Monitor replica catching up
    kubectl logs -n vapora surrealdb-replica -f

    # 4. Verify replica status
    kubectl exec -n vapora surrealdb-replica -- \
      surreal sql "SHOW REPLICATION STATUS"

## Database Health Checks

### Pre-Recovery Verification

    def verify_database_health [] {
      print "=== Database Health Check ==="

      # 1. Connection test
      let conn = (try (
        exec "surreal sql --conn ws://localhost:8000 \"SELECT 1\""
      ) catch {error make {msg: "Cannot connect to database"}})

      # 2. Data integrity test
      let integrity = (exec "surreal sql \"REBUILD INDEX\"")
      print "✓ Integrity check passed"

      # 3. Performance test
      let perf = (exec "surreal sql \"SELECT COUNT(*) FROM projects\"")
      print "✓ Performance acceptable"

      # 4. Replication lag (if applicable)
      # let lag = (exec "surreal sql \"SHOW REPLICATION STATUS\"")
      # print "✓ No replication lag"

      print "✓ All health checks passed"
    }

### Post-Recovery Verification

    def verify_recovery_success [] {
      print "=== Post-Recovery Verification ==="

      # 1. Database accessible
      kubectl exec -n vapora surrealdb-0 -- \
        surreal sql "SELECT 1"
      print "✓ Database accessible"

      # 2. All tables present
      kubectl exec -n vapora surrealdb-0 -- \
        surreal sql "SELECT table FROM meta::tb"
      print "✓ All tables present"

      # 3. Record counts reasonable
      kubectl exec -n vapora surrealdb-0 -- \
        surreal sql "SELECT table, count(*) FROM meta::tb"
      print "✓ Record counts verified"

      # 4. Application can connect
      kubectl logs -n vapora deployment/vapora-backend --tail=5 | grep -i connected
      print "✓ Application connected"

      # 5. API operational
      curl http://localhost:8001/api/projects
      print "✓ API operational"
    }

## Database Recovery Checklist

### Before Recovery

    □ Documented failure symptoms
    □ Determined root cause
    □ Selected appropriate recovery method
    □ Located backup to restore
    □ Verified backup integrity
    □ Notified relevant teams
    □ Have runbook available
    □ Test environment ready (for testing)

### During Recovery

    □ Followed procedure step-by-step
    □ Monitored each step completion
    □ Captured any error messages
    □ Took notes of timings
    □ Did NOT skip verification steps
    □ Had backup plans ready

### After Recovery

    □ Verified database accessible
    □ Verified data integrity
    □ Verified application can connect
    □ Checked API endpoints working
    □ Monitored error rates
    □ Waited for 30 min stability check
    □ Documented recovery procedure
    □ Identified improvements needed
    □ Updated runbooks if needed

## Recovery Troubleshooting

### Issue: "Cannot connect to database after restore"

Cause: Database not fully recovered, network issue

Solution:

    # 1. Wait longer (import can take 15+ minutes)
    sleep 60 && kubectl exec -n vapora surrealdb-0 -- surreal sql "SELECT 1"

    # 2. Check pod logs
    kubectl logs -n vapora surrealdb-0 | tail -50

    # 3. Restart pod
    kubectl delete pod -n vapora surrealdb-0

    # 4. Check network connectivity
    kubectl exec -n vapora surrealdb-0 -- ping localhost

### Issue: "Import corrupted data" error

Cause: Backup file corrupted or wrong format

Solution:

    # 1. Try different backup
    aws s3 ls s3://vapora-backups/database/ | sort | tail -5

    # 2. Verify backup format
    file vapora-db-backup.sql
    # Should show: text

    # 3. Manual inspection
    head -20 vapora-db-backup.sql
    # Should show SQL format

    # 4. Try with older backup

### Issue: "Database running but data seems wrong"

Cause: Restored wrong backup or partial restore

Solution:

    # 1. Verify record counts
    kubectl exec -n vapora surrealdb-0 -- \
      surreal sql "SELECT table, count(*) FROM meta::tb"

    # 2. Compare to pre-loss baseline
    # (from documentation or logs)

    # If counts don't match:
    # - Used wrong backup
    # - Restore incomplete
    # - Try again with correct backup

## Database Recovery Reference

Recovery Procedure Flowchart:

    Database Issue Detected
        ↓
    Is it just a pod restart?
      YES → kubectl delete pod surrealdb-0
      NO → Continue
        ↓
    Can queries connect and run?
      YES → Continue with application recovery
      NO → Continue
        ↓
    Is data corrupted (errors in queries)?
      YES → Try REBUILD INDEX
      NO → Continue
        ↓
    Still errors?
      YES → Scale replicas=0, clear PVC, restore from backup
      NO → Success, monitor for 30 min