## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# VAPORA Automated Backup & Recovery Automation

Automated backup and recovery procedures using Nushell scripts and Kubernetes CronJobs. Supports both direct S3 backups and Restic-based incremental backups.

## Overview

Backup Strategy:

- Hourly: Database export + Restic backup (1-hour RPO)
- Daily: Kubernetes config backup + Restic backup
- Monthly: Cleanup old snapshots and archive

Dual Backup Approach:

- S3 Direct: Simple file upload for quick recovery
- Restic: Incremental, deduplicated backups with integrated encryption

Recovery Procedures:

- One-command restore from S3 or Restic
- Verification before committing to production
- Automated database readiness checks

## Files and Components

### Backup Scripts

All scripts follow NUSHELL_GUIDELINES.md (0.109.0+) strictly.

#### `scripts/backup/database-backup.nu`

Direct S3 backup of SurrealDB with encryption.

    nu scripts/backup/database-backup.nu \
      --surreal-url "ws://localhost:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS" \
      --s3-bucket "vapora-backups" \
      --s3-prefix "backups/database" \
      --encryption-key "$ENCRYPTION_KEY_FILE"

Process:

1. Export SurrealDB to SQL
2. Compress with gzip
3. Encrypt with AES-256
4. Upload to S3 with metadata
5. Verify upload completed

Output: `s3://vapora-backups/backups/database/database-YYYYMMDD-HHMMSS.sql.gz.enc`

#### `scripts/backup/config-backup.nu`

Backup Kubernetes resources (ConfigMaps, Secrets, Deployments).

    nu scripts/backup/config-backup.nu \
      --namespace "vapora" \
      --s3-bucket "vapora-backups" \
      --s3-prefix "backups/config"

Process:

1. Export ConfigMaps from namespace
2. Export Secrets
3. Export Deployments, Services, Ingress
4. Compress all to tar.gz
5. Upload to S3

Output: `s3://vapora-backups/backups/config/configs-YYYYMMDD-HHMMSS.tar.gz`

#### `scripts/backup/restic-backup.nu`

Incremental, deduplicated backup using Restic.

    nu scripts/backup/restic-backup.nu \
      --repo "s3:s3.amazonaws.com/vapora-backups/restic" \
      --password "$RESTIC_PASSWORD" \
      --database-dir "/tmp/vapora-db-backup" \
      --k8s-dir "/tmp/vapora-k8s-backup" \
      --iac-dir "provisioning" \
      --backup-db \
      --backup-k8s \
      --backup-iac \
      --verify \
      --cleanup \
      --keep-daily 7 \
      --keep-weekly 4 \
      --keep-monthly 12

Features:

- Incremental backups (only changed data stored)
- Deduplication across snapshots
- Built-in compression and encryption
- Automatic retention policies
- Repository health verification

Output: Tagged snapshots in Restic repository with metadata

#### `scripts/orchestrate-backup-recovery.nu`

Coordinates all backup types (S3 + Restic).

    # Full backup cycle
    nu scripts/orchestrate-backup-recovery.nu \
      --operation backup \
      --mode full \
      --surreal-url "ws://localhost:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS" \
      --namespace "vapora" \
      --s3-bucket "vapora-backups" \
      --s3-prefix "backups/database" \
      --encryption-key "$ENCRYPTION_KEY_FILE" \
      --restic-repo "s3:s3.amazonaws.com/vapora-backups/restic" \
      --restic-password "$RESTIC_PASSWORD" \
      --iac-dir "provisioning"

Modes:

- `full`: Database export → S3 + Restic
- `database-only`: Database export only
- `config-only`: Kubernetes config only

### Recovery Scripts

#### `scripts/recovery/database-recovery.nu`

Restore SurrealDB from S3 backup (with decryption).

    nu scripts/recovery/database-recovery.nu \
      --s3-location "s3://vapora-backups/backups/database/database-20260112-010000.sql.gz.enc" \
      --encryption-key "$ENCRYPTION_KEY_FILE" \
      --surreal-url "ws://localhost:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS" \
      --namespace "vapora" \
      --statefulset "surrealdb" \
      --pvc "surrealdb-data-surrealdb-0" \
      --verify

Process:

1. Download encrypted backup from S3
2. Decrypt backup file
3. Decompress backup
4. Scale down StatefulSet (for PVC replacement)
5. Delete current PVC
6. Scale up StatefulSet (creates new PVC)
7. Wait for pod readiness
8. Import backup to database
9. Verify data integrity

Output: Restored database at specified SurrealDB URL

#### `scripts/orchestrate-backup-recovery.nu` (Recovery Mode)

One-command recovery from backup.

    nu scripts/orchestrate-backup-recovery.nu \
      --operation recovery \
      --s3-location "s3://vapora-backups/backups/database/database-20260112-010000.sql.gz.enc" \
      --encryption-key "$ENCRYPTION_KEY_FILE" \
      --surreal-url "ws://localhost:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS"

### Verification Scripts

#### `scripts/verify-backup-health.nu`

Health check for backup infrastructure.

    # Basic health check
    nu scripts/verify-backup-health.nu \
      --s3-bucket "vapora-backups" \
      --s3-prefix "backups/database" \
      --restic-repo "s3:s3.amazonaws.com/vapora-backups/restic" \
      --restic-password "$RESTIC_PASSWORD" \
      --surreal-url "ws://localhost:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS" \
      --max-age-hours 25

Checks Performed:

- ✓ S3 backups exist and have content
- ✓ Restic repository accessible and has snapshots
- ✓ Database connectivity verified
- ✓ Backup freshness (< 25 hours old)
- ✓ Backup rotation policy (daily, weekly, monthly)
- ✓ Restore test (if `--full-test` specified)

Output: Pass/fail for each check with detailed status

## Kubernetes Automation

### CronJob Configuration

File: `kubernetes/09-backup-cronjobs.yaml`

Defines four automated CronJobs:

#### 1. Hourly Database Backup

    schedule: "0 * * * *"  # Every hour
    timeout: 1800 seconds  # 30 minutes

Runs `orchestrate-backup-recovery.nu --operation backup --mode full`

Backups:

- SurrealDB to S3 (encrypted)
- SurrealDB to Restic (incremental)
- IaC to Restic

#### 2. Daily Configuration Backup

    schedule: "0 2 * * *"  # 02:00 UTC daily
    timeout: 3600 seconds  # 60 minutes

Runs `config-backup.nu` for Kubernetes resources.

#### 3. Daily Health Verification

    schedule: "0 3 * * *"  # 03:00 UTC daily
    timeout: 900 seconds   # 15 minutes

Runs `verify-backup-health.nu` to verify backup infrastructure.

Alerts if:

- No S3 backups found
- Restic repository inaccessible
- Database unreachable
- Backups older than 25 hours
- Rotation policy violated

#### 4. Monthly Backup Rotation

    schedule: "0 4 1 * *"  # First day of month, 04:00 UTC
    timeout: 3600 seconds

Cleans up old Restic snapshots per retention policy:

- Keep: 7 daily, 4 weekly, 12 monthly
- Prune: Remove unreferenced data

### Environment Configuration

CronJobs require these secrets and ConfigMaps:

ConfigMap: `vapora-config`

    backup_s3_bucket: "vapora-backups"
    restic_repo: "s3:s3.amazonaws.com/vapora-backups/restic"
    aws_region: "us-east-1"

Secret: `vapora-secrets`

    surreal_password: "<database-password>"
    restic_password: "<restic-encryption-password>"

Secret: `vapora-aws-credentials`

    access_key_id: "<aws-access-key>"
    secret_access_key: "<aws-secret-key>"

Secret: `vapora-encryption-key`

    # File containing AES-256 encryption key
    encryption.key: "<binary-key-data>"

### Deployment

1. Create secrets (if not existing):

    kubectl create secret generic vapora-secrets \
      --from-literal=surreal_password="$SURREAL_PASS" \
      --from-literal=restic_password="$RESTIC_PASSWORD" \
      -n vapora

    kubectl create secret generic vapora-aws-credentials \
      --from-literal=access_key_id="$AWS_ACCESS_KEY_ID" \
      --from-literal=secret_access_key="$AWS_SECRET_ACCESS_KEY" \
      -n vapora

    kubectl create secret generic vapora-encryption-key \
      --from-file=encryption.key=/path/to/encryption.key \
      -n vapora

2. Deploy CronJobs:

    kubectl apply -f kubernetes/09-backup-cronjobs.yaml

3. Verify CronJobs:

    kubectl get cronjobs -n vapora
    kubectl describe cronjob vapora-backup-database-hourly -n vapora

4. Monitor scheduled runs:

    # Watch CronJob executions
    kubectl get jobs -n vapora -l job-type=backup --watch

    # View logs from backup job
    kubectl logs -n vapora -l backup-type=database --tail=100 -f

## Setup Instructions

### Prerequisites

- Kubernetes 1.18+ with CronJob support
- Nushell 0.109.0+
- AWS CLI v2+
- Restic installed (or container image with restic)
- SurrealDB CLI (`surreal` command)
- `kubectl` with cluster access

### Local Testing

1. Setup environment variables:

    export SURREAL_URL="ws://localhost:8000"
    export SURREAL_USER="root"
    export SURREAL_PASS="password"
    export S3_BUCKET="vapora-backups"
    export ENCRYPTION_KEY_FILE="/path/to/encryption.key"
    export RESTIC_REPO="s3:s3.amazonaws.com/vapora-backups/restic"
    export RESTIC_PASSWORD="restic-password"
    export AWS_REGION="us-east-1"
    export AWS_ACCESS_KEY_ID="your-key"
    export AWS_SECRET_ACCESS_KEY="your-secret"

2. Run backup:

    nu scripts/orchestrate-backup-recovery.nu \
      --operation backup \
      --mode full \
      --surreal-url "$SURREAL_URL" \
      --surreal-user "$SURREAL_USER" \
      --surreal-pass "$SURREAL_PASS" \
      --s3-bucket "$S3_BUCKET" \
      --s3-prefix "backups/database" \
      --encryption-key "$ENCRYPTION_KEY_FILE" \
      --restic-repo "$RESTIC_REPO" \
      --restic-password "$RESTIC_PASSWORD" \
      --iac-dir "provisioning"

3. Verify backup:

    nu scripts/verify-backup-health.nu \
      --s3-bucket "$S3_BUCKET" \
      --s3-prefix "backups/database" \
      --restic-repo "$RESTIC_REPO" \
      --restic-password "$RESTIC_PASSWORD" \
      --surreal-url "$SURREAL_URL" \
      --surreal-user "$SURREAL_USER" \
      --surreal-pass "$SURREAL_PASS"

4. Test recovery:

    # First, list available backups
    aws s3 ls s3://$S3_BUCKET/backups/database/

    # Then recover from latest backup
    nu scripts/orchestrate-backup-recovery.nu \
      --operation recovery \
      --s3-location "s3://$S3_BUCKET/backups/database/database-20260112-010000.sql.gz.enc" \
      --encryption-key "$ENCRYPTION_KEY_FILE" \
      --surreal-url "$SURREAL_URL" \
      --surreal-user "$SURREAL_USER" \
      --surreal-pass "$SURREAL_PASS"

### Production Deployment

1. Create S3 bucket for backups:

    aws s3 mb s3://vapora-backups --region us-east-1

2. Enable bucket versioning for protection:

    aws s3api put-bucket-versioning \
      --bucket vapora-backups \
      --versioning-configuration Status=Enabled

3. Set lifecycle policy for Glacier archival (optional):

    # 30 days to standard-IA, 90 days to Glacier
    aws s3api put-bucket-lifecycle-configuration \
      --bucket vapora-backups \
      --lifecycle-configuration file://s3-lifecycle-policy.json

4. Create Restic repository:

    export RESTIC_REPO="s3:s3.amazonaws.com/vapora-backups/restic"
    export RESTIC_PASSWORD="your-restic-password"

    restic init

5. Deploy to Kubernetes:

    # 1. Create namespace
    kubectl create namespace vapora

    # 2. Create secrets
    kubectl create secret generic vapora-secrets \
      --from-literal=surreal_password="$SURREAL_PASS" \
      --from-literal=restic_password="$RESTIC_PASSWORD" \
      -n vapora

    # 3. Create ConfigMap
    kubectl create configmap vapora-config \
      --from-literal=backup_s3_bucket="vapora-backups" \
      --from-literal=restic_repo="s3:s3.amazonaws.com/vapora-backups/restic" \
      --from-literal=aws_region="us-east-1" \
      -n vapora

    # 4. Deploy CronJobs
    kubectl apply -f kubernetes/09-backup-cronjobs.yaml

6. Monitor:

    # Watch CronJobs
    kubectl get cronjobs -n vapora --watch

    # View backup logs
    kubectl logs -n vapora -l backup-type=database -f

    # Check health status
    kubectl get jobs -n vapora -l job-type=health-check -o wide

## Emergency Recovery

### Complete Database Loss

If production database is lost, restore from backup:

    # 1. Scale down StatefulSet
    kubectl scale statefulset surrealdb --replicas=0 -n vapora

    # 2. Delete current PVC
    kubectl delete pvc surrealdb-data-surrealdb-0 -n vapora

    # 3. Run recovery
    nu scripts/orchestrate-backup-recovery.nu \
      --operation recovery \
      --s3-location "s3://vapora-backups/backups/database/database-LATEST.sql.gz.enc" \
      --encryption-key "/path/to/encryption.key" \
      --surreal-url "ws://surrealdb:8000" \
      --surreal-user "root" \
      --surreal-pass "$SURREAL_PASS"

    # 4. Verify database restored
    kubectl exec -n vapora surrealdb-0 -- \
      surreal query \
        --conn ws://localhost:8000 \
        --user root \
        --pass "$SURREAL_PASS" \
        "SELECT COUNT() FROM projects"

### Backup Verification Failed

If health check fails:

1. Check Restic repository:

    export RESTIC_PASSWORD="$RESTIC_PASSWORD"
    restic -r "s3:s3.amazonaws.com/vapora-backups/restic" check

2. Force full verification (slow):

    restic -r "s3:s3.amazonaws.com/vapora-backups/restic" check --read-data

3. List recent snapshots:

    restic -r "s3:s3.amazonaws.com/vapora-backups/restic" snapshots --max 10

## Troubleshooting

Issue | Cause | Solution
--- | --- | ---
CronJob not running | Schedule incorrect | Check `kubectl get cronjobs` and verify schedule format
Backup file too large | Database growing | Check for old data that can be cleaned up
S3 upload fails | Credentials invalid | Verify `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
Restic backup slow | First backup or network latency | Expected on first run; use `--keep-*` flags to limit retention
Recovery fails | Database already running | Scale down StatefulSet before recovery
Encryption key missing | Secret not created | Create `vapora-encryption-key` secret in namespace

## Related Documentation

- Disaster Recovery Procedures: `docs/disaster-recovery/README.md`
- Backup Strategy: `docs/disaster-recovery/backup-strategy.md`
- Database Recovery: `docs/disaster-recovery/database-recovery-procedures.md`
- Operations Guide: `docs/operations/README.md`

Last Updated: January 12, 2026 Status: Production-Ready Automation: Full CronJob automation with health checks