## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Pre-Deployment Checklist

Critical verification steps before any VAPORA deployment to production or staging.

## 24 Hours Before Deployment

### Communication & Scheduling

- Schedule deployment with team (record in calendar/ticket)
- Post in #deployments channel: "Deployment scheduled for [DATE TIME UTC]"
- Identify on-call engineer for deployment period
- Brief on-call on deployment plan and rollback procedure
- Ensure affected teams (support, product, etc.) are notified
- Verify no other critical infrastructure changes scheduled same time window

### Change Documentation

- Create GitHub issue or ticket tracking the deployment
- Document: what's changing (configs, manifests, versions)
- Document: why (bug fix, feature, performance, security)
- Document: rollback plan (revision number or previous config)
- Document: success criteria (what indicates successful deployment)
- Document: estimated duration (usually 5-15 minutes)

### Code Review & Validation

- All provisioning changes merged and code reviewed
- Confirm `main` branch has latest changes
- Run validation locally: `nu scripts/validate-config.nu --mode enterprise`
- Verify all 3 modes validate without errors or critical warnings
- Check git log for unexpected commits
- Review artifact generation: ensure configs are correct

## 4 Hours Before Deployment

### Environment Verification

#### Staging Environment

- Access staging Kubernetes cluster: `kubectl cluster-info`
- Verify cluster is healthy: `kubectl get nodes` (all Ready)
- Check namespace exists: `kubectl get namespace vapora`
- Verify current deployments: `kubectl get deployments -n vapora`
- Check ConfigMap is up to date: `kubectl get configmap -n vapora -o yaml | head -20`

#### Production Environment (if applicable)

- Access production Kubernetes cluster: `kubectl cluster-info`
- Verify all nodes healthy: `kubectl get nodes` (all Ready)
- Check current resource usage: `kubectl top nodes` (not near capacity)
- Verify current deployments: `kubectl get deployments -n vapora`
- Check pod status: `kubectl get pods -n vapora` (all Running)
- Verify recent events: `kubectl get events -n vapora --sort-by='.lastTimestamp' | tail -10`

### Health Baseline

- Record current metrics before deployment

  - CPU usage per deployment
  - Memory usage per deployment
  - Request latency (p50, p95, p99)
  - Error rate (4xx, 5xx)
  - Queue depth (if applicable)

- Verify services are responsive:

      curl http://localhost:8001/health -H "Authorization: Bearer $TOKEN"
      curl http://localhost:8001/api/projects

- Check logs for recent errors:

      kubectl logs deployment/vapora-backend -n vapora --tail=50
      kubectl logs deployment/vapora-agents -n vapora --tail=50

### Infrastructure Check

- Verify storage is not near capacity: `df -h /var/lib/vapora`
- Check database health: `kubectl exec -n vapora <pod> -- surreal info`
- Verify backups are recent (within 24 hours)
- Check SSL certificate expiration: `openssl s_client -connect api.vapora.com:443 -showcerts | grep "Validity"`

## 2 Hours Before Deployment

### Artifact Preparation

- Trigger validation in CI/CD pipeline

- Wait for artifact generation to complete

- Download artifacts from pipeline:

      # From GitHub Actions or Woodpecker UI
      # Download: deployment-artifacts.zip

- Verify artifact contents:

      unzip deployment-artifacts.zip
      ls -la
      # Should contain:
      # - configmap.yaml
      # - deployment.yaml
      # - docker-compose.yml
      # - vapora-{solo,multiuser,enterprise}.{toml,yaml,json}

- Validate manifest syntax:

      yq eval '.' configmap.yaml > /dev/null && echo "✓ ConfigMap valid"
      yq eval '.' deployment.yaml > /dev/null && echo "✓ Deployment valid"

### Test in Staging

- Perform dry-run deployment to staging cluster:

      kubectl apply -f configmap.yaml --dry-run=server -n vapora
      kubectl apply -f deployment.yaml --dry-run=server -n vapora

- Review dry-run output for any warnings or errors

- If test deployment available, do actual staging deployment and verify:

      kubectl get deployments -n vapora
      kubectl get pods -n vapora
      kubectl logs deployment/vapora-backend -n vapora --tail=5

- Test health endpoints on staging

- Run smoke tests against staging (if available)

### Rollback Plan Verification

- Document current deployment revisions:

      kubectl rollout history deployment/vapora-backend -n vapora
      # Record the highest revision number

- Create backup of current ConfigMap:

      kubectl get configmap -n vapora vapora-config -o yaml > configmap-backup.yaml

- Test rollback procedure on staging (if safe):

      # Record current revision
      CURRENT_REV=$(kubectl rollout history deployment/vapora-backend -n vapora | tail -1 | awk '{print $1}')

      # Test undo
      kubectl rollout undo deployment/vapora-backend -n vapora

      # Verify rollback
      kubectl get deployment vapora-backend -n vapora -o yaml | grep image

      # Restore to current
      kubectl rollout undo deployment/vapora-backend -n vapora --to-revision=$CURRENT_REV

- Confirm rollback command is documented in ticket/issue

## 1 Hour Before Deployment

### Final Checks

- Confirm all prerequisites met:
  - Code merged to main
  - Artifacts generated and validated
  - Staging deployment tested
  - Rollback plan documented
  - Team notified

### Communication Setup

- Set status page to "Maintenance Mode" (if public)

      "VAPORA maintenance deployment starting at HH:MM UTC.
       Expected duration: 10 minutes. Services may be briefly unavailable."

- Join #deployments Slack channel

- Prepare message: "🚀 Deployment starting now. Will update every 2 minutes."

- Have on-call engineer monitoring

- Verify monitoring/alerting dashboards are accessible

### Access Verification

- Verify kubeconfig is valid and up to date:

      kubectl cluster-info
      kubectl get nodes

- Verify kubectl version compatibility:

      kubectl version
      # Should match server version reasonably (within 1 minor version)

- Test write access to cluster:

      kubectl auth can-i create deployments --namespace=vapora
      # Should return "yes"

- Verify docker/docker-compose access (if Docker deployment)

- Verify Slack webhook is working (test send message)

## 15 Minutes Before Deployment

### Final Go/No-Go Decision

STOP HERE and make final decision to proceed or reschedule:

Proceed IF:

- ✅ All checklist items above completed
- ✅ No critical issues found during testing
- ✅ Staging deployment successful
- ✅ Team ready and monitoring
- ✅ Rollback plan clear and tested
- ✅ Within designated maintenance window

RESCHEDULE IF:

- ❌ Any critical issues discovered
- ❌ Staging tests failed
- ❌ Team member unavailable
- ❌ Production issues detected
- ❌ Unexpected changes in code/configs

### Final Notifications

If proceeding:

- Post to #deployments: "🚀 Deployment starting in 5 minutes"
- Alert on-call engineer: "Ready to start - confirm you're monitoring"
- Have rollback plan visible and accessible
- Open monitoring dashboard showing current metrics

### Terminal Setup

- Open terminal with kubeconfig configured:

      export KUBECONFIG=/path/to/production/kubeconfig
      kubectl cluster-info  # Verify connected to production

- Open second terminal for tailing logs:

      kubectl logs -f deployment/vapora-backend -n vapora

- Have rollback commands ready:

      # For quick rollback if needed
      kubectl rollout undo deployment/vapora-backend -n vapora
      kubectl rollout undo deployment/vapora-agents -n vapora
      kubectl rollout undo deployment/vapora-llm-router -n vapora

- Prepare metrics check script:

      watch kubectl top pods -n vapora
      watch kubectl get pods -n vapora

## Success Criteria Verification

Document what "success" looks like for this deployment:

- All three deployments have updated image IDs
- All pods reach "Ready" state within 5 minutes
- No pod restarts: `kubectl get pods -n vapora --watch` (no restarts column increasing)
- No error logs in first 2 minutes
- Health endpoints respond (200 OK)
- API endpoints respond to test requests
- Metrics show normal resource usage
- No alerts triggered
- Support team reports no user impact

## Team Roles During Deployment

### Deployment Lead

- Executes deployment commands
- Monitors progress
- Communicates status updates
- Decides to proceed/rollback

### On-Call Engineer

- Monitors dashboards and alerts
- Watches for anomalies
- Prepares for rollback if needed
- Available for emergency decisions

### Communications Lead (optional)

- Updates #deployments channel
- Notifies support/product teams
- Updates status page if public
- Handles external communication

### Backup Person

- Monitors for issues
- Ready to assist with troubleshooting
- Prepares rollback procedures
- Escalates if needed

## Common Issues to Watch For

⚠️ Pod CrashLoopBackOff

- Indicates config or image issue
- Check pod logs: `kubectl logs <pod>`
- Check events: `kubectl describe pod <pod>`
- Action: Rollback immediately

⚠️ Pending Pods (not starting)

- Check resource availability: `kubectl describe pod <pod>`
- Check node capacity
- Action: Investigate or rollback if resource exhausted

⚠️ High Error Rate

- Check application logs
- Compare with baseline errors
- Action: If >10% error increase, rollback

⚠️ Database Connection Errors

- Check ConfigMap has correct database URL
- Verify network connectivity to database
- Action: Check ConfigMap, fix and reapply if needed

⚠️ Memory or CPU Spike

- Monitor trends (sudden spike vs gradual)
- Check if within expected range for new code
- Action: Rollback if resource limits exceeded

## Post-Deployment Documentation

After deployment completes, record:

- Deployment start time (UTC)
- Deployment end time (UTC)
- Total duration
- Any issues encountered and resolution
- Rollback performed (Y/N)
- Metrics before/after (CPU, memory, latency, errors)
- Team members involved
- Blockers or lessons learned

## Sign-Off

Use this template for deployment issue/ticket:

    DEPLOYMENT COMPLETED

    ✓ All checks passed
    ✓ Deployment successful
    ✓ All pods running
    ✓ Health checks passing
    ✓ No user impact

    Deployed by: [Name]
    Start time: [UTC]
    Duration: [X minutes]
    Rollback needed: No

    Metrics:
    - Latency (p99): [X]ms
    - Error rate: [X]%
    - Pod restarts: 0

    Next deployment: [Date/Time]