## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Incident Response Runbook

Procedures for responding to and resolving VAPORA production incidents.

## Incident Severity Levels

### Severity 1: Critical 🔴

Definition: Service completely down or severely degraded affecting all users

Examples:

- All backend pods crashed
- Database completely unreachable
- API returning 100% errors
- Frontend completely inaccessible

Response Time: Immediate (< 2 minutes) On-Call: Page immediately (not optional) Communication: Update status page every 2 minutes

### Severity 2: Major 🟠

Definition: Service partially down or significantly degraded

Examples:

- 50% of requests returning errors
- Latency 10x normal
- Some services down but others working
- Intermittent connectivity issues

Response Time: 5 minutes On-Call: Alert on-call engineer Communication: Internal updates every 5 minutes

### Severity 3: Minor 🟡

Definition: Service slow or minor issues affecting some users

Examples:

- 5-10% error rate
- Elevated latency (2x normal)
- One pod having issues, others recovering
- Non-critical features unavailable

Response Time: 15 minutes On-Call: Alert team, not necessarily emergency page Communication: Post-incident update

### Severity 4: Informational 🟢

Definition: No user impact, system anomalies or preventive issues

Examples:

- Disk usage trending high
- SSL cert expiring in 30 days
- Deployment taking longer than normal
- Non-critical service warnings

Response Time: During business hours On-Call: No alert needed Communication: Team Slack message

## Incident Response Process

### Step 1: Report & Assess (Immediately)

When incident reported (via alert, user report, or discovery):

    # 1. Create incident ticket
    # Title: "INCIDENT: [Service] - [Brief description]"
    # Example: "INCIDENT: API - 50% error rate since 14:30 UTC"
    # Severity: [1-4]
    # Reporter: [Your name]
    # Time Detected: [UTC time]

    # 2. Open dedicated Slack channel
    #slack /create #incident-20260112-backend
    # Then: /invite @on-call-engineer

    # 3. Post initial message
    # "🔴 INCIDENT DECLARED
    #  Service: VAPORA Backend
    #  Severity: 1 (Critical)
    #  Time Detected: 14:32 UTC
    #  Current Status: Unknown
    #  Next Update: 14:34 UTC"

### Step 2: Quick Diagnosis (First 2 minutes)

    # Establish facts quickly
    export NAMESPACE=vapora

    # Q1: Is the service actually down?
    curl -v http://api.vapora.com/health
    # If: Connection refused → Service down
    # If: 500 errors → Service crashed
    # If: Timeout → Service hung

    # Q2: What's the scope?
    kubectl get pods -n $NAMESPACE
    # Count Running vs non-Running pods
    # All down → Complete outage
    # Some down → Partial outage

    # Q3: What's happening right now?
    for deployment in vapora-backend vapora-agents vapora-llm-router; do
      echo "=== $deployment ==="
      kubectl get deployment $deployment -n $NAMESPACE
    done
    # Shows: DESIRED vs CURRENT vs AVAILABLE
    # Example: 3 DESIRED, 0 CURRENT, 0 AVAILABLE → Pod startup failure

    # Q4: Any obvious errors?
    kubectl logs deployment/vapora-backend -n $NAMESPACE --tail=20 | grep -i "error\|fatal"
    # Shows: What's in the logs right now

### Step 3: Escalate Decision

Based on quick diagnosis, decide next action:

    IF pods not starting (CrashLoopBackOff):
      → Likely config issue
      → Check ConfigMap values
      → Likely recent deployment
      → DECISION: Possible rollback

    IF pods pending (not scheduled):
      → Likely resource issue
      → Check node capacity
      → DECISION: Scale down workloads or investigate nodes

    IF pods running but unresponsive:
      → Likely application issue
      → Check application logs
      → DECISION: Investigate app logic

    IF network/database issues:
      → Check connectivity
      → Check credentials
      → DECISION: Infrastructure escalation

    IF unknown:
      → Ask: "What changed recently?"
      → Check deployment history
      → Check infrastructure changes

### Step 4: Initial Response Actions

For Severity 1 (Critical):

    # A. Escalate immediately
    - Page senior engineer if not already responding
    - Contact infrastructure team
    - Notify product/support managers

    # B. Buy time with failover if available
    - Switch to backup environment if configured
    - Scale to different region if multi-region

    # C. Gather data for debugging
    - Save current logs
    - Save pod events
    - Record current metrics
    - Take screenshot of dashboards

    # D. Keep team updated
    # Update #incident-* channel every 2 minutes

For Severity 2 (Major):

    # A. Alert on-call team
    # B. Gather same diagnostics
    # C. Start investigation
    # D. Update every 5 minutes

For Severity 3 (Minor):

    # A. Create ticket for later investigation
    # B. Monitor closely
    # C. Gather diagnostics
    # D. Plan fix during normal hours if not urgent

### Step 5: Detailed Diagnosis

Once immediate actions taken:

    # Get comprehensive view of system state
    kubectl describe node <nodename>      # Hardware/capacity issues
    kubectl describe pod <podname> -n $NAMESPACE  # Pod-specific issues
    kubectl events -n $NAMESPACE          # What happened recently
    kubectl top nodes                     # CPU/memory usage
    kubectl top pods -n $NAMESPACE        # Per-pod resource usage

    # Check recent changes
    git log -5 --oneline
    git diff HEAD~1 HEAD provisioning/

    # Check deployment history
    kubectl rollout history deployment/vapora-backend -n $NAMESPACE | tail -5

    # Timeline analysis
    # What happened at 14:30 UTC? (incident time)
    # Was there a deployment?
    # Did metrics change suddenly?
    # Any alerts triggered?

### Step 6: Implement Fix

Depending on root cause:

#### Root Cause: Recent Bad Deployment

    # Solution: Rollback
    # See: Rollback Runbook
    kubectl rollout undo deployment/vapora-backend -n $NAMESPACE
    kubectl rollout status deployment/vapora-backend --timeout=5m

    # Verify
    curl http://localhost:8001/health

#### Root Cause: Insufficient Resources

    # Solution: Either scale out or reduce load

    # Option A: Add more nodes
    kubectl scale nodes --increment=1
    # (Requires infrastructure access)

    # Option B: Scale down non-critical services
    kubectl scale deployment/vapora-agents --replicas=1 -n $NAMESPACE
    # Then scale back up when resolved

    # Option C: Temporarily scale down pod replicas
    kubectl scale deployment/vapora-backend --replicas=2 -n $NAMESPACE
    # (Trade: Reduced capacity but faster recovery)

#### Root Cause: Configuration Error

    # Solution: Fix ConfigMap

    # 1. Identify wrong value
    kubectl get configmap -n $NAMESPACE vapora-config -o yaml | grep -A 2 <suspicious-key>

    # 2. Fix value
    # Edit configmap in external editor or via kubectl patch:
    kubectl patch configmap vapora-config -n $NAMESPACE \
      --type merge \
      -p '{"data":{"vapora.toml":"[corrected content]"}}'

    # 3. Restart pods to pick up new config
    kubectl rollout restart deployment/vapora-backend -n $NAMESPACE
    kubectl rollout status deployment/vapora-backend --timeout=5m

#### Root Cause: Database Issues

    # Solution: Depends on specific issue

    # If database down:
    - Contact DBA or database team
    - Check database status: kubectl exec <pod> -- curl localhost:8000

    # If credentials wrong:
    kubectl patch configmap vapora-config -n $NAMESPACE \
      --type merge \
      -p '{"data":{"DB_PASSWORD":"[correct-password]"}}'

    # If database full:
    - Contact DBA for cleanup
    - Free up space on database volume

    # If connection pool exhausted:
    - Scale down services to reduce connections
    - Increase connection pool size if possible

#### Root Cause: External Service Down

    # Examples: Third-party API, external database

    # Solution: Depends on severity

    # If critical: Failover
    - Switch to backup provider if available
    - Route traffic differently

    # If non-critical: Degrade gracefully
    - Disable feature temporarily
    - Use cache if available
    - Return cached data

    # Communicate
    - Notify users of reduced functionality
    - Provide ETA for restoration

### Step 7: Verify Recovery

    # Once fix applied, verify systematically

    # 1. Pod health
    kubectl get pods -n $NAMESPACE
    # All should show: Running, 1/1 Ready

    # 2. Service endpoints
    kubectl get endpoints -n $NAMESPACE
    # All should have IP addresses

    # 3. Health endpoints
    curl http://localhost:8001/health
    # Should return: 200 OK

    # 4. Check errors
    kubectl logs deployment/vapora-backend -n $NAMESPACE --since=2m | grep -i error
    # Should return: few or no errors

    # 5. Monitor metrics
    kubectl top pods -n $NAMESPACE
    # CPU/Memory should be normal (not spiking)

    # 6. Check for new issues
    kubectl get events -n $NAMESPACE
    # Should show normal state, no warnings

### Step 8: Incident Closure

    # When everything verified healthy:

    # 1. Document resolution
    # Update incident ticket with:
    # - Root cause
    # - Fix applied
    # - Verification steps
    # - Resolution time
    # - Impact (how many users, how long)

    # 2. Post final update
    # "#incident channel:
    #  ✅ INCIDENT RESOLVED
    #
    #  Duration: [start] to [end] = [X minutes]
    #  Root Cause: [brief description]
    #  Fix Applied: [brief description]
    #  Impact: ~X users affected for X minutes
    #
    #  Status: All services healthy
    #  Monitoring: Continuing for 1 hour
    #  Post-mortem: Scheduled for [date]"

    # 3. Schedule post-mortem
    # Within 24 hours: review what happened and why
    # Document lessons learned

    # 4. Update dashboards
    # Document incident on status page history
    # If public incident: close status page incident

    # 5. Send all-clear message
    # Notify: support team, product team, key stakeholders

## Incident Response Roles & Responsibilities

### Incident Commander

- Overall control of incident response
- Makes critical decisions
- Drives decision-making speed
- Communicates status updates
- Calls when to escalate
- You if you discovered the incident and best understands it

### Technical Responders

- Investigate specific systems
- Implement fixes
- Report findings to commander
- Execute verified solutions

### Communication Lead (if Severity 1)

- Updates #incident channel every 2 minutes
- Updates status page every 5 minutes
- Fields questions from support/product
- Notifies key stakeholders

### On-Call Manager (if Severity 1)

- Pages additional resources if needed
- Escalates to senior engineers
- Engages infrastructure/DBA teams
- Tracks response timeline

## Common Incidents & Responses

### Incident Type: Service Unresponsive

    Detection: curl returns "Connection refused"
    Diagnosis Time: 1 minute
    Response:
    1. Check if pods are running: kubectl get pods
    2. If not running: likely crash → check logs
    3. If running but unresponsive: likely port/network issue
    4. Verify service exists: kubectl get service vapora-backend

    Solution:
    - If pods crashed: check logs, likely config or deployment issue
    - If pods hanging: restart pods: kubectl delete pods -l app=vapora-backend
    - If service/endpoints missing: apply service manifest

### Incident Type: High Error Rate

    Detection: Dashboard shows >10% 5xx errors
    Diagnosis Time: 2 minutes
    Response:
    1. Check which endpoint is failing
    2. Check logs for error pattern
    3. Identify affected service (backend, agents, router)
    4. Compare with baseline (worked X minutes ago)

    Solution:
    - If recent deployment: rollback
    - If config change: revert config
    - If database issue: contact DBA
    - If third-party down: implement fallback

### Incident Type: High Latency

    Detection: Dashboard shows p99 latency >2 seconds
    Diagnosis Time: 2 minutes
    Response:
    1. Check if requests still succeeding (is it slow or failing?)
    2. Check CPU/memory usage: kubectl top pods
    3. Check if database slow: run query diagnostics
    4. Check network: are there packet losses?

    Solution:
    - If resource exhausted: scale up or reduce load
    - If database slow: DBA investigation
    - If network issue: infrastructure team
    - If legitimate increased load: no action needed (expected)

### Incident Type: Pod Restarting Repeatedly

    Detection: kubectl get pods shows high RESTARTS count
    Diagnosis Time: 1 minute
    Response:
    1. Check restart count: kubectl get pods -n vapora
    2. Get pod logs: kubectl logs <pod-name> -n vapora --previous
    3. Get pod events: kubectl describe pod <pod-name> -n vapora

    Solution:
    - Application error: check logs, fix issue, redeploy
    - Config issue: fix ConfigMap, restart pods
    - Resource issue: increase limits or scale out
    - Liveness probe failing: adjust probe timing or fix health check

### Incident Type: Database Connectivity

    Detection: Logs show "database connection refused"
    Diagnosis Time: 2 minutes
    Response:
    1. Check database service running: kubectl get pod -n <db-namespace>
    2. Check database credentials in ConfigMap
    3. Test connectivity: kubectl exec <pod> -- psql $DB_URL
    4. Check firewall/network policy

    Solution:
    - If DB down: escalate to DBA, possibly restore from backup
    - If credentials wrong: fix ConfigMap, restart app pods
    - If network issue: network team investigation
    - If no space: DBA cleanup

## Communication During Incident

### Every 2 Minutes (Severity 1) or 5 Minutes (Severity 2)

Post update to #incident channel:

    ⏱️ 14:35 UTC UPDATE

    Status: Investigating
    Current Action: Checking pod logs
    Findings: Backend pods in CrashLoopBackOff
    Next Step: Review recent deployment
    ETA for Update: 14:37 UTC

    /cc @on-call-engineer

### Status Page Updates (If Public)

    INCIDENT: VAPORA API Partially Degraded

    Investigating: Our team is investigating elevated error rates
    Duration: 5 minutes
    Impact: ~30% of API requests failing

    Last Updated: 14:35 UTC
    Next Update: 14:37 UTC

### Escalation Communication

    If Severity 1 and unable to identify cause in 5 minutes:

    "Escalating to senior engineering team.
    Page @senior-engineer-on-call immediately.
    Activating Incident War Room."

    Include:
    - Service name
    - Duration so far
    - What's been tried
    - Current symptoms
    - Why stuck

## Incident Severity Decision Tree

    Question 1: Can any users access the service?
      NO → Severity 1 (Critical - complete outage)
      YES → Question 2

    Question 2: What percentage of requests are failing?
      >50% → Severity 1 (Critical)
      10-50% → Severity 2 (Major)
      5-10% → Severity 3 (Minor)
      <5% → Question 3

    Question 3: Is the service recovering on its own?
      NO (staying broken) → Severity 2
      YES (automatically recovering) → Question 4

    Question 4: Does it require any user action/data loss?
      YES → Severity 2
      NO → Severity 3

## Post-Incident Procedures

### Immediate (Within 30 minutes)

- Close incident ticket
- Post final update to #incident channel
- Save all logs and diagnostics
- Create post-mortem ticket
- Notify team: "incident resolved"

### Follow-Up (Within 24 hours)

- Schedule post-mortem meeting
- Identify root cause
- Document preventive measures
- Identify owner for each action item
- Create tickets for improvements

### Prevention (Within 1 week)

- Implement identified fixes
- Update monitoring/alerting
- Update runbooks with findings
- Conduct team training if needed
- Close post-mortem ticket

## Incident Checklist

    ☐ Incident severity determined
    ☐ Ticket created and updated
    ☐ #incident channel created
    ☐ On-call team alerted
    ☐ Initial diagnosis completed
    ☐ Fix identified and implemented
    ☐ Fix verified working
    ☐ Incident closed and communicated
    ☐ Post-mortem scheduled
    ☐ Team debriefed
    ☐ Root cause documented
    ☐ Prevention measures identified
    ☐ Tickets created for follow-up