provisioning/docs/src/infrastructure/infrastructure-management.md

# Infrastructure Management Guide

This comprehensive guide covers creating, managing, and maintaining infrastructure using Infrastructure Automation.

## What You'll Learn

- Infrastructure lifecycle management
- Server provisioning and management
- Task service installation and configuration
- Cluster deployment and orchestration
- Scaling and optimization strategies
- Monitoring and maintenance procedures
- Cost management and optimization

## Infrastructure Concepts

### Infrastructure Components

| Component | Description | Examples |
|-----------|-------------|----------|
| **Servers** | Virtual machines or containers | Web servers, databases, workers |
| **Task Services** | Software installed on servers | Kubernetes, Docker, databases |
| **Clusters** | Groups of related services | Web clusters, database clusters |
| **Networks** | Connectivity between resources | VPCs, subnets, load balancers |
| **Storage** | Persistent data storage | Block storage, object storage |

### Infrastructure Lifecycle

```plaintext
Plan → Create → Deploy → Monitor → Scale → Update → Retire
```plaintext

Each phase has specific commands and considerations.

## Server Management

### Understanding Server Configuration

Servers are defined in Nickel configuration files:

```nickel
# Example server configuration
import models.server

servers: [
    server.Server {
        name = "web-01"
        provider = "aws"          # aws, upcloud, local
        plan = "t3.medium"        # Instance type/plan
        os = "ubuntu-22.04"       # Operating system
        zone = "us-west-2a"       # Availability zone

        # Network configuration
        vpc = "main"
        subnet = "web"
        security_groups = ["web", "ssh"]

        # Storage configuration
        storage = {
            root_size = "50 GB"
            additional = [
                {name = "data", size = "100 GB", type = "gp3"}
            ]
        }

        # Task services to install
        taskservs = [
            "containerd",
            "kubernetes",
            "monitoring"
        ]

        # Tags for organization
        tags = {
            environment = "production"
            team = "platform"
            cost_center = "engineering"
        }
    }
]
```plaintext

### Server Lifecycle Commands

#### Creating Servers

```bash
# Plan server creation (dry run)
provisioning server create --infra my-infra --check

# Create servers
provisioning server create --infra my-infra

# Create with specific parameters
provisioning server create --infra my-infra --wait --yes

# Create single server type
provisioning server create web --infra my-infra
```plaintext

#### Managing Existing Servers

```bash
# List all servers
provisioning server list --infra my-infra

# Show detailed server information
provisioning show servers --infra my-infra

# Show specific server
provisioning show servers web-01 --infra my-infra

# Get server status
provisioning server status web-01 --infra my-infra
```plaintext

#### Server Operations

```bash
# Start/stop servers
provisioning server start web-01 --infra my-infra
provisioning server stop web-01 --infra my-infra

# Restart servers
provisioning server restart web-01 --infra my-infra

# Resize server
provisioning server resize web-01 --plan t3.large --infra my-infra

# Update server configuration
provisioning server update web-01 --infra my-infra
```plaintext

#### SSH Access

```bash
# SSH to server
provisioning server ssh web-01 --infra my-infra

# SSH with specific user
provisioning server ssh web-01 --user admin --infra my-infra

# Execute command on server
provisioning server exec web-01 "systemctl status kubernetes" --infra my-infra

# Copy files to/from server
provisioning server copy local-file.txt web-01:/tmp/ --infra my-infra
provisioning server copy web-01:/var/log/app.log ./logs/ --infra my-infra
```plaintext

#### Server Deletion

```bash
# Plan server deletion (dry run)
provisioning server delete --infra my-infra --check

# Delete specific server
provisioning server delete web-01 --infra my-infra

# Delete with confirmation
provisioning server delete web-01 --infra my-infra --yes

# Delete but keep storage
provisioning server delete web-01 --infra my-infra --keepstorage
```plaintext

## Task Service Management

### Understanding Task Services

Task services are software components installed on servers:

- **Container Runtimes**: containerd, cri-o, docker
- **Orchestration**: kubernetes, nomad
- **Networking**: cilium, calico, haproxy
- **Storage**: rook-ceph, longhorn, nfs
- **Databases**: postgresql, mysql, mongodb
- **Monitoring**: prometheus, grafana, alertmanager

### Task Service Configuration

```nickel
# Task service configuration example
taskservs: {
    kubernetes: {
        version = "1.28"
        network_plugin = "cilium"
        ingress_controller = "nginx"
        storage_class = "gp3"

        # Cluster configuration
        cluster = {
            name = "production"
            pod_cidr = "10.244.0.0/16"
            service_cidr = "10.96.0.0/12"
        }

        # Node configuration
        nodes = {
            control_plane = ["master-01", "master-02", "master-03"]
            workers = ["worker-01", "worker-02", "worker-03"]
        }
    }

    postgresql: {
        version = "15"
        port = 5432
        max_connections = 200
        shared_buffers = "256 MB"

        # High availability
        replication = {
            enabled = true
            replicas = 2
            sync_mode = "synchronous"
        }

        # Backup configuration
        backup = {
            enabled = true
            schedule = "0 2 * * *"  # Daily at 2 AM
            retention = "30d"
        }
    }
}
```plaintext

### Task Service Commands

#### Installing Services

```bash
# Install single service
provisioning taskserv create kubernetes --infra my-infra

# Install multiple services
provisioning taskserv create containerd kubernetes cilium --infra my-infra

# Install with specific version
provisioning taskserv create kubernetes --version 1.28 --infra my-infra

# Install on specific servers
provisioning taskserv create postgresql --servers db-01,db-02 --infra my-infra
```plaintext

#### Managing Services

```bash
# List available services
provisioning taskserv list

# List installed services
provisioning taskserv list --infra my-infra --installed

# Show service details
provisioning taskserv show kubernetes --infra my-infra

# Check service status
provisioning taskserv status kubernetes --infra my-infra

# Check service health
provisioning taskserv health kubernetes --infra my-infra
```plaintext

#### Service Operations

```bash
# Start/stop services
provisioning taskserv start kubernetes --infra my-infra
provisioning taskserv stop kubernetes --infra my-infra

# Restart services
provisioning taskserv restart kubernetes --infra my-infra

# Update services
provisioning taskserv update kubernetes --infra my-infra

# Configure services
provisioning taskserv configure kubernetes --config cluster.yaml --infra my-infra
```plaintext

#### Service Removal

```bash
# Remove service
provisioning taskserv delete kubernetes --infra my-infra

# Remove with data cleanup
provisioning taskserv delete postgresql --cleanup-data --infra my-infra

# Remove from specific servers
provisioning taskserv delete kubernetes --servers worker-03 --infra my-infra
```plaintext

### Version Management

```bash
# Check for updates
provisioning taskserv check-updates --infra my-infra

# Check specific service updates
provisioning taskserv check-updates kubernetes --infra my-infra

# Show available versions
provisioning taskserv versions kubernetes

# Upgrade to latest version
provisioning taskserv upgrade kubernetes --infra my-infra

# Upgrade to specific version
provisioning taskserv upgrade kubernetes --version 1.29 --infra my-infra
```plaintext

## Cluster Management

### Understanding Clusters

Clusters are collections of services that work together to provide functionality:

```nickel
# Cluster configuration example
clusters: {
    web_cluster: {
        name = "web-application"
        description = "Web application cluster"

        # Services in the cluster
        services = [
            {
                name = "nginx"
                replicas = 3
                image = "nginx:1.24"
                ports = [80, 443]
            }
            {
                name = "app"
                replicas = 5
                image = "myapp:latest"
                ports = [8080]
            }
        ]

        # Load balancer configuration
        load_balancer = {
            type = "application"
            health_check = "/health"
            ssl_cert = "wildcard.example.com"
        }

        # Auto-scaling
        auto_scaling = {
            min_replicas = 2
            max_replicas = 10
            target_cpu = 70
            target_memory = 80
        }
    }
}
```plaintext

### Cluster Commands

#### Creating Clusters

```bash
# Create cluster
provisioning cluster create web-cluster --infra my-infra

# Create with specific configuration
provisioning cluster create web-cluster --config cluster.yaml --infra my-infra

# Create and deploy
provisioning cluster create web-cluster --deploy --infra my-infra
```plaintext

#### Managing Clusters

```bash
# List available clusters
provisioning cluster list

# List deployed clusters
provisioning cluster list --infra my-infra --deployed

# Show cluster details
provisioning cluster show web-cluster --infra my-infra

# Get cluster status
provisioning cluster status web-cluster --infra my-infra
```plaintext

#### Cluster Operations

```bash
# Deploy cluster
provisioning cluster deploy web-cluster --infra my-infra

# Scale cluster
provisioning cluster scale web-cluster --replicas 10 --infra my-infra

# Update cluster
provisioning cluster update web-cluster --infra my-infra

# Rolling update
provisioning cluster update web-cluster --rolling --infra my-infra
```plaintext

#### Cluster Deletion

```bash
# Delete cluster
provisioning cluster delete web-cluster --infra my-infra

# Delete with data cleanup
provisioning cluster delete web-cluster --cleanup --infra my-infra
```plaintext

## Network Management

### Network Configuration

```nickel
# Network configuration
network: {
    vpc = {
        cidr = "10.0.0.0/16"
        enable_dns = true
        enable_dhcp = true
    }

    subnets = [
        {
            name = "web"
            cidr = "10.0.1.0/24"
            zone = "us-west-2a"
            public = true
        }
        {
            name = "app"
            cidr = "10.0.2.0/24"
            zone = "us-west-2b"
            public = false
        }
        {
            name = "data"
            cidr = "10.0.3.0/24"
            zone = "us-west-2c"
            public = false
        }
    ]

    security_groups = [
        {
            name = "web"
            rules = [
                {protocol = "tcp", port = 80, source = "0.0.0.0/0"}
                {protocol = "tcp", port = 443, source = "0.0.0.0/0"}
            ]
        }
        {
            name = "app"
            rules = [
                {protocol = "tcp", port = 8080, source = "10.0.1.0/24"}
            ]
        }
    ]

    load_balancers = [
        {
            name = "web-lb"
            type = "application"
            scheme = "internet-facing"
            subnets = ["web"]
            targets = ["web-01", "web-02"]
        }
    ]
}
```plaintext

### Network Commands

```bash
# Show network configuration
provisioning network show --infra my-infra

# Create network resources
provisioning network create --infra my-infra

# Update network configuration
provisioning network update --infra my-infra

# Test network connectivity
provisioning network test --infra my-infra
```plaintext

## Storage Management

### Storage Configuration

```nickel
# Storage configuration
storage: {
    # Block storage
    volumes = [
        {
            name = "app-data"
            size = "100 GB"
            type = "gp3"
            encrypted = true
        }
    ]

    # Object storage
    buckets = [
        {
            name = "app-assets"
            region = "us-west-2"
            versioning = true
            encryption = "AES256"
        }
    ]

    # Backup configuration
    backup = {
        schedule = "0 1 * * *"  # Daily at 1 AM
        retention = {
            daily = 7
            weekly = 4
            monthly = 12
        }
    }
}
```plaintext

### Storage Commands

```bash
# Create storage resources
provisioning storage create --infra my-infra

# List storage
provisioning storage list --infra my-infra

# Backup data
provisioning storage backup --infra my-infra

# Restore from backup
provisioning storage restore --backup latest --infra my-infra
```plaintext

## Monitoring and Observability

### Monitoring Setup

```bash
# Install monitoring stack
provisioning taskserv create prometheus --infra my-infra
provisioning taskserv create grafana --infra my-infra
provisioning taskserv create alertmanager --infra my-infra

# Configure monitoring
provisioning taskserv configure prometheus --config monitoring.yaml --infra my-infra
```plaintext

### Health Checks

```bash
# Check overall infrastructure health
provisioning health check --infra my-infra

# Check specific components
provisioning health check servers --infra my-infra
provisioning health check taskservs --infra my-infra
provisioning health check clusters --infra my-infra

# Continuous monitoring
provisioning health monitor --infra my-infra --watch
```plaintext

### Metrics and Alerting

```bash
# Get infrastructure metrics
provisioning metrics get --infra my-infra

# Set up alerts
provisioning alerts create --config alerts.yaml --infra my-infra

# List active alerts
provisioning alerts list --infra my-infra
```plaintext

## Cost Management

### Cost Monitoring

```bash
# Show current costs
provisioning cost show --infra my-infra

# Cost breakdown by component
provisioning cost breakdown --infra my-infra

# Cost trends
provisioning cost trends --period 30d --infra my-infra

# Set cost alerts
provisioning cost alert --threshold 1000 --infra my-infra
```plaintext

### Cost Optimization

```bash
# Analyze cost optimization opportunities
provisioning cost optimize --infra my-infra

# Show unused resources
provisioning cost unused --infra my-infra

# Right-size recommendations
provisioning cost recommendations --infra my-infra
```plaintext

## Scaling Strategies

### Manual Scaling

```bash
# Scale servers
provisioning server scale --count 5 --infra my-infra

# Scale specific service
provisioning taskserv scale kubernetes --nodes 3 --infra my-infra

# Scale cluster
provisioning cluster scale web-cluster --replicas 10 --infra my-infra
```plaintext

### Auto-scaling Configuration

```nickel
# Auto-scaling configuration
auto_scaling: {
    servers = {
        min_count = 2
        max_count = 10

        # Scaling metrics
        cpu_threshold = 70
        memory_threshold = 80

        # Scaling behavior
        scale_up_cooldown = "5m"
        scale_down_cooldown = "10m"
    }

    clusters = {
        web_cluster = {
            min_replicas = 3
            max_replicas = 20
            metrics = [
                {type = "cpu", target = 70}
                {type = "memory", target = 80}
                {type = "requests", target = 1000}
            ]
        }
    }
}
```plaintext

## Disaster Recovery

### Backup Strategies

```bash
# Full infrastructure backup
provisioning backup create --type full --infra my-infra

# Incremental backup
provisioning backup create --type incremental --infra my-infra

# Schedule automated backups
provisioning backup schedule --daily --time "02:00" --infra my-infra
```plaintext

### Recovery Procedures

```bash
# List available backups
provisioning backup list --infra my-infra

# Restore infrastructure
provisioning restore --backup latest --infra my-infra

# Partial restore
provisioning restore --backup latest --components servers --infra my-infra

# Test restore (dry run)
provisioning restore --backup latest --test --infra my-infra
```plaintext

## Advanced Infrastructure Patterns

### Multi-Region Deployment

```nickel
# Multi-region configuration
regions: {
    primary = {
        name = "us-west-2"
        servers = ["web-01", "web-02", "db-01"]
        availability_zones = ["us-west-2a", "us-west-2b"]
    }

    secondary = {
        name = "us-east-1"
        servers = ["web-03", "web-04", "db-02"]
        availability_zones = ["us-east-1a", "us-east-1b"]
    }

    # Cross-region replication
    replication = {
        database = {
            primary = "us-west-2"
            replicas = ["us-east-1"]
            sync_mode = "async"
        }

        storage = {
            sync_schedule = "*/15 * * * *"  # Every 15 minutes
        }
    }
}
```plaintext

### Blue-Green Deployment

```bash
# Create green environment
provisioning generate infra --from production --name production-green

# Deploy to green
provisioning server create --infra production-green
provisioning taskserv create --infra production-green
provisioning cluster deploy --infra production-green

# Switch traffic to green
provisioning network switch --from production --to production-green

# Decommission blue
provisioning server delete --infra production --yes
```plaintext

### Canary Deployment

```bash
# Create canary environment
provisioning cluster create web-cluster-canary --replicas 1 --infra my-infra

# Route small percentage of traffic
provisioning network route --target web-cluster-canary --weight 10 --infra my-infra

# Monitor canary metrics
provisioning metrics monitor web-cluster-canary --infra my-infra

# Promote or rollback
provisioning cluster promote web-cluster-canary --infra my-infra
# or
provisioning cluster rollback web-cluster-canary --infra my-infra
```plaintext

## Troubleshooting Infrastructure

### Common Issues

#### Server Creation Failures

```bash
# Check provider status
provisioning provider status aws

# Validate server configuration
provisioning server validate web-01 --infra my-infra

# Check quota limits
provisioning provider quota --infra my-infra

# Debug server creation
provisioning --debug server create web-01 --infra my-infra
```plaintext

#### Service Installation Failures

```bash
# Check service prerequisites
provisioning taskserv check kubernetes --infra my-infra

# Validate service configuration
provisioning taskserv validate kubernetes --infra my-infra

# Check service logs
provisioning taskserv logs kubernetes --infra my-infra

# Debug service installation
provisioning --debug taskserv create kubernetes --infra my-infra
```plaintext

#### Network Connectivity Issues

```bash
# Test network connectivity
provisioning network test --infra my-infra

# Check security groups
provisioning network security-groups --infra my-infra

# Trace network path
provisioning network trace --from web-01 --to db-01 --infra my-infra
```plaintext

### Performance Optimization

```bash
# Analyze performance bottlenecks
provisioning performance analyze --infra my-infra

# Get performance recommendations
provisioning performance recommendations --infra my-infra

# Monitor resource utilization
provisioning performance monitor --infra my-infra --duration 1h
```plaintext

## Testing Infrastructure

The provisioning system includes a comprehensive **Test Environment Service** for automated testing of infrastructure components before deployment.

### Why Test Infrastructure

Testing infrastructure before production deployment helps:

- **Validate taskserv configurations** before installing on production servers
- **Test integration** between multiple taskservs
- **Verify cluster topologies** (Kubernetes, etcd, etc.) before deployment
- **Catch configuration errors** early in the development cycle
- **Ensure compatibility** between components

### Test Environment Types

#### 1. Single Taskserv Testing

Test individual taskservs in isolated containers:

```bash
# Quick test (create, run, cleanup automatically)
provisioning test quick kubernetes

# Single taskserv with custom resources
provisioning test env single postgres \
  --cpu 2000 \
  --memory 4096 \
  --auto-start \
  --auto-cleanup

# Test with specific infrastructure context
provisioning test env single redis --infra my-infra
```plaintext

#### 2. Server Simulation

Test complete server configurations with multiple taskservs:

```bash
# Simulate web server with multiple taskservs
provisioning test env server web-01 [containerd kubernetes cilium] \
  --auto-start

# Simulate database server
provisioning test env server db-01 [postgres redis] \
  --infra prod-stack \
  --auto-start
```plaintext

#### 3. Multi-Node Cluster Testing

Test complex cluster topologies before production deployment:

```bash
# Test 3-node Kubernetes cluster
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# Test etcd cluster
provisioning test topology load etcd_cluster | \
  test env cluster etcd --auto-start

# Test single-node Kubernetes
provisioning test topology load kubernetes_single | \
  test env cluster kubernetes --auto-start
```plaintext

### Managing Test Environments

```bash
# List all test environments
provisioning test env list

# Check environment status
provisioning test env status <env-id>

# View environment logs
provisioning test env logs <env-id>

# Cleanup environment when done
provisioning test env cleanup <env-id>
```plaintext

### Available Topology Templates

Pre-configured multi-node cluster templates:

| Template | Description | Use Case |
|----------|-------------|----------|
| `kubernetes_3node` | 3-node HA K8s cluster | Production-like K8s testing |
| `kubernetes_single` | All-in-one K8s node | Development K8s testing |
| `etcd_cluster` | 3-member etcd cluster | Distributed consensus testing |
| `containerd_test` | Standalone containerd | Container runtime testing |
| `postgres_redis` | Database stack | Database integration testing |

### Test Environment Workflow

Typical testing workflow:

```bash
# 1. Test new taskserv before deploying
provisioning test quick kubernetes

# 2. If successful, test server configuration
provisioning test env server k8s-node [containerd kubernetes cilium] \
  --auto-start

# 3. Test complete cluster topology
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# 4. Deploy to production
provisioning server create --infra production
provisioning taskserv create kubernetes --infra production
```plaintext

### CI/CD Integration

Integrate infrastructure testing into CI/CD pipelines:

```yaml
# GitLab CI example
test-infrastructure:
  stage: test
  script:
    # Start orchestrator
    - ./scripts/start-orchestrator.nu --background

    # Test critical taskservs
    - provisioning test quick kubernetes
    - provisioning test quick postgres
    - provisioning test quick redis

    # Test cluster topology
    - provisioning test topology load kubernetes_3node |
        test env cluster kubernetes --auto-start

  artifacts:
    when: on_failure
    paths:
      - test-logs/
```plaintext

### Prerequisites

Test environments require:

1. **Docker Running**: Test environments use Docker containers

   ```bash
   docker ps  # Should work without errors
   ```

1. **Orchestrator Running**: The orchestrator manages test containers

   ```bash
   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   ```

### Advanced Testing

#### Custom Topology Testing

Create custom topology configurations:

```toml
# custom-topology.toml
[my_cluster]
name = "Custom Test Cluster"
cluster_type = "custom"

[[my_cluster.nodes]]
name = "node-01"
role = "primary"
taskservs = ["postgres", "redis"]
[my_cluster.nodes.resources]
cpu_millicores = 2000
memory_mb = 4096

[[my_cluster.nodes]]
name = "node-02"
role = "replica"
taskservs = ["postgres"]
[my_cluster.nodes.resources]
cpu_millicores = 1000
memory_mb = 2048
```plaintext

Load and test custom topology:

```bash
provisioning test env cluster custom-app custom-topology.toml --auto-start
```plaintext

#### Integration Testing

Test taskserv dependencies:

```bash
# Test Kubernetes dependencies in order
provisioning test quick containerd
provisioning test quick etcd
provisioning test quick kubernetes
provisioning test quick cilium

# Test complete stack
provisioning test env server k8s-stack \
  [containerd etcd kubernetes cilium] \
  --auto-start
```plaintext

### Documentation

For complete test environment documentation:

- **Test Environment Guide**: `docs/user/test-environment-guide.md`
- **Detailed Usage**: `docs/user/test-environment-usage.md`
- **Orchestrator README**: `provisioning/platform/orchestrator/README.md`

## Best Practices

### 1. Infrastructure Design

- **Principle of Least Privilege**: Grant minimal necessary access
- **Defense in Depth**: Multiple layers of security
- **High Availability**: Design for failure resilience
- **Scalability**: Plan for growth from the start

### 2. Operational Excellence

```bash
# Always validate before applying changes
provisioning validate config --infra my-infra

# Use check mode for dry runs
provisioning server create --check --infra my-infra

# Monitor continuously
provisioning health monitor --infra my-infra

# Regular backups
provisioning backup schedule --daily --infra my-infra
```plaintext

### 3. Security

```bash
# Regular security updates
provisioning taskserv update --security-only --infra my-infra

# Encrypt sensitive data
provisioning sops settings.ncl --infra my-infra

# Audit access
provisioning audit logs --infra my-infra
```plaintext

### 4. Cost Optimization

```bash
# Regular cost reviews
provisioning cost analyze --infra my-infra

# Right-size resources
provisioning cost optimize --apply --infra my-infra

# Use reserved instances for predictable workloads
provisioning server reserve --infra my-infra
```plaintext

## Next Steps

Now that you understand infrastructure management:

1. **Learn about extensions**: [Extension Development Guide](extension-development.md)
2. **Master configuration**: [Configuration Guide](configuration.md)
3. **Explore advanced examples**: [Examples and Tutorials](examples/)
4. **Set up monitoring and alerting**
5. **Implement automated scaling**
6. **Plan disaster recovery procedures**

You now have the knowledge to build and manage robust, scalable cloud infrastructure!