provisioning/docs/src/infrastructure/infrastructure-management.md

Infrastructure Management Guide

This comprehensive guide covers creating, managing, and maintaining infrastructure using Infrastructure Automation.

What You'll Learn

• Infrastructure lifecycle management
• Server provisioning and management
• Task service installation and configuration
• Cluster deployment and orchestration
• Scaling and optimization strategies
• Monitoring and maintenance procedures
• Cost management and optimization

Infrastructure Concepts

Infrastructure Components

Component	Description	Examples
Servers	Virtual machines or containers	Web servers, databases, workers
Task Services	Software installed on servers	Kubernetes, Docker, databases
Clusters	Groups of related services	Web clusters, database clusters
Networks	Connectivity between resources	VPCs, subnets, load balancers
Storage	Persistent data storage	Block storage, object storage

Infrastructure Lifecycle

Plan → Create → Deploy → Monitor → Scale → Update → Retire
```plaintext

Each phase has specific commands and considerations.

## Server Management

### Understanding Server Configuration

Servers are defined in Nickel configuration files:

```nickel
# Example server configuration
import models.server

servers: [
    server.Server {
        name = "web-01"
        provider = "aws"          # aws, upcloud, local
        plan = "t3.medium"        # Instance type/plan
        os = "ubuntu-22.04"       # Operating system
        zone = "us-west-2a"       # Availability zone

        # Network configuration
        vpc = "main"
        subnet = "web"
        security_groups = ["web", "ssh"]

        # Storage configuration
        storage = {
            root_size = "50 GB"
            additional = [
                {name = "data", size = "100 GB", type = "gp3"}
            ]
        }

        # Task services to install
        taskservs = [
            "containerd",
            "kubernetes",
            "monitoring"
        ]

        # Tags for organization
        tags = {
            environment = "production"
            team = "platform"
            cost_center = "engineering"
        }
    }
]
```plaintext

### Server Lifecycle Commands

#### Creating Servers

```bash
# Plan server creation (dry run)
provisioning server create --infra my-infra --check

# Create servers
provisioning server create --infra my-infra

# Create with specific parameters
provisioning server create --infra my-infra --wait --yes

# Create single server type
provisioning server create web --infra my-infra
```plaintext

#### Managing Existing Servers

```bash
# List all servers
provisioning server list --infra my-infra

# Show detailed server information
provisioning show servers --infra my-infra

# Show specific server
provisioning show servers web-01 --infra my-infra

# Get server status
provisioning server status web-01 --infra my-infra
```plaintext

#### Server Operations

```bash
# Start/stop servers
provisioning server start web-01 --infra my-infra
provisioning server stop web-01 --infra my-infra

# Restart servers
provisioning server restart web-01 --infra my-infra

# Resize server
provisioning server resize web-01 --plan t3.large --infra my-infra

# Update server configuration
provisioning server update web-01 --infra my-infra
```plaintext

#### SSH Access

```bash
# SSH to server
provisioning server ssh web-01 --infra my-infra

# SSH with specific user
provisioning server ssh web-01 --user admin --infra my-infra

# Execute command on server
provisioning server exec web-01 "systemctl status kubernetes" --infra my-infra

# Copy files to/from server
provisioning server copy local-file.txt web-01:/tmp/ --infra my-infra
provisioning server copy web-01:/var/log/app.log ./logs/ --infra my-infra
```plaintext

#### Server Deletion

```bash
# Plan server deletion (dry run)
provisioning server delete --infra my-infra --check

# Delete specific server
provisioning server delete web-01 --infra my-infra

# Delete with confirmation
provisioning server delete web-01 --infra my-infra --yes

# Delete but keep storage
provisioning server delete web-01 --infra my-infra --keepstorage
```plaintext

## Task Service Management

### Understanding Task Services

Task services are software components installed on servers:

- **Container Runtimes**: containerd, cri-o, docker
- **Orchestration**: kubernetes, nomad
- **Networking**: cilium, calico, haproxy
- **Storage**: rook-ceph, longhorn, nfs
- **Databases**: postgresql, mysql, mongodb
- **Monitoring**: prometheus, grafana, alertmanager

### Task Service Configuration

```nickel
# Task service configuration example
taskservs: {
    kubernetes: {
        version = "1.28"
        network_plugin = "cilium"
        ingress_controller = "nginx"
        storage_class = "gp3"

        # Cluster configuration
        cluster = {
            name = "production"
            pod_cidr = "10.244.0.0/16"
            service_cidr = "10.96.0.0/12"
        }

        # Node configuration
        nodes = {
            control_plane = ["master-01", "master-02", "master-03"]
            workers = ["worker-01", "worker-02", "worker-03"]
        }
    }

    postgresql: {
        version = "15"
        port = 5432
        max_connections = 200
        shared_buffers = "256 MB"

        # High availability
        replication = {
            enabled = true
            replicas = 2
            sync_mode = "synchronous"
        }

        # Backup configuration
        backup = {
            enabled = true
            schedule = "0 2 * * *"  # Daily at 2 AM
            retention = "30d"
        }
    }
}
```plaintext

### Task Service Commands

#### Installing Services

```bash
# Install single service
provisioning taskserv create kubernetes --infra my-infra

# Install multiple services
provisioning taskserv create containerd kubernetes cilium --infra my-infra

# Install with specific version
provisioning taskserv create kubernetes --version 1.28 --infra my-infra

# Install on specific servers
provisioning taskserv create postgresql --servers db-01,db-02 --infra my-infra
```plaintext

#### Managing Services

```bash
# List available services
provisioning taskserv list

# List installed services
provisioning taskserv list --infra my-infra --installed

# Show service details
provisioning taskserv show kubernetes --infra my-infra

# Check service status
provisioning taskserv status kubernetes --infra my-infra

# Check service health
provisioning taskserv health kubernetes --infra my-infra
```plaintext

#### Service Operations

```bash
# Start/stop services
provisioning taskserv start kubernetes --infra my-infra
provisioning taskserv stop kubernetes --infra my-infra

# Restart services
provisioning taskserv restart kubernetes --infra my-infra

# Update services
provisioning taskserv update kubernetes --infra my-infra

# Configure services
provisioning taskserv configure kubernetes --config cluster.yaml --infra my-infra
```plaintext

#### Service Removal

```bash
# Remove service
provisioning taskserv delete kubernetes --infra my-infra

# Remove with data cleanup
provisioning taskserv delete postgresql --cleanup-data --infra my-infra

# Remove from specific servers
provisioning taskserv delete kubernetes --servers worker-03 --infra my-infra
```plaintext

### Version Management

```bash
# Check for updates
provisioning taskserv check-updates --infra my-infra

# Check specific service updates
provisioning taskserv check-updates kubernetes --infra my-infra

# Show available versions
provisioning taskserv versions kubernetes

# Upgrade to latest version
provisioning taskserv upgrade kubernetes --infra my-infra

# Upgrade to specific version
provisioning taskserv upgrade kubernetes --version 1.29 --infra my-infra
```plaintext

## Cluster Management

### Understanding Clusters

Clusters are collections of services that work together to provide functionality:

```nickel
# Cluster configuration example
clusters: {
    web_cluster: {
        name = "web-application"
        description = "Web application cluster"

        # Services in the cluster
        services = [
            {
                name = "nginx"
                replicas = 3
                image = "nginx:1.24"
                ports = [80, 443]
            }
            {
                name = "app"
                replicas = 5
                image = "myapp:latest"
                ports = [8080]
            }
        ]

        # Load balancer configuration
        load_balancer = {
            type = "application"
            health_check = "/health"
            ssl_cert = "wildcard.example.com"
        }

        # Auto-scaling
        auto_scaling = {
            min_replicas = 2
            max_replicas = 10
            target_cpu = 70
            target_memory = 80
        }
    }
}
```plaintext

### Cluster Commands

#### Creating Clusters

```bash
# Create cluster
provisioning cluster create web-cluster --infra my-infra

# Create with specific configuration
provisioning cluster create web-cluster --config cluster.yaml --infra my-infra

# Create and deploy
provisioning cluster create web-cluster --deploy --infra my-infra
```plaintext

#### Managing Clusters

```bash
# List available clusters
provisioning cluster list

# List deployed clusters
provisioning cluster list --infra my-infra --deployed

# Show cluster details
provisioning cluster show web-cluster --infra my-infra

# Get cluster status
provisioning cluster status web-cluster --infra my-infra
```plaintext

#### Cluster Operations

```bash
# Deploy cluster
provisioning cluster deploy web-cluster --infra my-infra

# Scale cluster
provisioning cluster scale web-cluster --replicas 10 --infra my-infra

# Update cluster
provisioning cluster update web-cluster --infra my-infra

# Rolling update
provisioning cluster update web-cluster --rolling --infra my-infra
```plaintext

#### Cluster Deletion

```bash
# Delete cluster
provisioning cluster delete web-cluster --infra my-infra

# Delete with data cleanup
provisioning cluster delete web-cluster --cleanup --infra my-infra
```plaintext

## Network Management

### Network Configuration

```nickel
# Network configuration
network: {
    vpc = {
        cidr = "10.0.0.0/16"
        enable_dns = true
        enable_dhcp = true
    }

    subnets = [
        {
            name = "web"
            cidr = "10.0.1.0/24"
            zone = "us-west-2a"
            public = true
        }
        {
            name = "app"
            cidr = "10.0.2.0/24"
            zone = "us-west-2b"
            public = false
        }
        {
            name = "data"
            cidr = "10.0.3.0/24"
            zone = "us-west-2c"
            public = false
        }
    ]

    security_groups = [
        {
            name = "web"
            rules = [
                {protocol = "tcp", port = 80, source = "0.0.0.0/0"}
                {protocol = "tcp", port = 443, source = "0.0.0.0/0"}
            ]
        }
        {
            name = "app"
            rules = [
                {protocol = "tcp", port = 8080, source = "10.0.1.0/24"}
            ]
        }
    ]

    load_balancers = [
        {
            name = "web-lb"
            type = "application"
            scheme = "internet-facing"
            subnets = ["web"]
            targets = ["web-01", "web-02"]
        }
    ]
}
```plaintext

### Network Commands

```bash
# Show network configuration
provisioning network show --infra my-infra

# Create network resources
provisioning network create --infra my-infra

# Update network configuration
provisioning network update --infra my-infra

# Test network connectivity
provisioning network test --infra my-infra
```plaintext

## Storage Management

### Storage Configuration

```nickel
# Storage configuration
storage: {
    # Block storage
    volumes = [
        {
            name = "app-data"
            size = "100 GB"
            type = "gp3"
            encrypted = true
        }
    ]

    # Object storage
    buckets = [
        {
            name = "app-assets"
            region = "us-west-2"
            versioning = true
            encryption = "AES256"
        }
    ]

    # Backup configuration
    backup = {
        schedule = "0 1 * * *"  # Daily at 1 AM
        retention = {
            daily = 7
            weekly = 4
            monthly = 12
        }
    }
}
```plaintext

### Storage Commands

```bash
# Create storage resources
provisioning storage create --infra my-infra

# List storage
provisioning storage list --infra my-infra

# Backup data
provisioning storage backup --infra my-infra

# Restore from backup
provisioning storage restore --backup latest --infra my-infra
```plaintext

## Monitoring and Observability

### Monitoring Setup

```bash
# Install monitoring stack
provisioning taskserv create prometheus --infra my-infra
provisioning taskserv create grafana --infra my-infra
provisioning taskserv create alertmanager --infra my-infra

# Configure monitoring
provisioning taskserv configure prometheus --config monitoring.yaml --infra my-infra
```plaintext

### Health Checks

```bash
# Check overall infrastructure health
provisioning health check --infra my-infra

# Check specific components
provisioning health check servers --infra my-infra
provisioning health check taskservs --infra my-infra
provisioning health check clusters --infra my-infra

# Continuous monitoring
provisioning health monitor --infra my-infra --watch
```plaintext

### Metrics and Alerting

```bash
# Get infrastructure metrics
provisioning metrics get --infra my-infra

# Set up alerts
provisioning alerts create --config alerts.yaml --infra my-infra

# List active alerts
provisioning alerts list --infra my-infra
```plaintext

## Cost Management

### Cost Monitoring

```bash
# Show current costs
provisioning cost show --infra my-infra

# Cost breakdown by component
provisioning cost breakdown --infra my-infra

# Cost trends
provisioning cost trends --period 30d --infra my-infra

# Set cost alerts
provisioning cost alert --threshold 1000 --infra my-infra
```plaintext

### Cost Optimization

```bash
# Analyze cost optimization opportunities
provisioning cost optimize --infra my-infra

# Show unused resources
provisioning cost unused --infra my-infra

# Right-size recommendations
provisioning cost recommendations --infra my-infra
```plaintext

## Scaling Strategies

### Manual Scaling

```bash
# Scale servers
provisioning server scale --count 5 --infra my-infra

# Scale specific service
provisioning taskserv scale kubernetes --nodes 3 --infra my-infra

# Scale cluster
provisioning cluster scale web-cluster --replicas 10 --infra my-infra
```plaintext

### Auto-scaling Configuration

```nickel
# Auto-scaling configuration
auto_scaling: {
    servers = {
        min_count = 2
        max_count = 10

        # Scaling metrics
        cpu_threshold = 70
        memory_threshold = 80

        # Scaling behavior
        scale_up_cooldown = "5m"
        scale_down_cooldown = "10m"
    }

    clusters = {
        web_cluster = {
            min_replicas = 3
            max_replicas = 20
            metrics = [
                {type = "cpu", target = 70}
                {type = "memory", target = 80}
                {type = "requests", target = 1000}
            ]
        }
    }
}
```plaintext

## Disaster Recovery

### Backup Strategies

```bash
# Full infrastructure backup
provisioning backup create --type full --infra my-infra

# Incremental backup
provisioning backup create --type incremental --infra my-infra

# Schedule automated backups
provisioning backup schedule --daily --time "02:00" --infra my-infra
```plaintext

### Recovery Procedures

```bash
# List available backups
provisioning backup list --infra my-infra

# Restore infrastructure
provisioning restore --backup latest --infra my-infra

# Partial restore
provisioning restore --backup latest --components servers --infra my-infra

# Test restore (dry run)
provisioning restore --backup latest --test --infra my-infra
```plaintext

## Advanced Infrastructure Patterns

### Multi-Region Deployment

```nickel
# Multi-region configuration
regions: {
    primary = {
        name = "us-west-2"
        servers = ["web-01", "web-02", "db-01"]
        availability_zones = ["us-west-2a", "us-west-2b"]
    }

    secondary = {
        name = "us-east-1"
        servers = ["web-03", "web-04", "db-02"]
        availability_zones = ["us-east-1a", "us-east-1b"]
    }

    # Cross-region replication
    replication = {
        database = {
            primary = "us-west-2"
            replicas = ["us-east-1"]
            sync_mode = "async"
        }

        storage = {
            sync_schedule = "*/15 * * * *"  # Every 15 minutes
        }
    }
}
```plaintext

### Blue-Green Deployment

```bash
# Create green environment
provisioning generate infra --from production --name production-green

# Deploy to green
provisioning server create --infra production-green
provisioning taskserv create --infra production-green
provisioning cluster deploy --infra production-green

# Switch traffic to green
provisioning network switch --from production --to production-green

# Decommission blue
provisioning server delete --infra production --yes
```plaintext

### Canary Deployment

```bash
# Create canary environment
provisioning cluster create web-cluster-canary --replicas 1 --infra my-infra

# Route small percentage of traffic
provisioning network route --target web-cluster-canary --weight 10 --infra my-infra

# Monitor canary metrics
provisioning metrics monitor web-cluster-canary --infra my-infra

# Promote or rollback
provisioning cluster promote web-cluster-canary --infra my-infra
# or
provisioning cluster rollback web-cluster-canary --infra my-infra
```plaintext

## Troubleshooting Infrastructure

### Common Issues

#### Server Creation Failures

```bash
# Check provider status
provisioning provider status aws

# Validate server configuration
provisioning server validate web-01 --infra my-infra

# Check quota limits
provisioning provider quota --infra my-infra

# Debug server creation
provisioning --debug server create web-01 --infra my-infra
```plaintext

#### Service Installation Failures

```bash
# Check service prerequisites
provisioning taskserv check kubernetes --infra my-infra

# Validate service configuration
provisioning taskserv validate kubernetes --infra my-infra

# Check service logs
provisioning taskserv logs kubernetes --infra my-infra

# Debug service installation
provisioning --debug taskserv create kubernetes --infra my-infra
```plaintext

#### Network Connectivity Issues

```bash
# Test network connectivity
provisioning network test --infra my-infra

# Check security groups
provisioning network security-groups --infra my-infra

# Trace network path
provisioning network trace --from web-01 --to db-01 --infra my-infra
```plaintext

### Performance Optimization

```bash
# Analyze performance bottlenecks
provisioning performance analyze --infra my-infra

# Get performance recommendations
provisioning performance recommendations --infra my-infra

# Monitor resource utilization
provisioning performance monitor --infra my-infra --duration 1h
```plaintext

## Testing Infrastructure

The provisioning system includes a comprehensive **Test Environment Service** for automated testing of infrastructure components before deployment.

### Why Test Infrastructure

Testing infrastructure before production deployment helps:

- **Validate taskserv configurations** before installing on production servers
- **Test integration** between multiple taskservs
- **Verify cluster topologies** (Kubernetes, etcd, etc.) before deployment
- **Catch configuration errors** early in the development cycle
- **Ensure compatibility** between components

### Test Environment Types

#### 1. Single Taskserv Testing

Test individual taskservs in isolated containers:

```bash
# Quick test (create, run, cleanup automatically)
provisioning test quick kubernetes

# Single taskserv with custom resources
provisioning test env single postgres \
  --cpu 2000 \
  --memory 4096 \
  --auto-start \
  --auto-cleanup

# Test with specific infrastructure context
provisioning test env single redis --infra my-infra
```plaintext

#### 2. Server Simulation

Test complete server configurations with multiple taskservs:

```bash
# Simulate web server with multiple taskservs
provisioning test env server web-01 [containerd kubernetes cilium] \
  --auto-start

# Simulate database server
provisioning test env server db-01 [postgres redis] \
  --infra prod-stack \
  --auto-start
```plaintext

#### 3. Multi-Node Cluster Testing

Test complex cluster topologies before production deployment:

```bash
# Test 3-node Kubernetes cluster
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# Test etcd cluster
provisioning test topology load etcd_cluster | \
  test env cluster etcd --auto-start

# Test single-node Kubernetes
provisioning test topology load kubernetes_single | \
  test env cluster kubernetes --auto-start
```plaintext

### Managing Test Environments

```bash
# List all test environments
provisioning test env list

# Check environment status
provisioning test env status <env-id>

# View environment logs
provisioning test env logs <env-id>

# Cleanup environment when done
provisioning test env cleanup <env-id>
```plaintext

### Available Topology Templates

Pre-configured multi-node cluster templates:

| Template | Description | Use Case |
|----------|-------------|----------|
| `kubernetes_3node` | 3-node HA K8s cluster | Production-like K8s testing |
| `kubernetes_single` | All-in-one K8s node | Development K8s testing |
| `etcd_cluster` | 3-member etcd cluster | Distributed consensus testing |
| `containerd_test` | Standalone containerd | Container runtime testing |
| `postgres_redis` | Database stack | Database integration testing |

### Test Environment Workflow

Typical testing workflow:

```bash
# 1. Test new taskserv before deploying
provisioning test quick kubernetes

# 2. If successful, test server configuration
provisioning test env server k8s-node [containerd kubernetes cilium] \
  --auto-start

# 3. Test complete cluster topology
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# 4. Deploy to production
provisioning server create --infra production
provisioning taskserv create kubernetes --infra production
```plaintext

### CI/CD Integration

Integrate infrastructure testing into CI/CD pipelines:

```yaml
# GitLab CI example
test-infrastructure:
  stage: test
  script:
    # Start orchestrator
    - ./scripts/start-orchestrator.nu --background

    # Test critical taskservs
    - provisioning test quick kubernetes
    - provisioning test quick postgres
    - provisioning test quick redis

    # Test cluster topology
    - provisioning test topology load kubernetes_3node |
        test env cluster kubernetes --auto-start

  artifacts:
    when: on_failure
    paths:
      - test-logs/
```plaintext

### Prerequisites

Test environments require:

1. **Docker Running**: Test environments use Docker containers

   ```bash
   docker ps  # Should work without errors

1. Orchestrator Running: The orchestrator manages test containers

   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   

Advanced Testing

Custom Topology Testing

Create custom topology configurations:

# custom-topology.toml
[my_cluster]
name = "Custom Test Cluster"
cluster_type = "custom"

[[my_cluster.nodes]]
name = "node-01"
role = "primary"
taskservs = ["postgres", "redis"]
[my_cluster.nodes.resources]
cpu_millicores = 2000
memory_mb = 4096

[[my_cluster.nodes]]
name = "node-02"
role = "replica"
taskservs = ["postgres"]
[my_cluster.nodes.resources]
cpu_millicores = 1000
memory_mb = 2048
```plaintext

Load and test custom topology:

```bash
provisioning test env cluster custom-app custom-topology.toml --auto-start
```plaintext

#### Integration Testing

Test taskserv dependencies:

```bash
# Test Kubernetes dependencies in order
provisioning test quick containerd
provisioning test quick etcd
provisioning test quick kubernetes
provisioning test quick cilium

# Test complete stack
provisioning test env server k8s-stack \
  [containerd etcd kubernetes cilium] \
  --auto-start
```plaintext

### Documentation

For complete test environment documentation:

- **Test Environment Guide**: `docs/user/test-environment-guide.md`
- **Detailed Usage**: `docs/user/test-environment-usage.md`
- **Orchestrator README**: `provisioning/platform/orchestrator/README.md`

## Best Practices

### 1. Infrastructure Design

- **Principle of Least Privilege**: Grant minimal necessary access
- **Defense in Depth**: Multiple layers of security
- **High Availability**: Design for failure resilience
- **Scalability**: Plan for growth from the start

### 2. Operational Excellence

```bash
# Always validate before applying changes
provisioning validate config --infra my-infra

# Use check mode for dry runs
provisioning server create --check --infra my-infra

# Monitor continuously
provisioning health monitor --infra my-infra

# Regular backups
provisioning backup schedule --daily --infra my-infra
```plaintext

### 3. Security

```bash
# Regular security updates
provisioning taskserv update --security-only --infra my-infra

# Encrypt sensitive data
provisioning sops settings.ncl --infra my-infra

# Audit access
provisioning audit logs --infra my-infra
```plaintext

### 4. Cost Optimization

```bash
# Regular cost reviews
provisioning cost analyze --infra my-infra

# Right-size resources
provisioning cost optimize --apply --infra my-infra

# Use reserved instances for predictable workloads
provisioning server reserve --infra my-infra
```plaintext

## Next Steps

Now that you understand infrastructure management:

1. **Learn about extensions**: [Extension Development Guide](extension-development.md)
2. **Master configuration**: [Configuration Guide](configuration.md)
3. **Explore advanced examples**: [Examples and Tutorials](examples/)
4. **Set up monitoring and alerting**
5. **Implement automated scaling**
6. **Plan disaster recovery procedures**

You now have the knowledge to build and manage robust, scalable cloud infrastructure!