provisioning/docs/src/development/extensions.md

# Extension Development Guide

This document provides comprehensive guidance on creating providers, task services, and clusters for provisioning, including templates, testing frameworks, publishing, and best practices.

## Table of Contents

1. [Overview](#overview)
2. [Extension Types](#extension-types)
3. [Provider Development](#provider-development)
4. [Task Service Development](#task-service-development)
5. [Cluster Development](#cluster-development)
6. [Testing and Validation](#testing-and-validation)
7. [Publishing and Distribution](#publishing-and-distribution)
8. [Best Practices](#best-practices)
9. [Troubleshooting](#troubleshooting)

## Overview

Provisioning supports three types of extensions that enable customization and expansion of functionality:

- **Providers**: Cloud provider implementations for resource management
- **Task Services**: Infrastructure service components (databases, monitoring, etc.)
- **Clusters**: Complete deployment solutions combining multiple services

**Key Features**:

- **Template-Based Development**: Comprehensive templates for all extension types
- **Workspace Integration**: Extensions developed in isolated workspace environments
- **Configuration-Driven**: KCL schemas for type-safe configuration
- **Version Management**: GitHub integration for version tracking
- **Testing Framework**: Comprehensive testing and validation tools
- **Hot Reloading**: Development-time hot reloading support

**Location**: `workspace/extensions/`

## Extension Types

### Extension Architecture

```plaintext
Extension Ecosystem
├── Providers                    # Cloud resource management
│   ├── AWS                     # Amazon Web Services
│   ├── UpCloud                 # UpCloud platform
│   ├── Local                   # Local development
│   └── Custom                  # User-defined providers
├── Task Services               # Infrastructure components
│   ├── Kubernetes             # Container orchestration
│   ├── Database Services      # PostgreSQL, MongoDB, etc.
│   ├── Monitoring            # Prometheus, Grafana, etc.
│   ├── Networking            # Cilium, CoreDNS, etc.
│   └── Custom Services       # User-defined services
└── Clusters                   # Complete solutions
    ├── Web Stack             # Web application deployment
    ├── CI/CD Pipeline        # Continuous integration/deployment
    ├── Data Platform         # Data processing and analytics
    └── Custom Clusters       # User-defined clusters
```plaintext

### Extension Discovery

**Discovery Order**:

1. `workspace/extensions/{type}/{user}/{name}` - User-specific extensions
2. `workspace/extensions/{type}/{name}` - Workspace shared extensions
3. `workspace/extensions/{type}/template` - Templates
4. Core system paths (fallback)

**Path Resolution**:

```nushell
# Automatic extension discovery
use workspace/lib/path-resolver.nu

# Find provider extension
let provider_path = (path-resolver resolve_extension "providers" "my-aws-provider")

# List all available task services
let taskservs = (path-resolver list_extensions "taskservs" --include-core)

# Resolve cluster definition
let cluster_path = (path-resolver resolve_extension "clusters" "web-stack")
```plaintext

## Provider Development

### Provider Architecture

Providers implement cloud resource management through a standardized interface that supports multiple cloud platforms while maintaining consistent APIs.

**Core Responsibilities**:

- **Authentication**: Secure API authentication and credential management
- **Resource Management**: Server creation, deletion, and lifecycle management
- **Configuration**: Provider-specific settings and validation
- **Error Handling**: Comprehensive error handling and recovery
- **Rate Limiting**: API rate limiting and retry logic

### Creating a New Provider

**1. Initialize from Template**:

```bash
# Copy provider template
cp -r workspace/extensions/providers/template workspace/extensions/providers/my-cloud

# Navigate to new provider
cd workspace/extensions/providers/my-cloud
```plaintext

**2. Update Configuration**:

```bash
# Initialize provider metadata
nu init-provider.nu \
    --name "my-cloud" \
    --display-name "MyCloud Provider" \
    --author "$USER" \
    --description "MyCloud platform integration"
```plaintext

### Provider Structure

```plaintext
my-cloud/
├── README.md                    # Provider documentation
├── schemas/                     # Nickel configuration schemas
│   ├── settings.ncl            # Provider settings schema
│   ├── servers.ncl             # Server configuration schema
│   ├── networks.ncl            # Network configuration schema
│   └── manifest.toml           # Nickel module dependencies
├── nulib/                      # Nushell implementation
│   ├── provider.nu             # Main provider interface
│   ├── servers/                # Server management
│   │   ├── create.nu           # Server creation logic
│   │   ├── delete.nu           # Server deletion logic
│   │   ├── list.nu             # Server listing
│   │   ├── status.nu           # Server status checking
│   │   └── utils.nu            # Server utilities
│   ├── auth/                   # Authentication
│   │   ├── client.nu           # API client setup
│   │   ├── tokens.nu           # Token management
│   │   └── validation.nu       # Credential validation
│   └── utils/                  # Provider utilities
│       ├── api.nu              # API interaction helpers
│       ├── config.nu           # Configuration helpers
│       └── validation.nu       # Input validation
├── templates/                  # Jinja2 templates
│   ├── server-config.j2        # Server configuration
│   ├── cloud-init.j2           # Cloud initialization
│   └── network-config.j2       # Network configuration
├── generate/                   # Code generation
│   ├── server-configs.nu       # Generate server configurations
│   └── infrastructure.nu      # Generate infrastructure
└── tests/                      # Testing framework
    ├── unit/                   # Unit tests
    │   ├── test-auth.nu        # Authentication tests
    │   ├── test-servers.nu     # Server management tests
    │   └── test-validation.nu  # Validation tests
    ├── integration/            # Integration tests
    │   ├── test-lifecycle.nu   # Complete lifecycle tests
    │   └── test-api.nu         # API integration tests
    └── mock/                   # Mock data and services
        ├── api-responses.json  # Mock API responses
        └── test-configs.toml   # Test configurations
```plaintext

### Provider Implementation

**Main Provider Interface** (`nulib/provider.nu`):

```nushell
#!/usr/bin/env nu
# MyCloud Provider Implementation

# Provider metadata
export const PROVIDER_NAME = "my-cloud"
export const PROVIDER_VERSION = "1.0.0"
export const API_VERSION = "v1"

# Main provider initialization
export def "provider init" [
    --config-path: string = ""     # Path to provider configuration
    --validate: bool = true        # Validate configuration on init
] -> record {
    let config = if $config_path == "" {
        load_provider_config
    } else {
        open $config_path | from toml
    }

    if $validate {
        validate_provider_config $config
    }

    # Initialize API client
    let client = (setup_api_client $config)

    # Return provider instance
    {
        name: $PROVIDER_NAME,
        version: $PROVIDER_VERSION,
        config: $config,
        client: $client,
        initialized: true
    }
}

# Server management interface
export def "provider create-server" [
    name: string                   # Server name
    plan: string                   # Server plan/size
    --zone: string = "auto"        # Deployment zone
    --template: string = "ubuntu22" # OS template
    --dry-run: bool = false        # Show what would be created
] -> record {
    let provider = (provider init)

    # Validate inputs
    if ($name | str length) == 0 {
        error make {msg: "Server name cannot be empty"}
    }

    if not (is_valid_plan $plan) {
        error make {msg: $"Invalid server plan: ($plan)"}
    }

    # Build server configuration
    let server_config = {
        name: $name,
        plan: $plan,
        zone: (resolve_zone $zone),
        template: $template,
        provider: $PROVIDER_NAME
    }

    if $dry_run {
        return {action: "create", config: $server_config, status: "dry-run"}
    }

    # Create server via API
    let result = try {
        create_server_api $server_config $provider.client
    } catch { |e|
        error make {
            msg: $"Server creation failed: ($e.msg)",
            help: "Check provider credentials and quota limits"
        }
    }

    {
        server: $name,
        status: "created",
        id: $result.id,
        ip_address: $result.ip_address,
        created_at: (date now)
    }
}

export def "provider delete-server" [
    name: string                   # Server name or ID
    --force: bool = false          # Force deletion without confirmation
] -> record {
    let provider = (provider init)

    # Find server
    let server = try {
        find_server $name $provider.client
    } catch {
        error make {msg: $"Server not found: ($name)"}
    }

    if not $force {
        let confirm = (input $"Delete server '($name)' (y/N)? ")
        if $confirm != "y" and $confirm != "yes" {
            return {action: "delete", server: $name, status: "cancelled"}
        }
    }

    # Delete server
    let result = try {
        delete_server_api $server.id $provider.client
    } catch { |e|
        error make {msg: $"Server deletion failed: ($e.msg)"}
    }

    {
        server: $name,
        status: "deleted",
        deleted_at: (date now)
    }
}

export def "provider list-servers" [
    --zone: string = ""            # Filter by zone
    --status: string = ""          # Filter by status
    --format: string = "table"     # Output format: table, json, yaml
] -> list<record> {
    let provider = (provider init)

    let servers = try {
        list_servers_api $provider.client
    } catch { |e|
        error make {msg: $"Failed to list servers: ($e.msg)"}
    }

    # Apply filters
    let filtered = $servers
        | if $zone != "" { filter {|s| $s.zone == $zone} } else { $in }
        | if $status != "" { filter {|s| $s.status == $status} } else { $in }

    match $format {
        "json" => ($filtered | to json),
        "yaml" => ($filtered | to yaml),
        _ => $filtered
    }
}

# Provider testing interface
export def "provider test" [
    --test-type: string = "basic"  # Test type: basic, full, integration
] -> record {
    match $test_type {
        "basic" => test_basic_functionality,
        "full" => test_full_functionality,
        "integration" => test_integration,
        _ => (error make {msg: $"Unknown test type: ($test_type)"})
    }
}
```plaintext

**Authentication Module** (`nulib/auth/client.nu`):

```nushell
# API client setup and authentication

export def setup_api_client [config: record] -> record {
    # Validate credentials
    if not ("api_key" in $config) {
        error make {msg: "API key not found in configuration"}
    }

    if not ("api_secret" in $config) {
        error make {msg: "API secret not found in configuration"}
    }

    # Setup HTTP client with authentication
    let client = {
        base_url: ($config.api_url? | default "https://api.my-cloud.com"),
        api_key: $config.api_key,
        api_secret: $config.api_secret,
        timeout: ($config.timeout? | default 30),
        retries: ($config.retries? | default 3)
    }

    # Test authentication
    try {
        test_auth_api $client
    } catch { |e|
        error make {
            msg: $"Authentication failed: ($e.msg)",
            help: "Check your API credentials and network connectivity"
        }
    }

    $client
}

def test_auth_api [client: record] -> bool {
    let response = http get $"($client.base_url)/auth/test" --headers {
        "Authorization": $"Bearer ($client.api_key)",
        "Content-Type": "application/json"
    }

    $response.status == "success"
}
```plaintext

**Nickel Configuration Schema** (`schemas/settings.ncl`):

```nickel
# MyCloud Provider Configuration Schema

let MyCloudConfig = {
    # MyCloud provider configuration
    api_url | string | default = "https://api.my-cloud.com",
    api_key | string,
    api_secret | string,
    timeout | number | default = 30,
    retries | number | default = 3,

    # Rate limiting
    rate_limit | {
        requests_per_minute | number | default = 60,
        burst_size | number | default = 10,
    } | default = {},

    # Default settings
    defaults | {
        zone | string | default = "us-east-1",
        template | string | default = "ubuntu-22.04",
        network | string | default = "default",
    } | default = {},
} in
MyCloudConfig

let MyCloudServerConfig = {
    # MyCloud server configuration
    name | string,
    plan | string,
    zone | string | optional,
    template | string | default = "ubuntu-22.04",
    storage | number | default = 25,
    tags | { } | default = {},

    # Network configuration
    network | {
        vpc_id | string | optional,
        subnet_id | string | optional,
        public_ip | bool | default = true,
        firewall_rules | array | default = [],
    } | optional,
} in
MyCloudServerConfig

let FirewallRule = {
    # Firewall rule configuration
    port | (number | string),
    protocol | string | default = "tcp",
    source | string | default = "0.0.0.0/0",
    description | string | optional,
} in
FirewallRule
```plaintext

### Provider Testing

**Unit Testing** (`tests/unit/test-servers.nu`):

```nushell
# Unit tests for server management

use ../../../nulib/provider.nu

def test_server_creation [] {
    # Test valid server creation
    let result = (provider create-server "test-server" "small" --dry-run)

    assert ($result.action == "create")
    assert ($result.config.name == "test-server")
    assert ($result.config.plan == "small")
    assert ($result.status == "dry-run")

    print "✅ Server creation test passed"
}

def test_invalid_server_name [] {
    # Test invalid server name
    try {
        provider create-server "" "small" --dry-run
        assert false "Should have failed with empty name"
    } catch { |e|
        assert ($e.msg | str contains "Server name cannot be empty")
    }

    print "✅ Invalid server name test passed"
}

def test_invalid_plan [] {
    # Test invalid server plan
    try {
        provider create-server "test" "invalid-plan" --dry-run
        assert false "Should have failed with invalid plan"
    } catch { |e|
        assert ($e.msg | str contains "Invalid server plan")
    }

    print "✅ Invalid plan test passed"
}

def main [] {
    print "Running server management unit tests..."
    test_server_creation
    test_invalid_server_name
    test_invalid_plan
    print "✅ All server management tests passed"
}
```plaintext

**Integration Testing** (`tests/integration/test-lifecycle.nu`):

```nushell
# Integration tests for complete server lifecycle

use ../../../nulib/provider.nu

def test_complete_lifecycle [] {
    let test_server = $"test-server-(date now | format date '%Y%m%d%H%M%S')"

    try {
        # Test server creation (dry run)
        let create_result = (provider create-server $test_server "small" --dry-run)
        assert ($create_result.status == "dry-run")

        # Test server listing
        let servers = (provider list-servers --format json)
        assert ($servers | length) >= 0

        # Test provider info
        let provider_info = (provider init)
        assert ($provider_info.name == "my-cloud")
        assert $provider_info.initialized

        print $"✅ Complete lifecycle test passed for ($test_server)"
    } catch { |e|
        print $"❌ Integration test failed: ($e.msg)"
        exit 1
    }
}

def main [] {
    print "Running provider integration tests..."
    test_complete_lifecycle
    print "✅ All integration tests passed"
}
```plaintext

## Task Service Development

### Task Service Architecture

Task services are infrastructure components that can be deployed and managed across different environments. They provide standardized interfaces for installation, configuration, and lifecycle management.

**Core Responsibilities**:

- **Installation**: Service deployment and setup
- **Configuration**: Dynamic configuration management
- **Health Checking**: Service status monitoring
- **Version Management**: Automatic version updates from GitHub
- **Integration**: Integration with other services and clusters

### Creating a New Task Service

**1. Initialize from Template**:

```bash
# Copy task service template
cp -r workspace/extensions/taskservs/template workspace/extensions/taskservs/my-service

# Navigate to new service
cd workspace/extensions/taskservs/my-service
```plaintext

**2. Initialize Service**:

```bash
# Initialize service metadata
nu init-service.nu \
    --name "my-service" \
    --display-name "My Custom Service" \
    --type "database" \
    --github-repo "myorg/my-service"
```plaintext

### Task Service Structure

```plaintext
my-service/
├── README.md                    # Service documentation
├── schemas/                     # Nickel schemas
│   ├── version.ncl             # Version and GitHub integration
│   ├── config.ncl              # Service configuration schema
│   └── manifest.toml           # Module dependencies
├── nushell/                    # Nushell implementation
│   ├── taskserv.nu             # Main service interface
│   ├── install.nu              # Installation logic
│   ├── uninstall.nu            # Removal logic
│   ├── config.nu               # Configuration management
│   ├── status.nu               # Status and health checking
│   ├── versions.nu             # Version management
│   └── utils.nu                # Service utilities
├── templates/                  # Jinja2 templates
│   ├── deployment.yaml.j2      # Kubernetes deployment
│   ├── service.yaml.j2         # Kubernetes service
│   ├── configmap.yaml.j2       # Configuration
│   ├── install.sh.j2           # Installation script
│   └── systemd.service.j2      # Systemd service
├── manifests/                  # Static manifests
│   ├── rbac.yaml               # RBAC definitions
│   ├── pvc.yaml                # Persistent volume claims
│   └── ingress.yaml            # Ingress configuration
├── generate/                   # Code generation
│   ├── manifests.nu            # Generate Kubernetes manifests
│   ├── configs.nu              # Generate configurations
│   └── docs.nu                 # Generate documentation
└── tests/                      # Testing framework
    ├── unit/                   # Unit tests
    ├── integration/            # Integration tests
    └── fixtures/               # Test fixtures and data
```plaintext

### Task Service Implementation

**Main Service Interface** (`nushell/taskserv.nu`):

```nushell
#!/usr/bin/env nu
# My Custom Service Task Service Implementation

export const SERVICE_NAME = "my-service"
export const SERVICE_TYPE = "database"
export const SERVICE_VERSION = "1.0.0"

# Service installation
export def "taskserv install" [
    target: string                 # Target server or cluster
    --config: string = ""          # Custom configuration file
    --dry-run: bool = false        # Show what would be installed
    --wait: bool = true            # Wait for installation to complete
] -> record {
    # Load service configuration
    let service_config = if $config != "" {
        open $config | from toml
    } else {
        load_default_config
    }

    # Validate target environment
    let target_info = validate_target $target
    if not $target_info.valid {
        error make {msg: $"Invalid target: ($target_info.reason)"}
    }

    if $dry_run {
        let install_plan = generate_install_plan $target $service_config
        return {
            action: "install",
            service: $SERVICE_NAME,
            target: $target,
            plan: $install_plan,
            status: "dry-run"
        }
    }

    # Perform installation
    print $"Installing ($SERVICE_NAME) on ($target)..."

    let install_result = try {
        install_service $target $service_config $wait
    } catch { |e|
        error make {
            msg: $"Installation failed: ($e.msg)",
            help: "Check target connectivity and permissions"
        }
    }

    {
        service: $SERVICE_NAME,
        target: $target,
        status: "installed",
        version: $install_result.version,
        endpoint: $install_result.endpoint?,
        installed_at: (date now)
    }
}

# Service removal
export def "taskserv uninstall" [
    target: string                 # Target server or cluster
    --force: bool = false          # Force removal without confirmation
    --cleanup-data: bool = false   # Remove persistent data
] -> record {
    let target_info = validate_target $target
    if not $target_info.valid {
        error make {msg: $"Invalid target: ($target_info.reason)"}
    }

    # Check if service is installed
    let status = get_service_status $target
    if $status.status != "installed" {
        error make {msg: $"Service ($SERVICE_NAME) is not installed on ($target)"}
    }

    if not $force {
        let confirm = (input $"Remove ($SERVICE_NAME) from ($target)? (y/N) ")
        if $confirm != "y" and $confirm != "yes" {
            return {action: "uninstall", service: $SERVICE_NAME, status: "cancelled"}
        }
    }

    print $"Removing ($SERVICE_NAME) from ($target)..."

    let removal_result = try {
        uninstall_service $target $cleanup_data
    } catch { |e|
        error make {msg: $"Removal failed: ($e.msg)"}
    }

    {
        service: $SERVICE_NAME,
        target: $target,
        status: "uninstalled",
        data_removed: $cleanup_data,
        uninstalled_at: (date now)
    }
}

# Service status checking
export def "taskserv status" [
    target: string                 # Target server or cluster
    --detailed: bool = false       # Show detailed status information
] -> record {
    let target_info = validate_target $target
    if not $target_info.valid {
        error make {msg: $"Invalid target: ($target_info.reason)"}
    }

    let status = get_service_status $target

    if $detailed {
        let health = check_service_health $target
        let metrics = get_service_metrics $target

        $status | merge {
            health: $health,
            metrics: $metrics,
            checked_at: (date now)
        }
    } else {
        $status
    }
}

# Version management
export def "taskserv check-updates" [
    --target: string = ""          # Check updates for specific target
] -> record {
    let current_version = get_current_version
    let latest_version = get_latest_version_from_github

    let update_available = $latest_version != $current_version

    {
        service: $SERVICE_NAME,
        current_version: $current_version,
        latest_version: $latest_version,
        update_available: $update_available,
        target: $target,
        checked_at: (date now)
    }
}

export def "taskserv update" [
    target: string                 # Target to update
    --version: string = "latest"   # Specific version to update to
    --dry-run: bool = false        # Show what would be updated
] -> record {
    let current_status = (taskserv status $target)
    if $current_status.status != "installed" {
        error make {msg: $"Service not installed on ($target)"}
    }

    let target_version = if $version == "latest" {
        get_latest_version_from_github
    } else {
        $version
    }

    if $dry_run {
        return {
            action: "update",
            service: $SERVICE_NAME,
            target: $target,
            from_version: $current_status.version,
            to_version: $target_version,
            status: "dry-run"
        }
    }

    print $"Updating ($SERVICE_NAME) on ($target) to version ($target_version)..."

    let update_result = try {
        update_service $target $target_version
    } catch { |e|
        error make {msg: $"Update failed: ($e.msg)"}
    }

    {
        service: $SERVICE_NAME,
        target: $target,
        status: "updated",
        from_version: $current_status.version,
        to_version: $target_version,
        updated_at: (date now)
    }
}

# Service testing
export def "taskserv test" [
    target: string = "local"       # Target for testing
    --test-type: string = "basic"  # Test type: basic, integration, full
] -> record {
    match $test_type {
        "basic" => test_basic_functionality $target,
        "integration" => test_integration $target,
        "full" => test_full_functionality $target,
        _ => (error make {msg: $"Unknown test type: ($test_type)"})
    }
}
```plaintext

**Version Configuration** (`schemas/version.ncl`):

```nickel
# Version management with GitHub integration

let version_config = {
    service_name = "my-service",

    # GitHub repository for version checking
    github = {
        owner = "myorg",
        repo = "my-service",

        # Release configuration
        release = {
            tag_prefix = "v",
            prerelease = false,
            draft = false,
        },

        # Asset patterns for different platforms
        assets = {
            linux_amd64 = "my-service-{version}-linux-amd64.tar.gz",
            darwin_amd64 = "my-service-{version}-darwin-amd64.tar.gz",
            windows_amd64 = "my-service-{version}-windows-amd64.zip",
        },
    },

    # Version constraints and compatibility
    compatibility = {
        min_kubernetes_version = "1.20.0",
        max_kubernetes_version = "1.28.*",

        # Dependencies
        requires = {
            "cert-manager" = ">=1.8.0",
            "ingress-nginx" = ">=1.0.0",
        },

        # Conflicts
        conflicts = {
            "old-my-service" = "*",
        },
    },

    # Installation configuration
    installation = {
        default_namespace = "my-service",
        create_namespace = true,

        # Resource requirements
        resources = {
            requests = {
                cpu = "100m",
                memory = "128Mi",
            },
            limits = {
                cpu = "500m",
                memory = "512Mi",
            },
        },

        # Persistence
        persistence = {
            enabled = true,
            storage_class = "default",
            size = "10Gi",
        },
    },

    # Health check configuration
    health_check = {
        initial_delay_seconds = 30,
        period_seconds = 10,
        timeout_seconds = 5,
        failure_threshold = 3,

        # Health endpoints
        endpoints = {
            liveness = "/health/live",
            readiness = "/health/ready",
        },
    },
} in
version_config
```plaintext

## Cluster Development

### Cluster Architecture

Clusters represent complete deployment solutions that combine multiple task services, providers, and configurations to create functional environments.

**Core Responsibilities**:

- **Service Orchestration**: Coordinate multiple task service deployments
- **Dependency Management**: Handle service dependencies and startup order
- **Configuration Management**: Manage cross-service configuration
- **Health Monitoring**: Monitor overall cluster health
- **Scaling**: Handle cluster scaling operations

### Creating a New Cluster

**1. Initialize from Template**:

```bash
# Copy cluster template
cp -r workspace/extensions/clusters/template workspace/extensions/clusters/my-stack

# Navigate to new cluster
cd workspace/extensions/clusters/my-stack
```plaintext

**2. Initialize Cluster**:

```bash
# Initialize cluster metadata
nu init-cluster.nu \
    --name "my-stack" \
    --display-name "My Application Stack" \
    --type "web-application"
```plaintext

### Cluster Implementation

**Main Cluster Interface** (`nushell/cluster.nu`):

```nushell
#!/usr/bin/env nu
# My Application Stack Cluster Implementation

export const CLUSTER_NAME = "my-stack"
export const CLUSTER_TYPE = "web-application"
export const CLUSTER_VERSION = "1.0.0"

# Cluster creation
export def "cluster create" [
    target: string                 # Target infrastructure
    --config: string = ""          # Custom configuration file
    --dry-run: bool = false        # Show what would be created
    --wait: bool = true            # Wait for cluster to be ready
] -> record {
    let cluster_config = if $config != "" {
        open $config | from toml
    } else {
        load_default_cluster_config
    }

    if $dry_run {
        let deployment_plan = generate_deployment_plan $target $cluster_config
        return {
            action: "create",
            cluster: $CLUSTER_NAME,
            target: $target,
            plan: $deployment_plan,
            status: "dry-run"
        }
    }

    print $"Creating cluster ($CLUSTER_NAME) on ($target)..."

    # Deploy services in dependency order
    let services = get_service_deployment_order $cluster_config.services
    let deployment_results = []

    for service in $services {
        print $"Deploying service: ($service.name)"

        let result = try {
            deploy_service $service $target $wait
        } catch { |e|
            # Rollback on failure
            rollback_cluster $target $deployment_results
            error make {msg: $"Service deployment failed: ($e.msg)"}
        }

        $deployment_results = ($deployment_results | append $result)
    }

    # Configure inter-service communication
    configure_service_mesh $target $deployment_results

    {
        cluster: $CLUSTER_NAME,
        target: $target,
        status: "created",
        services: $deployment_results,
        created_at: (date now)
    }
}

# Cluster deletion
export def "cluster delete" [
    target: string                 # Target infrastructure
    --force: bool = false          # Force deletion without confirmation
    --cleanup-data: bool = false   # Remove persistent data
] -> record {
    let cluster_status = get_cluster_status $target
    if $cluster_status.status != "running" {
        error make {msg: $"Cluster ($CLUSTER_NAME) is not running on ($target)"}
    }

    if not $force {
        let confirm = (input $"Delete cluster ($CLUSTER_NAME) from ($target)? (y/N) ")
        if $confirm != "y" and $confirm != "yes" {
            return {action: "delete", cluster: $CLUSTER_NAME, status: "cancelled"}
        }
    }

    print $"Deleting cluster ($CLUSTER_NAME) from ($target)..."

    # Delete services in reverse dependency order
    let services = get_service_deletion_order $cluster_status.services
    let deletion_results = []

    for service in $services {
        print $"Removing service: ($service.name)"

        let result = try {
            remove_service $service $target $cleanup_data
        } catch { |e|
            print $"Warning: Failed to remove service ($service.name): ($e.msg)"
        }

        $deletion_results = ($deletion_results | append $result)
    }

    {
        cluster: $CLUSTER_NAME,
        target: $target,
        status: "deleted",
        services_removed: $deletion_results,
        data_removed: $cleanup_data,
        deleted_at: (date now)
    }
}
```plaintext

## Testing and Validation

### Testing Framework

**Test Types**:

- **Unit Tests**: Individual function and module testing
- **Integration Tests**: Cross-component interaction testing
- **End-to-End Tests**: Complete workflow testing
- **Performance Tests**: Load and performance validation
- **Security Tests**: Security and vulnerability testing

### Extension Testing Commands

**Workspace Testing Tools**:

```bash
# Validate extension syntax and structure
nu workspace.nu tools validate-extension providers/my-cloud

# Run extension unit tests
nu workspace.nu tools test-extension taskservs/my-service --test-type unit

# Integration testing with real infrastructure
nu workspace.nu tools test-extension clusters/my-stack --test-type integration --target test-env

# Performance testing
nu workspace.nu tools test-extension providers/my-cloud --test-type performance --duration 5m
```plaintext

### Automated Testing

**Test Runner** (`tests/run-tests.nu`):

```nushell
#!/usr/bin/env nu
# Automated test runner for extensions

def main [
    extension_type: string         # Extension type: providers, taskservs, clusters
    extension_name: string         # Extension name
    --test-types: string = "all"   # Test types to run: unit, integration, e2e, all
    --target: string = "local"     # Test target environment
    --verbose: bool = false        # Verbose test output
    --parallel: bool = true        # Run tests in parallel
] -> record {
    let extension_path = $"workspace/extensions/($extension_type)/($extension_name)"

    if not ($extension_path | path exists) {
        error make {msg: $"Extension not found: ($extension_path)"}
    }

    let test_types = if $test_types == "all" {
        ["unit", "integration", "e2e"]
    } else {
        $test_types | split row ","
    }

    print $"Running tests for ($extension_type)/($extension_name)..."

    let test_results = []

    for test_type in $test_types {
        print $"Running ($test_type) tests..."

        let result = try {
            run_test_suite $extension_path $test_type $target $verbose
        } catch { |e|
            {
                test_type: $test_type,
                status: "failed",
                error: $e.msg,
                duration: 0
            }
        }

        $test_results = ($test_results | append $result)
    }

    let total_tests = ($test_results | length)
    let passed_tests = ($test_results | where status == "passed" | length)
    let failed_tests = ($test_results | where status == "failed" | length)

    {
        extension: $"($extension_type)/($extension_name)",
        test_results: $test_results,
        summary: {
            total: $total_tests,
            passed: $passed_tests,
            failed: $failed_tests,
            success_rate: ($passed_tests / $total_tests * 100)
        },
        completed_at: (date now)
    }
}
```plaintext

## Publishing and Distribution

### Extension Publishing

**Publishing Process**:

1. **Validation**: Comprehensive testing and validation
2. **Documentation**: Complete documentation and examples
3. **Packaging**: Create distribution packages
4. **Registry**: Publish to extension registry
5. **Versioning**: Semantic version tagging

### Publishing Commands

```bash
# Validate extension for publishing
nu workspace.nu tools validate-for-publish providers/my-cloud

# Create distribution package
nu workspace.nu tools package-extension providers/my-cloud --version 1.0.0

# Publish to registry
nu workspace.nu tools publish-extension providers/my-cloud --registry official

# Tag version
nu workspace.nu tools tag-extension providers/my-cloud --version 1.0.0 --push
```plaintext

### Extension Registry

**Registry Structure**:

```plaintext
Extension Registry
├── providers/
│   ├── aws/              # Official AWS provider
│   ├── upcloud/          # Official UpCloud provider
│   └── community/        # Community providers
├── taskservs/
│   ├── kubernetes/       # Official Kubernetes service
│   ├── databases/        # Database services
│   └── monitoring/       # Monitoring services
└── clusters/
    ├── web-stacks/       # Web application stacks
    ├── data-platforms/   # Data processing platforms
    └── ci-cd/            # CI/CD pipelines
```plaintext

## Best Practices

### Code Quality

**Function Design**:

```nushell
# Good: Single responsibility, clear parameters, comprehensive error handling
export def "provider create-server" [
    name: string                   # Server name (must be unique in region)
    plan: string                   # Server plan (see list-plans for options)
    --zone: string = "auto"        # Deployment zone (auto-selects optimal zone)
    --dry-run: bool = false        # Preview changes without creating resources
] -> record {                      # Returns creation result with server details
    # Validate inputs first
    if ($name | str length) == 0 {
        error make {
            msg: "Server name cannot be empty"
            help: "Provide a unique name for the server"
        }
    }

    # Implementation with comprehensive error handling
    # ...
}

# Bad: Unclear parameters, no error handling
def create [n, p] {
    # Missing validation and error handling
    api_call $n $p
}
```plaintext

**Configuration Management**:

```nushell
# Good: Configuration-driven with validation
def get_api_endpoint [provider: string] -> string {
    let config = get-config-value $"providers.($provider).api_url"

    if ($config | is-empty) {
        error make {
            msg: $"API URL not configured for provider ($provider)",
            help: $"Add 'api_url' to providers.($provider) configuration"
        }
    }

    $config
}

# Bad: Hardcoded values
def get_api_endpoint [] {
    "https://api.provider.com"  # Never hardcode!
}
```plaintext

### Error Handling

**Comprehensive Error Context**:

```nushell
def create_server_with_context [name: string, config: record] -> record {
    try {
        # Validate configuration
        validate_server_config $config
    } catch { |e|
        error make {
            msg: $"Invalid server configuration: ($e.msg)",
            label: {text: "configuration error", span: $e.span?},
            help: "Check configuration syntax and required fields"
        }
    }

    try {
        # Create server via API
        let result = api_create_server $name $config
        return $result
    } catch { |e|
        match $e.msg {
            $msg if ($msg | str contains "quota") => {
                error make {
                    msg: $"Server creation failed: quota limit exceeded",
                    help: "Contact support to increase quota or delete unused servers"
                }
            },
            $msg if ($msg | str contains "auth") => {
                error make {
                    msg: "Server creation failed: authentication error",
                    help: "Check API credentials and permissions"
                }
            },
            _ => {
                error make {
                    msg: $"Server creation failed: ($e.msg)",
                    help: "Check network connectivity and try again"
                }
            }
        }
    }
}
```plaintext

### Testing Practices

**Test Organization**:

```nushell
# Organize tests by functionality
# tests/unit/server-creation-test.nu

def test_valid_server_creation [] {
    # Test valid cases with various inputs
    let valid_configs = [
        {name: "test-1", plan: "small"},
        {name: "test-2", plan: "medium"},
        {name: "test-3", plan: "large"}
    ]

    for config in $valid_configs {
        let result = create_server $config.name $config.plan --dry-run
        assert ($result.status == "dry-run")
        assert ($result.config.name == $config.name)
    }
}

def test_invalid_inputs [] {
    # Test error conditions
    let invalid_cases = [
        {name: "", plan: "small", error: "empty name"},
        {name: "test", plan: "invalid", error: "invalid plan"},
        {name: "test with spaces", plan: "small", error: "invalid characters"}
    ]

    for case in $invalid_cases {
        try {
            create_server $case.name $case.plan --dry-run
            assert false $"Should have failed: ($case.error)"
        } catch { |e|
            # Verify specific error message
            assert ($e.msg | str contains $case.error)
        }
    }
}
```plaintext

### Documentation Standards

**Function Documentation**:

```nushell
# Comprehensive function documentation
def "provider create-server" [
    name: string                   # Server name - must be unique within the provider
    plan: string                   # Server size plan (run 'provider list-plans' for options)
    --zone: string = "auto"        # Target zone - 'auto' selects optimal zone based on load
    --template: string = "ubuntu22" # OS template - see 'provider list-templates' for options
    --storage: int = 25             # Storage size in GB (minimum 10, maximum 2048)
    --dry-run: bool = false        # Preview mode - shows what would be created without creating
] -> record {                      # Returns server creation details including ID and IP
    """
    Creates a new server instance with the specified configuration.

    This function provisions a new server using the provider's API, configures
    basic security settings, and returns the server details upon successful creation.

    Examples:
      # Create a small server with default settings
      provider create-server "web-01" "small"

      # Create with specific zone and storage
      provider create-server "db-01" "large" --zone "us-west-2" --storage 100

      # Preview what would be created
      provider create-server "test" "medium" --dry-run

    Error conditions:
      - Invalid server name (empty, invalid characters)
      - Invalid plan (not in supported plans list)
      - Insufficient quota or permissions
      - Network connectivity issues

    Returns:
      Record with keys: server, status, id, ip_address, created_at
    """

    # Implementation...
}
```plaintext

## Troubleshooting

### Common Development Issues

#### Extension Not Found

**Error**: `Extension 'my-provider' not found`

```bash
# Solution: Check extension location and structure
ls -la workspace/extensions/providers/my-provider
nu workspace/lib/path-resolver.nu resolve_extension "providers" "my-provider"

# Validate extension structure
nu workspace.nu tools validate-extension providers/my-provider
```plaintext

#### Configuration Errors

**Error**: `Invalid Nickel configuration`

```bash
# Solution: Validate Nickel syntax
nickel check workspace/extensions/providers/my-provider/schemas/

# Format Nickel files
nickel fmt workspace/extensions/providers/my-provider/schemas/

# Test with example data
nickel eval workspace/extensions/providers/my-provider/schemas/settings.ncl
```plaintext

#### API Integration Issues

**Error**: `Authentication failed`

```bash
# Solution: Test credentials and connectivity
curl -H "Authorization: Bearer $API_KEY" https://api.provider.com/auth/test

# Debug API calls
export PROVISIONING_DEBUG=true
export PROVISIONING_LOG_LEVEL=debug
nu workspace/extensions/providers/my-provider/nulib/provider.nu test --test-type basic
```plaintext

### Debug Mode

**Enable Extension Debugging**:

```bash
# Set debug environment
export PROVISIONING_DEBUG=true
export PROVISIONING_LOG_LEVEL=debug
export PROVISIONING_WORKSPACE_USER=$USER

# Run extension with debug
nu workspace/extensions/providers/my-provider/nulib/provider.nu create-server test-server small --dry-run
```plaintext

### Performance Optimization

**Extension Performance**:

```bash
# Profile extension performance
time nu workspace/extensions/providers/my-provider/nulib/provider.nu list-servers

# Monitor resource usage
nu workspace/tools/runtime-manager.nu monitor --duration 1m --interval 5s

# Optimize API calls (use caching)
export PROVISIONING_CACHE_ENABLED=true
export PROVISIONING_CACHE_TTL=300  # 5 minutes
```plaintext

This extension development guide provides a comprehensive framework for creating high-quality, maintainable extensions that integrate seamlessly with provisioning's architecture and workflows.