# Nickel Package and Module Loader System\n\nThis document describes the package-based architecture implemented for the provisioning system, replacing hardcoded extension paths with a\nflexible module discovery and loading system using Nickel for type-safe configuration.\n\n## Architecture Overview\n\nThe system consists of two main components:\n\n1. **Core Nickel Package**: Distributable core provisioning schemas with type safety\n2. **Module Loader System**: Dynamic discovery and loading of extensions\n\n### Benefits\n\n- **Type-Safe Configuration**: Nickel ensures configuration validity at evaluation time\n- **Clean Separation**: Core package is self-contained and distributable\n- **Plug-and-Play Extensions**: Taskservs, providers, and clusters can be loaded dynamically\n- **Version Management**: Core package and extensions can be versioned independently\n- **Developer Friendly**: Easy workspace setup and module management with lazy evaluation\n\n## Components\n\n### 1. Core Nickel Package (`/provisioning/schemas/`)\n\nContains fundamental schemas for provisioning:\n\n- `main.ncl` - Primary provisioning configuration\n- `server.ncl` - Server definitions and schemas\n- `defaults.ncl` - Default configurations\n- `lib.ncl` - Common library schemas\n- `dependencies.ncl` - Dependency management schemas\n\n**Key Features:**\n\n- No hardcoded extension paths\n- Self-contained and distributable\n- Type-safe package-based imports\n- Lazy evaluation of expensive computations\n\n### 2. Module Discovery System\n\n#### Discovery Commands\n\n```\n# Discover available modules\nmodule-loader discover taskservs              # List all taskservs\nmodule-loader discover providers --format yaml # List providers as YAML\nmodule-loader discover clusters redis          # Search for redis clusters\n```\n\n#### Supported Module Types\n\n- **Taskservs**: Infrastructure services (kubernetes, redis, postgres, etc.)\n- **Providers**: Cloud providers (upcloud, aws, local)\n- **Clusters**: Complete configurations (buildkit, web, oci-reg)\n\n### 3. Module Loading System\n\n#### Loading Commands\n\n```\n# Load modules into workspace\nmodule-loader load taskservs . [kubernetes, cilium, containerd]\nmodule-loader load providers . [upcloud]\nmodule-loader load clusters . [buildkit]\n\n# Initialize workspace with modules\nmodule-loader init workspace/infra/production \\n    --taskservs [kubernetes, cilium] \\n    --providers [upcloud]\n```\n\n#### Generated Files\n\n- `taskservs.ncl` - Auto-generated taskserv imports\n- `providers.ncl` - Auto-generated provider imports\n- `clusters.ncl` - Auto-generated cluster imports\n- `.manifest/*.yaml` - Module loading manifests\n\n## Workspace Structure\n\n### New Workspace Layout\n\n```\nworkspace/infra/my-project/\n├── kcl.mod                    # Package dependencies\n├── servers.ncl                  # Main server configuration\n├── taskservs.ncl               # Auto-generated taskserv imports\n├── providers.ncl               # Auto-generated provider imports\n├── clusters.ncl                # Auto-generated cluster imports\n├── .taskservs/               # Loaded taskserv modules\n│   ├── kubernetes/\n│   ├── cilium/\n│   └── containerd/\n├── .providers/               # Loaded provider modules\n│   └── upcloud/\n├── .clusters/                # Loaded cluster modules\n│   └── buildkit/\n├── .manifest/                # Module manifests\n│   ├── taskservs.yaml\n│   ├── providers.yaml\n│   └── clusters.yaml\n├── data/                     # Runtime data\n├── tmp/                      # Temporary files\n├── resources/                # Resource definitions\n└── clusters/                 # Cluster configurations\n```\n\n### Import Patterns\n\n#### Before (Old System)\n\n```\n# Hardcoded relative paths\nimport ../../../kcl/server as server\nimport ../../../extensions/taskservs/kubernetes/kcl/kubernetes as k8s\n```\n\n#### After (New System)\n\n```\n# Package-based imports\nimport provisioning.server as server\n\n# Auto-generated module imports (after loading)\nimport .taskservs.nclubernetes.kubernetes as k8s\n```\n\n## Package Distribution\n\n### Building Core Package\n\n```\n# Build distributable package\n./provisioning/tools/kcl-packager.nu build --version 1.0.0\n\n# Install locally\n./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz\n\n# Create release\n./provisioning/tools/kcl-packager.nu build --format tar.gz --include-docs\n```\n\n### Package Installation Methods\n\n#### Method 1: Local Installation (Recommended for development)\n\n```\n[dependencies]\nprovisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }\n```\n\n#### Method 2: Git Repository (For distributed teams)\n\n```\n[dependencies]\nprovisioning = { git = "https://github.com/your-org/provisioning-kcl", version = "v0.0.1" }\n```\n\n#### Method 3: KCL Registry (When available)\n\n```\n[dependencies]\nprovisioning = { version = "0.0.1" }\n```\n\n## Developer Workflows\n\n### 1. New Project Setup\n\n```\n# Create workspace from template\ncp -r provisioning/templates/workspaces/kubernetes ./my-k8s-cluster\ncd my-k8s-cluster\n\n# Initialize with modules\nworkspace-init.nu . init\n\n# Load required modules\nmodule-loader load taskservs . [kubernetes, cilium, containerd]\nmodule-loader load providers . [upcloud]\n\n# Validate and deploy\nkcl run servers.ncl\nprovisioning server create --infra . --check\n```\n\n### 2. Extension Development\n\n```\n# Create new taskserv\nmkdir -p extensions/taskservs/my-service/kcl\ncd extensions/taskservs/my-service/kcl\n\n# Initialize KCL module\nkcl mod init my-service\necho 'provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }' >> kcl.mod\n\n# Develop and test\nmodule-loader discover taskservs   # Should find your service\n```\n\n### 3. Workspace Migration\n\n```\n# Analyze existing workspace\nworkspace-migrate.nu workspace/infra/old-project dry-run\n\n# Perform migration\nworkspace-migrate.nu workspace/infra/old-project\n\n# Verify migration\nmodule-loader validate workspace/infra/old-project\n```\n\n### 4. Multi-Environment Management\n\n```\n# Development environment\ncd workspace/infra/dev\nmodule-loader load taskservs . [redis, postgres]\nmodule-loader load providers . [local]\n\n# Production environment\ncd workspace/infra/prod\nmodule-loader load taskservs . [redis, postgres, kubernetes, monitoring]\nmodule-loader load providers . [upcloud, aws]  # Multi-cloud\n```\n\n## Module Management\n\n### Listing and Validation\n\n```\n# List loaded modules\nmodule-loader list taskservs .\nmodule-loader list providers .\nmodule-loader list clusters .\n\n# Validate workspace\nmodule-loader validate .\n\n# Show workspace info\nworkspace-init.nu . info\n```\n\n### Unloading Modules\n\n```\n# Remove specific modules\nmodule-loader unload taskservs . redis\nmodule-loader unload providers . aws\n\n# This regenerates import files automatically\n```\n\n### Module Information\n\n```\n# Get detailed module info\nmodule-loader info taskservs kubernetes\nmodule-loader info providers upcloud\nmodule-loader info clusters buildkit\n```\n\n## CI/CD Integration\n\n### Pipeline Example\n\n```\n#!/usr/bin/env nu\n# deploy-pipeline.nu\n\n# Install specific versions\nkcl-packager.nu install --version $env.PROVISIONING_VERSION\n\n# Load production modules\nmodule-loader init $env.WORKSPACE_PATH \\n    --taskservs $env.REQUIRED_TASKSERVS \\n    --providers [$env.CLOUD_PROVIDER]\n\n# Validate configuration\nmodule-loader validate $env.WORKSPACE_PATH\n\n# Deploy infrastructure\nprovisioning server create --infra $env.WORKSPACE_PATH\n```\n\n## Troubleshooting\n\n### Common Issues\n\n#### Module Import Errors\n\n```\nError: module not found\n```\n\n**Solution**: Verify modules are loaded and regenerate imports\n\n```\nmodule-loader list taskservs .\nmodule-loader load taskservs . [kubernetes, cilium, containerd]\n```\n\n#### Provider Configuration Issues\n\n**Solution**: Check provider-specific configuration in `.providers/` directory\n\n#### KCL Compilation Errors\n\n**Solution**: Verify core package installation and kcl.mod configuration\n\n```\nkcl-packager.nu install --version latest\nkcl run --dry-run servers.ncl\n```\n\n### Debug Commands\n\n```\n# Show workspace structure\ntree -a workspace/infra/my-project\n\n# Check generated imports\ncat workspace/infra/my-project/taskservs.ncl\n\n# Validate KCL files\nnickel typecheck workspace/infra/my-project/*.ncl\n\n# Show module manifests\ncat workspace/infra/my-project/.manifest/taskservs.yaml\n```\n\n## Best Practices\n\n### 1. Version Management\n\n- Pin core package versions in production\n- Use semantic versioning for extensions\n- Test compatibility before upgrading\n\n### 2. Module Organization\n\n- Load only required modules to keep workspaces clean\n- Use meaningful workspace names\n- Document required modules in README\n\n### 3. Security\n\n- Exclude `.manifest/` and `data/` from version control\n- Use secrets management for sensitive configuration\n- Validate modules before loading in production\n\n### 4. Performance\n\n- Load modules at workspace initialization, not runtime\n- Cache discovery results when possible\n- Use parallel loading for multiple modules\n\n## Migration Guide\n\nFor existing workspaces, follow these steps:\n\n### 1. Backup Current Workspace\n\n```\ncp -r workspace/infra/existing workspace/infra/existing-backup\n```\n\n### 2. Analyze Migration Requirements\n\n```\nworkspace-migrate.nu workspace/infra/existing dry-run\n```\n\n### 3. Perform Migration\n\n```\nworkspace-migrate.nu workspace/infra/existing\n```\n\n### 4. Load Required Modules\n\n```\ncd workspace/infra/existing\nmodule-loader load taskservs . [kubernetes, cilium]\nmodule-loader load providers . [upcloud]\n```\n\n### 5. Test and Validate\n\n```\nkcl run servers.ncl\nmodule-loader validate .\n```\n\n### 6. Deploy\n\n```\nprovisioning server create --infra . --check\n```\n\n## Future Enhancements\n\n- Registry-based module distribution\n- Module dependency resolution\n- Automatic version updates\n- Module templates and scaffolding\n- Integration with external package managers