provisioning/docs/src/infrastructure/workspace-config-architecture.md

Workspace Configuration Architecture

Version: 2.0.0 Date: 2025-10-06 Status: Implemented

Overview

The provisioning system now uses a workspace-based configuration architecture where each workspace has its own complete configuration structure. This replaces the old ENV-based and template-only system.

Critical Design Principle

config.defaults.toml is ONLY a template, NEVER loaded at runtime

This file exists solely as a reference template for generating workspace configurations. The system does NOT load it during operation.

Configuration Hierarchy

Configuration is loaded in the following order (lowest to highest priority):

1. Workspace Config (Base): {workspace}/config/provisioning.yaml
2. Provider Configs: {workspace}/config/providers/*.toml
3. Platform Configs: {workspace}/config/platform/*.toml
4. User Context: ~/Library/Application Support/provisioning/ws_{name}.yaml
5. Environment Variables: PROVISIONING_* (highest priority)

Workspace Structure

When a workspace is initialized, the following structure is created:

{workspace}/
├── config/
│   ├── provisioning.yaml       # Main workspace config (generated from template)
│   ├── providers/              # Provider-specific configs
│   │   ├── aws.toml
│   │   ├── local.toml
│   │   └── upcloud.toml
│   ├── platform/               # Platform service configs
│   │   ├── orchestrator.toml
│   │   └── mcp.toml
│   └── kms.toml                # KMS configuration
├── infra/                      # Infrastructure definitions
├── .cache/                     # Cache directory
├── .runtime/                   # Runtime data
│   ├── taskservs/
│   └── clusters/
├── .providers/                 # Provider state
├── .kms/                       # Key management
│   └── keys/
├── generated/                  # Generated files
└── .gitignore                  # Workspace gitignore
```plaintext

## Template System

Templates are located at: `/Users/Akasha/project-provisioning/provisioning/config/templates/`

### Available Templates

1. **workspace-provisioning.yaml.template** - Main workspace configuration
2. **provider-aws.toml.template** - AWS provider configuration
3. **provider-local.toml.template** - Local provider configuration
4. **provider-upcloud.toml.template** - UpCloud provider configuration
5. **kms.toml.template** - KMS configuration
6. **user-context.yaml.template** - User context configuration

### Template Variables

Templates support the following interpolation variables:

- `{{workspace.name}}` - Workspace name
- `{{workspace.path}}` - Absolute path to workspace
- `{{now.iso}}` - Current timestamp in ISO format
- `{{env.HOME}}` - User's home directory
- `{{env.*}}` - Environment variables (safe list only)
- `{{paths.base}}` - Base path (after config load)

## Workspace Initialization

### Command

```bash
# Using the workspace init function
nu -c "use provisioning/core/nulib/lib_provisioning/workspace/init.nu *; workspace-init 'my-workspace' '/path/to/workspace' --providers ['aws' 'local'] --activate"
```plaintext

### Process

1. **Create Directory Structure**: All necessary directories
2. **Generate Config from Template**: Creates `config/provisioning.yaml`
3. **Generate Provider Configs**: For each specified provider
4. **Generate KMS Config**: Security configuration
5. **Create User Context** (if --activate): User-specific overrides
6. **Create .gitignore**: Ignore runtime/cache files

## User Context

User context files are stored per workspace:

**Location**: `~/Library/Application Support/provisioning/ws_{workspace_name}.yaml`

### Purpose

- Store user-specific overrides (debug settings, output preferences)
- Mark active workspace
- Override workspace paths if needed

### Example

```yaml
workspace:
  name: "my-workspace"
  path: "/path/to/my-workspace"
  active: true

debug:
  enabled: true
  log_level: "debug"

output:
  format: "json"

providers:
  default: "aws"
```plaintext

## Configuration Loading Process

### 1. Determine Active Workspace

```nushell
# Check user config directory for active workspace
let user_config_dir = ~/Library/Application Support/provisioning/
let active_workspace = (find workspace with active: true in ws_*.yaml files)
```plaintext

### 2. Load Workspace Config

```nushell
# Load main workspace config
let workspace_config = {workspace.path}/config/provisioning.yaml
```plaintext

### 3. Load Provider Configs

```nushell
# Merge all provider configs
for provider in {workspace.path}/config/providers/*.toml {
  merge provider config
}
```plaintext

### 4. Load Platform Configs

```nushell
# Merge all platform configs
for platform in {workspace.path}/config/platform/*.toml {
  merge platform config
}
```plaintext

### 5. Apply User Context

```nushell
# Apply user-specific overrides
let user_context = ~/Library/Application Support/provisioning/ws_{name}.yaml
merge user_context (highest config priority)
```plaintext

### 6. Apply Environment Variables

```nushell
# Final overrides from environment
PROVISIONING_DEBUG=true
PROVISIONING_LOG_LEVEL=debug
PROVISIONING_PROVIDER=aws
# etc.
```plaintext

## Migration from Old System

### Before (ENV-based)

```bash
export PROVISIONING=/usr/local/provisioning
export PROVISIONING_INFRA_PATH=/path/to/infra
export PROVISIONING_DEBUG=true
# ... many ENV variables
```plaintext

### After (Workspace-based)

```bash
# Initialize workspace
workspace-init "production" "/workspaces/prod" --providers ["aws"] --activate

# All config is now in workspace
# No ENV variables needed (except for overrides)
```plaintext

### Breaking Changes

1. **`config.defaults.toml` NOT loaded** - Only used as template
2. **Workspace required** - Must have active workspace or be in workspace directory
3. **New config locations** - User config in `~/Library/Application Support/provisioning/`
4. **YAML main config** - `provisioning.yaml` instead of TOML

## Workspace Management Commands

### Initialize Workspace

```nushell
use provisioning/core/nulib/lib_provisioning/workspace/init.nu *
workspace-init "my-workspace" "/path/to/workspace" --providers ["aws" "local"] --activate
```plaintext

### List Workspaces

```nushell
workspace-list
```plaintext

### Activate Workspace

```nushell
workspace-activate "my-workspace"
```plaintext

### Get Active Workspace

```nushell
workspace-get-active
```plaintext

## Implementation Files

### Core Files

1. **Template Directory**: `/Users/Akasha/project-provisioning/provisioning/config/templates/`
2. **Workspace Init**: `/Users/Akasha/project-provisioning/provisioning/core/nulib/lib_provisioning/workspace/init.nu`
3. **Config Loader**: `/Users/Akasha/project-provisioning/provisioning/core/nulib/lib_provisioning/config/loader.nu`

### Key Changes in Config Loader

#### Removed

- `get-defaults-config-path()` - No longer loads config.defaults.toml
- Old hierarchy with user/project/infra TOML files

#### Added

- `get-active-workspace()` - Finds active workspace from user config
- Support for YAML config files
- Provider and platform config merging
- User context loading

## Configuration Schema

### Main Workspace Config (provisioning.yaml)

```yaml
workspace:
  name: string
  version: string
  created: timestamp

paths:
  base: string
  infra: string
  cache: string
  runtime: string
  # ... all paths

core:
  version: string
  name: string

debug:
  enabled: bool
  log_level: string
  # ... debug settings

providers:
  active: [string]
  default: string

# ... all other sections
```plaintext

### Provider Config (providers/*.toml)

```toml
[provider]
name = "aws"
enabled = true
workspace = "workspace-name"

[provider.auth]
profile = "default"
region = "us-east-1"

[provider.paths]
base = "{workspace}/.providers/aws"
cache = "{workspace}/.providers/aws/cache"
```plaintext

### User Context (ws_{name}.yaml)

```yaml
workspace:
  name: string
  path: string
  active: bool

debug:
  enabled: bool
  log_level: string

output:
  format: string
```plaintext

## Benefits

1. **No Template Loading**: config.defaults.toml is template-only
2. **Workspace Isolation**: Each workspace is self-contained
3. **Explicit Configuration**: No hidden defaults from ENV
4. **Clear Hierarchy**: Predictable override behavior
5. **Multi-Workspace Support**: Easy switching between workspaces
6. **User Overrides**: Per-workspace user preferences
7. **Version Control**: Workspace configs can be committed (except secrets)

## Security Considerations

### Generated .gitignore

The workspace .gitignore excludes:

- `.cache/` - Cache files
- `.runtime/` - Runtime data
- `.providers/` - Provider state
- `.kms/keys/` - Secret keys
- `generated/` - Generated files
- `*.log` - Log files

### Secret Management

- KMS keys stored in `.kms/keys/` (gitignored)
- SOPS config references keys, doesn't store them
- Provider credentials in user-specific locations (not workspace)

## Troubleshooting

### No Active Workspace Error

```plaintext
Error: No active workspace found. Please initialize or activate a workspace.
```plaintext

**Solution**: Initialize or activate a workspace:

```bash
workspace-init "my-workspace" "/path/to/workspace" --activate
```plaintext

### Config File Not Found

```plaintext
Error: Required configuration file not found: {workspace}/config/provisioning.yaml
```plaintext

**Solution**: The workspace config is corrupted or deleted. Re-initialize:

```bash
workspace-init "workspace-name" "/existing/path" --providers ["aws"]
```plaintext

### Provider Not Configured

**Solution**: Add provider config to workspace:

```bash
# Generate provider config manually
generate-provider-config "/workspace/path" "workspace-name" "aws"
```plaintext

## Future Enhancements

1. **Workspace Templates**: Pre-configured workspace templates (dev, prod, test)
2. **Workspace Import/Export**: Share workspace configurations
3. **Remote Workspace**: Load workspace from remote Git repository
4. **Workspace Validation**: Comprehensive workspace health checks
5. **Config Migration Tool**: Automated migration from old ENV-based system

## Summary

- **config.defaults.toml is ONLY a template** - Never loaded at runtime
- **Workspaces are self-contained** - Complete config structure generated from templates
- **New hierarchy**: Workspace → Provider → Platform → User Context → ENV
- **User context for overrides** - Stored in ~/Library/Application Support/provisioning/
- **Clear, explicit configuration** - No hidden defaults

## Related Documentation

- Template files: `provisioning/config/templates/`
- Workspace init: `provisioning/core/nulib/lib_provisioning/workspace/init.nu`
- Config loader: `provisioning/core/nulib/lib_provisioning/config/loader.nu`
- User guide: `docs/user/workspace-management.md`