provisioning/docs/src/operations/orchestrator-system.md

# Hybrid Orchestrator Architecture (v3.0.0)

## 🚀 Orchestrator Implementation Completed (2025-09-25)

A production-ready hybrid Rust/Nushell orchestrator has been implemented to solve deep call stack limitations while preserving all Nushell business logic.

## Architecture Overview

- **Rust Orchestrator**: High-performance coordination layer with REST API
- **Nushell Business Logic**: All existing scripts preserved and enhanced
- **File-based Persistence**: Reliable task queue using lightweight file storage
- **Priority Processing**: Intelligent task scheduling with retry logic
- **Deep Call Stack Solution**: Eliminates template.nu:71 "Type not supported" errors

## Orchestrator Management

```bash
# Start orchestrator in background
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background --provisioning-path "/usr/local/bin/provisioning"

# Check orchestrator status
./scripts/start-orchestrator.nu --check

# Stop orchestrator
./scripts/start-orchestrator.nu --stop

# View logs
tail -f ./data/orchestrator.log
```

## Workflow System

The orchestrator provides comprehensive workflow management:

### Server Workflows

```bash
# Submit server creation workflow
nu -c "use core/nulib/workflows/server_create.nu *; server_create_workflow 'wuji' '' [] --check"

# Traditional orchestrated server creation
provisioning servers create --orchestrated --check
```

### Taskserv Workflows

```bash
# Create taskserv workflow
nu -c "use core/nulib/workflows/taskserv.nu *; taskserv create 'kubernetes' 'wuji' --check"

# Other taskserv operations
nu -c "use core/nulib/workflows/taskserv.nu *; taskserv delete 'kubernetes' 'wuji' --check"
nu -c "use core/nulib/workflows/taskserv.nu *; taskserv generate 'kubernetes' 'wuji'"
nu -c "use core/nulib/workflows/taskserv.nu *; taskserv check-updates"
```

### Cluster Workflows

```bash
# Create cluster workflow
nu -c "use core/nulib/workflows/cluster.nu *; cluster create 'buildkit' 'wuji' --check"

# Delete cluster workflow
nu -c "use core/nulib/workflows/cluster.nu *; cluster delete 'buildkit' 'wuji' --check"
```

### Workflow Management

```bash
# List all workflows
nu -c "use core/nulib/workflows/management.nu *; workflow list"

# Get workflow statistics
nu -c "use core/nulib/workflows/management.nu *; workflow stats"

# Monitor workflow in real-time
nu -c "use core/nulib/workflows/management.nu *; workflow monitor <task_id>"

# Check orchestrator health
nu -c "use core/nulib/workflows/management.nu *; workflow orchestrator"

# Get specific workflow status
nu -c "use core/nulib/workflows/management.nu *; workflow status <task_id>"
```

## REST API Endpoints

The orchestrator exposes HTTP endpoints for external integration:

- **Health**: `GET http://localhost:9090/v1/health`
- **List Tasks**: `GET http://localhost:9090/v1/tasks`
- **Task Status**: `GET http://localhost:9090/v1/tasks/{id}`
- **Server Workflow**: `POST http://localhost:9090/v1/workflows/servers/create`
- **Taskserv Workflow**: `POST http://localhost:9090/v1/workflows/taskserv/create`
- **Cluster Workflow**: `POST http://localhost:9090/v1/workflows/cluster/create`