# Orchestrator Integration Model - Deep Dive\n\n**Date:** 2025-10-01\n**Status:** Clarification Document\n**Related:** [Multi-Repo Strategy](multi-repo-strategy.md), [Hybrid Orchestrator v3.0](../user/hybrid-orchestrator.md)\n\n## Executive Summary\n\nThis document clarifies **how the Rust orchestrator integrates with Nushell core** in both monorepo and multi-repo architectures. The orchestrator is\na **critical performance layer** that coordinates Nushell business logic execution, solving deep call stack limitations while preserving all existing\nfunctionality.\n\n---\n\n## Current Architecture (Hybrid Orchestrator v3.0)\n\n### The Problem Being Solved\n\n**Original Issue:**\n\n```\nDeep call stack in Nushell (template.nu:71)\n→ "Type not supported" errors\n→ Cannot handle complex nested workflows\n→ Performance bottlenecks with recursive calls\n```\n\n**Solution:** Rust orchestrator provides:\n\n1. **Task queue management** (file-based, reliable)\n2. **Priority scheduling** (intelligent task ordering)\n3. **Deep call stack elimination** (Rust handles recursion)\n4. **Performance optimization** (async/await, parallel execution)\n5. **State management** (workflow checkpointing)\n\n### How It Works Today (Monorepo)\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                        User                                  │\n└───────────────────────────┬─────────────────────────────────┘\n                            │ calls\n                            ↓\n                    ┌───────────────┐\n                    │ provisioning  │ (Nushell CLI)\n                    │      CLI      │\n                    └───────┬───────┘\n                            │\n        ┌───────────────────┼───────────────────┐\n        │                   │                   │\n        ↓                   ↓                   ↓\n┌───────────────┐   ┌───────────────┐   ┌──────────────┐\n│ Direct Mode   │   │Orchestrated   │   │ Workflow     │\n│ (Simple ops)  │   │ Mode          │   │ Mode         │\n└───────────────┘   └───────┬───────┘   └──────┬───────┘\n                            │                   │\n                            ↓                   ↓\n                    ┌────────────────────────────────┐\n                    │   Rust Orchestrator Service    │\n                    │   (Background daemon)           │\n                    │                                 │\n                    │ • Task Queue (file-based)      │\n                    │ • Priority Scheduler           │\n                    │ • Workflow Engine              │\n                    │ • REST API Server              │\n                    └────────┬───────────────────────┘\n                            │ spawns\n                            ↓\n                    ┌────────────────┐\n                    │ Nushell        │\n                    │ Business Logic │\n                    │                │\n                    │ • servers.nu   │\n                    │ • taskservs.nu │\n                    │ • clusters.nu  │\n                    └────────────────┘\n```\n\n### Three Execution Modes\n\n#### Mode 1: Direct Mode (Simple Operations)\n\n```\n# No orchestrator needed\nprovisioning server list\nprovisioning env\nprovisioning help\n\n# Direct Nushell execution\nprovisioning (CLI) → Nushell scripts → Result\n```\n\n#### Mode 2: Orchestrated Mode (Complex Operations)\n\n```\n# Uses orchestrator for coordination\nprovisioning server create --orchestrated\n\n# Flow:\nprovisioning CLI → Orchestrator API → Task Queue → Nushell executor\n                                                 ↓\n                                            Result back to user\n```\n\n#### Mode 3: Workflow Mode (Batch Operations)\n\n```\n# Complex workflows with dependencies\nprovisioning workflow submit server-cluster.ncl\n\n# Flow:\nprovisioning CLI → Orchestrator Workflow Engine → Dependency Graph\n                                                 ↓\n                                            Parallel task execution\n                                                 ↓\n                                            Nushell scripts for each task\n                                                 ↓\n                                            Checkpoint state\n```\n\n---\n\n## Integration Patterns\n\n### Pattern 1: CLI Submits Tasks to Orchestrator\n\n**Current Implementation:**\n\n**Nushell CLI (`core/nulib/workflows/server_create.nu`):**\n\n```\n# Submit server creation workflow to orchestrator\nexport def server_create_workflow [\n    infra_name: string\n    --orchestrated\n] {\n    if $orchestrated {\n        # Submit task to orchestrator\n        let task = {\n            type: "server_create"\n            infra: $infra_name\n            params: { ... }\n        }\n\n        # POST to orchestrator REST API\n        http post http://localhost:9090/workflows/servers/create $task\n    } else {\n        # Direct execution (old way)\n        do-server-create $infra_name\n    }\n}\n```\n\n**Rust Orchestrator (`platform/orchestrator/src/api/workflows.rs`):**\n\n```\n// Receive workflow submission from Nushell CLI\n#[axum::debug_handler]\nasync fn create_server_workflow(\n    State(state): State<Arc<AppState>>,\n    Json(request): Json<ServerCreateRequest>,\n) -> Result<Json<WorkflowResponse>, ApiError> {\n    // Create task\n    let task = Task {\n        id: Uuid::new_v4(),\n        task_type: TaskType::ServerCreate,\n        payload: serde_json::to_value(&request)?,\n        priority: Priority::Normal,\n        status: TaskStatus::Pending,\n        created_at: Utc::now(),\n    };\n\n    // Queue task\n    state.task_queue.enqueue(task).await?;\n\n    // Return immediately (async execution)\n    Ok(Json(WorkflowResponse {\n        workflow_id: task.id,\n        status: "queued",\n    }))\n}\n```\n\n**Flow:**\n\n```\nUser → provisioning server create --orchestrated\n     ↓\nNushell CLI prepares task\n     ↓\nHTTP POST to orchestrator (localhost:9090)\n     ↓\nOrchestrator queues task\n     ↓\nReturns workflow ID immediately\n     ↓\nUser can monitor: provisioning workflow monitor <id>\n```\n\n### Pattern 2: Orchestrator Executes Nushell Scripts\n\n**Orchestrator Task Executor (`platform/orchestrator/src/executor.rs`):**\n\n```\n// Orchestrator spawns Nushell to execute business logic\npub async fn execute_task(task: Task) -> Result<TaskResult> {\n    match task.task_type {\n        TaskType::ServerCreate => {\n            // Orchestrator calls Nushell script via subprocess\n            let output = Command::new("nu")\n                .arg("-c")\n                .arg(format!(\n                    "use {}/servers/create.nu; create-server '{}'",\n                    PROVISIONING_LIB_PATH,\n                    task.payload.infra_name\n                ))\n                .output()\n                .await?;\n\n            // Parse Nushell output\n            let result = parse_nushell_output(&output)?;\n\n            Ok(TaskResult {\n                task_id: task.id,\n                status: if result.success { "completed" } else { "failed" },\n                output: result.data,\n            })\n        }\n        // Other task types...\n    }\n}\n```\n\n**Flow:**\n\n```\nOrchestrator task queue has pending task\n     ↓\nExecutor picks up task\n     ↓\nSpawns Nushell subprocess: nu -c "use servers/create.nu; create-server 'wuji'"\n     ↓\nNushell executes business logic\n     ↓\nReturns result to orchestrator\n     ↓\nOrchestrator updates task status\n     ↓\nUser monitors via: provisioning workflow status <id>\n```\n\n### Pattern 3: Bidirectional Communication\n\n**Nushell Calls Orchestrator API:**\n\n```\n# Nushell script checks orchestrator status during execution\nexport def check-orchestrator-health [] {\n    let response = (http get http://localhost:9090/health)\n\n    if $response.status != "healthy" {\n        error make { msg: "Orchestrator not available" }\n    }\n\n    $response\n}\n\n# Nushell script reports progress to orchestrator\nexport def report-progress [task_id: string, progress: int] {\n    http post http://localhost:9090/tasks/$task_id/progress {\n        progress: $progress\n        status: "in_progress"\n    }\n}\n```\n\n**Orchestrator Monitors Nushell Execution:**\n\n```\n// Orchestrator tracks Nushell subprocess\npub async fn execute_with_monitoring(task: Task) -> Result<TaskResult> {\n    let mut child = Command::new("nu")\n        .arg("-c")\n        .arg(&task.script)\n        .stdout(Stdio::piped())\n        .stderr(Stdio::piped())\n        .spawn()?;\n\n    // Monitor stdout/stderr in real-time\n    let stdout = child.stdout.take().unwrap();\n    tokio::spawn(async move {\n        let reader = BufReader::new(stdout);\n        let mut lines = reader.lines();\n\n        while let Some(line) = lines.next_line().await.unwrap() {\n            // Parse progress updates from Nushell\n            if line.contains("PROGRESS:") {\n                update_task_progress(&line);\n            }\n        }\n    });\n\n    // Wait for completion with timeout\n    let result = tokio::time::timeout(\n        Duration::from_secs(3600),\n        child.wait()\n    ).await??;\n\n    Ok(TaskResult::from_exit_status(result))\n}\n```\n\n---\n\n## Multi-Repo Architecture Impact\n\n### Repository Split Doesn't Change Integration Model\n\n**In Multi-Repo Setup:**\n\n**Repository: `provisioning-core`**\n\n- Contains: Nushell business logic\n- Installs to: `/usr/local/lib/provisioning/`\n- Package: `provisioning-core-3.2.1.tar.gz`\n\n**Repository: `provisioning-platform`**\n\n- Contains: Rust orchestrator\n- Installs to: `/usr/local/bin/provisioning-orchestrator`\n- Package: `provisioning-platform-2.5.3.tar.gz`\n\n**Runtime Integration (Same as Monorepo):**\n\n```\nUser installs both packages:\n  provisioning-core-3.2.1     → /usr/local/lib/provisioning/\n  provisioning-platform-2.5.3 → /usr/local/bin/provisioning-orchestrator\n\nOrchestrator expects core at:  /usr/local/lib/provisioning/\nCore expects orchestrator at:  http://localhost:9090/\n\nNo code dependencies, just runtime coordination!\n```\n\n### Configuration-Based Integration\n\n**Core Package (`provisioning-core`) config:**\n\n```\n# /usr/local/share/provisioning/config/config.defaults.toml\n\n[orchestrator]\nenabled = true\nendpoint = "http://localhost:9090"\ntimeout = 60\nauto_start = true  # Start orchestrator if not running\n\n[execution]\ndefault_mode = "orchestrated"  # Use orchestrator by default\nfallback_to_direct = true      # Fall back if orchestrator down\n```\n\n**Platform Package (`provisioning-platform`) config:**\n\n```\n# /usr/local/share/provisioning/platform/config.toml\n\n[orchestrator]\nhost = "127.0.0.1"\nport = 8080\ndata_dir = "/var/lib/provisioning/orchestrator"\n\n[executor]\nnushell_binary = "nu"  # Expects nu in PATH\nprovisioning_lib = "/usr/local/lib/provisioning"\nmax_concurrent_tasks = 10\ntask_timeout_seconds = 3600\n```\n\n### Version Compatibility\n\n**Compatibility Matrix (`provisioning-distribution/versions.toml`):**\n\n```\n[compatibility.platform."2.5.3"]\ncore = "^3.2"  # Platform 2.5.3 compatible with core 3.2.x\nmin-core = "3.2.0"\napi-version = "v1"\n\n[compatibility.core."3.2.1"]\nplatform = "^2.5"  # Core 3.2.1 compatible with platform 2.5.x\nmin-platform = "2.5.0"\norchestrator-api = "v1"\n```\n\n---\n\n## Execution Flow Examples\n\n### Example 1: Simple Server Creation (Direct Mode)\n\n**No Orchestrator Needed:**\n\n```\nprovisioning server list\n\n# Flow:\nCLI → servers/list.nu → Query state → Return results\n(Orchestrator not involved)\n```\n\n### Example 2: Server Creation with Orchestrator\n\n**Using Orchestrator:**\n\n```\nprovisioning server create --orchestrated --infra wuji\n\n# Detailed Flow:\n1. User executes command\n   ↓\n2. Nushell CLI (provisioning binary)\n   ↓\n3. Reads config: orchestrator.enabled = true\n   ↓\n4. Prepares task payload:\n   {\n     type: "server_create",\n     infra: "wuji",\n     params: { ... }\n   }\n   ↓\n5. HTTP POST → http://localhost:9090/workflows/servers/create\n   ↓\n6. Orchestrator receives request\n   ↓\n7. Creates task with UUID\n   ↓\n8. Enqueues to task queue (file-based: /var/lib/provisioning/queue/)\n   ↓\n9. Returns immediately: { workflow_id: "abc-123", status: "queued" }\n   ↓\n10. User sees: "Workflow submitted: abc-123"\n   ↓\n11. Orchestrator executor picks up task\n   ↓\n12. Spawns Nushell subprocess:\n    nu -c "use /usr/local/lib/provisioning/servers/create.nu; create-server 'wuji'"\n   ↓\n13. Nushell executes business logic:\n    - Reads Nickel config\n    - Calls provider API (UpCloud/AWS)\n    - Creates server\n    - Returns result\n   ↓\n14. Orchestrator captures output\n   ↓\n15. Updates task status: "completed"\n   ↓\n16. User monitors: provisioning workflow status abc-123\n    → Shows: "Server wuji created successfully"\n```\n\n### Example 3: Batch Workflow with Dependencies\n\n**Complex Workflow:**\n\n```\nprovisioning batch submit multi-cloud-deployment.ncl\n\n# Workflow contains:\n- Create 5 servers (parallel)\n- Install Kubernetes on servers (depends on server creation)\n- Deploy applications (depends on Kubernetes)\n\n# Detailed Flow:\n1. CLI submits Nickel workflow to orchestrator\n   ↓\n2. Orchestrator parses workflow\n   ↓\n3. Builds dependency graph using petgraph (Rust)\n   ↓\n4. Topological sort determines execution order\n   ↓\n5. Creates tasks for each operation\n   ↓\n6. Executes in parallel where possible:\n\n   [Server 1] [Server 2] [Server 3] [Server 4] [Server 5]\n       ↓          ↓          ↓          ↓          ↓\n   (All execute in parallel via Nushell subprocesses)\n       ↓          ↓          ↓          ↓          ↓\n       └──────────┴──────────┴──────────┴──────────┘\n                           │\n                           ↓\n                    [All servers ready]\n                           ↓\n                  [Install Kubernetes]\n                  (Nushell subprocess)\n                           ↓\n                  [Kubernetes ready]\n                           ↓\n                  [Deploy applications]\n                  (Nushell subprocess)\n                           ↓\n                       [Complete]\n\n7. Orchestrator checkpoints state at each step\n   ↓\n8. If failure occurs, can retry from checkpoint\n   ↓\n9. User monitors real-time: provisioning batch monitor <id>\n```\n\n---\n\n## Why This Architecture\n\n### Orchestrator Benefits\n\n1. **Eliminates Deep Call Stack Issues**\n\n   ```text\n\n   Without Orchestrator:\n   template.nu → calls → cluster.nu → calls → taskserv.nu → calls → provider.nu\n   (Deep nesting causes "Type not supported" errors)\n\n   With Orchestrator:\n   Orchestrator → spawns → Nushell subprocess (flat execution)\n   (No deep nesting, fresh Nushell context for each task)\n\n   ```\n\n2. **Performance Optimization**\n\n   ```rust\n   // Orchestrator executes tasks in parallel\n   let tasks = vec![task1, task2, task3, task4, task5];\n\n   let results = futures::future::join_all(\n       tasks.iter().map(|t| execute_task(t))\n   ).await;\n\n   // 5 Nushell subprocesses run concurrently\n   ```\n\n1. **Reliable State Management**\n\n```\n   Orchestrator maintains:\n   - Task queue (survives crashes)\n   - Workflow checkpoints (resume on failure)\n   - Progress tracking (real-time monitoring)\n   - Retry logic (automatic recovery)\n```\n\n1. **Clean Separation**\n\n```\n   Orchestrator (Rust):     Performance, concurrency, state\n   Business Logic (Nushell): Providers, taskservs, workflows\n\n   Each does what it's best at!\n```\n\n### Why NOT Pure Rust\n\n**Question:** Why not implement everything in Rust?\n\n**Answer:**\n\n1. **Nushell is perfect for infrastructure automation:**\n   - Shell-like scripting for system operations\n   - Built-in structured data handling\n   - Easy template rendering\n   - Readable business logic\n\n2. **Rapid iteration:**\n   - Change Nushell scripts without recompiling\n   - Community can contribute Nushell modules\n   - Template-based configuration generation\n\n3. **Best of both worlds:**\n   - Rust: Performance, type safety, concurrency\n   - Nushell: Flexibility, readability, ease of use\n\n---\n\n## Multi-Repo Integration Example\n\n### Installation\n\n**User installs bundle:**\n\n```\ncurl -fsSL https://get.provisioning.io | sh\n\n# Installs:\n1. provisioning-core-3.2.1.tar.gz\n   → /usr/local/bin/provisioning (Nushell CLI)\n   → /usr/local/lib/provisioning/ (Nushell libraries)\n   → /usr/local/share/provisioning/ (configs, templates)\n\n2. provisioning-platform-2.5.3.tar.gz\n   → /usr/local/bin/provisioning-orchestrator (Rust binary)\n   → /usr/local/share/provisioning/platform/ (platform configs)\n\n3. Sets up systemd/launchd service for orchestrator\n```\n\n### Runtime Coordination\n\n**Core package expects orchestrator:**\n\n```\n# core/nulib/lib_provisioning/orchestrator/client.nu\n\n# Check if orchestrator is running\nexport def orchestrator-available [] {\n    let config = (load-config)\n    let endpoint = $config.orchestrator.endpoint\n\n    try {\n        let response = (http get $"($endpoint)/health")\n        $response.status == "healthy"\n    } catch {\n        false\n    }\n}\n\n# Auto-start orchestrator if needed\nexport def ensure-orchestrator [] {\n    if not (orchestrator-available) {\n        if (load-config).orchestrator.auto_start {\n            print "Starting orchestrator..."\n            ^provisioning-orchestrator --daemon\n            sleep 2sec\n        }\n    }\n}\n```\n\n**Platform package executes core scripts:**\n\n```\n// platform/orchestrator/src/executor/nushell.rs\n\npub struct NushellExecutor {\n    provisioning_lib: PathBuf,  // /usr/local/lib/provisioning\n    nu_binary: PathBuf,          // nu (from PATH)\n}\n\nimpl NushellExecutor {\n    pub async fn execute_script(&self, script: &str) -> Result<Output> {\n        Command::new(&self.nu_binary)\n            .env("NU_LIB_DIRS", &self.provisioning_lib)\n            .arg("-c")\n            .arg(script)\n            .output()\n            .await\n    }\n\n    pub async fn execute_module_function(\n        &self,\n        module: &str,\n        function: &str,\n        args: &[String],\n    ) -> Result<Output> {\n        let script = format!(\n            "use {}/{}; {} {}",\n            self.provisioning_lib.display(),\n            module,\n            function,\n            args.join(" ")\n        );\n\n        self.execute_script(&script).await\n    }\n}\n```\n\n---\n\n## Configuration Examples\n\n### Core Package Config\n\n**`/usr/local/share/provisioning/config/config.defaults.toml`:**\n\n```\n[orchestrator]\nenabled = true\nendpoint = "http://localhost:9090"\ntimeout_seconds = 60\nauto_start = true\nfallback_to_direct = true\n\n[execution]\n# Modes: "direct", "orchestrated", "auto"\ndefault_mode = "auto"  # Auto-detect based on complexity\n\n# Operations that always use orchestrator\nforce_orchestrated = [\n    "server.create",\n    "cluster.create",\n    "batch.*",\n    "workflow.*"\n]\n\n# Operations that always run direct\nforce_direct = [\n    "*.list",\n    "*.show",\n    "help",\n    "version"\n]\n```\n\n### Platform Package Config\n\n**`/usr/local/share/provisioning/platform/config.toml`:**\n\n```\n[server]\nhost = "127.0.0.1"\nport = 8080\n\n[storage]\nbackend = "filesystem"  # or "surrealdb"\ndata_dir = "/var/lib/provisioning/orchestrator"\n\n[executor]\nmax_concurrent_tasks = 10\ntask_timeout_seconds = 3600\ncheckpoint_interval_seconds = 30\n\n[nushell]\nbinary = "nu"  # Expects nu in PATH\nprovisioning_lib = "/usr/local/lib/provisioning"\nenv_vars = { NU_LIB_DIRS = "/usr/local/lib/provisioning" }\n```\n\n---\n\n## Key Takeaways\n\n### 1. **Orchestrator is Essential**\n\n- Solves deep call stack problems\n- Provides performance optimization\n- Enables complex workflows\n- NOT optional for production use\n\n### 2. **Integration is Loose but Coordinated**\n\n- No code dependencies between repos\n- Runtime integration via CLI + REST API\n- Configuration-driven coordination\n- Works in both monorepo and multi-repo\n\n### 3. **Best of Both Worlds**\n\n- Rust: High-performance coordination\n- Nushell: Flexible business logic\n- Clean separation of concerns\n- Each technology does what it's best at\n\n### 4. **Multi-Repo Doesn't Change Integration**\n\n- Same runtime model as monorepo\n- Package installation sets up paths\n- Configuration enables discovery\n- Versioning ensures compatibility\n\n---\n\n## Conclusion\n\nThe confusing example in the multi-repo doc was **oversimplified**. The real architecture is:\n\n```\n✅ Orchestrator IS USED and IS ESSENTIAL\n✅ Platform (Rust) coordinates Core (Nushell) execution\n✅ Loose coupling via CLI + REST API (not code dependencies)\n✅ Works identically in monorepo and multi-repo\n✅ Configuration-based integration (no hardcoded paths)\n```\n\nThe orchestrator provides:\n\n- Performance layer (async, parallel execution)\n- Workflow engine (complex dependencies)\n- State management (checkpoints, recovery)\n- Task queue (reliable execution)\n\nWhile Nushell provides:\n\n- Business logic (providers, taskservs, clusters)\n- Template rendering (Jinja2 via nu_plugin_tera)\n- Configuration management (KCL integration)\n- User-facing scripting\n\n**Multi-repo just splits WHERE the code lives, not HOW it works together.**