provisioning/docs/src/api-reference/rest-api.md

# REST API Reference

This document provides comprehensive documentation for all REST API endpoints in provisioning.

## Overview

Provisioning exposes two main REST APIs:

- **Orchestrator API** (Port 8080): Core workflow management and batch operations
- **Control Center API** (Port 9080): Authentication, authorization, and policy management

## Base URLs

- **Orchestrator**: `http://localhost:9090`
- **Control Center**: `http://localhost:9080`

## Authentication

### JWT Authentication

All API endpoints (except health checks) require JWT authentication via the Authorization header:

```http
Authorization: Bearer <jwt_token>
```plaintext

### Getting Access Token

```http
POST /auth/login
Content-Type: application/json

{
  "username": "admin",
  "password": "password",
  "mfa_code": "123456"
}
```plaintext

## Orchestrator API Endpoints

### Health Check

#### GET /health

Check orchestrator health status.

**Response:**

```json
{
  "success": true,
  "data": "Orchestrator is healthy"
}
```plaintext

### Task Management

#### GET /tasks

List all workflow tasks.

**Query Parameters:**

- `status` (optional): Filter by task status (Pending, Running, Completed, Failed, Cancelled)
- `limit` (optional): Maximum number of results
- `offset` (optional): Pagination offset

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "id": "uuid-string",
      "name": "create_servers",
      "command": "/usr/local/provisioning servers create",
      "args": ["--infra", "production", "--wait"],
      "dependencies": [],
      "status": "Completed",
      "created_at": "2025-09-26T10:00:00Z",
      "started_at": "2025-09-26T10:00:05Z",
      "completed_at": "2025-09-26T10:05:30Z",
      "output": "Successfully created 3 servers",
      "error": null
    }
  ]
}
```plaintext

#### GET /tasks/{id}

Get specific task status and details.

**Path Parameters:**

- `id`: Task UUID

**Response:**

```json
{
  "success": true,
  "data": {
    "id": "uuid-string",
    "name": "create_servers",
    "command": "/usr/local/provisioning servers create",
    "args": ["--infra", "production", "--wait"],
    "dependencies": [],
    "status": "Running",
    "created_at": "2025-09-26T10:00:00Z",
    "started_at": "2025-09-26T10:00:05Z",
    "completed_at": null,
    "output": null,
    "error": null
  }
}
```plaintext

### Workflow Submission

#### POST /workflows/servers/create

Submit server creation workflow.

**Request Body:**

```json
{
  "infra": "production",
  "settings": "config.ncl",
  "check_mode": false,
  "wait": true
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "uuid-task-id"
}
```plaintext

#### POST /workflows/taskserv/create

Submit task service workflow.

**Request Body:**

```json
{
  "operation": "create",
  "taskserv": "kubernetes",
  "infra": "production",
  "settings": "config.ncl",
  "check_mode": false,
  "wait": true
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "uuid-task-id"
}
```plaintext

#### POST /workflows/cluster/create

Submit cluster workflow.

**Request Body:**

```json
{
  "operation": "create",
  "cluster_type": "buildkit",
  "infra": "production",
  "settings": "config.ncl",
  "check_mode": false,
  "wait": true
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "uuid-task-id"
}
```plaintext

### Batch Operations

#### POST /batch/execute

Execute batch workflow operation.

**Request Body:**

```json
{
  "name": "multi_cloud_deployment",
  "version": "1.0.0",
  "storage_backend": "surrealdb",
  "parallel_limit": 5,
  "rollback_enabled": true,
  "operations": [
    {
      "id": "upcloud_servers",
      "type": "server_batch",
      "provider": "upcloud",
      "dependencies": [],
      "server_configs": [
        {"name": "web-01", "plan": "1xCPU-2 GB", "zone": "de-fra1"},
        {"name": "web-02", "plan": "1xCPU-2 GB", "zone": "us-nyc1"}
      ]
    },
    {
      "id": "aws_taskservs",
      "type": "taskserv_batch",
      "provider": "aws",
      "dependencies": ["upcloud_servers"],
      "taskservs": ["kubernetes", "cilium", "containerd"]
    }
  ]
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "batch_id": "uuid-string",
    "status": "Running",
    "operations": [
      {
        "id": "upcloud_servers",
        "status": "Pending",
        "progress": 0.0
      },
      {
        "id": "aws_taskservs",
        "status": "Pending",
        "progress": 0.0
      }
    ]
  }
}
```plaintext

#### GET /batch/operations

List all batch operations.

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "batch_id": "uuid-string",
      "name": "multi_cloud_deployment",
      "status": "Running",
      "created_at": "2025-09-26T10:00:00Z",
      "operations": [...]
    }
  ]
}
```plaintext

#### GET /batch/operations/{id}

Get batch operation status.

**Path Parameters:**

- `id`: Batch operation ID

**Response:**

```json
{
  "success": true,
  "data": {
    "batch_id": "uuid-string",
    "name": "multi_cloud_deployment",
    "status": "Running",
    "operations": [
      {
        "id": "upcloud_servers",
        "status": "Completed",
        "progress": 100.0,
        "results": {...}
      }
    ]
  }
}
```plaintext

#### POST /batch/operations/{id}/cancel

Cancel running batch operation.

**Path Parameters:**

- `id`: Batch operation ID

**Response:**

```json
{
  "success": true,
  "data": "Operation cancelled"
}
```plaintext

### State Management

#### GET /state/workflows/{id}/progress

Get real-time workflow progress.

**Path Parameters:**

- `id`: Workflow ID

**Response:**

```json
{
  "success": true,
  "data": {
    "workflow_id": "uuid-string",
    "progress": 75.5,
    "current_step": "Installing Kubernetes",
    "total_steps": 8,
    "completed_steps": 6,
    "estimated_time_remaining": 180
  }
}
```plaintext

#### GET /state/workflows/{id}/snapshots

Get workflow state snapshots.

**Path Parameters:**

- `id`: Workflow ID

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "snapshot_id": "uuid-string",
      "timestamp": "2025-09-26T10:00:00Z",
      "state": "running",
      "details": {...}
    }
  ]
}
```plaintext

#### GET /state/system/metrics

Get system-wide metrics.

**Response:**

```json
{
  "success": true,
  "data": {
    "total_workflows": 150,
    "active_workflows": 5,
    "completed_workflows": 140,
    "failed_workflows": 5,
    "system_load": {
      "cpu_usage": 45.2,
      "memory_usage": 2048,
      "disk_usage": 75.5
    }
  }
}
```plaintext

#### GET /state/system/health

Get system health status.

**Response:**

```json
{
  "success": true,
  "data": {
    "overall_status": "Healthy",
    "components": {
      "storage": "Healthy",
      "batch_coordinator": "Healthy",
      "monitoring": "Healthy"
    },
    "last_check": "2025-09-26T10:00:00Z"
  }
}
```plaintext

#### GET /state/statistics

Get state manager statistics.

**Response:**

```json
{
  "success": true,
  "data": {
    "total_workflows": 150,
    "active_snapshots": 25,
    "storage_usage": "245 MB",
    "average_workflow_duration": 300
  }
}
```plaintext

### Rollback and Recovery

#### POST /rollback/checkpoints

Create new checkpoint.

**Request Body:**

```json
{
  "name": "before_major_update",
  "description": "Checkpoint before deploying v2.0.0"
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "checkpoint-uuid"
}
```plaintext

#### GET /rollback/checkpoints

List all checkpoints.

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "id": "checkpoint-uuid",
      "name": "before_major_update",
      "description": "Checkpoint before deploying v2.0.0",
      "created_at": "2025-09-26T10:00:00Z",
      "size": "150 MB"
    }
  ]
}
```plaintext

#### GET /rollback/checkpoints/{id}

Get specific checkpoint details.

**Path Parameters:**

- `id`: Checkpoint ID

**Response:**

```json
{
  "success": true,
  "data": {
    "id": "checkpoint-uuid",
    "name": "before_major_update",
    "description": "Checkpoint before deploying v2.0.0",
    "created_at": "2025-09-26T10:00:00Z",
    "size": "150 MB",
    "operations_count": 25
  }
}
```plaintext

#### POST /rollback/execute

Execute rollback operation.

**Request Body:**

```json
{
  "checkpoint_id": "checkpoint-uuid"
}
```plaintext

Or for partial rollback:

```json
{
  "operation_ids": ["op-1", "op-2", "op-3"]
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "rollback_id": "rollback-uuid",
    "success": true,
    "operations_executed": 25,
    "operations_failed": 0,
    "duration": 45.5
  }
}
```plaintext

#### POST /rollback/restore/{id}

Restore system state from checkpoint.

**Path Parameters:**

- `id`: Checkpoint ID

**Response:**

```json
{
  "success": true,
  "data": "State restored from checkpoint checkpoint-uuid"
}
```plaintext

#### GET /rollback/statistics

Get rollback system statistics.

**Response:**

```json
{
  "success": true,
  "data": {
    "total_checkpoints": 10,
    "total_rollbacks": 3,
    "success_rate": 100.0,
    "average_rollback_time": 30.5
  }
}
```plaintext

## Control Center API Endpoints

### Authentication

#### POST /auth/login

Authenticate user and get JWT token.

**Request Body:**

```json
{
  "username": "admin",
  "password": "secure_password",
  "mfa_code": "123456"
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "token": "jwt-token-string",
    "expires_at": "2025-09-26T18:00:00Z",
    "user": {
      "id": "user-uuid",
      "username": "admin",
      "email": "admin@example.com",
      "roles": ["admin", "operator"]
    }
  }
}
```plaintext

#### POST /auth/refresh

Refresh JWT token.

**Request Body:**

```json
{
  "token": "current-jwt-token"
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "token": "new-jwt-token",
    "expires_at": "2025-09-26T18:00:00Z"
  }
}
```plaintext

#### POST /auth/logout

Logout and invalidate token.

**Response:**

```json
{
  "success": true,
  "data": "Successfully logged out"
}
```plaintext

### User Management

#### GET /users

List all users.

**Query Parameters:**

- `role` (optional): Filter by role
- `enabled` (optional): Filter by enabled status

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "id": "user-uuid",
      "username": "admin",
      "email": "admin@example.com",
      "roles": ["admin"],
      "enabled": true,
      "created_at": "2025-09-26T10:00:00Z",
      "last_login": "2025-09-26T12:00:00Z"
    }
  ]
}
```plaintext

#### POST /users

Create new user.

**Request Body:**

```json
{
  "username": "newuser",
  "email": "newuser@example.com",
  "password": "secure_password",
  "roles": ["operator"],
  "enabled": true
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "id": "new-user-uuid",
    "username": "newuser",
    "email": "newuser@example.com",
    "roles": ["operator"],
    "enabled": true
  }
}
```plaintext

#### PUT /users/{id}

Update existing user.

**Path Parameters:**

- `id`: User ID

**Request Body:**

```json
{
  "email": "updated@example.com",
  "roles": ["admin", "operator"],
  "enabled": false
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "User updated successfully"
}
```plaintext

#### DELETE /users/{id}

Delete user.

**Path Parameters:**

- `id`: User ID

**Response:**

```json
{
  "success": true,
  "data": "User deleted successfully"
}
```plaintext

### Policy Management

#### GET /policies

List all policies.

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "id": "policy-uuid",
      "name": "admin_access_policy",
      "version": "1.0.0",
      "rules": [...],
      "created_at": "2025-09-26T10:00:00Z",
      "enabled": true
    }
  ]
}
```plaintext

#### POST /policies

Create new policy.

**Request Body:**

```json
{
  "name": "new_policy",
  "version": "1.0.0",
  "rules": [
    {
      "effect": "Allow",
      "resource": "servers:*",
      "action": ["create", "read"],
      "condition": "user.role == 'admin'"
    }
  ]
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": {
    "id": "new-policy-uuid",
    "name": "new_policy",
    "version": "1.0.0"
  }
}
```plaintext

#### PUT /policies/{id}

Update policy.

**Path Parameters:**

- `id`: Policy ID

**Request Body:**

```json
{
  "name": "updated_policy",
  "rules": [...]
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": "Policy updated successfully"
}
```plaintext

### Audit Logging

#### GET /audit/logs

Get audit logs.

**Query Parameters:**

- `user_id` (optional): Filter by user
- `action` (optional): Filter by action
- `resource` (optional): Filter by resource
- `from` (optional): Start date (ISO 8601)
- `to` (optional): End date (ISO 8601)
- `limit` (optional): Maximum results
- `offset` (optional): Pagination offset

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "id": "audit-log-uuid",
      "timestamp": "2025-09-26T10:00:00Z",
      "user_id": "user-uuid",
      "action": "server.create",
      "resource": "servers/web-01",
      "result": "success",
      "details": {...}
    }
  ]
}
```plaintext

## Error Responses

All endpoints may return error responses in this format:

```json
{
  "success": false,
  "error": "Detailed error message"
}
```plaintext

### HTTP Status Codes

- `200 OK`: Successful request
- `201 Created`: Resource created successfully
- `400 Bad Request`: Invalid request parameters
- `401 Unauthorized`: Authentication required or invalid
- `403 Forbidden`: Permission denied
- `404 Not Found`: Resource not found
- `422 Unprocessable Entity`: Validation error
- `500 Internal Server Error`: Server error

## Rate Limiting

API endpoints are rate-limited:

- Authentication: 5 requests per minute per IP
- General APIs: 100 requests per minute per user
- Batch operations: 10 requests per minute per user

Rate limit headers are included in responses:

```http
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1632150000
```plaintext

## Monitoring Endpoints

### GET /metrics

Prometheus-compatible metrics endpoint.

**Response:**

```plaintext
# HELP orchestrator_tasks_total Total number of tasks
# TYPE orchestrator_tasks_total counter
orchestrator_tasks_total{status="completed"} 150
orchestrator_tasks_total{status="failed"} 5

# HELP orchestrator_task_duration_seconds Task execution duration
# TYPE orchestrator_task_duration_seconds histogram
orchestrator_task_duration_seconds_bucket{le="10"} 50
orchestrator_task_duration_seconds_bucket{le="30"} 120
orchestrator_task_duration_seconds_bucket{le="+Inf"} 155
```plaintext

### WebSocket /ws

Real-time event streaming via WebSocket connection.

**Connection:**

```javascript
const ws = new WebSocket('ws://localhost:9090/ws?token=jwt-token');

ws.onmessage = function(event) {
  const data = JSON.parse(event.data);
  console.log('Event:', data);
};
```plaintext

**Event Format:**

```json
{
  "event_type": "TaskStatusChanged",
  "timestamp": "2025-09-26T10:00:00Z",
  "data": {
    "task_id": "uuid-string",
    "status": "completed"
  },
  "metadata": {
    "task_id": "uuid-string",
    "status": "completed"
  }
}
```plaintext

## SDK Examples

### Python SDK Example

```python
import requests

class ProvisioningClient:
    def __init__(self, base_url, token):
        self.base_url = base_url
        self.headers = {
            'Authorization': f'Bearer {token}',
            'Content-Type': 'application/json'
        }

    def create_server_workflow(self, infra, settings, check_mode=False):
        payload = {
            'infra': infra,
            'settings': settings,
            'check_mode': check_mode,
            'wait': True
        }
        response = requests.post(
            f'{self.base_url}/workflows/servers/create',
            json=payload,
            headers=self.headers
        )
        return response.json()

    def get_task_status(self, task_id):
        response = requests.get(
            f'{self.base_url}/tasks/{task_id}',
            headers=self.headers
        )
        return response.json()

# Usage
client = ProvisioningClient('http://localhost:9090', 'your-jwt-token')
result = client.create_server_workflow('production', 'config.ncl')
print(f"Task ID: {result['data']}")
```plaintext

### JavaScript/Node.js SDK Example

```javascript
const axios = require('axios');

class ProvisioningClient {
  constructor(baseUrl, token) {
    this.client = axios.create({
      baseURL: baseUrl,
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      }
    });
  }

  async createServerWorkflow(infra, settings, checkMode = false) {
    const response = await this.client.post('/workflows/servers/create', {
      infra,
      settings,
      check_mode: checkMode,
      wait: true
    });
    return response.data;
  }

  async getTaskStatus(taskId) {
    const response = await this.client.get(`/tasks/${taskId}`);
    return response.data;
  }
}

// Usage
const client = new ProvisioningClient('http://localhost:9090', 'your-jwt-token');
const result = await client.createServerWorkflow('production', 'config.ncl');
console.log(`Task ID: ${result.data}`);
```plaintext

## Webhook Integration

The system supports webhooks for external integrations:

### Webhook Configuration

Configure webhooks in the system configuration:

```toml
[webhooks]
enabled = true
endpoints = [
  {
    url = "https://your-system.com/webhook"
    events = ["task.completed", "task.failed", "batch.completed"]
    secret = "webhook-secret"
  }
]
```plaintext

### Webhook Payload

```json
{
  "event": "task.completed",
  "timestamp": "2025-09-26T10:00:00Z",
  "data": {
    "task_id": "uuid-string",
    "status": "completed",
    "output": "Task completed successfully"
  },
  "signature": "sha256=calculated-signature"
}
```plaintext

## Pagination

For endpoints that return lists, use pagination parameters:

- `limit`: Maximum number of items per page (default: 50, max: 1000)
- `offset`: Number of items to skip

Pagination metadata is included in response headers:

```http
X-Total-Count: 1500
X-Limit: 50
X-Offset: 100
Link: </api/endpoint?offset=150&limit=50>; rel="next"
```plaintext

## API Versioning

The API uses header-based versioning:

```http
Accept: application/vnd.provisioning.v1+json
```plaintext

Current version: v1

## Testing

Use the included test suite to validate API functionality:

```bash
# Run API integration tests
cd src/orchestrator
cargo test --test api_tests

# Run load tests
cargo test --test load_tests --release
```plaintext