provisioning/docs/src/testing/taskserv-validation-guide.md

# Taskserv Validation and Testing Guide

**Version**: 1.0.0
**Date**: 2025-10-06
**Status**: Production Ready

---

## Overview

The taskserv validation and testing system provides comprehensive evaluation of infrastructure services before deployment, reducing errors and
increasing confidence in deployments.

## Validation Levels

### 1. Static Validation

Validates configuration files, templates, and scripts without requiring infrastructure access.

**What it checks:**

- KCL schema syntax and semantics
- Jinja2 template syntax
- Shell script syntax (with shellcheck if available)
- File structure and naming conventions

**Command:**

```bash
provisioning taskserv validate kubernetes --level static
```

### 2. Dependency Validation

Checks taskserv dependencies, conflicts, and requirements.

**What it checks:**

- Required dependencies are available
- Optional dependencies status
- Conflicting taskservs
- Resource requirements (memory, CPU, disk)
- Health check configuration

**Command:**

```bash
provisioning taskserv validate kubernetes --level dependencies
```

**Check against infrastructure:**

```bash
provisioning taskserv check-deps kubernetes --infra my-project
```

### 3. Check Mode (Dry-Run)

Enhanced check mode that performs validation and previews deployment without making changes.

**What it does:**

- Runs static validation
- Validates dependencies
- Previews configuration generation
- Lists files to be deployed
- Checks prerequisites (without SSH in check mode)

**Command:**

```bash
provisioning taskserv create kubernetes --check
```

### 4. Sandbox Testing

Tests taskserv in isolated container environment before actual deployment.

**What it tests:**

- Package prerequisites
- Configuration validity
- Script execution
- Health check simulation

**Command:**

```bash
# Test with Docker
provisioning taskserv test kubernetes --runtime docker

# Test with Podman
provisioning taskserv test kubernetes --runtime podman

# Keep container for inspection
provisioning taskserv test kubernetes --runtime docker --keep
```

---

## Complete Validation Workflow

### Recommended Validation Sequence

```bash
# 1. Static validation (fastest, no infrastructure needed)
provisioning taskserv validate kubernetes --level static -v

# 2. Dependency validation
provisioning taskserv check-deps kubernetes --infra my-project

# 3. Check mode (dry-run with full validation)
provisioning taskserv create kubernetes --check -v

# 4. Sandbox testing (optional, requires Docker/Podman)
provisioning taskserv test kubernetes --runtime docker

# 5. Actual deployment (after all validations pass)
provisioning taskserv create kubernetes
```

### Quick Validation (All Levels)

```bash
# Run all validation levels
provisioning taskserv validate kubernetes --level all -v
```

---

## Validation Commands Reference

### `provisioning taskserv validate <taskserv>`

Multi-level validation framework.

**Options:**

- `--level <level>` - Validation level: static, dependencies, health, all (default: all)
- `--infra <name>` - Infrastructure context
- `--settings <path>` - Settings file path
- `--verbose` - Verbose output
- `--out <format>` - Output format: json, yaml, text

**Examples:**

```bash
# Complete validation
provisioning taskserv validate kubernetes

# Only static validation
provisioning taskserv validate kubernetes --level static

# With verbose output
provisioning taskserv validate kubernetes -v

# JSON output
provisioning taskserv validate kubernetes --out json
```

### `provisioning taskserv check-deps <taskserv>`

Check dependencies against infrastructure.

**Options:**

- `--infra <name>` - Infrastructure context
- `--settings <path>` - Settings file path
- `--verbose` - Verbose output

**Examples:**

```bash
# Check dependencies
provisioning taskserv check-deps kubernetes --infra my-project

# Verbose output
provisioning taskserv check-deps kubernetes --infra my-project -v
```

### `provisioning taskserv create <taskserv> --check`

Enhanced check mode with full validation and preview.

**Options:**

- `--check` - Enable check mode (no actual deployment)
- `--verbose` - Verbose output
- All standard create options

**Examples:**

```bash
# Check mode with verbose output
provisioning taskserv create kubernetes --check -v

# Check specific server
provisioning taskserv create kubernetes server-01 --check
```

### `provisioning taskserv test <taskserv>`

Sandbox testing in isolated environment.

**Options:**

- `--runtime <name>` - Runtime: docker, podman, native (default: docker)
- `--infra <name>` - Infrastructure context
- `--settings <path>` - Settings file path
- `--keep` - Keep container after test
- `--verbose` - Verbose output

**Examples:**

```bash
# Test with Docker
provisioning taskserv test kubernetes --runtime docker

# Test with Podman
provisioning taskserv test kubernetes --runtime podman

# Keep container for debugging
provisioning taskserv test kubernetes --keep -v

# Connect to kept container
docker exec -it taskserv-test-kubernetes bash
```

---

## Validation Output

### Static Validation

```bash
Taskserv Validation
Taskserv: kubernetes
Level: static

Validating Nickel schemas for kubernetes...
  Checking main.ncl...
    ✓ Valid
  Checking version.ncl...
    ✓ Valid
  Checking dependencies.ncl...
    ✓ Valid

Validating templates for kubernetes...
  Checking env-kubernetes.j2...
    ✓ Basic syntax OK
  Checking install-kubernetes.sh...
    ✓ Basic syntax OK

Validation Summary
✓ nickel: 0 errors, 0 warnings
✓ templates: 0 errors, 0 warnings
✓ scripts: 0 errors, 0 warnings

Overall Status
✓ VALID - 0 warnings
```

### Dependency Validation

```bash
Dependency Validation Report
Taskserv: kubernetes

Status: VALID

Required Dependencies:
  • containerd
  • etcd
  • os

Optional Dependencies:
  • cilium
  • helm

Conflicts:
  • docker
  • podman
```

### Check Mode Output

```bash
Check Mode: kubernetes on server-01

→ Running static validation...
  ✓ Static validation passed

→ Checking dependencies...
  ✓ Dependencies OK
    Required: containerd, etcd, os

→ Previewing configuration generation...
  ✓ Configuration preview generated
    Files to process: 15

→ Checking prerequisites...
  ℹ Prerequisite checks (preview mode):
    ⊘ Server accessibility: Check mode - SSH not tested
    ℹ Directory /tmp: Would verify directory exists
    ℹ Command bash: Would verify command is available

Check Mode Summary
✓ All validations passed

💡 Taskserv can be deployed with: provisioning taskserv create kubernetes
```

### Test Output

```bash
Taskserv Sandbox Testing
Taskserv: kubernetes
Runtime: docker

→ Running pre-test validation...
✓ Validation passed

→ Preparing sandbox environment...
  Using base image: ubuntu:22.04
✓ Sandbox prepared: a1b2c3d4e5f6

→ Running tests in sandbox...
  Test 1: Package prerequisites...
  Test 2: Configuration validity...
  Test 3: Script execution...
  Test 4: Health check simulation...

Test Summary
Total tests: 4
Passed: 4
Failed: 0
Skipped: 0

Detailed Results:
  ✓ Package prerequisites: Package manager accessible
  ✓ Configuration validity: 3 configuration files validated
  ✓ Script execution: 2 scripts validated
  ✓ Health check: Health check configuration valid: http://localhost:6443/healthz

✓ All tests passed
```

---

## Integration with CI/CD

### GitLab CI Example

```bash
validate-taskservs:
  stage: validate
  script:
    - provisioning taskserv validate kubernetes --level all --out json
    - provisioning taskserv check-deps kubernetes --infra production

test-taskservs:
  stage: test
  script:
    - provisioning taskserv test kubernetes --runtime docker
  dependencies:
    - validate-taskservs

deploy-taskservs:
  stage: deploy
  script:
    - provisioning taskserv create kubernetes
  dependencies:
    - test-taskservs
  only:
    - main
```

### GitHub Actions Example

```bash
name: Taskserv Validation

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate Taskservs
        run: |
          provisioning taskserv validate kubernetes --level all -v

      - name: Check Dependencies
        run: |
          provisioning taskserv check-deps kubernetes --infra production

      - name: Test in Sandbox
        run: |
          provisioning taskserv test kubernetes --runtime docker
```

---

## Troubleshooting

### shellcheck not found

If shellcheck is not available, script validation will be skipped with a warning.

**Install shellcheck:**

```bash
# macOS
brew install shellcheck

# Ubuntu/Debian
apt install shellcheck

# Fedora
dnf install shellcheck
```

### Docker/Podman not available

Sandbox testing requires Docker or Podman.

**Check runtime:**

```bash
# Docker
docker ps

# Podman
podman ps

# Use native mode (limited testing)
provisioning taskserv test kubernetes --runtime native
```

### Nickel type checking errors

Nickel type checking errors indicate syntax or type problems.

**Common fixes:**

- Check schema syntax in `.ncl` files
- Validate imports and dependencies
- Run `nickel format` to format files
- Check `manifest.toml` dependencies

### Dependency conflicts

If conflicting taskservs are detected:

- Remove conflicting taskserv first
- Check infrastructure configuration
- Review dependency declarations in `dependencies.ncl`

---

## Advanced Usage

### Custom Validation Scripts

You can create custom validation scripts by extending the validation framework:

```bash
# custom_validation.nu
use provisioning/core/nulib/taskservs/validate.nu *

def custom-validate [taskserv: string] {
    # Custom validation logic
    let result = (validate-nickel-schemas $taskserv --verbose=true)

    # Additional custom checks
    # ...

    return $result
}
```

### Batch Validation

Validate multiple taskservs:

```bash
# Validate all taskservs in infrastructure
for taskserv in (provisioning taskserv list | get name) {
    provisioning taskserv validate $taskserv
}
```

### Automated Testing

Create test suite for all taskservs:

```nushell
#!/usr/bin/env nu

let taskservs = ["kubernetes", "containerd", "cilium", "etcd"]

for ts in $taskservs {
    print $"Testing ($ts)..."
    provisioning taskserv test $ts --runtime docker
}
```

---

## Best Practices

### Before Deployment

1. **Always validate** before deploying to production
2. **Run check mode** to preview changes
3. **Test in sandbox** for critical services
4. **Check dependencies** in infrastructure context

### During Development

1. **Validate frequently** during taskserv development
2. **Use verbose mode** to understand validation details
3. **Fix warnings** even if validation passes
4. **Keep containers** for debugging test failures

### In CI/CD

1. **Fail fast** on validation errors
2. **Require all tests pass** before merge
3. **Generate reports** in JSON format for analysis
4. **Archive test results** for audit trail

---

## Related Documentation

- [Taskserv Development Guide](taskserv-development-guide.md)
- KCL Schema Reference
- [Dependency Management](dependency-management.md)
- [CI/CD Integration](cicd-integration.md)

---

## Version History

| Version | Date | Changes |
| --------- | ------ | --------- |
| 1.0.0 | 2025-10-06 | Initial validation and testing guide |

---

**Maintained By**: Infrastructure Team
**Review Cycle**: Quarterly