provisioning/docs/src/development/extension-registry.md

Extension Registry Service

A high-performance Rust microservice that provides a unified REST API for extension discovery, versioning, and download from multiple Git-based sources and OCI registries.

Source: provisioning/platform/crates/extension-registry/

Features

• Multi-Backend Source Support: Fetch extensions from Gitea, Forgejo, and GitHub releases
• Multi-Registry Distribution Support: Distribute extensions to Zot, Harbor, Docker Hub, GHCR, Quay, and other OCI-compliant registries
• Unified REST API: Single API for all extension operations across all backends
• Smart Caching: LRU cache with TTL to reduce backend API calls
• Prometheus Metrics: Built-in metrics for monitoring
• Health Monitoring: Parallel health checks for all backends with aggregated status
• Aggregation & Fallback: Intelligent request routing with aggregation and fallback strategies
• Type-Safe: Strong typing for extension metadata
• Async/Await: High-performance async operations with Tokio
• Backward Compatible: Old single-instance configs auto-migrate to new multi-instance format

Architecture

Dual-Trait System

The extension registry uses a trait-based architecture separating source and distribution backends:

┌────────────────────────────────────────────────────────────────────┐
│                    Extension Registry API                           │
│                          (axum)                                     │
├────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─ SourceClients ────────────┐  ┌─ DistributionClients ────────┐  │
│  │                             │  │                              │  │
│  │ • Gitea (Git releases)      │  │ • OCI Registries             │  │
│  │ • Forgejo (Git releases)    │  │   - Zot                      │  │
│  │ • GitHub (Releases API)     │  │   - Harbor                   │  │
│  │                             │  │   - Docker Hub               │  │
│  │ Strategy: Aggregation +     │  │   - GHCR / Quay              │  │
│  │ Fallback across all sources │  │   - Any OCI-compliant        │  │
│  │                             │  │                              │  │
│  └─────────────────────────────┘  └──────────────────────────────┘  │
│                                                                      │
│  ┌─ LRU Cache ───────────────────────────────────────────────────┐  │
│  │ • Metadata cache (with TTL)                                   │  │
│  │ • List cache (with TTL)                                       │  │
│  │ • Version cache (version strings only)                        │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                                                                      │
└────────────────────────────────────────────────────────────────────┘
```plaintext

### Request Strategies

#### Aggregation Strategy (list_extensions, list_versions, search)

1. **Parallel Execution**: Spawn concurrent tasks for all source and distribution clients
2. **Merge Results**: Combine results from all backends
3. **Deduplication**: Remove duplicates, preferring more recent versions
4. **Pagination**: Apply limit/offset to merged results
5. **Caching**: Store merged results with composite cache key

#### Fallback Strategy (get_extension, download_extension)

1. **Sequential Retry**: Try source clients first (in configured order)
2. **Distribution Fallback**: If all sources fail, try distribution clients
3. **Return First Success**: Return result from first successful client
4. **Caching**: Cache successful result with backend-specific key

## Installation

```bash
cd provisioning/platform/extension-registry
cargo build --release
```plaintext

## Configuration

### Single-Instance Configuration (Legacy - Auto-Migrated)

Old format is automatically migrated to new multi-instance format:

```toml
[server]
host = "0.0.0.0"
port = 8082

# Single Gitea instance (auto-migrated to sources.gitea[0])
[gitea]
url = "https://gitea.example.com"
organization = "provisioning-extensions"
token_path = "/path/to/gitea-token.txt"

# Single OCI registry (auto-migrated to distributions.oci[0])
[oci]
registry = "registry.example.com"
namespace = "provisioning"
auth_token_path = "/path/to/oci-token.txt"

[cache]
capacity = 1000
ttl_seconds = 300
```plaintext

### Multi-Instance Configuration (Recommended)

New format supporting multiple backends of each type:

```toml
[server]
host = "0.0.0.0"
port = 8082
workers = 4
enable_cors = false
enable_compression = true

# Multiple Gitea sources
[sources.gitea]

[[sources.gitea]]
id = "internal-gitea"
url = "https://gitea.internal.example.com"
organization = "provisioning"
token_path = "/etc/secrets/gitea-internal-token.txt"
timeout_seconds = 30
verify_ssl = true

[[sources.gitea]]
id = "public-gitea"
url = "https://gitea.public.example.com"
organization = "extensions"
token_path = "/etc/secrets/gitea-public-token.txt"
timeout_seconds = 30
verify_ssl = true

# Forgejo sources (API compatible with Gitea)
[sources.forgejo]

[[sources.forgejo]]
id = "community-forgejo"
url = "https://forgejo.community.example.com"
organization = "provisioning"
token_path = "/etc/secrets/forgejo-token.txt"
timeout_seconds = 30
verify_ssl = true

# GitHub sources
[sources.github]

[[sources.github]]
id = "org-github"
organization = "my-organization"
token_path = "/etc/secrets/github-token.txt"
timeout_seconds = 30
verify_ssl = true

# Multiple OCI distribution registries
[distributions.oci]

[[distributions.oci]]
id = "internal-zot"
registry = "zot.internal.example.com"
namespace = "extensions"
timeout_seconds = 30
verify_ssl = true

[[distributions.oci]]
id = "public-harbor"
registry = "harbor.public.example.com"
namespace = "extensions"
auth_token_path = "/etc/secrets/harbor-token.txt"
timeout_seconds = 30
verify_ssl = true

[[distributions.oci]]
id = "docker-hub"
registry = "docker.io"
namespace = "myorg"
auth_token_path = "/etc/secrets/docker-hub-token.txt"
timeout_seconds = 30
verify_ssl = true

# Cache configuration
[cache]
capacity = 1000
ttl_seconds = 300
enable_metadata_cache = true
enable_list_cache = true
```plaintext

### Configuration Notes

- **Backend Identifiers**: Use `id` field to uniquely identify each backend instance (auto-generated if omitted)
- **Gitea/Forgejo Compatible**: Both use same config format; organization field is required for Git repos
- **GitHub Configuration**: Uses organization as owner; token_path points to GitHub Personal Access Token
- **OCI Registries**: Support any OCI-compliant registry (Zot, Harbor, Docker Hub, GHCR, Quay, etc.)
- **Optional Fields**: `id`, `verify_ssl`, `timeout_seconds` have sensible defaults
- **Token Files**: Should contain only the token with no extra whitespace; permissions should be `0600`

### Environment Variable Overrides

Legacy environment variable support (for backward compatibility):

```bash
REGISTRY_SERVER_HOST=127.0.0.1
REGISTRY_SERVER_PORT=8083
REGISTRY_SERVER_WORKERS=8
REGISTRY_GITEA_URL=https://gitea.example.com
REGISTRY_GITEA_ORG=extensions
REGISTRY_GITEA_TOKEN_PATH=/path/to/token
REGISTRY_OCI_REGISTRY=registry.example.com
REGISTRY_OCI_NAMESPACE=extensions
REGISTRY_CACHE_CAPACITY=2000
REGISTRY_CACHE_TTL=600
```plaintext

## API Endpoints

### Extension Operations

#### List Extensions

```bash
GET /api/v1/extensions?type=provider&limit=10
```plaintext

#### Get Extension

```bash
GET /api/v1/extensions/{type}/{name}
```plaintext

#### List Versions

```bash
GET /api/v1/extensions/{type}/{name}/versions
```plaintext

#### Download Extension

```bash
GET /api/v1/extensions/{type}/{name}/{version}
```plaintext

#### Search Extensions

```bash
GET /api/v1/extensions/search?q=kubernetes&type=taskserv
```plaintext

### System Endpoints

#### Health Check

```bash
GET /api/v1/health
```plaintext

**Response** (with multi-backend aggregation):

```json
{
  "status": "healthy|degraded|unhealthy",
  "version": "0.1.0",
  "uptime": 3600,
  "backends": {
    "gitea": {
      "enabled": true,
      "healthy": true,
      "error": null
    },
    "oci": {
      "enabled": true,
      "healthy": true,
      "error": null
    }
  }
}
```plaintext

**Status Values**:
- `healthy`: All configured backends are healthy
- `degraded`: At least one backend is healthy, but some are failing
- `unhealthy`: No backends are responding

#### Metrics

```bash
GET /api/v1/metrics
```plaintext

#### Cache Statistics

```bash
GET /api/v1/cache/stats
```plaintext

**Response**:

```json
{
  "metadata_hits": 1024,
  "metadata_misses": 256,
  "list_hits": 512,
  "list_misses": 128,
  "version_hits": 2048,
  "version_misses": 512,
  "size": 4096
}
```plaintext

## Extension Naming Conventions

### Gitea Repositories

- **Providers**: `{name}_prov` (for example, `aws_prov`)
- **Task Services**: `{name}_taskserv` (for example, `kubernetes_taskserv`)
- **Clusters**: `{name}_cluster` (for example, `buildkit_cluster`)

### OCI Artifacts

- **Providers**: `{namespace}/{name}-provider`
- **Task Services**: `{namespace}/{name}-taskserv`
- **Clusters**: `{namespace}/{name}-cluster`

## Deployment

### Docker

```bash
docker build -t extension-registry:latest .
docker run -d -p 8082:8082 -v $(pwd)/config.toml:/app/config.toml:ro extension-registry:latest
```plaintext

### Kubernetes

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: extension-registry
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: extension-registry
        image: extension-registry:latest
        ports:
        - containerPort: 8082
```plaintext

## Migration Guide: Single to Multi-Instance

### Automatic Migration

Old single-instance configs are automatically detected and migrated to the new multi-instance format during startup:

1. **Detection**: Registry checks if old-style fields (`gitea`, `oci`) contain values
2. **Migration**: Single instances are moved to new Vec-based format (`sources.gitea[0]`, `distributions.oci[0]`)
3. **Logging**: Migration event is logged for audit purposes
4. **Transparency**: No user action required; old configs continue to work

### Before Migration

```toml
[gitea]
url = "https://gitea.example.com"
organization = "extensions"
token_path = "/path/to/token"

[oci]
registry = "registry.example.com"
namespace = "extensions"
```plaintext

### After Migration (Automatic)

```toml
[sources.gitea]
[[sources.gitea]]
url = "https://gitea.example.com"
organization = "extensions"
token_path = "/path/to/token"

[distributions.oci]
[[distributions.oci]]
registry = "registry.example.com"
namespace = "extensions"
```plaintext

### Gradual Upgrade Path

To adopt the new format manually:

1. **Backup current config** - Keep old format as reference
2. **Adopt new format** - Replace old fields with new structure
3. **Test** - Verify all backends are reachable and extensions are discovered
4. **Add new backends** - Use new format to add Forgejo, GitHub, or additional OCI registries
5. **Remove old fields** - Delete deprecated `gitea` and `oci` top-level sections

### Benefits of Upgrading

- **Multiple Sources**: Support Gitea, Forgejo, and GitHub simultaneously
- **Multiple Registries**: Distribute to multiple OCI registries
- **Better Resilience**: If one backend fails, others continue to work
- **Flexible Configuration**: Each backend can have different credentials and timeouts
- **Future-Proof**: New backends can be added without config restructuring

## Related Documentation

- **Extension Development**: [Module System](../development/extensions.md)
- **Extension Development Quickstart**: [Getting Started Guide](../guides/extension-development-quickstart.md)
- **ADR-005**: [Extension Framework Architecture](../architecture/adr/adr-005-extension-framework.md)
- **OCI Registry Integration**: [OCI Registry Guide](../integration/oci-registry-guide.md)