provisioning/docs/src/security/secrets-management-guide.md

# Secrets Management System - Configuration Guide

**Status**: Production Ready
**Date**: 2025-11-19
**Version**: 1.0.0

## Overview

The provisioning system supports secure SSH key retrieval from multiple secret sources, eliminating hardcoded filesystem dependencies and enabling
enterprise-grade security. SSH keys are retrieved from configured secret sources (SOPS, KMS, RustyVault) with automatic fallback to local-dev mode for
development environments.

## Secret Sources

### 1. SOPS (Secrets Operations)

Age-based encrypted secrets file with YAML structure.

**Pros**:

- ✅ Age encryption (modern, performant)
- ✅ Easy to version in Git (encrypted)
- ✅ No external services required
- ✅ Simple YAML structure

**Cons**:

- ❌ Requires Age key management
- ❌ No key rotation automation

**Environment Variables**:

```
PROVISIONING_SECRET_SOURCE=sops
PROVISIONING_SOPS_ENABLED=true
PROVISIONING_SOPS_SECRETS_FILE=/path/to/secrets.enc.yaml
PROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning
```

**Secrets File Structure** (provisioning/secrets.enc.yaml):

```
# Encrypted with sops
ssh:
  web-01:
    ubuntu: /path/to/id_rsa
    root: /path/to/root_id_rsa
  db-01:
    postgres: /path/to/postgres_id_rsa
```

**Setup Instructions**:

```
# 1. Install sops and age
brew install sops age

# 2. Generate Age key (store securely!)
age-keygen -o $HOME/.age/provisioning

# 3. Create encrypted secrets file
cat > secrets.yaml << 'EOF'
ssh:
  web-01:
    ubuntu: ~/.ssh/provisioning_web01
  db-01:
    postgres: ~/.ssh/provisioning_db01
EOF

# 4. Encrypt with sops
sops -e -i secrets.yaml

# 5. Rename to enc version
mv secrets.yaml provisioning/secrets.enc.yaml

# 6. Configure environment
export PROVISIONING_SECRET_SOURCE=sops
export PROVISIONING_SOPS_SECRETS_FILE=$(pwd)/provisioning/secrets.enc.yaml
export PROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning
```

### 2. KMS (Key Management Service)

AWS KMS or compatible key management service.

**Pros**:

- ✅ Cloud-native security
- ✅ Automatic key rotation
- ✅ Audit logging built-in
- ✅ High availability

**Cons**:

- ❌ Requires AWS account/credentials
- ❌ API calls add latency (~50 ms)
- ❌ Cost per API call

**Environment Variables**:

```
PROVISIONING_SECRET_SOURCE=kms
PROVISIONING_KMS_ENABLED=true
PROVISIONING_KMS_REGION=us-east-1
```

**Secret Storage Pattern**:

```
provisioning/ssh-keys/{hostname}/{username}
```

**Setup Instructions**:

```
# 1. Create KMS key (one-time)
aws kms create-key \
    --description "Provisioning SSH Keys" \
    --region us-east-1

# 2. Store SSH keys in Secrets Manager
aws secretsmanager create-secret \
    --name provisioning/ssh-keys/web-01/ubuntu \
    --secret-string "$(cat ~/.ssh/provisioning_web01)" \
    --region us-east-1

# 3. Configure environment
export PROVISIONING_SECRET_SOURCE=kms
export PROVISIONING_KMS_REGION=us-east-1

# 4. Ensure AWS credentials available
export AWS_PROFILE=provisioning
# or
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```

### 3. RustyVault (Hashicorp Vault-Compatible)

Self-hosted or managed Vault instance for secrets.

**Pros**:

- ✅ Self-hosted option
- ✅ Fine-grained access control
- ✅ Multiple authentication methods
- ✅ Easy key rotation

**Cons**:

- ❌ Requires Vault instance
- ❌ More operational overhead
- ❌ Network latency

**Environment Variables**:

```
PROVISIONING_SECRET_SOURCE=vault
PROVISIONING_VAULT_ENABLED=true
PROVISIONING_VAULT_ADDRESS=http://localhost:8200
PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...
```

**Secret Storage Pattern**:

```
GET /v1/secret/ssh-keys/{hostname}/{username}
# Returns: {"key_content": "-----BEGIN OPENSSH PRIVATE KEY-----..."}
```

**Setup Instructions**:

```
# 1. Start Vault (if not already running)
docker run -p 8200:8200 \
    -e VAULT_DEV_ROOT_TOKEN_ID=provisioning \
    vault server -dev

# 2. Create KV v2 mount (if not exists)
vault secrets enable -version=2 -path=secret kv

# 3. Store SSH key
vault kv put secret/ssh-keys/web-01/ubuntu \
    key_content=@~/.ssh/provisioning_web01

# 4. Configure environment
export PROVISIONING_SECRET_SOURCE=vault
export PROVISIONING_VAULT_ADDRESS=http://localhost:8200
export PROVISIONING_VAULT_TOKEN=provisioning

# 5. Create AppRole for production
vault auth enable approle
vault write auth/approle/role/provisioning \
    token_ttl=1h \
    token_max_ttl=4h
vault read auth/approle/role/provisioning/role-id
vault write -f auth/approle/role/provisioning/secret-id
```

### 4. Local-Dev (Fallback)

Local filesystem SSH keys (development only).

**Pros**:

- ✅ No setup required
- ✅ Fast (local filesystem)
- ✅ Works offline

**Cons**:

- ❌ NOT for production
- ❌ Hardcoded filesystem dependency
- ❌ No key rotation

**Environment Variables**:

```
PROVISIONING_ENVIRONMENT=local-dev
```

**Behavior**:

Standard paths checked (in order):

1. `$HOME/.ssh/id_rsa`
2. `$HOME/.ssh/id_ed25519`
3. `$HOME/.ssh/provisioning`
4. `$HOME/.ssh/provisioning_rsa`

## Auto-Detection Logic

When `PROVISIONING_SECRET_SOURCE` is not explicitly set, the system auto-detects in this order:

```
1. PROVISIONING_SOPS_ENABLED=true or PROVISIONING_SOPS_SECRETS_FILE set?
   → Use SOPS
2. PROVISIONING_KMS_ENABLED=true or PROVISIONING_KMS_REGION set?
   → Use KMS
3. PROVISIONING_VAULT_ENABLED=true or both VAULT_ADDRESS and VAULT_TOKEN set?
   → Use Vault
4. Otherwise
   → Use local-dev (with warnings in production environments)
```

## Configuration Matrix

| Secret Source | Env Variables | Enabled in |
| --- | --- | --- |
| **SOPS** | `PROVISIONING_SOPS_*` | Development, Staging, Production |
| **KMS** | `PROVISIONING_KMS_*` | Staging, Production (with AWS) |
| **Vault** | `PROVISIONING_VAULT_*` | Development, Staging, Production |
| **Local-dev** | `PROVISIONING_ENVIRONMENT=local-dev` | Development only |

## Production Recommended Setup

### Minimal Setup (Single Source)

```
# Using Vault (recommended for self-hosted)
export PROVISIONING_SECRET_SOURCE=vault
export PROVISIONING_VAULT_ADDRESS=https://vault.example.com:8200
export PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...
export PROVISIONING_ENVIRONMENT=production
```

### Enhanced Setup (Fallback Chain)

```
# Primary: Vault
export PROVISIONING_VAULT_ADDRESS=https://vault.primary.com:8200
export PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...

# Fallback: SOPS
export PROVISIONING_SOPS_SECRETS_FILE=/etc/provisioning/secrets.enc.yaml
export PROVISIONING_SOPS_AGE_KEY_FILE=/etc/provisioning/.age/key

# Environment
export PROVISIONING_ENVIRONMENT=production
export PROVISIONING_SECRET_SOURCE=vault  # Explicit: use Vault first
```

### High-Availability Setup

```
# Use KMS (managed service)
export PROVISIONING_SECRET_SOURCE=kms
export PROVISIONING_KMS_REGION=us-east-1
export AWS_PROFILE=provisioning-admin

# Or use Vault with HA
export PROVISIONING_VAULT_ADDRESS=https://vault-ha.example.com:8200
export PROVISIONING_VAULT_NAMESPACE=provisioning
export PROVISIONING_ENVIRONMENT=production
```

## Validation & Testing

### Check Configuration

```
# Nushell
provisioning secrets status

# Show secret source and configuration
provisioning secrets validate

# Detailed diagnostics
provisioning secrets diagnose
```

### Test SSH Key Retrieval

```
# Test specific host/user
provisioning secrets get-key web-01 ubuntu

# Test all configured hosts
provisioning secrets validate-all

# Dry-run SSH with retrieved key
provisioning ssh --test-key web-01 ubuntu
```

## Migration Path

### From Local-Dev to SOPS

```
# 1. Create SOPS secrets file with existing keys
cat > secrets.yaml << 'EOF'
ssh:
  web-01:
    ubuntu: ~/.ssh/provisioning_web01
  db-01:
    postgres: ~/.ssh/provisioning_db01
EOF

# 2. Encrypt with Age
sops -e -i secrets.yaml

# 3. Move to repo
mv secrets.yaml provisioning/secrets.enc.yaml

# 4. Update environment
export PROVISIONING_SECRET_SOURCE=sops
export PROVISIONING_SOPS_SECRETS_FILE=$(pwd)/provisioning/secrets.enc.yaml
export PROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning
```

### From SOPS to Vault

```
# 1. Decrypt SOPS file
sops -d provisioning/secrets.enc.yaml > /tmp/secrets.yaml

# 2. Import to Vault
vault kv put secret/ssh-keys/web-01/ubuntu key_content=@~/.ssh/provisioning_web01

# 3. Update environment
export PROVISIONING_SECRET_SOURCE=vault
export PROVISIONING_VAULT_ADDRESS=http://vault.example.com:8200
export PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...

# 4. Validate retrieval works
provisioning secrets validate-all
```

## Security Best Practices

### 1. Never Commit Secrets

```
# Add to .gitignore
echo "provisioning/secrets.enc.yaml" >> .gitignore
echo ".age/provisioning" >> .gitignore
echo ".vault-token" >> .gitignore
```

### 2. Rotate Keys Regularly

```
# SOPS: Rotate Age key
age-keygen -o ~/.age/provisioning.new
# Update all secrets with new key

# KMS: Enable automatic rotation
aws kms enable-key-rotation --key-id alias/provisioning

# Vault: Set TTL on secrets
vault write -f secret/metadata/ssh-keys/web-01/ubuntu \
    delete_version_after=2160h  # 90 days
```

### 3. Restrict Access

```
# SOPS: Protect Age key
chmod 600 ~/.age/provisioning

# KMS: Restrict IAM permissions
aws iam put-user-policy --user-name provisioning \
    --policy-name ProvisioningSecretsAccess \
    --policy-document file://kms-policy.json

# Vault: Use AppRole for applications
vault write auth/approle/role/provisioning \
    token_ttl=1h \
    secret_id_ttl=30m
```

### 4. Audit Logging

```
# KMS: Enable CloudTrail
aws cloudtrail put-event-selectors \
    --trail-name provisioning-trail \
    --event-selectors ReadWriteType=All

# Vault: Check audit logs
vault audit list

# SOPS: Version control (encrypted)
git log -p provisioning/secrets.enc.yaml
```

## Troubleshooting

### SOPS Issues

```
# Test Age decryption
sops -d provisioning/secrets.enc.yaml

# Verify Age key
age-keygen -l ~/.age/provisioning

# Regenerate if needed
rm ~/.age/provisioning
age-keygen -o ~/.age/provisioning
```

### KMS Issues

```
# Test AWS credentials
aws sts get-caller-identity

# Check KMS key permissions
aws kms describe-key --key-id alias/provisioning

# List secrets
aws secretsmanager list-secrets --filters Name=name,Values=provisioning
```

### Vault Issues

```
# Check Vault status
vault status

# Test authentication
vault token lookup

# List secrets
vault kv list secret/ssh-keys/

# Check audit logs
vault audit list
vault read sys/audit
```

## FAQ

**Q: Can I use multiple secret sources simultaneously?**
A: Yes, configure multiple sources and set `PROVISIONING_SECRET_SOURCE` to specify primary. If primary fails, manual fallback to secondary is supported.

**Q: What happens if secret retrieval fails?**
A: System logs the error and fails fast. No automatic fallback to local filesystem (for security).

**Q: Can I cache SSH keys?**
A: Currently not, keys are retrieved fresh for each operation. Use local caching at OS level (ssh-agent) if needed.

**Q: How do I rotate keys?**
A: Update the secret in your configured source (SOPS/KMS/Vault) and retrieve fresh on next operation.

**Q: Is local-dev mode secure?**
A: No - it's development only. Production requires SOPS/KMS/Vault.

## Architecture

```
SSH Operation
    ↓
SecretsManager (Nushell/Rust)
    ↓
[Detect Source]
    ↓
┌─────────────────────────────────────┐
│ SOPS          KMS      Vault   LocalDev
│ (Encrypted    (AWS KMS (Self-  (Filesystem
│  Secrets)     Service)  Hosted) Dev Only)
│
└─────────────────────────────────────┘
    ↓
Return SSH Key Path/Content
    ↓
SSH Operation Completes
```

## Integration with SSH Utilities

SSH operations automatically use secrets manager:

```
# Automatic secret retrieval
ssh-cmd-smart $settings $server false "command" $ip
# Internally:
#   1. Determine secret source
#   2. Retrieve SSH key for server.installer_user@ip
#   3. Execute SSH with retrieved key
#   4. Cleanup sensitive data

# Batch operations also integrate
ssh-batch-execute $servers $settings "command"
# Per-host: Retrieves key → executes → cleans up
```

---

**For Support**: See `docs/user/TROUBLESHOOTING_GUIDE.md`
**For Integration**: See `provisioning/core/nulib/lib_provisioning/platform/secrets.nu`