# Troubleshooting Guide\n\nThis comprehensive troubleshooting guide helps you diagnose and resolve common issues with Infrastructure Automation.\n\n## What You'll Learn\n\n- Common issues and their solutions\n- Diagnostic commands and techniques\n- Error message interpretation\n- Performance optimization\n- Recovery procedures\n- Prevention strategies\n\n## General Troubleshooting Approach\n\n### 1. Identify the Problem\n\n```\n# Check overall system status\nprovisioning env\nprovisioning validate config\n\n# Check specific component status\nprovisioning show servers --infra my-infra\nprovisioning taskserv list --infra my-infra --installed\n```\n\n### 2. Gather Information\n\n```\n# Enable debug mode for detailed output\nprovisioning --debug <command>\n\n# Check logs and errors\nprovisioning show logs --infra my-infra\n```\n\n### 3. Use Diagnostic Commands\n\n```\n# Validate configuration\nprovisioning validate config --detailed\n\n# Test connectivity\nprovisioning provider test aws\nprovisioning network test --infra my-infra\n```\n\n## Installation and Setup Issues\n\n### Issue: Installation Fails\n\n**Symptoms:**\n\n- Installation script errors\n- Missing dependencies\n- Permission denied errors\n\n**Diagnosis:**\n\n```\n# Check system requirements\nuname -a\ndf -h\nwhoami\n\n# Check permissions\nls -la /usr/local/\nsudo -l\n```\n\n**Solutions:**\n\n#### Permission Issues\n\n```\n# Run installer with sudo\nsudo ./install-provisioning\n\n# Or install to user directory\n./install-provisioning --prefix=$HOME/provisioning\nexport PATH="$HOME/provisioning/bin:$PATH"\n```\n\n#### Missing Dependencies\n\n```\n# Ubuntu/Debian\nsudo apt update\nsudo apt install -y curl wget tar build-essential\n\n# RHEL/CentOS\nsudo dnf install -y curl wget tar gcc make\n```\n\n#### Architecture Issues\n\n```\n# Check architecture\nuname -m\n\n# Download correct architecture package\n# x86_64: Intel/AMD 64-bit\n# arm64: ARM 64-bit (Apple Silicon)\nwget https://releases.example.com/provisioning-linux-x86_64.tar.gz\n```\n\n### Issue: Command Not Found\n\n**Symptoms:**\n\n```\nbash: provisioning: command not found\n```\n\n**Diagnosis:**\n\n```\n# Check if provisioning is installed\nwhich provisioning\nls -la /usr/local/bin/provisioning\n\n# Check PATH\necho $PATH\n```\n\n**Solutions:**\n\n```\n# Add to PATH\nexport PATH="/usr/local/bin:$PATH"\n\n# Make permanent (add to shell profile)\necho 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc\nsource ~/.bashrc\n\n# Create symlink if missing\nsudo ln -sf /usr/local/provisioning/core/nulib/provisioning /usr/local/bin/provisioning\n```\n\n### Issue: Nushell Plugin Errors\n\n**Symptoms:**\n\n```\nPlugin not found: nu_plugin_kcl\nPlugin registration failed\n```\n\n**Diagnosis:**\n\n```\n# Check Nushell version\nnu --version\n\n# Check KCL installation (required for nu_plugin_kcl)\nkcl version\n\n# Check plugin registration\nnu -c "version | get installed_plugins"\n```\n\n**Solutions:**\n\n```\n# Install KCL CLI (required for nu_plugin_kcl)\n# Download from: https://github.com/kcl-lang/cli/releases\n\n# Re-register plugins\nnu -c "plugin add /usr/local/provisioning/plugins/nu_plugin_kcl"\nnu -c "plugin add /usr/local/provisioning/plugins/nu_plugin_tera"\n\n# Restart Nushell after plugin registration\n```\n\n## Configuration Issues\n\n### Issue: Configuration Not Found\n\n**Symptoms:**\n\n```\nConfiguration file not found\nFailed to load configuration\n```\n\n**Diagnosis:**\n\n```\n# Check configuration file locations\nprovisioning env | grep config\n\n# Check if files exist\nls -la ~/.config/provisioning/\nls -la /usr/local/provisioning/config.defaults.toml\n```\n\n**Solutions:**\n\n```\n# Initialize user configuration\nprovisioning init config\n\n# Create missing directories\nmkdir -p ~/.config/provisioning\n\n# Copy template\ncp /usr/local/provisioning/config-examples/config.user.toml ~/.config/provisioning/config.toml\n\n# Verify configuration\nprovisioning validate config\n```\n\n### Issue: Configuration Validation Errors\n\n**Symptoms:**\n\n```\nConfiguration validation failed\nInvalid configuration value\nMissing required field\n```\n\n**Diagnosis:**\n\n```\n# Detailed validation\nprovisioning validate config --detailed\n\n# Check specific sections\nprovisioning config show --section paths\nprovisioning config show --section providers\n```\n\n**Solutions:**\n\n#### Path Configuration Issues\n\n```\n# Check base path exists\nls -la /path/to/provisioning\n\n# Update configuration\nnano ~/.config/provisioning/config.toml\n\n# Fix paths section\n[paths]\nbase = "/correct/path/to/provisioning"\n```\n\n#### Provider Configuration Issues\n\n```\n# Test provider connectivity\nprovisioning provider test aws\n\n# Check credentials\naws configure list  # For AWS\nupcloud-cli config  # For UpCloud\n\n# Update provider configuration\n[providers.aws]\ninterface = "CLI"  # or "API"\n```\n\n### Issue: Interpolation Failures\n\n**Symptoms:**\n\n```\nInterpolation pattern not resolved: {{env.VARIABLE}}\nTemplate rendering failed\n```\n\n**Diagnosis:**\n\n```\n# Test interpolation\nprovisioning validate interpolation test\n\n# Check environment variables\nenv | grep VARIABLE\n\n# Debug interpolation\nprovisioning --debug validate interpolation validate\n```\n\n**Solutions:**\n\n```\n# Set missing environment variables\nexport MISSING_VARIABLE="value"\n\n# Use fallback values in configuration\nconfig_value = "{{env.VARIABLE || 'default_value'}}"\n\n# Check interpolation syntax\n# Correct: {{env.HOME}}\n# Incorrect: ${HOME} or $HOME\n```\n\n## Server Management Issues\n\n### Issue: Server Creation Fails\n\n**Symptoms:**\n\n```\nFailed to create server\nProvider API error\nInsufficient quota\n```\n\n**Diagnosis:**\n\n```\n# Check provider status\nprovisioning provider status aws\n\n# Test connectivity\nping api.provider.com\ncurl -I https://api.provider.com\n\n# Check quota\nprovisioning provider quota --infra my-infra\n\n# Debug server creation\nprovisioning --debug server create web-01 --infra my-infra --check\n```\n\n**Solutions:**\n\n#### API Authentication Issues\n\n```\n# AWS\naws configure list\naws sts get-caller-identity\n\n# UpCloud\nupcloud-cli account show\n\n# Update credentials\naws configure  # For AWS\nexport UPCLOUD_USERNAME="your-username"\nexport UPCLOUD_PASSWORD="your-password"\n```\n\n#### Quota/Limit Issues\n\n```\n# Check current usage\nprovisioning show costs --infra my-infra\n\n# Request quota increase from provider\n# Or reduce resource requirements\n\n# Use smaller instance types\n# Reduce number of servers\n```\n\n#### Network/Connectivity Issues\n\n```\n# Test network connectivity\ncurl -v https://api.aws.amazon.com\ncurl -v https://api.upcloud.com\n\n# Check DNS resolution\nnslookup api.aws.amazon.com\n\n# Check firewall rules\n# Ensure outbound HTTPS (port 443) is allowed\n```\n\n### Issue: SSH Access Fails\n\n**Symptoms:**\n\n```\nConnection refused\nPermission denied\nHost key verification failed\n```\n\n**Diagnosis:**\n\n```\n# Check server status\nprovisioning server list --infra my-infra\n\n# Test SSH manually\nssh -v user@server-ip\n\n# Check SSH configuration\nprovisioning show servers web-01 --infra my-infra\n```\n\n**Solutions:**\n\n#### Connection Issues\n\n```\n# Wait for server to be fully ready\nprovisioning server list --infra my-infra --status\n\n# Check security groups/firewall\n# Ensure SSH (port 22) is allowed\n\n# Use correct IP address\nprovisioning show servers web-01 --infra my-infra | grep ip\n```\n\n#### Authentication Issues\n\n```\n# Check SSH key\nls -la ~/.ssh/\nssh-add -l\n\n# Generate new key if needed\nssh-keygen -t ed25519 -f ~/.ssh/provisioning_key\n\n# Use specific key\nprovisioning server ssh web-01 --key ~/.ssh/provisioning_key --infra my-infra\n```\n\n#### Host Key Issues\n\n```\n# Remove old host key\nssh-keygen -R server-ip\n\n# Accept new host key\nssh -o StrictHostKeyChecking=accept-new user@server-ip\n```\n\n## Task Service Issues\n\n### Issue: Service Installation Fails\n\n**Symptoms:**\n\n```\nService installation failed\nPackage not found\nDependency conflicts\n```\n\n**Diagnosis:**\n\n```\n# Check service prerequisites\nprovisioning taskserv check kubernetes --infra my-infra\n\n# Debug installation\nprovisioning --debug taskserv create kubernetes --infra my-infra --check\n\n# Check server resources\nprovisioning server ssh web-01 --command "free -h && df -h" --infra my-infra\n```\n\n**Solutions:**\n\n#### Resource Issues\n\n```\n# Check available resources\nprovisioning server ssh web-01 --command "\n    echo 'Memory:' && free -h\n    echo 'Disk:' && df -h\n    echo 'CPU:' && nproc\n" --infra my-infra\n\n# Upgrade server if needed\nprovisioning server resize web-01 --plan larger-plan --infra my-infra\n```\n\n#### Package Repository Issues\n\n```\n# Update package lists\nprovisioning server ssh web-01 --command "\n    sudo apt update && sudo apt upgrade -y\n" --infra my-infra\n\n# Check repository connectivity\nprovisioning server ssh web-01 --command "\n    curl -I https://download.docker.com/linux/ubuntu/\n" --infra my-infra\n```\n\n#### Dependency Issues\n\n```\n# Install missing dependencies\nprovisioning taskserv create containerd --infra my-infra\n\n# Then install dependent service\nprovisioning taskserv create kubernetes --infra my-infra\n```\n\n### Issue: Service Not Running\n\n**Symptoms:**\n\n```\nService status: failed\nService not responding\nHealth check failures\n```\n\n**Diagnosis:**\n\n```\n# Check service status\nprovisioning taskserv status kubernetes --infra my-infra\n\n# Check service logs\nprovisioning taskserv logs kubernetes --infra my-infra\n\n# SSH and check manually\nprovisioning server ssh web-01 --command "\n    sudo systemctl status kubernetes\n    sudo journalctl -u kubernetes --no-pager -n 50\n" --infra my-infra\n```\n\n**Solutions:**\n\n#### Configuration Issues\n\n```\n# Reconfigure service\nprovisioning taskserv configure kubernetes --infra my-infra\n\n# Reset to defaults\nprovisioning taskserv reset kubernetes --infra my-infra\n```\n\n#### Port Conflicts\n\n```\n# Check port usage\nprovisioning server ssh web-01 --command "\n    sudo netstat -tulpn | grep :6443\n    sudo ss -tulpn | grep :6443\n" --infra my-infra\n\n# Change port configuration or stop conflicting service\n```\n\n#### Permission Issues\n\n```\n# Fix permissions\nprovisioning server ssh web-01 --command "\n    sudo chown -R kubernetes:kubernetes /var/lib/kubernetes\n    sudo chmod 600 /etc/kubernetes/admin.conf\n" --infra my-infra\n```\n\n## Cluster Management Issues\n\n### Issue: Cluster Deployment Fails\n\n**Symptoms:**\n\n```\nCluster deployment failed\nPod creation errors\nService unavailable\n```\n\n**Diagnosis:**\n\n```\n# Check cluster status\nprovisioning cluster status web-cluster --infra my-infra\n\n# Check Kubernetes cluster\nprovisioning server ssh master-01 --command "\n    kubectl get nodes\n    kubectl get pods --all-namespaces\n" --infra my-infra\n\n# Check cluster logs\nprovisioning cluster logs web-cluster --infra my-infra\n```\n\n**Solutions:**\n\n#### Node Issues\n\n```\n# Check node status\nprovisioning server ssh master-01 --command "\n    kubectl describe nodes\n" --infra my-infra\n\n# Drain and rejoin problematic nodes\nprovisioning server ssh master-01 --command "\n    kubectl drain worker-01 --ignore-daemonsets\n    kubectl delete node worker-01\n" --infra my-infra\n\n# Rejoin node\nprovisioning taskserv configure kubernetes --infra my-infra --servers worker-01\n```\n\n#### Resource Constraints\n\n```\n# Check resource usage\nprovisioning server ssh master-01 --command "\n    kubectl top nodes\n    kubectl top pods --all-namespaces\n" --infra my-infra\n\n# Scale down or add more nodes\nprovisioning cluster scale web-cluster --replicas 3 --infra my-infra\nprovisioning server create worker-04 --infra my-infra\n```\n\n#### Network Issues\n\n```\n# Check network plugin\nprovisioning server ssh master-01 --command "\n    kubectl get pods -n kube-system | grep cilium\n" --infra my-infra\n\n# Restart network plugin\nprovisioning taskserv restart cilium --infra my-infra\n```\n\n## Performance Issues\n\n### Issue: Slow Operations\n\n**Symptoms:**\n\n- Commands take very long to complete\n- Timeouts during operations\n- High CPU/memory usage\n\n**Diagnosis:**\n\n```\n# Check system resources\ntop\nhtop\nfree -h\ndf -h\n\n# Check network latency\nping api.aws.amazon.com\ntraceroute api.aws.amazon.com\n\n# Profile command execution\ntime provisioning server list --infra my-infra\n```\n\n**Solutions:**\n\n#### Local System Issues\n\n```\n# Close unnecessary applications\n# Upgrade system resources\n# Use SSD storage if available\n\n# Increase timeout values\nexport PROVISIONING_TIMEOUT=600  # 10 minutes\n```\n\n#### Network Issues\n\n```\n# Use region closer to your location\n[providers.aws]\nregion = "us-west-1"  # Closer region\n\n# Enable connection pooling/caching\n[cache]\nenabled = true\n```\n\n#### Large Infrastructure Issues\n\n```\n# Use parallel operations\nprovisioning server create --infra my-infra --parallel 4\n\n# Filter results\nprovisioning server list --infra my-infra --filter "status == 'running'"\n```\n\n### Issue: High Memory Usage\n\n**Symptoms:**\n\n- System becomes unresponsive\n- Out of memory errors\n- Swap usage high\n\n**Diagnosis:**\n\n```\n# Check memory usage\nfree -h\nps aux --sort=-%mem | head\n\n# Check for memory leaks\nvalgrind provisioning server list --infra my-infra\n```\n\n**Solutions:**\n\n```\n# Increase system memory\n# Close other applications\n# Use streaming operations for large datasets\n\n# Enable garbage collection\nexport PROVISIONING_GC_ENABLED=true\n\n# Reduce concurrent operations\nexport PROVISIONING_MAX_PARALLEL=2\n```\n\n## Network and Connectivity Issues\n\n### Issue: API Connectivity Problems\n\n**Symptoms:**\n\n```\nConnection timeout\nDNS resolution failed\nSSL certificate errors\n```\n\n**Diagnosis:**\n\n```\n# Test basic connectivity\nping 8.8.8.8\ncurl -I https://api.aws.amazon.com\nnslookup api.upcloud.com\n\n# Check SSL certificates\nopenssl s_client -connect api.aws.amazon.com:443 -servername api.aws.amazon.com\n```\n\n**Solutions:**\n\n#### DNS Issues\n\n```\n# Use alternative DNS\necho 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf\n\n# Clear DNS cache\nsudo systemctl restart systemd-resolved  # Ubuntu\nsudo dscacheutil -flushcache             # macOS\n```\n\n#### Proxy/Firewall Issues\n\n```\n# Configure proxy if needed\nexport HTTP_PROXY=http://proxy.company.com:9090\nexport HTTPS_PROXY=http://proxy.company.com:9090\n\n# Check firewall rules\nsudo ufw status  # Ubuntu\nsudo firewall-cmd --list-all  # RHEL/CentOS\n```\n\n#### Certificate Issues\n\n```\n# Update CA certificates\nsudo apt update && sudo apt install ca-certificates  # Ubuntu\nbrew install ca-certificates                         # macOS\n\n# Skip SSL verification (temporary)\nexport PROVISIONING_SKIP_SSL_VERIFY=true\n```\n\n## Security and Encryption Issues\n\n### Issue: SOPS Decryption Fails\n\n**Symptoms:**\n\n```\nSOPS decryption failed\nAge key not found\nInvalid key format\n```\n\n**Diagnosis:**\n\n```\n# Check SOPS configuration\nprovisioning sops config\n\n# Test SOPS manually\nsops -d encrypted-file.ncl\n\n# Check Age keys\nls -la ~/.config/sops/age/keys.txt\nage-keygen -y ~/.config/sops/age/keys.txt\n```\n\n**Solutions:**\n\n#### Missing Keys\n\n```\n# Generate new Age key\nage-keygen -o ~/.config/sops/age/keys.txt\n\n# Update SOPS configuration\nprovisioning sops config --key-file ~/.config/sops/age/keys.txt\n```\n\n#### Key Permissions\n\n```\n# Fix key file permissions\nchmod 600 ~/.config/sops/age/keys.txt\nchown $(whoami) ~/.config/sops/age/keys.txt\n```\n\n#### Configuration Issues\n\n```\n# Update SOPS configuration in ~/.config/provisioning/config.toml\n[sops]\nuse_sops = true\nkey_search_paths = [\n    "~/.config/sops/age/keys.txt",\n    "/path/to/your/key.txt"\n]\n```\n\n### Issue: Access Denied Errors\n\n**Symptoms:**\n\n```\nPermission denied\nAccess denied\nInsufficient privileges\n```\n\n**Diagnosis:**\n\n```\n# Check user permissions\nid\ngroups\n\n# Check file permissions\nls -la ~/.config/provisioning/\nls -la /usr/local/provisioning/\n\n# Test with sudo\nsudo provisioning env\n```\n\n**Solutions:**\n\n```\n# Fix file ownership\nsudo chown -R $(whoami):$(whoami) ~/.config/provisioning/\n\n# Fix permissions\nchmod -R 755 ~/.config/provisioning/\nchmod 600 ~/.config/provisioning/config.toml\n\n# Add user to required groups\nsudo usermod -a -G docker $(whoami)  # For Docker access\n```\n\n## Data and Storage Issues\n\n### Issue: Disk Space Problems\n\n**Symptoms:**\n\n```\nNo space left on device\nWrite failed\nDisk full\n```\n\n**Diagnosis:**\n\n```\n# Check disk usage\ndf -h\ndu -sh ~/.config/provisioning/\ndu -sh /usr/local/provisioning/\n\n# Find large files\nfind /usr/local/provisioning -type f -size +100M\n```\n\n**Solutions:**\n\n```\n# Clean up cache files\nrm -rf ~/.config/provisioning/cache/*\nrm -rf /usr/local/provisioning/.cache/*\n\n# Clean up logs\nfind /usr/local/provisioning -name "*.log" -mtime +30 -delete\n\n# Clean up temporary files\nrm -rf /tmp/provisioning-*\n\n# Compress old backups\ngzip ~/.config/provisioning/backups/*.yaml\n```\n\n## Recovery Procedures\n\n### Configuration Recovery\n\n```\n# Restore from backup\nprovisioning config restore --backup latest\n\n# Reset to defaults\nprovisioning config reset\n\n# Recreate configuration\nprovisioning init config --force\n```\n\n### Infrastructure Recovery\n\n```\n# Check infrastructure status\nprovisioning show servers --infra my-infra\n\n# Recover failed servers\nprovisioning server create failed-server --infra my-infra\n\n# Restore from backup\nprovisioning restore --backup latest --infra my-infra\n```\n\n### Service Recovery\n\n```\n# Restart failed services\nprovisioning taskserv restart kubernetes --infra my-infra\n\n# Reinstall corrupted services\nprovisioning taskserv delete kubernetes --infra my-infra\nprovisioning taskserv create kubernetes --infra my-infra\n```\n\n## Prevention Strategies\n\n### Regular Maintenance\n\n```\n# Weekly maintenance script\n#!/bin/bash\n\n# Update system\nprovisioning update --check\n\n# Validate configuration\nprovisioning validate config\n\n# Check for service updates\nprovisioning taskserv check-updates\n\n# Clean up old files\nprovisioning cleanup --older-than 30d\n\n# Create backup\nprovisioning backup create --name "weekly-$(date +%Y%m%d)"\n```\n\n### Monitoring Setup\n\n```\n# Set up health monitoring\n#!/bin/bash\n\n# Check system health every hour\n0 * * * * /usr/local/bin/provisioning health check || echo "Health check failed" | mail -s "Provisioning Alert" admin@company.com\n\n# Weekly cost reports\n0 9 * * 1 /usr/local/bin/provisioning show costs --all | mail -s "Weekly Cost Report" finance@company.com\n```\n\n### Best Practices\n\n1. **Configuration Management**\n   - Version control all configuration files\n   - Use check mode before applying changes\n   - Regular validation and testing\n\n2. **Security**\n   - Regular key rotation\n   - Principle of least privilege\n   - Audit logs review\n\n3. **Backup Strategy**\n   - Automated daily backups\n   - Test restore procedures\n   - Off-site backup storage\n\n4. **Documentation**\n   - Document custom configurations\n   - Keep troubleshooting logs\n   - Share knowledge with team\n\n## Getting Additional Help\n\n### Debug Information Collection\n\n```\n#!/bin/bash\n# Collect debug information\n\necho "Collecting provisioning debug information..."\n\nmkdir -p /tmp/provisioning-debug\ncd /tmp/provisioning-debug\n\n# System information\nuname -a > system-info.txt\nfree -h >> system-info.txt\ndf -h >> system-info.txt\n\n# Provisioning information\nprovisioning --version > provisioning-info.txt\nprovisioning env >> provisioning-info.txt\nprovisioning validate config --detailed > config-validation.txt 2>&1\n\n# Configuration files\ncp ~/.config/provisioning/config.toml user-config.toml 2>/dev/null || echo "No user config" > user-config.toml\n\n# Logs\nprovisioning show logs > system-logs.txt 2>&1\n\n# Create archive\ncd /tmp\ntar czf provisioning-debug-$(date +%Y%m%d_%H%M%S).tar.gz provisioning-debug/\n\necho "Debug information collected in: provisioning-debug-*.tar.gz"\n```\n\n### Support Channels\n\n1. **Built-in Help**\n\n   ```bash\n   provisioning help\n   provisioning help <command>\n   ```\n\n1. **Documentation**\n   - User guides in `docs/user/`\n   - CLI reference: `docs/user/cli-reference.md`\n   - Configuration guide: `docs/user/configuration.md`\n\n2. **Community Resources**\n   - Project repository issues\n   - Community forums\n   - Documentation wiki\n\n3. **Enterprise Support**\n   - Professional services\n   - Priority support\n   - Custom development\n\nRemember: When reporting issues, always include the debug information collected above and specific error messages.