# Retrieval-Augmented Generation (RAG) System\n\n**Status**: ✅ Production-Ready (SurrealDB 1.5.0+, 22/22 tests passing)\n\nThe RAG system enables the AI service to access, retrieve, and reason over infrastructure documentation, schemas, and past configurations. This allows\nthe AI to generate contextually accurate infrastructure configurations and provide intelligent troubleshooting advice grounded in actual platform\nknowledge.\n\n## Architecture Overview\n\nThe RAG system consists of:\n\n1. **Document Store**: SurrealDB vector store with semantic indexing\n2. **Hybrid Search**: Vector similarity + BM25 keyword search\n3. **Chunk Management**: Intelligent document chunking for code and markdown\n4. **Context Ranking**: Relevance scoring for retrieved documents\n5. **Semantic Cache**: Deduplication of repeated queries\n\n## Core Components\n\n### 1. Vector Embeddings\n\nThe system uses embedding models to convert documents into vector representations:\n\n```\n┌─────────────────────┐\n│ Document Source     │\n│ (Markdown, Code)    │\n└──────────┬──────────┘\n           │\n           ▼\n┌──────────────────────────────────┐\n│ Chunking & Tokenization          │\n│ - Code-aware splits              │\n│ - Markdown aware                 │\n│ - Preserves context              │\n└──────────┬───────────────────────┘\n           │\n           ▼\n┌──────────────────────────────────┐\n│ Embedding Model                  │\n│ (OpenAI Ada, Anthropic, Local)   │\n└──────────┬───────────────────────┘\n           │\n           ▼\n┌──────────────────────────────────┐\n│ Vector Storage (SurrealDB)       │\n│ - Vector index                   │\n│ - Metadata indexed               │\n│ - BM25 index for keywords        │\n└──────────────────────────────────┘\n```\n\n### 2. SurrealDB Integration\n\nSurrealDB serves as the vector database and knowledge store:\n\n```\n# Configuration in provisioning/schemas/ai.ncl\nlet {\n  rag = {\n    enabled = true,\n    db_url = "surreal://localhost:8000",\n    namespace = "provisioning",\n    database = "ai_rag",\n    \n    # Collections for different document types\n    collections = {\n      documentation = {\n        chunking_strategy = "markdown",\n        chunk_size = 1024,\n        overlap = 256,\n      },\n      schemas = {\n        chunking_strategy = "code",\n        chunk_size = 512,\n        overlap = 128,\n      },\n      deployments = {\n        chunking_strategy = "json",\n        chunk_size = 2048,\n        overlap = 512,\n      },\n    },\n    \n    # Embedding configuration\n    embedding = {\n      provider = "openai",  # or "anthropic", "local"\n      model = "text-embedding-3-small",\n      cache_vectors = true,\n    },\n    \n    # Search configuration\n    search = {\n      hybrid_enabled = true,\n      vector_weight = 0.7,\n      keyword_weight = 0.3,\n      top_k = 5,  # Number of results to return\n      semantic_cache = true,\n    },\n  }\n}\n```\n\n### 3. Document Chunking\n\nIntelligent chunking preserves context while managing token limits:\n\n#### Markdown Chunking Strategy\n\n```\nInput Document: provisioning/docs/src/guides/from-scratch.md\n\nChunks:\n  [1] Header + first section (up to 1024 tokens)\n  [2] Next logical section + overlap with [1]\n  [3] Code examples preserve as atomic units\n  [4] Continue with overlap...\n\nEach chunk includes:\n  - Original section heading (for context)\n  - Content\n  - Source file and line numbers\n  - Metadata (doctype, category, version)\n```\n\n#### Code Chunking Strategy\n\n```\nInput Document: provisioning/schemas/main.ncl\n\nChunks:\n  [1] Top-level let binding + comments\n  [2] Function definition (atomic, preserves signature)\n  [3] Type definition (atomic, preserves interface)\n  [4] Implementation blocks with context overlap\n\nEach chunk preserves:\n  - Type signatures\n  - Function signatures\n  - Import statements needed for context\n  - Comments and docstrings\n```\n\n## Hybrid Search\n\nThe system implements dual search strategy for optimal results:\n\n### Vector Similarity Search\n\n```\n// Find semantically similar documents\nasync fn vector_search(query: &str, top_k: usize) -> Vec<Document> {\n    let embedding = embed(query).await?;\n    \n    // L2 distance in SurrealDB\n    db.query("\n        SELECT *, vector::similarity::cosine(embedding, $embedding) AS score\n        FROM documents\n        WHERE embedding <~> $embedding\n        ORDER BY score DESC\n        LIMIT $top_k\n    ")\n    .bind(("embedding", embedding))\n    .bind(("top_k", top_k))\n    .await\n}\n```\n\n**Use case**: Semantic understanding of intent\n- Query: "How to configure PostgreSQL"\n- Finds: Documents about database configuration, examples, schemas\n\n### BM25 Keyword Search\n\n```\n// Find documents with matching keywords\nasync fn keyword_search(query: &str, top_k: usize) -> Vec<Document> {\n    // BM25 full-text search in SurrealDB\n    db.query("\n        SELECT *, search::bm25(.) AS score\n        FROM documents\n        WHERE text @@ $query\n        ORDER BY score DESC\n        LIMIT $top_k\n    ")\n    .bind(("query", query))\n    .bind(("top_k", top_k))\n    .await\n}\n```\n\n**Use case**: Exact term matching\n- Query: "SurrealDB configuration"\n- Finds: Documents mentioning SurrealDB specifically\n\n### Hybrid Results\n\n```\nasync fn hybrid_search(\n    query: &str,\n    vector_weight: f32,\n    keyword_weight: f32,\n    top_k: usize,\n) -> Vec<Document> {\n    let vector_results = vector_search(query, top_k * 2).await?;\n    let keyword_results = keyword_search(query, top_k * 2).await?;\n    \n    let mut scored = HashMap::new();\n    \n    // Score from vector search\n    for (i, doc) in vector_results.iter().enumerate() {\n        *scored.entry(doc.id).or_insert(0.0) +=\n            vector_weight * (1.0 - (i as f32 / top_k as f32));\n    }\n    \n    // Score from keyword search\n    for (i, doc) in keyword_results.iter().enumerate() {\n        *scored.entry(doc.id).or_insert(0.0) +=\n            keyword_weight * (1.0 - (i as f32 / top_k as f32));\n    }\n    \n    // Return top-k by combined score\n    let mut results: Vec<_> = scored.into_iter().collect();\n| results.sort_by( | a, b | b.1.partial_cmp(&a.1).unwrap()); |\n| Ok(results.into_iter().take(top_k).map( | (id, _) | ...).collect()) |\n}\n```\n\n## Semantic Caching\n\nReduces API calls by caching embeddings of repeated queries:\n\n```\nstruct SemanticCache {\n    queries: Arc<DashMap<Vec<f32>, CachedResult>>,\n    similarity_threshold: f32,\n}\n\nimpl SemanticCache {\n    async fn get(&self, query: &str) -> Option<CachedResult> {\n        let embedding = embed(query).await?;\n        \n        // Find cached query with similar embedding\n        // (cosine distance < threshold)\n        for entry in self.queries.iter() {\n            let distance = cosine_distance(&embedding, entry.key());\n            if distance < self.similarity_threshold {\n                return Some(entry.value().clone());\n            }\n        }\n        None\n    }\n    \n    async fn insert(&self, query: &str, result: CachedResult) {\n        let embedding = embed(query).await?;\n        self.queries.insert(embedding, result);\n    }\n}\n```\n\n**Benefits**:\n- 50-80% reduction in embedding API calls\n- Identical queries return in <10ms\n- Similar queries reuse cached context\n\n## Ingestion Workflow\n\n### Document Indexing\n\n```\n# Index all documentation\nprovisioning ai index-docs provisioning/docs/src\n\n# Index schemas\nprovisioning ai index-schemas provisioning/schemas\n\n# Index past deployments\nprovisioning ai index-deployments workspaces/*/deployments\n\n# Watch directory for changes (development mode)\nprovisioning ai watch docs provisioning/docs/src\n```\n\n### Programmatic Indexing\n\n```\n// In ai-service on startup\nasync fn initialize_rag() -> Result<()> {\n    let rag = RAGSystem::new(&config.rag).await?;\n    \n    // Index documentation\n    let docs = load_markdown_docs("provisioning/docs/src")?;\n    for doc in docs {\n        rag.ingest_document(&doc).await?;\n    }\n    \n    // Index schemas\n    let schemas = load_nickel_schemas("provisioning/schemas")?;\n    for schema in schemas {\n        rag.ingest_schema(&schema).await?;\n    }\n    \n    Ok(())\n}\n```\n\n## Usage Examples\n\n### Query the RAG System\n\n```\n# Search for context-aware information\nprovisioning ai query "How do I configure PostgreSQL with encryption?"\n\n# Get configuration template\nprovisioning ai template "Describe production Kubernetes on AWS"\n\n# Interactive mode\nprovisioning ai chat\n> What are the best practices for database backup?\n```\n\n### AI Service Integration\n\n```\n// AI service uses RAG to enhance generation\nasync fn generate_config(user_request: &str) -> Result<String> {\n    // Retrieve relevant context\n    let context = rag.search(user_request, top_k=5).await?;\n    \n    // Build prompt with context\n    let prompt = build_prompt_with_context(user_request, &context);\n    \n    // Generate configuration\n    let config = llm.generate(&prompt).await?;\n    \n    // Validate against schemas\n    validate_nickel_config(&config)?;\n    \n    Ok(config)\n}\n```\n\n### Form Assistance Integration\n\n```\n// In typdialog-ai (JavaScript/TypeScript)\nasync function suggestFieldValue(fieldName, currentInput) {\n    // Query RAG for similar configurations\n    const context = await rag.search(\n        `Field: ${fieldName}, Input: ${currentInput}`,\n        { topK: 3, semantic: true }\n    );\n    \n    // Generate suggestion using context\n    const suggestion = await ai.suggest({\n        field: fieldName,\n        input: currentInput,\n        context: context,\n    });\n    \n    return suggestion;\n}\n```\n\n## Performance Characteristics\n\n|  | Operation | Time | Cache Hit |  |\n|  | ----------- | ------ | ----------- |  |\n|  | Vector embedding | 200-500ms | N/A |  |\n|  | Vector search (cold) | 300-800ms | N/A |  |\n|  | Keyword search | 50-200ms | N/A |  |\n|  | Hybrid search | 500-1200ms | <100ms cached |  |\n|  | Semantic cache hit | 10-50ms | Always |  |\n\n**Typical query flow**:\n1. Embedding: 300ms\n2. Vector search: 400ms\n3. Keyword search: 100ms\n4. Ranking: 50ms\n5. **Total**: ~850ms (first call), <100ms (cached)\n\n## Configuration\n\nSee [Configuration Guide](configuration.md) for detailed RAG setup:\n\n- LLM provider for embeddings\n- SurrealDB connection\n- Chunking strategies\n- Search weights and limits\n- Cache settings and TTLs\n\n## Limitations and Considerations\n\n### Document Freshness\n\n- RAG indexes static snapshots\n- Changes to documentation require re-indexing\n- Use watch mode during development\n\n### Token Limits\n\n- Large documents chunked to fit LLM context\n- Some context may be lost in chunking\n- Adjustable chunk size vs. context trade-off\n\n### Embedding Quality\n\n- Quality depends on embedding model\n- Domain-specific models perform better\n- Fine-tuning possible for specialized vocabularies\n\n## Monitoring and Debugging\n\n### Query Metrics\n\n```\n# View RAG search metrics\nprovisioning ai metrics show rag\n\n# Analysis of search quality\nprovisioning ai eval-rag --sample-queries 100\n```\n\n### Debug Mode\n\n```\n# In provisioning/config/ai.toml\n[ai.rag.debug]\nenabled = true\nlog_embeddings = true      # Log embedding vectors\nlog_search_scores = true   # Log relevance scores\nlog_context_used = true    # Log context retrieved\n```\n\n## Related Documentation\n\n- [Architecture](architecture.md) - AI system overview\n- [MCP Integration](mcp-integration.md) - RAG access via MCP\n- [Configuration](configuration.md) - RAG setup guide\n- [API Reference](api-reference.md) - RAG API endpoints\n- [ADR-015](../architecture/adr/adr-015-ai-integration-architecture.md) - Design decisions\n\n---\n\n**Last Updated**: 2025-01-13\n**Status**: ✅ Production-Ready\n**Test Coverage**: 22/22 tests passing\n**Database**: SurrealDB 1.5.0+