Quickstart: Build a Memory-Enabled AI Assistant with RAG in a Weekend

Introduction

The promise of AI assistants has always been personalization—systems that remember your preferences, learn from past interactions, and provide contextually relevant responses. Yet most chatbot implementations today suffer from amnesia, treating every conversation as a fresh start and lacking awareness of domain-specific knowledge. Retrieval-Augmented Generation (RAG) solves this by combining large language models with external memory systems, enabling assistants to access relevant information on-demand rather than relying solely on parametric knowledge baked into model weights.

This guide presents a weekend-worthy implementation roadmap for building a memory-enabled AI assistant using RAG. We'll cover the essential components—document ingestion, vector-based retrieval, conversation state management, and observability—while maintaining a minimal architecture that can scale from prototype to production. By the end, you'll have a working system that can ingest documents, answer questions with source attribution, maintain conversation context, and provide visibility into retrieval quality. More importantly, you'll understand the engineering trade-offs that matter when moving beyond toy examples.

Understanding RAG and Memory Architecture

Retrieval-Augmented Generation represents a fundamental shift in how we build AI applications. Traditional fine-tuning approaches burn knowledge into model parameters, requiring expensive retraining cycles whenever information changes. RAG instead treats knowledge as data: documents are encoded into vector representations and stored in searchable indexes, then dynamically retrieved at query time to augment LLM prompts. This separation of knowledge from reasoning enables continuous updates, source attribution, and more reliable answers grounded in verifiable information rather than statistical patterns.

The architecture consists of two distinct phases: an offline ingestion pipeline and an online retrieval loop. During ingestion, source documents are split into chunks, converted to vector embeddings using an encoder model, and stored in a vector database alongside metadata. At query time, user questions are similarly embedded and used to search for semantically similar chunks via nearest-neighbor search. Retrieved context is then injected into the LLM prompt, allowing the model to generate answers based on specific source material rather than general training data.

Memory in this context operates at multiple layers. Short-term memory lives in conversation state—the rolling window of recent messages that provides dialogue coherence. Long-term memory resides in the vector store, containing all ingested documents that might be relevant across sessions. Effective RAG systems balance both: retrieving pertinent long-term knowledge while maintaining sufficient conversation context to handle follow-up questions and references. Getting this balance right requires careful prompt engineering, chunk sizing, and retrieval strategies that we'll explore in depth.

Core Components of a Memory-Enabled Assistant

A production-ready RAG assistant requires four foundational components: a document ingestion pipeline, a vector storage and retrieval layer, a conversation state manager, and an observability framework. Each component addresses a specific failure mode common in naive implementations. The ingestion pipeline ensures consistent document processing and embedding generation. The vector layer provides low-latency semantic search at scale. Conversation state prevents context window overflow while maintaining dialogue coherence. Observability exposes retrieval quality issues before users notice them.

The ingestion pipeline transforms unstructured source material into retrievable chunks. This involves more than simple text splitting—effective chunking requires understanding document structure, preserving semantic boundaries, and adding metadata that improves retrieval precision. A PDF whitepaper might be split by section headings with chapter metadata, while API documentation might be chunked by endpoint with method and path tags. The embedding model converts these chunks into dense vectors, typically 384 to 1536 dimensions, capturing semantic meaning in a form amenable to efficient nearest-neighbor search.

Vector storage presents unique engineering challenges distinct from traditional databases. Vector databases like Pinecone, Weaviate, Qdrant, or pgvector extension for PostgreSQL optimize for approximate nearest neighbor (ANN) search using algorithms like HNSW or IVF. The choice matters: HNSW offers excellent query performance at the cost of higher memory usage, while IVF trades some accuracy for better memory efficiency. For weekend projects, managed solutions reduce operational complexity, but understanding the underlying index structures helps debug retrieval quality issues and capacity planning decisions.

Conversation state management determines how much dialogue history to maintain and how to represent it efficiently. Naive approaches dump entire conversation histories into prompts, quickly exhausting context windows on multi-turn dialogues. Better strategies include sliding windows that retain recent messages, summarization that compresses older turns, and hybrid approaches that keep recent verbatim messages while summarizing earlier context. The right strategy depends on your use case: customer support might prioritize recent exchanges, while research assistants benefit from longer context retention with aggressive summarization.

Weekend Implementation Guide

Building a functional RAG assistant in a weekend requires ruthless scope management. The goal is a working end-to-end system demonstrating core concepts, not a production-hardened service. Start with a narrow knowledge domain—perhaps your team's internal documentation or a well-structured dataset like Wikipedia articles on a specific topic. Constrain the problem to a single document type (Markdown or plain text) to avoid format parsing complexity. Use managed services for infrastructure concerns: OpenAI or Anthropic for LLM access, a hosted vector database, and simple file-based conversation state. This foundation can be enhanced incrementally after validating the core architecture.

Day one focuses on the ingestion pipeline. Write a script that reads documents, splits them into chunks with overlap to preserve context across boundaries, generates embeddings using a model like OpenAI's text-embedding-3-small or sentence-transformers, and uploads vectors to your chosen database. Implement idempotency so re-running ingestion updates existing chunks rather than duplicating them. Add basic metadata tagging—document source, timestamps, and any categorical information useful for filtering retrieval results. By end of day one, you should have a populated vector store and a repeatable ingestion process.

Day two builds the retrieval and response loop. Create an API endpoint or CLI that accepts user queries, embeds them, performs similarity search against your vector store, constructs a prompt combining retrieved context with conversation history, and streams the LLM response. Start with synchronous request handling—streaming and caching optimizations come later. Implement basic conversation state by storing message histories in memory or a simple key-value store keyed by session ID. Add minimal observability: log query latency, retrieved chunk IDs, and token usage. By the end of day two, you have a working assistant that answers questions using your knowledge base while maintaining conversation context.

Building the Ingestion Pipeline

Document chunking strategy profoundly impacts retrieval quality. Chunks must be large enough to contain meaningful semantic units but small enough to fit efficiently in prompts alongside other context. A common starting point is 500-1000 tokens per chunk with 100-200 token overlap, but optimal values depend on your content structure. Technical documentation with clear section boundaries might use larger chunks aligned to headings, while conversational transcripts might need smaller chunks to isolate specific exchanges. The overlap ensures concepts mentioned near chunk boundaries appear in multiple chunks, improving recall for queries that match boundary regions.

Metadata enrichment during ingestion dramatically improves retrieval precision by enabling hybrid search strategies. Beyond basic source identifiers, consider adding structural metadata (section hierarchy, document type), temporal metadata (creation date, last modified), and content-derived tags (automatically extracted keywords, categories). Many vector databases support metadata filtering, allowing you to scope searches to specific document types or time ranges before computing semantic similarity. This reduces noise and improves relevance, especially in large knowledge bases where semantic similarity alone may surface tangentially related content.

import os
from typing import List, Dict
from openai import OpenAI
import tiktoken
from datetime import datetime

class DocumentIngestionPipeline:
    def __init__(self, vector_db_client, embedding_model="text-embedding-3-small"):
        self.client = OpenAI()
        self.vector_db = vector_db_client
        self.embedding_model = embedding_model
        self.encoder = tiktoken.get_encoding("cl100k_base")
        
    def chunk_document(
        self, 
        text: str, 
        chunk_size: int = 800, 
        overlap: int = 150
    ) -> List[Dict[str, any]]:
        """Split document into overlapping chunks with token counting."""
        tokens = self.encoder.encode(text)
        chunks = []
        
        start = 0
        while start < len(tokens):
            end = min(start + chunk_size, len(tokens))
            chunk_tokens = tokens[start:end]
            chunk_text = self.encoder.decode(chunk_tokens)
            
            chunks.append({
                "text": chunk_text,
                "start_token": start,
                "end_token": end,
                "token_count": len(chunk_tokens)
            })
            
            start += chunk_size - overlap
            
        return chunks
    
    def generate_embeddings(self, texts: List[str]) -> List[List[float]]:
        """Generate embeddings for multiple text chunks."""
        response = self.client.embeddings.create(
            model=self.embedding_model,
            input=texts
        )
        return [item.embedding for item in response.data]
    
    def ingest_document(
        self, 
        document_path: str, 
        metadata: Dict[str, any] = None
    ) -> Dict[str, any]:
        """Complete ingestion pipeline for a single document."""
        # Read document
        with open(document_path, 'r', encoding='utf-8') as f:
            content = f.read()
        
        # Chunk with overlap
        chunks = self.chunk_document(content)
        
        # Generate embeddings in batches
        chunk_texts = [c["text"] for c in chunks]
        embeddings = self.generate_embeddings(chunk_texts)
        
        # Prepare records with metadata
        records = []
        base_metadata = metadata or {}
        base_metadata.update({
            "source": document_path,
            "ingested_at": datetime.utcnow().isoformat(),
            "total_chunks": len(chunks)
        })
        
        for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
            record = {
                "id": f"{document_path}_{idx}",
                "vector": embedding,
                "metadata": {
                    **base_metadata,
                    "chunk_index": idx,
                    "token_count": chunk["token_count"]
                },
                "text": chunk["text"]
            }
            records.append(record)
        
        # Upsert to vector database (idempotent)
        self.vector_db.upsert(records)
        
        return {
            "document": document_path,
            "chunks_created": len(records),
            "total_tokens": sum(c["token_count"] for c in chunks)
        }

Implementing Retrieval and Context Injection

Retrieval strategy determines what information reaches your LLM and consequently the quality of generated responses. The simplest approach—embedding the user query and returning top-k similar chunks—works surprisingly well for many use cases. However, production systems often benefit from hybrid retrieval combining semantic similarity with keyword matching (BM25) or metadata filters. For example, a user asking "What changed in the latest release?" benefits from both semantic understanding of "change" and temporal filtering to recent documentation versions.

Query embedding requires careful consideration of the semantic gap between user questions and stored content. Users ask questions ("How do I authenticate?") while documents contain declarative statements ("Authentication requires an API key"). Some systems address this by query expansion—generating multiple phrasings or adding domain-specific terms before embedding. Others use asymmetric embeddings where queries and documents are encoded differently to optimize for matching questions to answers rather than statements to statements. For weekend implementations, symmetric encoders like text-embedding-3-small work well enough to validate the architecture.

Context injection is where retrieval meets generation. The retrieved chunks must be formatted into a prompt that clearly distinguishes between system instructions, retrieved knowledge, conversation history, and the current user query. A typical structure places system instructions first (defining behavior and constraints), followed by retrieved context marked as reference material, then recent conversation history for continuity, and finally the user's current question. Token budget allocation is critical: reserve sufficient tokens for the response while balancing retrieved context (typically 2-4 chunks of ~500 tokens each) against conversation history (last 5-10 turns).

import { OpenAI } from 'openai';
import { VectorDBClient } from './vector-db';

interface RetrievalResult {
  text: string;
  metadata: Record<string, any>;
  score: number;
}

interface Message {
  role: 'user' | 'assistant' | 'system';
  content: string;
}

class RAGAssistant {
  private openai: OpenAI;
  private vectorDB: VectorDBClient;
  private embeddingModel = 'text-embedding-3-small';
  private chatModel = 'gpt-4o-mini';

  constructor(vectorDB: VectorDBClient) {
    this.openai = new OpenAI();
    this.vectorDB = vectorDB;
  }

  async embedQuery(query: string): Promise<number[]> {
    const response = await this.openai.embeddings.create({
      model: this.embeddingModel,
      input: query,
    });
    return response.data[0].embedding;
  }

  async retrieveContext(
    query: string,
    topK: number = 4,
    filters?: Record<string, any>
  ): Promise<RetrievalResult[]> {
    const queryEmbedding = await this.embedQuery(query);
    
    const results = await this.vectorDB.query({
      vector: queryEmbedding,
      topK,
      filter: filters,
      includeMetadata: true,
    });

    return results.map((r: any) => ({
      text: r.text,
      metadata: r.metadata,
      score: r.score,
    }));
  }

  constructPrompt(
    query: string,
    retrievedContext: RetrievalResult[],
    conversationHistory: Message[]
  ): Message[] {
    // System instruction with role definition
    const systemMessage: Message = {
      role: 'system',
      content: `You are a helpful AI assistant with access to a knowledge base. 
Answer questions using the provided context. If the context doesn't contain 
relevant information, say so clearly. Always cite sources by referencing 
the document name when using retrieved information.`,
    };

    // Format retrieved context
    const contextText = retrievedContext
      .map(
        (ctx, idx) =>
          `[Source ${idx + 1}: ${ctx.metadata.source}]\n${ctx.text}\n`
      )
      .join('\n---\n');

    const contextMessage: Message = {
      role: 'system',
      content: `# Retrieved Context\n\n${contextText}`,
    };

    // Combine: system prompt + context + history + current query
    return [
      systemMessage,
      contextMessage,
      ...conversationHistory.slice(-10), // Last 10 turns
      { role: 'user', content: query },
    ];
  }

  async query(
    userQuery: string,
    conversationHistory: Message[] = [],
    retrievalFilters?: Record<string, any>
  ): Promise<{ response: string; sources: RetrievalResult[] }> {
    // Retrieve relevant context
    const retrievedContext = await this.retrieveContext(
      userQuery,
      4,
      retrievalFilters
    );

    // Construct prompt with context
    const messages = this.constructPrompt(
      userQuery,
      retrievedContext,
      conversationHistory
    );

    // Generate response
    const completion = await this.openai.chat.completions.create({
      model: this.chatModel,
      messages,
      temperature: 0.7,
      max_tokens: 800,
    });

    return {
      response: completion.choices[0].message.content || '',
      sources: retrievedContext,
    };
  }
}

Managing Conversation State

Conversation state management sits at the intersection of user experience and technical constraints. Users expect assistants to remember earlier parts of the dialogue, handling pronouns ("What about the other option you mentioned?") and building on established context. But LLMs have finite context windows—even long-context models become less effective as prompt length grows, and costs scale linearly with tokens. Effective state management compresses conversation history while preserving semantic continuity.

A practical approach for weekend implementations uses a sliding window with selective retention. Store the complete conversation history in your backend, but only send recent messages to the LLM—typically the last 5-10 turns depending on average message length. For longer conversations, prepend a summary of earlier turns generated by the LLM itself. This hybrid strategy keeps recent context verbatim (preserving exact phrasings and details) while compressing distant history. Implement this as a background task: after every N turns, asynchronously generate a summary of turns N-10 through N, then remove those raw messages from the active context window.

interface ConversationTurn {
  role: 'user' | 'assistant';
  content: string;
  timestamp: Date;
  metadata?: Record<string, any>;
}

class ConversationStateManager {
  private conversations: Map<string, ConversationTurn[]> = new Map();
  private maxActiveHistory = 10;
  private summaryThreshold = 20;

  addTurn(sessionId: string, turn: ConversationTurn): void {
    if (!this.conversations.has(sessionId)) {
      this.conversations.set(sessionId, []);
    }
    this.conversations.get(sessionId)!.push(turn);

    // Trigger summarization if needed
    if (this.conversations.get(sessionId)!.length >= this.summaryThreshold) {
      this.compressHistory(sessionId);
    }
  }

  getActiveHistory(sessionId: string): Message[] {
    const history = this.conversations.get(sessionId) || [];
    
    // Return last N turns for prompt construction
    return history.slice(-this.maxActiveHistory).map((turn) => ({
      role: turn.role,
      content: turn.content,
    }));
  }

  private async compressHistory(sessionId: string): Promise<void> {
    const history = this.conversations.get(sessionId)!;
    const toSummarize = history.slice(0, -this.maxActiveHistory);
    
    if (toSummarize.length < 5) return; // Not enough to compress

    // Generate summary of older turns
    const summaryPrompt = `Summarize the key points and context from this conversation:\n\n${
      toSummarize.map((t) => `${t.role}: ${t.content}`).join('\n')
    }`;

    const openai = new OpenAI();
    const summary = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: summaryPrompt }],
      max_tokens: 300,
    });

    // Replace old history with summary + recent turns
    const summaryTurn: ConversationTurn = {
      role: 'system',
      content: `Previous conversation summary: ${summary.choices[0].message.content}`,
      timestamp: new Date(),
      metadata: { type: 'summary', compressed_turns: toSummarize.length },
    };

    this.conversations.set(sessionId, [
      summaryTurn,
      ...history.slice(-this.maxActiveHistory),
    ]);
  }
}

Adding Observability and Monitoring

Observability in RAG systems differs from traditional application monitoring because failure modes are subtle. The system may appear to work—returning responses with normal latency—while providing irrelevant or incorrect information due to poor retrieval. Effective observability tracks both system health metrics (latency, error rates) and quality metrics (retrieval relevance, answer groundedness). For weekend implementations, focus on instrumenting three critical paths: embedding generation, vector search, and LLM completion.

Start with structured logging that captures the retrieval pipeline. For each query, log the user's question, the embedding latency, the retrieved chunk IDs with similarity scores, the constructed prompt token count, the LLM response time, and completion token usage. This creates an audit trail enabling post-hoc analysis: if users report poor answers, you can replay the retrieval step to see what context was actually provided. Include correlation IDs that link all operations for a single request, enabling distributed tracing even in a monolithic weekend prototype.

Retrieval quality metrics require a feedback mechanism. The simplest approach adds thumbs up/down buttons to responses, storing the user's rating alongside the query and retrieved chunk IDs. This creates a labeled dataset for evaluating retrieval precision: what percentage of "thumbs down" responses had low similarity scores? Are certain document types consistently rated poorly? More sophisticated approaches compute answer relevance automatically using a second LLM call that scores whether the response actually addresses the query, but manual feedback suffices for initial validation.

import logging
import json
import time
from typing import Dict, List, Optional
from dataclasses import dataclass, asdict
from datetime import datetime
import uuid

@dataclass
class QueryMetrics:
    query_id: str
    session_id: str
    query: str
    embedding_latency_ms: float
    retrieval_latency_ms: float
    retrieved_chunks: List[Dict]
    prompt_tokens: int
    completion_tokens: int
    llm_latency_ms: float
    total_latency_ms: float
    timestamp: str
    user_feedback: Optional[int] = None  # 1 for positive, -1 for negative
    
class RAGObservability:
    def __init__(self, log_file: str = "rag_metrics.jsonl"):
        self.log_file = log_file
        self.logger = logging.getLogger("rag_assistant")
        self.logger.setLevel(logging.INFO)
        
        # JSON Lines format for easy analysis
        handler = logging.FileHandler(log_file)
        handler.setFormatter(logging.Formatter('%(message)s'))
        self.logger.addHandler(handler)
    
    def create_query_id(self) -> str:
        return str(uuid.uuid4())
    
    def log_query_metrics(self, metrics: QueryMetrics) -> None:
        """Log structured query metrics as JSON Lines."""
        self.logger.info(json.dumps(asdict(metrics)))
    
    def record_user_feedback(self, query_id: str, rating: int) -> None:
        """Record user feedback for a specific query."""
        feedback_record = {
            "event_type": "user_feedback",
            "query_id": query_id,
            "rating": rating,
            "timestamp": datetime.utcnow().isoformat()
        }
        self.logger.info(json.dumps(feedback_record))
    
    def analyze_retrieval_quality(self, min_score_threshold: float = 0.7) -> Dict:
        """Analyze retrieval quality from logged metrics."""
        total_queries = 0
        low_score_retrievals = 0
        latency_sum = 0
        token_sum = 0
        
        with open(self.log_file, 'r') as f:
            for line in f:
                try:
                    record = json.loads(line)
                    if 'query_id' in record and 'retrieved_chunks' in record:
                        total_queries += 1
                        latency_sum += record['total_latency_ms']
                        token_sum += record['prompt_tokens'] + record['completion_tokens']
                        
                        # Check if any retrieved chunk has low similarity
                        max_score = max(
                            chunk.get('score', 0) 
                            for chunk in record['retrieved_chunks']
                        )
                        if max_score < min_score_threshold:
                            low_score_retrievals += 1
                except json.JSONDecodeError:
                    continue
        
        return {
            "total_queries": total_queries,
            "avg_latency_ms": latency_sum / max(total_queries, 1),
            "avg_tokens": token_sum / max(total_queries, 1),
            "low_confidence_retrieval_rate": low_score_retrievals / max(total_queries, 1),
        }

# Usage in RAG pipeline
observability = RAGObservability()

def handle_query_with_metrics(user_query: str, session_id: str) -> str:
    query_id = observability.create_query_id()
    start_time = time.time()
    
    # Embedding phase
    embed_start = time.time()
    query_embedding = generate_embedding(user_query)
    embed_latency = (time.time() - embed_start) * 1000
    
    # Retrieval phase
    retrieval_start = time.time()
    retrieved_chunks = vector_search(query_embedding, top_k=4)
    retrieval_latency = (time.time() - retrieval_start) * 1000
    
    # LLM generation phase
    prompt = construct_prompt(user_query, retrieved_chunks)
    llm_start = time.time()
    response = call_llm(prompt)
    llm_latency = (time.time() - llm_start) * 1000
    
    total_latency = (time.time() - start_time) * 1000
    
    # Log comprehensive metrics
    metrics = QueryMetrics(
        query_id=query_id,
        session_id=session_id,
        query=user_query,
        embedding_latency_ms=embed_latency,
        retrieval_latency_ms=retrieval_latency,
        retrieved_chunks=[
            {"id": c.id, "score": c.score, "source": c.metadata["source"]}
            for c in retrieved_chunks
        ],
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
        llm_latency_ms=llm_latency,
        total_latency_ms=total_latency,
        timestamp=datetime.utcnow().isoformat()
    )
    
    observability.log_query_metrics(metrics)
    
    return response.choices[0].message.content

Trade-offs and Pitfalls

Chunk size presents a fundamental trade-off between precision and recall. Small chunks (200-300 tokens) provide precise attribution and fit more discrete pieces of information in the same token budget, but risk fragmenting concepts that span multiple chunks. Large chunks (1000+ tokens) preserve more context and reduce the chance of splitting related information, but decrease the number of distinct sources you can retrieve and may include irrelevant material. The optimal size depends on your source material structure and query patterns—start with 500-800 tokens and adjust based on observed retrieval quality.

Embedding model selection impacts both cost and retrieval quality. Larger embedding dimensions (1536 for text-embedding-3-large vs 256 for text-embedding-3-small) capture more nuanced semantic distinctions but increase storage costs and query latency. For most applications, the smaller models provide excellent results at a fraction of the cost. However, specialized domains with subtle semantic distinctions (legal documents, medical literature) may benefit from larger embeddings. Run comparative evaluations on a sample of representative queries before committing to infrastructure that locks you into a specific dimension.

A common pitfall is neglecting the cold start problem. When users ask questions on topics not covered by your knowledge base, retrieval returns the "closest" matches even if they're completely irrelevant. Set a minimum similarity threshold (typically 0.7-0.8 for cosine similarity) below which the system should acknowledge it lacks relevant information rather than hallucinating based on tangentially related context. This trades off recall for precision, preventing confidently wrong answers that erode user trust more than admissions of ignorance.

Conversation history management creates subtle bugs. Storing the assistant's own retrieval-augmented responses in conversation history means future turns include that generated content, potentially compounding errors or hallucinations across multiple turns. Some systems store only user queries in long-term history, generating fresh responses based on current retrieval without carrying forward previous generations. Others include assistant responses but periodically validate them against source material. For weekend implementations, the simplest approach that maintains acceptable quality is to store everything but rely on fresh retrieval to ground each response in source documents.

Best Practices for Production

Caching dramatically improves both cost and latency for repeated queries. Implement semantic caching by storing query embeddings alongside their retrieved results and checking for similar recent queries before performing vector search. If a new query's embedding is within a small distance threshold of a cached query (cosine similarity > 0.95), return the cached results. Similarly, cache LLM responses for identical context-query pairs. This is particularly valuable for FAQ-style queries where multiple users ask essentially the same question. Use a TTL-based invalidation strategy to ensure cached results expire when underlying documents update.

Implement incremental ingestion rather than full rebuilds. As your knowledge base grows, re-embedding every document becomes prohibitively expensive. Design your ingestion pipeline to detect changed documents using checksums or modification timestamps, only re-processing deltas. Store document-to-chunk mappings so updating a document can efficiently delete old chunks and insert new ones. This enables continuous integration of new knowledge without scheduled maintenance windows—documentation updates can flow into the assistant within minutes.

import crypto from 'crypto';

interface CacheEntry {
  queryEmbedding: number[];
  results: RetrievalResult[];
  timestamp: Date;
}

class SemanticCache {
  private cache: Map<string, CacheEntry> = new Map();
  private ttlMs: number = 3600000; // 1 hour
  private similarityThreshold: number = 0.95;

  private computeSimilarity(vec1: number[], vec2: number[]): number {
    // Cosine similarity
    const dotProduct = vec1.reduce((sum, a, idx) => sum + a * vec2[idx], 0);
    const mag1 = Math.sqrt(vec1.reduce((sum, a) => sum + a * a, 0));
    const mag2 = Math.sqrt(vec2.reduce((sum, a) => sum + a * a, 0));
    return dotProduct / (mag1 * mag2);
  }

  private isExpired(entry: CacheEntry): boolean {
    return Date.now() - entry.timestamp.getTime() > this.ttlMs;
  }

  findSimilarQuery(
    queryEmbedding: number[]
  ): RetrievalResult[] | null {
    // Check cache for semantically similar recent queries
    for (const [_key, entry] of this.cache.entries()) {
      if (this.isExpired(entry)) {
        continue;
      }

      const similarity = this.computeSimilarity(
        queryEmbedding,
        entry.queryEmbedding
      );

      if (similarity >= this.similarityThreshold) {
        console.log(`Cache hit with similarity: ${similarity.toFixed(3)}`);
        return entry.results;
      }
    }
    return null;
  }

  store(
    query: string,
    queryEmbedding: number[],
    results: RetrievalResult[]
  ): void {
    const key = crypto.createHash('md5').update(query).digest('hex');
    this.cache.set(key, {
      queryEmbedding,
      results,
      timestamp: new Date(),
    });

    // Prune expired entries
    for (const [k, entry] of this.cache.entries()) {
      if (this.isExpired(entry)) {
        this.cache.delete(k);
      }
    }
  }
}

// Integration with RAG assistant
class CachedRAGAssistant extends RAGAssistant {
  private cache = new SemanticCache();

  async retrieveContext(
    query: string,
    topK: number = 4,
    filters?: Record<string, any>
  ): Promise<RetrievalResult[]> {
    // Generate query embedding
    const queryEmbedding = await this.embedQuery(query);

    // Check cache first
    const cachedResults = this.cache.findSimilarQuery(queryEmbedding);
    if (cachedResults) {
      return cachedResults;
    }

    // Cache miss - perform retrieval
    const results = await super.retrieveContext(query, topK, filters);

    // Store in cache
    this.cache.store(query, queryEmbedding, results);

    return results;
  }
}

Advanced Patterns for Scale

As your RAG system matures beyond the weekend prototype, several advanced patterns become relevant. Reranking improves retrieval precision by performing a two-stage search: first retrieve a large candidate set (k=20-50) using fast vector search, then rerank using a more sophisticated model that scores query-document pairs. Cross-encoders like those from sentence-transformers evaluate the relevance of each retrieved chunk to the specific query, often improving top-k precision by 10-20% compared to embedding similarity alone.

Query decomposition handles complex multi-part questions by breaking them into atomic sub-queries. A question like "Compare authentication methods in version 2 vs version 3" benefits from separate retrievals for each version's authentication documentation, then combining results. Implement this using an LLM-powered query planner that identifies sub-queries, executes them in parallel, and synthesizes results. This pattern significantly improves answer quality for comparative or multi-aspect questions at the cost of additional retrieval latency and token usage.

Analogies & Mental Models

Think of RAG as building an assistant with a well-organized reference library instead of memorizing every book. Traditional LLMs are like students who've memorized textbooks—they have vast knowledge but no way to verify facts or update information without retraining. RAG systems are like researchers with instant access to a searchable library: they retrieve relevant sources, cite them in answers, and incorporate new materials without relearning everything. The vector database is the card catalog, embeddings are the semantic indexing system, and retrieval is the reference librarian finding relevant texts.

Conversation state management resembles taking meeting notes. You don't transcribe every word from a three-hour meeting—you capture key decisions, action items, and critical context while summarizing less important details. Similarly, RAG conversation managers keep recent exchanges verbatim (like meeting notes from the last 10 minutes) while summarizing older context. This balances fidelity for immediate follow-ups with efficiency for long-running dialogues, just as good meeting notes balance completeness with usability.

Chunking strategy is like creating index entries for a technical manual. Too granular (indexing every sentence) creates noise and loses context. Too coarse (indexing whole chapters) makes finding specific information difficult. Effective chunking, like good indexing, groups related concepts together while maintaining clear boundaries. The overlap between chunks mirrors cross-references in indexes, ensuring concepts mentioned near section breaks remain discoverable regardless of where the user's query lands.

80/20 Insight

Twenty percent of RAG implementation effort produces eighty percent of the value, and that twenty percent focuses on retrieval quality. Specifically: chunk size tuning, metadata enrichment, and hybrid search strategies. A system with perfectly tuned chunking and rich metadata outperforms sophisticated architectures with poor retrieval by orders of magnitude. Before investing in rerankers, query decomposition, or advanced prompt engineering, ensure your baseline retrieval returns genuinely relevant chunks for typical queries. Run evaluations on 20-30 representative questions, manually reviewing retrieved chunks. If precision is below 80%, optimize chunking and metadata before adding complexity.

The corollary is that conversation state complexity matters far less than expected. A simple sliding window with the last 10 turns handles the vast majority of use cases. Sophisticated summarization, semantic compression, and context planning provide marginal improvements for most applications. Focus on retrieval first, then response quality, and only then optimize state management if specific user patterns demand it. This ordering prevents premature optimization while ensuring effort focuses on user-facing quality.

Key Takeaways

1. Start with naive retrieval and iterate based on data. Embed queries, retrieve top-k chunks by similarity, inject into prompts. This baseline often suffices, and observability will reveal when sophistication is actually needed rather than prematurely optimizing.

2. Instrument everything from day one. Log query embeddings, retrieved chunk IDs, similarity scores, and token counts. When users report issues, your logs should enable complete reconstruction of what context the LLM saw. Observability is not optional for RAG.

3. Metadata filtering eliminates entire classes of retrieval errors. Structural tags (document type, section), temporal tags (version, date), and categorical tags (product, topic) enable scoped searches that dramatically improve precision when combined with semantic similarity.

4. Manage context window budgets explicitly. Define token allocations: system prompt (200), retrieved context (2000), conversation history (1000), response (1000). Enforce these limits in code to prevent silent truncation that corrupts prompts or loses critical context.

5. Validate retrieval quality before optimizing generation. If the LLM receives irrelevant chunks, no amount of prompt engineering fixes the response. Build evaluation harnesses that let you quickly assess whether retrieved context actually answers test questions before tuning other components.

Conclusion

Building a memory-enabled AI assistant with RAG in a weekend is achievable by focusing on the minimal viable architecture: a document ingestion pipeline that chunks and embeds content, a vector store for semantic search, basic conversation state management, and structured logging for observability. This foundation provides genuine utility—answering domain-specific questions with source attribution while maintaining dialogue context—without the complexity of production-scale systems. More importantly, it creates a working baseline for data-driven optimization based on actual usage patterns rather than theoretical concerns.

The architecture presented here scales surprisingly well beyond weekend prototypes. The same ingestion, retrieval, and state management patterns underpin production RAG systems handling millions of queries, with differences primarily in infrastructure choices (managed vs self-hosted vector databases, caching layers, distributed tracing) rather than fundamental design. Start simple, instrument comprehensively, and let real-world usage data guide optimization efforts. The result is an assistant that provides reliable, grounded answers while maintaining conversation context—exactly what users expect from "memory-enabled" AI, delivered in a weekend of focused engineering.

References

OpenAI Embeddings API Documentation. OpenAI Platform Docs. https://platform.openai.com/docs/guides/embeddings
Anthropic Prompt Engineering Guide. Anthropic Documentation. https://docs.anthropic.com/claude/docs/prompt-engineering
Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Proceedings of NeurIPS 2020. https://arxiv.org/abs/2005.11401
Pinecone Vector Database Documentation. Pinecone Docs. https://docs.pinecone.io/
Weaviate Vector Search Documentation. Weaviate Docs. https://weaviate.io/developers/weaviate
LangChain Documentation - Retrieval. LangChain Python Docs. https://python.langchain.com/docs/modules/data_connection/
Sentence-Transformers Documentation. SBERT.net. https://www.sbert.net/
Malkov, Y. A., & Yashunin, D. A. (2018). "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." IEEE Transactions on Pattern Analysis and Machine Intelligence. https://arxiv.org/abs/1603.09320
OpenAI Tokenization Guide. OpenAI Cookbook. https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
Qdrant Vector Database Documentation. Qdrant Docs. https://qdrant.tech/documentation/
PostgreSQL pgvector Extension. GitHub Repository. https://github.com/pgvector/pgvector
Reimers, N., & Gurevych, I. (2019). "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of EMNLP 2019. https://arxiv.org/abs/1908.10084
Structured Logging Best Practices. Google Cloud Logging Documentation. https://cloud.google.com/logging/docs/structured-logging
Context Window Management Strategies. Anthropic Engineering Blog. https://www.anthropic.com/index/claude-2-1-prompting
Robertson, S., & Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval. Foundation for hybrid search understanding.