Paul SerbanSoftware Engineer

CQRS in AI systems: why separating reads from writes is the mental model prompt engineers have been missingHow a battle-tested software architecture pattern maps surprisingly cleanly onto inference pipelines, prompt design, and AI system boundaries

Introduction: The Architectural Debt of "Chat"

In the early days of any new technology, we favor simplicity over structure. For AI engineering, this manifested as the "monolithic prompt"—a single, massive string containing instructions, few-shot examples, and raw data context. While this works for a weekend prototype, it creates a maintenance nightmare in production. Developers often find themselves struggling with "prompt drift," where changing a single instruction to improve output format accidentally breaks the model's ability to reason over the provided data. This fragility stems from a fundamental lack of separation between different types of system operations.

The industry is currently transitioning from "AI wrappers" to "AI systems." As we build complex agents and RAG (Retrieval-Augmented Generation) pipelines, the need for a formal architectural mental model becomes urgent. Surprisingly, the solution isn't found in a new AI-native paper, but in a decades-old pattern from the world of distributed systems: Command Query Responsibility Segregation (CQRS). By viewing AI interactions through the lens of Commands (mutations) and Queries (reads), we can build systems that are significantly more observable, testable, and scalable.

The Problem: The "Everything, Everywhere" Context Window

Standard AI implementations often treat the LLM as a black-box processor that handles state management, data retrieval, and logic simultaneously. In a typical RAG flow, a single request might ask the model to: "Check the user's subscription status, summarize these three documents, and update the interaction log." This forces the model to act as a database coordinator, a summarizer, and a state machine all at once. Because the "Read" (fetching documents) and the "Write" (updating logs/status) are entangled in one execution path, the cognitive load on the model increases, leading to higher latency and more frequent hallucinations.

This entanglement also creates a massive hurdle for evaluation (Evals). When a system fails, it is difficult to discern if the failure occurred because the retrieval was poor (the Read), the instructions were ignored (the Logic), or the state update failed (the Write). In traditional software, we solved this by ensuring that an object used to display data didn't have the same schema or requirements as the object used to change data. Applying this to AI means recognizing that the data structure optimized for "feeding the context window" is rarely the same structure used for "storing the system truth."

Deep Technical Explanation: Mapping CQRS to AI

At its core, CQRS separates the data models used for reading (Queries) from the models used for updating (Commands). In an AI context, a Command is an instruction meant to change the state of the world or the conversation—for example, "Extract entities and save them to the CRM." A Query is the act of gathering the necessary context to inform a response—such as "Search the vector database for relevant whitepapers." When we separate these, we no longer send the entire database schema to the LLM; we send a highly optimized "Read Model" that contains only what is necessary for that specific turn of the conversation.

This separation allows for Asymmetric Scaling. In many AI systems, the Read path is much more frequent and computationally expensive (involving embedding searches and long-context processing) than the Write path (which might just be a structured JSON output). By decoupling them, you can optimize your "Read Model"—perhaps using a smaller, faster model or a cache—while reserving your most capable, expensive model for the "Command" logic that requires high-level reasoning. This mimics the traditional CQRS benefit where the read database is a denormalized, high-speed replica of the normalized write database.

Practical Implementation: A TypeScript Example

To implement CQRS in an AI agent, you should avoid a single "processPrompt" function. Instead, define separate handlers for Commands and Queries. In the example below, notice how the ProcessDocumentCommand focuses on data integrity and mutation, while the GetContextualSummaryQuery focuses on assembling the perfect context for the user.

// The Write Model: Optimized for Structured Extraction
interface ProcessDocumentCommand {
  type: 'PROCESS_DOCUMENT';
  payload: { rawText: string; documentId: string };
}

async function handleDocumentCommand(command: ProcessDocumentCommand) {
  const extractionPrompt = `Extract key entities from the following text and return JSON: ${command.payload.rawText}`;
  const response = await llm.generateStructured(extractionPrompt, entitySchema);
  
  // Update the "Source of Truth" (Write Database)
  await db.entities.upsertMany(response.entities);
  // Invalidate or update the Read Model (Vector Index)
  await vectorStore.reindex(command.payload.documentId);
}

// The Read Model: Optimized for Contextual Inference
interface GetContextualSummaryQuery {
  type: 'GET_SUMMARY';
  userId: string;
  topic: string;
}

async function handleSummaryQuery(query: GetContextualSummaryQuery) {
  // Fetch only the pre-processed "Read Model" data
  const context = await vectorStore.search(query.topic, { limit: 5 });
  
  const summaryPrompt = `Based on the following context, summarize the topic for the user: ${JSON.stringify(context)}`;
  return await llm.generate(summaryPrompt);
}

This approach ensures that the LLM performing the summary isn't also burdened with the logic of how entities should be structured or saved. The "Write" logic is encapsulated, meaning if your CRM schema changes, you only update the Command handler, leaving your Query/Summary logic completely untouched. This modularity is what allows AI systems to scale beyond simple demos.

Trade-offs and Pitfalls: The Cost of Complexity

The most immediate drawback of applying CQRS to AI is the increased boilerplate. For a simple chatbot, creating separate models for reads and writes is undoubtedly overkill. You introduce Eventual Consistency issues; for instance, if a user uploads a file (Command) and immediately asks a question about it (Query), the vector index might not have finished updating. Handling these race conditions requires more sophisticated engineering, such as implementing a temporary "hot cache" or using polling mechanisms.

Furthermore, there is the risk of "Model Fragmentation." If the Read Model and Write Model diverge too far, the AI might reason about data in a way that doesn't align with how that data is actually stored. This is particularly dangerous in AI because the "schema" is often natural language. Maintaining parity between what the "Command" handler extracts and what the "Query" handler retrieves requires rigorous versioning of prompts and embedding models.

Key Takeaways for Prompt Engineers

  1. Stop "Over-Prompting": Don't ask one prompt to both analyze data and decide how to save it. Break these into a "Logic" step and a "State Change" step.
  2. Optimize the Read Model: Your RAG context shouldn't just be raw chunks; it should be a "Read-optimized" version of your data (e.g., summaries or tagged snippets).
  3. Audit the Command Path: Use strict JSON schema validation for any prompt that acts as a "Command" to ensure system integrity.
  4. Decouple Models: Consider using a cheaper model (like GPT-4o-mini) for the "Read/Query" context assembly and a smarter model (like Claude 3.5 Sonnet) for the "Command/Reasoning" execution.
  5. Version Separately: Track changes to your Retrieval logic independently of your Generation logic.

Conclusion: Engineering the Future of AI

As we move toward autonomous agents, the "prompt" will increasingly be seen as a component of a larger system rather than the system itself. CQRS provides a robust framework for this evolution. By treating AI interactions as distinct reads and writes, we move away from unpredictable "magic boxes" and toward reliable, scalable software. This separation doesn't just make the code cleaner; it makes the AI smarter by allowing it to focus on one specific task at a time.

Next time you find yourself adding "And also make sure to..." to a 2,000-word prompt, stop and ask: Is this a Command or a Query? If it’s both, it’s time to refactor. Separating these concerns is the first step toward building AI that doesn't just work in a demo, but thrives in production.

Would you like me to generate a specific Python implementation of a CQRS-based RAG pipeline using LangGraph or Haystack?

References