You shipped an AI chatbot. Users tried it. And now you're dealing with some combination of these complaints:
"It made something up and it sounded completely confident."
"It gave me information that's six months out of date."
"How do I know if this answer is actually correct?"
"Wait, can everyone see our internal documents through this thing?"
These aren't edge cases. They're the predictable failure modes of any chatbot built on a raw LLM without proper retrieval architecture. And they're all fixable.
Let me walk through each failure mode and the architectural pattern that addresses it.
Failure 1: Hallucination
The problem. LLMs generate plausible text. That's what they're trained to do. When they don't have relevant information, they don't say "I don't know." They construct an answer that sounds authoritative and is completely fabricated.
This isn't a bug. It's the fundamental architecture of language models. They're optimizing for coherent next-token prediction, not factual accuracy.
The fix: Ground every response in retrieved documents.
Retrieval-Augmented Generation works by searching a knowledge base for relevant documents before generating a response. The LLM receives those documents as context and generates an answer based on what it found, not what it "knows" from training.
The key architectural decision: constrain the model's response to the retrieved content. Your system prompt should explicitly instruct the model to only answer based on the provided context and to say "I don't have information on that" when the retrieved documents don't cover the question.
This doesn't eliminate hallucination entirely. Models can still misinterpret retrieved content or over-extrapolate. But it reduces the failure rate dramatically because the model is working from specific source material rather than its parametric memory.
What this looks like in practice. We built FARbot, a free chatbot for searching the Federal Acquisition Regulation, using this exact pattern. The FAR is a massive, complex regulatory document. Getting an answer wrong isn't just unhelpful, it could lead to compliance violations. Every FARbot response is grounded in specific FAR sections that were retrieved based on the user's question.
Failure 2: Stale and Outdated Information
The problem. LLMs have a training cutoff. GPT-4's knowledge ends at some point in the past. Claude's does too. If your chatbot is answering questions about current policies, recent product changes, or evolving regulations, the base model's knowledge is already wrong.
Fine-tuning helps marginally, but it's expensive, slow, and still creates a new cutoff date.
The fix: Decouple knowledge from the model.
RAG separates the knowledge layer from the reasoning layer. Your documents live in a vector database, indexed and searchable. When those documents change, you re-index them. The model's training data becomes irrelevant for domain-specific questions because it's always reading from your current document set.
The practical architecture:
Ingest pipeline. Documents are chunked, embedded, and stored in a vector database. We use Pinecone for production workloads, but Qdrant, Weaviate, and pgvector all work depending on your requirements.
Incremental updates. When documents change, you re-process only the changed documents. You don't need to re-embed your entire corpus.
Metadata timestamps. Attach last-updated timestamps to your chunks. Surface these in responses so users know how current the information is.
In FARbot, when FAR clauses are updated, we re-ingest the affected sections. Users always get answers based on the current regulation, not whatever version existed when the underlying model was trained.
Failure 3: No Source Citations
The problem. Your chatbot gives an answer. The user asks, "Where did you get that?" And there's no good response.
For internal tools, this erodes trust. For anything customer-facing, it's a liability. For government and regulated industries, it's a disqualifier.
The fix: Track and surface retrieval provenance.
Every RAG response should include the source documents that informed it. Not as an afterthought, but as a core feature of the response architecture.
This requires:
Chunk-level attribution. When the retrieval system returns relevant passages, maintain references to the original documents, sections, and page numbers. Pass these through the generation step.
Source panel in the UI. Don't bury citations in footnotes. Give them a dedicated, prominent place in the interface. Users should be able to click through to the original document.
Retrieval logs. Log which documents were retrieved for each query, their relevance scores, and which ones the model actually referenced in its response. This is invaluable for debugging answer quality and for audit purposes.
FARbot implements all three. Every answer includes a source panel showing the specific FAR sections that were retrieved. Users can see exactly which regulatory text informed the response. The retrieval logs record every search, so we can analyze patterns and improve retrieval quality over time.
Failure 4: No Access Control
The problem. You built a chatbot over your company's internal documents. Sales proposals, HR policies, financial reports, engineering specs. A summer intern asks it a question, and the RAG system happily retrieves from the CFO's confidential board presentation.
The LLM doesn't understand permissions. It doesn't know that Document A is for executives only. It retrieved the most semantically relevant chunks, regardless of who was asking.
The fix: Permission-aware retrieval.
Access control in RAG must happen at the retrieval layer, before the LLM ever sees the documents. This means:
Per-document (or per-collection) permissions. When documents are ingested, they're tagged with access control metadata. User roles, groups, classification levels, whatever your permission model requires.
Filtered vector search. When a user submits a query, the vector search includes a permission filter. The query is: "Find the most relevant documents that this specific user is authorized to access." Not: "Find the most relevant documents, and we'll filter the display later."
That distinction matters. If you filter after retrieval, the LLM has already seen unauthorized content and may reference it in the response even if you strip the citations. The filter must happen before content reaches the model.
Hierarchical access models. Most organizations need more than flat roles. Team-level access, project-level access, classification-level access. These all need to compose correctly with your retrieval filters.
In Knowledge Spaces, we implement this with per-collection RBAC that's enforced at the vector query layer. An analyst in Division A only retrieves from collections they're authorized to access. The model never sees content from Division B's restricted collections. This is also why we log 64+ audit events per interaction. When you're handling sensitive data, you need to prove that access controls are working, not just assert it.
The Architecture That Ties It All Together
These four fixes aren't independent features you bolt on. They're layers of a coherent retrieval architecture:
Ingestion layer. Documents are chunked, embedded, tagged with metadata (source, timestamps, permissions), and stored in a vector database with the right connectors for your data sources.
Retrieval layer. Queries are embedded, permission filters are applied, and semantically relevant chunks are retrieved with full provenance tracking.
Generation layer. The LLM receives retrieved context with explicit instructions to ground responses in that context, cite sources, and acknowledge gaps.
Presentation layer. Responses are displayed with source citations, confidence indicators, and links to original documents.
Observability layer. Every step is logged. Retrieval scores, model selection, token usage, permission evaluations, response latency. All queryable, all auditable.
When people ask why their chatbot fails, the answer is almost always that they skipped one or more of these layers. They went straight from "user question" to "LLM response" and hoped for the best.
Getting Started
If you have a failing chatbot, you don't need to rebuild from scratch. Start with the retrieval layer. Add a vector database, index your documents, and inject retrieved context into your existing prompts. That alone fixes the hallucination and staleness problems.
Then add source tracking. Then add permission filters. Each layer compounds the reliability of the one before it.
RAG isn't magic. It's plumbing. But it's the plumbing that makes the difference between a demo that impresses people in a meeting and a product that people actually trust with real work.
Jamie Thompson is the Founder and CEO of Sprinklenet AI, where he builds enterprise AI platforms for government and commercial clients. He writes weekly at newsletter.sprinklenet.com.
Top comments (0)