How to Use RAG Architecture for Better Information Retrieval

If you’re an AI engineer trying to understand and build with GenAI, RAG (Retrieval-Augmented Generation) is one of the most essential components to master. It’s the backbone of any LLM system that needs fresh, accurate, and context-aware outputs. Let’s break down how RAG works, step by step, from an engineering lens, not a hype one: 🧠 How RAG Works (Under the Hood) 1. Embed your knowledge base → Start with unstructured sources - docs, PDFs, internal wikis, etc. → Convert them into semantic vector representations using embedding models (e.g., OpenAI, Cohere, or HuggingFace models) → Output: N-dimensional vectors that preserve meaning across contexts 2. Store in a vector database → Use a vector store like Pinecone, Weaviate, or FAISS → Index embeddings to enable fast similarity search (cosine, dot-product, etc.) 3. Query comes in - embed that too → The user prompt is embedded using the same embedding model → Perform a top-k nearest neighbor search to fetch the most relevant document chunks 4. Context injection → Combine retrieved chunks with the user query → Format this into a structured prompt for the generation model (e.g., Mistral, Claude, Llama) 5. Generate the final output → LLM uses both the query and retrieved context to generate a grounded, context-rich response → Minimizes hallucinations and improves factuality at inference time 📚 What changes with RAG? Without RAG: 🧠 “I don’t have data on that.” With RAG: 🤖 “Based on [retrieved source], here’s what’s currently known…” Same model, drastically improved quality. 🔍 Why this matters You need RAG when: → Your data changes daily (support tickets, news, policies) → You can’t afford hallucinations (legal, finance, compliance) → You want your LLMs to access your private knowledge base without retraining It’s the most flexible, production-grade approach to bridge static models with dynamic information. 🛠️ Arvind and I are kicking off a hands-on workshop on RAG This first session is designed for beginner to intermediate practitioners who want to move beyond theory and actually build. Here’s what you’ll learn: → How RAG enhances LLMs with real-time, contextual data → Core concepts: vector DBs, indexing, reranking, fusion → Build a working RAG pipeline using LangChain + Pinecone → Explore no-code/low-code setups and real-world use cases If you're serious about building with LLMs, this is where you start. 📅 Save your seat and join us live: https://lnkd.in/gS_B7_7d

Many companies have started experimenting with simple RAG systems, probably as their first use case, to test the effectiveness of generative AI in extracting knowledge from unstructured data like PDFs, text files, and PowerPoint files. If you've used basic RAG architectures with tools like LlamaIndex or LangChain, you might have already encountered three key problems: 𝟭. 𝗜𝗻𝗮𝗱𝗲𝗾𝘂𝗮𝘁𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: Existing metrics fail to catch subtle errors like unsupported claims or hallucinations, making it hard to accurately assess and enhance system performance. 𝟮. 𝗗𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝘆 𝗛𝗮𝗻𝗱𝗹𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝗹𝗲𝘅 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀: Standard RAG methods often struggle to find and combine information from multiple sources effectively, leading to slower responses and less relevant results. 𝟯. 𝗦𝘁𝗿𝘂𝗴𝗴𝗹𝗶𝗻𝗴 𝘁𝗼 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗮𝗻𝗱 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: Basic RAG approaches often miss the deeper relationships between information pieces, resulting in incomplete or inaccurate answers that don't fully meet user needs. In this post I will introduce three useful papers to address these gaps: 𝟭. 𝗥𝗔𝗚𝗖𝗵𝗲𝗸𝗲𝗿: introduces a new framework for evaluating RAG systems with a focus on fine-grained, claim-level metrics. It proposes a comprehensive set of metrics: claim-level precision, recall, and F1 score to measure the correctness and completeness of responses; claim recall and context precision to evaluate the effectiveness of the retriever; and faithfulness, noise sensitivity, hallucination rate, self-knowledge reliance, and context utilization to diagnose the generator's performance. Consider using these metrics to help identify errors, enhance accuracy, and reduce hallucinations in generated outputs. 𝟮. 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗥𝗔𝗚: It uses a labeler and filter mechanism to identify and retain only the most relevant parts of retrieved information, reducing the need for repeated large language model calls. This iterative approach refines search queries efficiently, lowering latency and costs while maintaining high accuracy for complex, multi-hop questions. 𝟯. 𝗚𝗿𝗮𝗽𝗵𝗥𝗔𝗚: By leveraging structured data from knowledge graphs, GraphRAG methods enhance the retrieval process, capturing complex relationships and dependencies between entities that traditional text-based retrieval methods often miss. This approach enables the generation of more precise and context-aware content, making it particularly valuable for applications in domains that require a deep understanding of interconnected data, such as scientific research, legal documentation, and complex question answering. For example, in tasks such as query-focused summarization, GraphRAG demonstrates substantial gains by effectively leveraging graph structures to capture local and global relationships within documents. It's encouraging to see how quickly gaps are identified and improvements are made in the GenAI world.

TL;DR: RAG (Retrieval Augmented Generation) is the most common GenAI pattern but getting it work for enterprise use cases is not easy at all. With the latest release, Amazon Web Services (AWS) Bedrock’s knowledge bases (for RAG) maybe the best managed RAG offering that overcomes the most common RAG blockers. Naive RAG has 4 phases: 𝟎. 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠 – Create an (vector) index of data 𝟏. 𝐐𝐮𝐞𝐫𝐲 – User issues Query 𝟐. 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 – Data is retrieved based on query 𝟑. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 – Data is fed to LLM to generate a response. But naive RAG has 7 failure points: (https://lnkd.in/ehAqbYbj) 𝟏. 𝐌𝐢𝐬𝐬𝐢𝐧𝐠 𝐂𝐨𝐧𝐭𝐞𝐧𝐭 – When user query is not in index, it can hallucinate a response 𝟐. 𝐌𝐢𝐬𝐬𝐞𝐝 𝐭𝐡𝐞 𝐓𝐨𝐩 𝐑𝐚𝐧𝐤𝐞𝐝 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐬 - The answer to the question is in the document but did not rank highly enough to be returned. 𝟑. 𝐍𝐨𝐭 𝐢𝐧 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 - Docs with the answer were retrieved from the database but did not make it into the context for generating an answer. 𝟒. 𝐍𝐨𝐭 𝐄𝐱𝐭𝐫𝐚𝐜𝐭𝐞𝐝 - The answer is present in the context, but the LLM failed to extract out the correct answer. 𝟓. 𝐖𝐫𝐨𝐧𝐠 𝐅𝐨𝐫𝐦𝐚𝐭 - The question involved extracting information in a certain format such as a table or list & the LLM ignored the instruction. 𝟔. 𝐈𝐧𝐜𝐨𝐫𝐫𝐞𝐜𝐭 𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜𝐢𝐭𝐲 - The answer is returned in the response but is not specific enough or is too specific to address the query. 𝟕. 𝐈𝐧𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞 response Amazon Bedrock’s Knowledge Base (KB) has grown to address all the above & then some. Here is the latest that Bedrock offers at each RAG stage: • 𝐃𝐚𝐭𝐚 𝐒𝐨𝐮𝐫𝐜𝐞𝐬 – S3, Web, Salesforce, SharePoint, Confluence     • 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠 – Source documents/data is chunked for retrieval & chunking strategies can significantly impact quality. Bedrock supports multiple chunking techniques – 𝐅𝐢𝐱𝐞𝐝, 𝐇𝐢𝐞𝐫𝐚𝐫𝐜𝐡𝐢𝐜𝐚𝐥, 𝐒𝐞𝐦𝐚𝐧𝐭𝐢𝐜 & even 𝐂𝐮𝐬𝐭𝐨𝐦 𝐂𝐡𝐮𝐧𝐤𝐢𝐧𝐠 via Lambda(!!) • 𝐐𝐮𝐞𝐫𝐲 𝐑𝐞𝐟𝐨𝐫𝐦𝐮𝐥𝐚𝐭𝐢𝐨𝐧 – Bedrock takes a complex input query & breaks it into multiple sub-queries. These sub-queries will then separately go through their own retrieval steps to find relevant chunks. • 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 •  𝐇𝐲𝐛𝐫𝐢𝐝 𝐒𝐞𝐚𝐫𝐜𝐡 – Combine Keyword, Semantic or Hybrid search of data sources to improve retrieval quality •  𝐕𝐞𝐜𝐭𝐨𝐫 𝐃𝐁 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 – OpenSearch, Pinecone, Mongo, Redis & Aurora •  𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐏𝐚𝐫𝐬𝐢𝐧𝐠 – Bedrock provides the option to use FMs for parsing complex documents such as .pdf files with nested tables or text within images. •  𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐟𝐢𝐥𝐭𝐞𝐫𝐢𝐧𝐠 to limit search aperture •  𝐂𝐢𝐭𝐚𝐭𝐢𝐨𝐧 𝐓𝐫𝐚𝐜𝐤𝐢𝐧𝐠 when providing responses •  𝐂𝐨𝐧𝐭𝐞𝐱𝐭𝐮𝐚𝐥 𝐆𝐫𝐨𝐮𝐧𝐝𝐢𝐧𝐠 – Combined with Guardrails Bedrock can reduce hallucinations even further KBs are not perfect but if you want to RAG on AWS then KBs today are your best bet! (Listen to Matt Wood talk about KBs: https://bit.ly/3S57FOq)

Most people do not look beyond the basic RAG pipeline, and it rarely works out as expected! RAG is known to lack robustness due to the LLM weaknesses, but it doesn't mean we cannot build robust pipelines! Here is how we can improve them. The RAG pipeline, in its simplest form, is composed of a retriever and a generator. The user question is used to retrieve the database data that could be used as context to answer the question better. The retrieved data is used as context in a prompt for an LLM to answer the question. Instead of using the original user question as a query to the database, it is typical to rewrite the question for optimized retrieval. Instead of blindly returning the answer to the user, we better assess the generated answer. That is the idea behind Self-RAG. We can check for hallucinations and relevance to the question. If the model hallucinates, we are going to try again the generation, and if the answer doesn't address the question, we are going to restart the retrieval by rewriting the query. If the answer passes the validation, we can return it to the user. It might be better to provide feedback for the new retrieval and the new generation to be performed in a more educated manner. In the case we have too many iterations, we are going to assume that we just reach a state where the model will apologize for not being able to provide an answer to the question. When we are retrieving the documents, we are likely retrieving irrelevant documents, so it could be a good idea to filter only the relevant ones before providing them to the generator. Once the documents are filtered, it is likely that a lot of the information contained in the documents is irrelevant, so it is also good to extract only what could be useful to answer the question from the documents. This way, the generator will only see relevant information to answer the question. The assumption in typical RAG is that the question will be about the data stored in the database, but this is a very rigid assumption. We can use the idea behind Adaptive-RAG, where we are going to assess the question first and route to a datastore RAG, a websearch or a simple LLM. It is possible that we realize that none of the documents are actually relevant to the question, and we better reroute the question back to the web search. That is part of the idea behind Corrective RAG. If we reach the maximum of web search retries, we can give up and apologize to the user. Here is how I implemented this pipeline with LangGraph: https://lnkd.in/g8AAF7Fw

In organizations where RAG is already delivering big value, the best teams operate more like product builders than AI researchers. They don’t start with papers; they start with people. They talk to users, comb through logs, and reverse-engineer what success looks like from the end-user’s perspective. They ask: - What are people trying to do? - What business metric changes if they succeed? From there, they build prototypes that map AI behavior directly to business outcomes. It’s user intent first, model second. But even the best product thinking can't succeed if the foundation is weak and that foundation is content. Most enterprises are sitting on decades of documents, buried in SharePoint, Confluence, or Google Drive, with no consistent structure and little usable metadata. The best teams know that you can't retrieve what isn't there, and even if it's technically present, you can't reliably retrieve it unless it’s been tagged, cleaned, and organized for machine consumption. So they start there: restructuring knowledge bases, tagging documents with business-critical metadata, and fixing the information architecture before a single embedding is generated. When it comes time to build, they make a crucial tradeoff that many others miss: they prioritize precision over recall. In consumer AI, partial answers are better than nothing. BUT in enterprise, that logic flips. Getting the right answer 80% of the time beats giving some answer 100% of the time. Why? Because hallucinations destroy trust. So the best teams enforce tight scope: they limit the document corpus, use smaller language models when needed, and tune for predictability. This isn’t about building the most powerful general system but it’s about building the most reliable system for the task at hand. And perhaps most importantly, they treat RAG like a product, not AI. That means every interaction is logged, every poor answer is investigated, and every week there's a review: what got retrieved, why, and how to improve it. This is the product feedback loop, re-applied to AI systems. The goal isn’t model improvement, it’s system improvement and the system includes everything from content quality to user feedback to evaluation metrics. And then they iterate like it’s their core product because for the most forward-looking enterprises, it already is.