## **I love thinking in frameworks.** It’s the best way to break down complex systems into something practical. I also prefer cutting straight to the point—so let’s do exactly that for a typical retrieval-augmented generation (RAG) pipeline. At a high level, a RAG pipeline revolves around three core components: Ingestion, Retrieval, and Generation. To understand how these pieces fit together, we’ll explore each independently, starting with a high-level overview before diving into the sub-components, tools, strategies, and practical considerations within each. Lastly, we’ll walk through an example that ties everything together. But before we get too far, let me address something that always bugs me about technical blogs: the way they always throw around specialized terms, assuming everyone just knows what they mean. What’s even worse? Flipping between synonyms for the same concept. To avoid falling victim to my own annoyances, here’s a dictionary of terms to keep things clear. If I miss a synonym or two, just send me hate mail (don’t do that, but maybe let me know). ### Dictionary of Terms - **Retrieval-Augmented Generation (RAG) System:** A system design pattern that integrates retrieval and generation components to enhance an LLM’s performance by incorporating external knowledge. It typically consists of ingestion, retrieval, and generation phases, often leveraging vector databases for efficient search. - **Source Document:** Any raw piece of information ingested into the system. Examples include PDFs, web pages, spreadsheets, emails, Slack threads, or even database entries. - **Source Node:** A smaller, chunked section of a source document, optimized for retrieval and downstream generation tasks. - **Vector Embedding (Dense Embedding):** A high-dimensional representation where most values are non-zero, generated by transformer models. Used for semantic search, it captures meaning and context - **Sparse Embedding:** A vector representation with mostly zero values, optimized for keyword search. Generated by algorithms such as SPARTA or BLADE. - **Embedding Model:** A machine learning model, often based on transformers, used to generate embeddings. Examples include OpenAI’s Ada-002, SBERT, or SPARTA. - **Vector Search (Semantic Search):** A search method that uses vector embeddings to find results based on semantic similarity, retrieving information aligned with the meaning and context of a query. - **Keyword Search:** A search method that uses sparse embeddings to identify matches based on exact terms in the query, often powered by algorithms like BM25. - **Hybrid Search:** Combines **dense** and **sparse** **embeddings** to balance semantic understanding and keyword relevance in search. - **Fine-Tuning:** The process of training a pre-trained model on specific domain data to improve its performance for a specialized task. - **K-Nearest Neighbor Algorithm (KNN):** A basic algorithm that finds the K closest points to a query point by measuring similarity or distance (e.g., cosine similarity or Euclidean distance). It’s straightforward but can be slow for large datasets since it compares the query to every single point. - **Approximate Nearest Neighbor Algorithm (ANN):** A faster alternative to KNN that skips finding exact matches and instead focuses on finding points that are *close enough* to the query. ANN uses smarter techniques such as HNSW or Product Quantization to save time. - **Hybrid Search Weighting:** The strategy of assigning different importance scores to sparse and dense retrieval methods in hybrid search. This allows fine-tuning of search results to better balance keyword precision and semantic understanding. - **Namespace Partitioning:** A technique in vector databases that organizes documents into separate retrieval spaces based on domain, user access, or data segmentation strategies. - **Sharding:** A method for distributing vector database storage across multiple nodes to improve scalability and reduce query latency. Often used in large-scale, multi-tenant RAG systems. - **Function Calling (legacy term: Tool Use):** A capability in modern LLMs that allows them to call external functions provided by the user. Instead of just generating text, the model can return structured API calls or trigger predefined actions in a system. Common use cases include database updates, API requests, and multi-agent orchestration. - **Context Injection:** The process of expanding or modifying a query by adding relevant information before retrieval. This can improve search performance and ensure that the retrieved results are more aligned with the original intent. - **HyDE (Hypothetical Document Embeddings):** A retrieval technique that generates a hypothetical response based on a query and embeds it to improve search performance. It helps bridge gaps between ambiguous queries and structured knowledge. - **Rerankers:** Models or algorithms that reorder retrieved results based on relevance to the query. Examples include transformer-based rerankers (ColBERT, Cohere Rerank) and rule-based ranking heuristics. - **Metadata Filtering:** The process of narrowing down retrieval results by applying constraints on metadata fields such as timestamps, document sources, or authorship. - **Long-Context Reordering:** A post-processing technique that rearranges retrieved documents to ensure that the most relevant context appears at the beginning or end of the input, optimizing LLM performance. Now that’s out of the way, let’s start with Ingestion and dive right in: ## Ingestion Ingestion is the process of transforming any raw source of information—what I call source documents—into individual knowledge entries, or what I refer to as source nodes. These source documents can be anything: PDFs, Slack messages, web pages, books, source code, spreadsheets, database entries, email threads, or virtually any other raw information you need to work with. Ingestion relies on a combination of tools and strategies. At a high level, here’s what that includes: ### **1. Source Document Aggregation and Parsing Pipelines** This is where raw data from various formats and platforms is pulled into the system, either as raw content in a usable format or as contextually enriched information. Think PDF parsers for extracting text from files, web scrapers for grabbing HTML, image processors for pulling text or contextual cues from visuals, video ingestors for summarizing content, code ingestors for parsing repositories, or email extractors for breaking down threads. Wow, okay, that’s probably way too many examples—but you get the idea. The goal here is to capture every potential source of knowledge and transform it into a format that’s ready for your use case. ### **2. Source Document Chunking Strategies** This is the step where source documents are transformed into source nodes, breaking larger pieces of content into smaller, more manageable, and retrievable sections. But why is chunking even necessary? Imagine you’re at a bookstore (or scrolling online) looking for information on how to bake sourdough bread. Instead of reading an entire cookbook cover to cover, you’d naturally focus on specific sections or recipes—like “starter preparation” or “kneading techniques”—that are directly relevant to what you need right now. Think about how inefficient it would be to skim through the entire book just to learn how to complete one part of a single recipe. In a RAG system, chunking serves the same purpose. When you submit a query such as *"how to bake sourdough bread,"* the system should optimally minimize the search space to be as relevant as possible without overfitting. Instead of searching the entire source document, it should compare the query against smaller, more targeted source nodes, making the search space more focused and efficient. The way documents are chunked depends on the content type and the use case. Basic strategies might involve splitting content by characters, sentences, or paragraphs. More advanced techniques, like embedding-based clustering, group semantically related content to improve precision. More sophisticated methods may even use LLMs to create contextually aware chunks that adapt to the structure and meaning of the document, ensuring the most relevant information is surfaced during retrieval. For one of the most fundamental reads in this space, check out Greg Kamradt’s article on the [5 levels of text splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb). It provides a high-level overview of increasingly complex approaches to chunking text, from basic rule-based methods to advanced, agentic-driven strategies. There’s a ton of nuance here, especially when you start tweaking hyperparameters that can completely change how chunking plays out. Things like chunk overlap (how much content carries over between chunks), semantic similarity thresholds (used in embedding-based clustering), and a whole lot more that we’re not gonna dive into right now. Your key takeaway from this section is that chunking is far from a one-size-fits-all solution. The best approach depends entirely on your data, your end use case, and—let’s be real—you’re probably gonna have to run evals on different strategies and hyperparameters to actually dial it in. ### **3. Source Node Augmentation Strategies** Immediately after the chunking step, source nodes often undergo an enrichment process designed to optimize them for retrieval and generation. This process ensures the nodes remain highly searchable, contextually relevant, and strategically aligned with potential user queries. Key enrichment methods include hypothetical question generation strategies, which create synthetic questions that the source node effectively answers, improving symmetry between a user query and the searchable nodes. Node Summarization strategies distill the key points within a node to ensure it’s easily retrieved for broader tasks that require concise overviews or summaries. Lastly, topic assignment strategies categorize nodes into high-level themes, enabling more targeted retrieval for queries tied to specific concepts or subjects. Beyond these enhancements, hard metadata—such as creation date, author, source document ID, content type, or even logs of past successful queries—can be incorporated to give each source node a significant edge in search flexibility. For example, picture a situation where you’re highly confident that the user’s query is specific to a given author. Why would you waste time searching across all of your source nodes when you can immediately filter down to those associated with this author? By narrowing the scope upfront, you make the retrieval process faster and greatly improve the likelihood that the returned documents are accurate and aligned with the user’s intent. ### **4. Embedding Model Overview and Strategies** Before we even start this section, let’s clear something up—I’m talking specifically about dense embeddings here, not sparse embeddings (check the dictionary of terms if you need a refresher). Remember, embeddings transform raw data into high-dimensional vectors that capture meaning, relationships, and context. In a retrieval-augmented generation (RAG) pipeline, these embeddings are what allow us to search and retrieve relevant information efficiently, even when the exact words in a query don’t match the words in a document. Choosing the right embedding model is a deep topic on its own. To really get into it, we’d have to talk about dimensionality, sequence length, model size, and a whole lot more. That’s a bit beyond the scope of this article, so instead, I’ll point you to two great resources: - This [article](https://www.pinecone.io/learn/series/rag/embedding-models-rundown/) from Pinecone for a solid overview of embedding models. - The [Massive Text Embedding Leaderboard (MTEB)](https://huggingface.co/spaces/mteb/leaderboard) for a ranking of the best-performing models. ### **Embedding Model Strategies** Now, let’s focus on two high-level strategies: 1. What should we embed? (deciding which data gets converted into embeddings) 2. Fine-tuning an embedding model (optimizing for domain-specific retrieval). ### **What Should We Embed?** Arguably even more important as picking the right embedding model is deciding what actually gets embedded. This can have a massive impact on retrieval quality and efficiency. - Should embeddings focus only on the raw node content? - Should enriched elements—like topics, summaries, or hypothetical questions—be embedded alongside the raw content, or even replace it entirely? - Should we embed only specific enriched content, like hypothetical questions, while ignoring everything else? - What about hard metadata like creation dates or author names? Could embedding these improve retrieval? If so, how should they be formatted to make search as effective as possible? ### **Fine-tuning An Embedding Model** The decision to fine-tune an embedding model on specific knowledge sources or rely on a pre-trained base model depends largely on the domain’s complexity and how much domain-specific knowledge is necessary to understand and process its concepts accurately. Alright, maybe I went a little too far into the weeds there—so what am I actually saying? Let’s look at an example. Say you’re building a RAG system for med students to help them study. If you’re using a base embedding model that hasn’t been fine-tuned on medical data, it’s probably gonna struggle to accurately capture the semantic meaning of the ingested course materials. The reason is actually surprisingly simple—it hasn’t been exposed to the domain-specific terminology enough during its original training to fully understand the nuances needed for context-rich tasks. Think about a freshman med student trying to explain a complex organic chemistry concept versus a senior who’s spent years studying it. All else being equal, the senior is gonna do a much better job because they’ve been exposed to the material way more times. Embedding models fine-tuned on domain-specific examples work the same way. However, fine-tuning an embedding model is often impractical due to constraints like limited labeled query data, a lack of machine learning expertise, or the high costs of training and deployment. For an interesting read on an alternative approach, check out this [article](https://research.trychroma.com/embedding-adapters) by [ChromaDB](https://www.trychroma.com/). It explores how applying a linear transformation to only the query vector—on top of an existing embedding model—can improve retrieval accuracy even with a limited dataset. ### **5. Database Selection & Architectural Design Strategies** Selecting the right vector database relies on understanding its underlying architecture and capabilities. A database optimized for approximate nearest neighbor (ANN) accuracy and retrieval is non-negotiable for a highly scalable RAG system. This often involves advanced indexing techniques like HNSW (Hierarchical Navigable Small World) or Product Quantization, which drastically improve performance and mitigate latency issues as embeddings scale. In my personal opinion, an often overlooked consideration is hybrid search. It’s important to assess whether the database supports hybrid search natively and how it combines dense and sparse embeddings to implement this functionality under the hood. I’ll dive deeper into hybrid search later and why [Pinecone](https://pinecone.io/) is an excellent example of this in practice. Key database architecture features like namespaces for organizing data, sharding for reducing search space and latency, and serverless architectures for decoupling storage from compute can make a huge difference. These components play a vital role in controlling costs, improving data organization, and ensuring your system can scale—especially in multi-tenant applications. ## Retrieval Now that your source documents have been transformed into enriched, optimized source nodes and stored in a vector database, let's tackle the next phase: retrieval. This is where we fetch the most relevant source nodes for any task presented to an LLM-based system. Like ingestion, retrieval combines various tools and strategies. Here's what that involves at a high level: ### **1. Search Query Augmentation** The retrieval process always begins with a query, which is the starting point for finding relevant information. An LLM would actually handle this initial step much better than a human. Why? People tend to write vague and imprecise queries—partly due to laziness (guilty), partly due to not understanding effective querying strategies. Let's look at an example: when a user interacts with a RAG-based chatbot built for the Python `requests` library documentation to learn about implementing proxies, their query might look like: > *Proxies in Python requests? How do I use a proxy? Proxy setup?* > While the intent is clear, the phrasing is vague and disconnected from the documentation's structure. An LLM, on the other hand, could refine this into a precise query, such as: *"Retrieve documentation examples for implementing proxies using the `requests` library, including parameter details and code snippets."* This approach ensures the system retrieves relevant, actionable information with minimal ambiguity. Query augmentation works similarly by refining rough, human-generated queries into structured, retrieval-friendly versions using an LLM. Techniques like context injection, Hypothetical Document Embeddings (HyDE), and sub-question decomposition align the query with the database’s structure, effectively closing the gap between user query and intent. It's like having an LLM ask the right question from the start. ### 2. Raw Retrieval Strategies When I say raw retrieval strategies, I’m referring to the method used to pull source nodes from your vector database. This step is critical because these nodes provide the context—or at least the seeds of context—that your LLM will rely on to answer a query or complete a task. Let’s start with pure semantic search, arguably the most naive approach for many vector database setups. Semantic search uses dense embeddings and similarity metrics, like cosine similarity, to identify source nodes that are semantically closest in meaning to a query. While it excels at capturing linguistic subtleties and broader intent, it often struggles when it needs to make direct references to specific keywords or phrases. Then there’s keyword search, which relies on sparse embeddings and algorithms like Best Match 25 (commonly referred to as BM25). BM25 ranks documents based on term frequency (TF) and inverse document frequency (IDF), normalized by document length. It’s incredibly effective for exact matches and structured queries and outperforms in scenarios where precise language is critical. For instance, in highly technical datasets, BM25 might outperform semantic search by surfacing source nodes that explicitly reference the exact terms in a query. However, it lacks the broader contextual understanding that semantic search provides, making it less effective for ambiguous queries Both dense vector search (semantic) and sparse keyword search (BM25) have clear strengths and weaknesses. As highlighted in this Pinecone technical blog post: > *Vector search unlocks incredible and intelligent retrieval but struggles to adapt to new domains. Whereas traditional search can cope with new domains but is fundamentally limited to a set performance level.* > So, what if we could combine the best of both worlds? That’s exactly where hybrid search comes in—and why I think choosing a vector database with native support for it, like [Pinecone](https://pinecone.io), should be a top consideration for anyone building RAG systems. Hybrid search leverages the strengths of both dense and sparse vector embeddings, effectively compensating for each other’s weaknesses. Dense embeddings are great at capturing semantic meaning and broader intent. On the other hand, sparse embeddings, rooted in traditional lexical matching methods like BM25, excel at pinpointing exact keyword relevance. (To clarify, BM25 itself doesn’t produce sparse embeddings; rather, sparse embeddings may come from models like SPARTA or SPLADE, which encode term-document relevance into sparse vectors.) Now, since I promised not to go too deep, I’ll save my detailed breakdown of how Pinecone implements hybrid search for another article (yes, I wrote it—Karn made me cut it because I went *way* too deep, but *whatever*). I’m also going to skip over other advanced strategies like recursive retrieval, hierarchical retrieval, metadata filtering, and agentic retrieval systems. While these are incredibly powerful, I think we’ve covered enough here to give you a strong foundation on raw retrieval strategies and how they fit into RAG pipelines. ### 3. Postprocessors Postprocessing refers to the process of refining and potentially augmenting the retrieved source nodes and the content ultimately passed to the LLM during the generation step. It’s an expansive topic, encompassing a range of techniques designed to enhance the quality, relevance, and safety of the retrieved data. Rerankers are a major component in postprocessing that deserve their own deep dive. They reorder retrieved results based on query alignment with a simple goal: *given X results for this query, return the Y best ones*. Rerankers can take various forms, from rule-based systems that apply predefined criteria to sort and filter results, to agentic rerankers that leverage LLMs to analyze the initial query, evaluate returned results, and optimize rankings. Additionally, transformer-based rerankers—such as Cohere or ColBERT—leverage dedicated pre-trained transformer models to process the search results and generate an optimized, reordered result set. Beyond reranking, metadata replacement strategies refine the retrieval-to-generation pipeline by swapping out the data used for retrieval with specific metadata fields during generation—such as the augmented context approaches we discussed earlier (for a refresher, see *Source Node Augmentation Strategies*). For example, assume we leverage the synthetic question generation strategy, where retrieval is based on comparing the user query vector against a list of generated questions. However, just because these questions were effective for retrieval doesn’t mean they should be passed to the LLM for generation. Why? Well, it's pretty obvious—a question that aligns with the query won’t necessarily contain the actual information needed to answer it. Instead, we might replace it with more relevant metadata—such as a content summary or the raw content itself. This strategy allows us to optimize retrieval independently from generation, ensuring each step gets the most useful input. For a simple example of metadata replacement, check out this [Metadata Replacement Demo](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo/) from [LlamaIndex](https://docs.llamaindex.ai/en/stable/). Aside from rerankers and metadata replacement strategies, here are other ways to refine the context passed to the LLM for generation: 1. **Long-context reordering** tackles the challenge of information loss in extended contexts, where models often struggle to retain details buried in the middle. Research shows key information is best placed at the start or end of the context, while performance declines as length increases (see this [research paper](https://arxiv.org/abs/2307.03172) for more information). This technique restructures retrieved nodes to prioritize the most critical details upfront, improving coherence and accuracy, especially when using a large top-k. 2. **Keyword-based postprocessors** refine results by emphasizing specific terms or phrases critical to the query's intent. For example, a query asking for "best practices" might prioritize nodes containing that exact phrase while downweighting less precise matches. Essentially, this layers a keyword retrieval step on top of the initial semantic retrieval to further refine relevance. 3. **PII postprocessors** protect sensitive data by detecting and redacting personally identifiable information like names, addresses, and account details before passing content to the LLM. I threw this one in here to highlight how postprocessing can handle a wide range of unique tasks, showcasing the versatility of this step in refining and safeguarding data. This list is far from exhaustive, but these examples show how postprocessing can refine retrieval, ensuring the LLM gets optimized inputs for generation. ## Generation So, we’ve ingested our source documents and transformed them into source nodes optimized for retrieval, selected a retrieval process that best suits our use case, and even refined—or, if we were feeling bold, augmented—our retrieved nodes through postprocessing. At this stage, the pipeline has two critical pieces: 1. **The task** – What the LLM has been instructed to do (e.g., answer a question, summarize, generate code, call an API, etc). 2. **The context** – The information retrieved from the RAG pipeline to help the LLM complete that task accurately. This is where the generation step kicks in. The LLM takes the task, integrates the retrieved context, and produces the final output. But one thing I think people often overlook is the structure of that output—because having a deep understanding of it inevitably shapes key architectural decisions. ### **What Even Is Generation?** At its core, generation is just taking input (a task + retrieved context) and producing output. If you're building something like a Q&A chatbot, this is pretty straightforward: the task is the user's question, and the retrieved documents are the most relevant source nodes. The generation step simply pulls from those source nodes to construct a response. Something like this: ``` You are an AI assistant answering user queries based on the provided context. Use the information given to generate the best possible response. If the context does not contain enough information, say so insteadÏ of making up an answer. Context: {retrieved_documents} User Query: {user_question} Response: ``` But in more advanced applications, generation goes far beyond just returning text. It can involve generating structured outputs, executing external actions, or even orchestrating complex workflows. Let’s go over a few of these common generation strategies and how they impact the way we pass retrieved context to the LLM—just some quick, top-of-mind examples. ### 1. Generating Structured Outputs The most basic evolution beyond plain text generation is structured output generation. If you’re not familiar, most state-of-the-art (SOTA) models—like GPT-4o, Anthropic’s Claude models, and reasoning-focused models like o1 (although not DeepSeek, yet!)—can natively generate structured JSON objects instead of raw text. This unlocks interesting use cases where an application requires highly structured data, especially when feeding a downstream system like writing to a spreadsheet, putting data into a database, or calling an API. In these cases, we can take advantage of several techniques: - Passing a schema at runtime to guide the LLM’s output. - Using another LLM to validate or refine the output, with built-in retry logic in case of failure. - Dynamically adjusting the structure based on retrieved context—especially useful when the final shape of the data isn’t known upfront. And this is really just some possibilities off the top of my head. Structured generation opens up way more possibilities than just a plain text solution can provide. ### 2. Executing External Actions In the structured outputs section, we saw how models can generate parsable objects to make downstream tasks more reliable and easier to handle from a software perspective. But what if we wanted to move beyond just structuring data and actually execute actions? Surprisingly, this is entirely possible. Just like SOTA models can output structured JSON, many are also fine-tuned to execute external commands through a process known as function calling. On the engineering side, all we need to do is expose the available functions to the LLM when initializing it, along with their corresponding schemas (parameters). This takes things a step further—now the LLM isn’t just returning information, it’s actually doing things within a system. A few quick examples: - **Automating system updates** – The LLM can trigger database writes when specific conditions are met, like logging new customer inquiries or updating order statuses. - **Orchestrating API calls** – Instead of just returning API request details, the model can directly make the call and handle the response, like fetching live stock prices or querying an internal knowledge base. So while structured output generation allows us to format responses for downstream use, function calling lets us give the LLM direct access to execute specific tasks. ### **3. Orchestrating Multi-Step Workflows** This last example builds on the previous generation strategies, scaling to meet the demands of more complex workflows. Let’s say we’re designing an automated, agentic system where different parts of the application need access to external knowledge scattered across various vector DB namespaces. Since we already know that LLMs can generate structured outputs, return text, and call external functions, we now have the flexibility to orchestrate entire multi-step workflows. With this, we can do things like: - **Calling other agents** – Each agent can have specialized access to certain knowledge or actions, allowing for more modular and intelligent decision-making. - **Chaining multiple function calls** – An LLM can act as an orchestrator, dynamically executing a sequence of actions where each step requires unique knowledge, like retrieving data, analyzing it, and then taking an action based on the insights. - **Adaptive workflows** – Instead of rigid, pre-defined logic, workflows can adjust in real time based on retrieved context, ensuring the right actions are triggered at the right moments. ## **Walkthrough: Building an OpenAI API Call Agent** Let’s make this stick with an example. Say we’re building an application to make API calls to OpenAI—nothing like an LLM calling another LLM, right? ### **Ingesting the Right Source Documents** To ensure the LLM can properly debug and optimize API calls, we source documents from: - OpenAI documentation (pretty important, I’d say). - Solved examples from StackOverflow, where users posted questions with incorrect implementations and others responded with correct solutions. The StackOverflow examples are particularly intriguing as source documents because they often reflect real-world implementation issues paired with varying levels of feedback from the community. Typically, these documents include: - **A question with code**, where the user is likely implementing something incorrectly (e.g., a poorly constructed API call or outdated syntax). - **Answers and suggestions**, ideally with corrected code, explanations, or alternative approaches to solve the issue. - **Potential challenges**, such as conflicting suggestions, incomplete explanations, or answers that reflect bad practices or outdated information. ### **Why a Basic Ingestion Pipeline Probably Won’t Work** Because of this interesting mix of useful implementations and the usual StackOverflow ego-fueled crap (sorry, I mean noise), a basic ingestion pipeline just won’t cut it. Instead, we build a custom ingestion tool that scrapes, cleans, and augments these examples into a structured, high-signal format that’s actually useful. Without getting too deep into the weeds, here’s an example of what the formatted output might look like: ```

Question Here


	Incorrect Code here

Error message here


	Correct code here

Correct code explanation here ``` ### **Optimizing for Retrieval and Generation** Now that we’ve got our StackOverflow source documents that actually make sense for our use case—what’s next? Well here’s a curveball, we don’t even bother chunking these into separate source nodes. Why? Because keeping the full context intact just works better. Each example is a self-contained unit, with the incorrect implementation, the error it threw, the user’s question, and the correct fix with an explanation. Splitting that up would just make retrieval harder for no good reason. However, just because we’re not chunking doesn’t mean we should embed everything. So, how do we optimize these documents (which are also our nodes) for both retrieval and generation? ### **Retrieval: Remembering What Matters** Let’s think about what the system is actually doing during retrieval, and let’s start simple. When will the LLM most likely need the StackOverflow information? The answer is pretty obvious—when either a human (or an AI agent, who knows) screws up an API call. At this point in time, they have accessible to them the following information: - The error message - The current (broken) code implementation - Maybe even the ability to generate a question about what went wrong This is exactly why the StackOverflow data is so valuable—it’s structured to handle scenarios just like this. It’s got the incorrect implementation, the error it threw, and the user’s question, all lined up to guide the AI in debugging. So why not only embed that? (hint: do it) ### **Metadata Replacement for Generation** And what about the correct implementation and explanation? That gets stored as metadata. When the system retrieves this node, we use a metadata replacement strategy, swapping out the incorrect implementation for the correct one before passing it to the LLM. This way, the AI gets the best possible answer without sacrificing relevance or context. ### Bringing It All Together If you haven’t noticed by now, this is a pretty advanced example that ties together everything we’ve covered so far: - Building a custom ingestion tool to process unstructured data into structured nodes - Augmenting the source document to enhance retrieval quality - Skipping chunking because full-context retrieval is more effective - Embedding only the parts relevant for retrieval (incorrect implementation, error, question) - Storing the correct fix as metadata for post-retrieval replacement - Using semantic or hybrid search to ensure high-quality retrieval - Applying metadata replacement at generation time to optimize context ### **Still Have Questions?** If some of this went over your head, no worries—I kinda threw you into the deep end on purpose. I wanted to push the boundaries a bit with this example to emphasize how crucial it is to understand your end application when designing a system. If anything feels unclear, feel free to hit me up at **chris@architects.dev**—always happy to dive deeper.