A team we spoke with showed a RAG demo that answered every question correctly for their board of directors. Three weeks after the official launch, the system cited a policy that had been replaced eight months earlier. It also omitted information about a particular product line because the corresponding PDF files had not been processed correctly. Furthermore, the system took four seconds to deliver answers that should have been provided in four hundred milliseconds.
The model stayed unchanged. No one had constructed the surrounding architecture to work correctly with real data.
RAG architecture comprises the components ingestion, chunking, embeddings, a vector store, retrieval, reranking, and generation, which convert a language model into a system anchored in your own data instead of its training set.
Most production failures stem from the way data moves between stages rather than from the model at the pipeline's end.
The structure of a RAG pipeline
A RAG system functions as a data pipeline that employs a language model at the final stage. In the ingestion step, documents are split into chunks and transformed into embeddings prior to storage in a vector store. When a user poses a question, the query is likewise turned into an embedding. The vector store finds the most similar chunks, and a reranker then orders them by relevance. Finally, the generator produces an answer using the information that remains after this filtering.
The sequence of operations matters because errors build up through the process. If a document is parsed incorrectly during ingestion, the resulting text chunks become incoherent. These irregular chunks are then turned into vectors and placed in incorrect locations within the vector space. During retrieval, the system will choose incorrect context with high confidence. The generator then generates a fluent response based on inaccurate source material. You should see this as a pipeline that requires quality maintenance at each step. These issues cannot be resolved by tweaking a single component.
Ingestion: where most of the damage happens before a query is ever asked
Ingestion is the least attractive part of building a system, yet it is the most critical part to address first. It entails retrieving content from its source, such as PDFs, Confluence, support ticket systems, or product databases, and converting it into clean text while maintaining its structure. PDFs often pose challenges: tables may turn into unreadable text strings, headers can merge with body text, and multi-column layouts might be read in the wrong order if the parser is not specifically designed to handle them.
Effective ingestion retains metadata attached to each piece of content it produces: source system, author, last-updated date, access permissions, document type. That metadata distinguishes a retriever that can filter by recency or limit results to what a user may see from one that treats every chunk as equally valid regardless of its creation date.
You must decide early how to handle ongoing data ingestion. A single import becomes outdated as soon as source documents change, which occurs more often than most teams anticipate. You can perform re-ingestion on a schedule, via a webhook, or as a nightly rebuild. This decision determines the reliability of your answers and how accurately they reflect the business's current state.
Chunking: the decision everyone underestimates
Chunking splits documents into segments that are small enough to be embedded and retrieved. The most common method splits text at fixed intervals of a few hundred characters. This approach often causes problems because it breaks sentences and tables apart without regard to context. When a chunk ends in the middle of a point, the retriever gets incomplete information and the generator receives partial facts to process.
Structure-aware chunking, which splits on headings, paragraphs, list items, and table boundaries, creates chunks that read as complete thoughts, which both the embedding model and the reader require. Semantic chunking goes further by using the embedding model itself to locate natural breakpoints where the topic shifts, at the expense of extra processing time.
Choosing a chunk size involves a direct trade-off instead of following a standard rule. Small chunks enable precise retrieval but often miss surrounding context. This can produce an answer that is accurate yet omits important details found in the next paragraph. Large chunks preserve context but render the embedding less specific. A vector that represents an entire page acts as an average of all content, leading to poor retrieval performance for specific questions. Most teams use sizes ranging from a few hundred to two thousand tokens. They also add a small overlap between consecutive chunks to ensure that information split across a boundary stays searchable.
Embeddings and the vector store
The embedding model converts each chunk and query into a vector. This vector captures the meaning so that related ideas are placed close together in the vector space. A general-purpose embedding model works for many tasks. However, content with specialised terminology often performs better when a model trained on similar text is used. This applies to legal contracts, clinical notes, and engineering specifications. Generic models can fail to distinguish specific terms, which reduces retrieval accuracy.
Treat changing your chosen model as a significant event rather than a simple configuration change. Every data chunk in the store was created using the specific language understanding of that model. If you switch models, you must recompute every vector. A query processed by a new model will not have a reliable relationship to the vectors generated by the previous model.
A vector store must do more than return nearest neighbors quickly. Metadata filtering lets a query be limited to a specific tenant, document type, or date range before the similarity search starts. This process is required for both accuracy and speed. Many production systems also employ hybrid search to combine vector similarity with keyword search methods such as BM25. Pure vector search often fails to identify exact terms like product codes, error messages, or names. A keyword index finds these strings instantly, while a similarity search may miss them.
Retrieval and reranking
Naive retrieval embeds the user's question and fetches the chunks with the most similar vectors. This method works well when questions use the same phrasing as the source documents. It is less effective for most real-world questions, which are usually phrased differently. Query rewriting helps by having a model restate the question in a style that matches the source material. Multi-query retrieval also assists by generating several versions of the question and merging the results.
Reranking adds a second processing step after the initial retrieval. Rather than relying solely on vector similarity, a cross-encoder model evaluates the query and each candidate chunk simultaneously. This model scores the actual relevance of each pair. This process is slower than comparing two vectors, yet it yields more accurate results. Most systems use this pattern to retrieve a larger set of thirty or fifty candidates and then rerank them. The final output is a small number of chunks sent to the generator. This approach slightly increases latency to achieve a significant improvement in precision.
You should periodically review the number of candidates you retrieve and retain after reranking as your corpus grows. Settings that work for one thousand documents may fail to find relevant material when the store reaches one hundred thousand documents. This occurs because the same top-k value represents a smaller percentage of the total available data.
Gathering context and producing the response
By the time chunks reach the generator, they still need arranging into something the model can use well. Context windows are finite, so every chunk included is budget spent, and stuffing in every marginally relevant result usually produces a worse answer than a tightly curated few, because the model has to work harder to find the signal among the padding. Ordering matters too: many models weight the start and end of a prompt more heavily than the middle, so burying the most relevant chunk in the centre of a long context can cost accuracy.
Grounding is the second part of this stage. A well-designed generation step requires the model to answer using only the provided context. It must cite which specific chunk supports each claim. It must also state when the retrieved material does not answer the question instead of providing a plausible but incorrect response. Most demonstrations ignore this final behaviour. However, most production incidents involve this issue because a confident answer without evidence looks like a correct answer until someone acts on the information.
Where RAG architectures break in production
The most common failure is silent staleness. The vector store continues to provide answers based on documents that were correct during ingestion but have since been replaced. This occurs because the architecture lacks a mechanism to signal when a source document has changed.
A crashed pipeline produces an error, but this process generates a confident answer based on outdated information. This lack of a failure signal is the reason the issue often reaches the customer.
Permission leakage is the second issue that affects teams.
A retriever designed without access control will retrieve segments from restricted documents if they match the query, even if the user lacks permission to view them. Any RAG system that uses content with specific access levels must enforce those boundaries during the retrieval process. You cannot rely on the generator to manage these permissions.
The remaining issues are routine but expensive. Latency and costs increase as the database expands because retrieval settings are not updated for the larger scale. Original chunking methods become less effective when new document formats are introduced. Without an evaluation set, developers cannot see if a change that helps one query type harms another until users report the problem.
Making effective design decisions
Orchestration frameworks like LangChain and LlamaIndex are useful during the initial stages of development. They allow teams to build working pipelines quickly using standard components. As a system becomes more advanced, developers often replace specific framework functions with custom code. This typically happens for critical stages like chunking and retrieval. Teams often continue using the framework for tasks that do not require custom solutions, such as data connectors and prompt templating.
The success of this approach depends on treating RAG as a deliberate architecture. This requires an evaluation set, monitoring for retrieval quality, and a plan for managing changes in source data. It should not be treated as a library that is installed and tuned until a demo appears to work. This distinction determines whether a system can handle difficult datasets or if it will require rebuilding after six months. Our our search and retrieval work focuses on this process. We design ingestion, chunking, and retrieval based on your actual data instead of the simplified datasets used for demonstrations.
Many teams complete the theoretical phase of a project but stop before the building stage. Our AI development service is designed to assist with that specific step.
If you want to collaborate on the design rather than outsourcing the work, our guide to what a RAG consulting engagement covers explains the project stages, the evaluation criteria, and the questions you should ask before you sign a contract.
Frequently asked questions
What is RAG architecture?
RAG architecture is the arrangement of components, ingestion, chunking, embeddings, a vector store, retrieval, reranking and generation, that lets a language model answer from your own data. Each stage feeds the next, so the quality of the final answer depends on every step in the chain.
What is the difference between RAG architecture and fine-tuning?
Fine-tuning modifies the weights of a model through additional training on your data. This process is costly to update and does not provide the source of an answer. RAG architecture leaves the model as it is and retrieves relevant information when a query is made. This method is much less expensive to maintain and allows you to identify the specific document used to support an answer.
How do you choose a chunk size in a RAG pipeline?
Chunk sizes are not fixed. The choice involves a trade-off between precision and context. Small chunks allow for more precise retrieval, but they may lack the specific details required to answer a question. Large chunks preserve more context, but they result in less accurate embeddings that are harder to retrieve. Most systems use chunks between a few hundred and two thousand tokens. These chunks usually include a small overlap. You should adjust these settings based on how the system performs when tested against actual questions.
Why does RAG architecture need a reranker?
Initial vector retrieval ranks candidates based on similarity. This process is fast but can be inaccurate because a text segment may be near a query in vector space without providing a good answer. A reranker evaluates the query and each candidate together to score actual relevance. Retrieving a large set of candidates and then reranking them into a smaller group usually produces better results than relying only on the first retrieval step.
Why does a RAG pipeline that worked in testing break in production?
These issues typically occur when assumptions from a small test set do not apply to a larger system. Source documents may change and cause the index to become outdated. New document types might be incompatible with the initial chunking method. Additionally, access permissions are often not enforced during the retrieval process. These problems do not trigger system errors, so they usually appear as incorrect answers when real users interact with the application.
If you want to identify the specific areas where AI can benefit your business, Talk to us to begin with a clear assessment.