A team we spoke with had a RAG demo that answered every question flawlessly in front of the board. Three weeks after go-live, it was quoting a policy that had been superseded eight months earlier, missing an entire product line because those PDFs had never parsed properly, and taking four seconds to answer questions that should have taken four hundred milliseconds. The model had not changed. Nobody had designed the architecture around it to survive contact with real data.
RAG architecture is the set of components, ingestion, chunking, embeddings, a vector store, retrieval, reranking and generation, that turns a language model into a system grounded in your own data rather than its training set. Most production failures trace back to how data moves between these stages, not to the model sitting at the end of the pipeline.
The shape of a RAG pipeline
Strip away the vendor diagrams and a RAG system is really a data pipeline with a language model bolted onto the end. Documents come in through ingestion, get split into chunks, get converted into embeddings, and land in a vector store. At query time, the same embedding step runs on the user's question, the vector store returns the closest matching chunks, a reranker often reorders them for relevance, and the generator assembles an answer from whatever survived that filtering.
The order matters because errors compound downstream rather than cancel out. A document parsed badly at ingestion produces chunks that read strangely. Oddly shaped chunks embed into vectors sitting in the wrong part of the space. Retrieval then pulls back the wrong context with total confidence, and the generator writes a fluent answer from material that was never right to begin with. Treat it as a pipeline where quality has to be protected at every stage, not a single component you can tune your way out of trouble with.
Ingestion: where most of the damage happens before a query is ever asked
Ingestion is the least glamorous part of the architecture and the part most worth getting right first. It covers pulling content from wherever it lives, PDFs, Confluence, a support ticket system, a product database, and turning it into clean text with its structure intact. PDFs are the usual trap: tables collapse into unreadable strings, headers bleed into body text, and multi-column layouts read in the wrong order unless the parser is built to handle them.
Good ingestion keeps metadata attached to every piece of content it produces: source system, author, last-updated date, access permissions, document type. That metadata is the difference between a retriever that can filter by recency or restrict results to what a user is allowed to see, and one that treats every chunk as equally valid regardless of when it was written.
The other decision to make early is how ingestion runs on an ongoing basis. A one-off import gets stale the moment source documents change, and most teams underestimate how quickly that happens. Whether re-ingestion runs on a schedule, on a webhook, or as a nightly rebuild determines how confident you can be that an answer reflects the current state of the business rather than a snapshot from launch day.
Chunking: the decision everyone underrates
Chunking splits ingested documents into pieces small enough to embed and retrieve individually. The obvious approach, fixed-size splitting every few hundred characters, is also the one that causes the most quiet damage, because it cuts sentences and tables in half with no regard for what the text is actually saying. A chunk that ends mid-argument gives the retriever half an idea to work with and the generator half a fact to reason over.
Structure-aware chunking, splitting on headings, paragraphs, list items and table boundaries, produces chunks that read as complete thoughts, which is what both the embedding model and the reader need. Semantic chunking goes further, using the embedding model itself to find natural breakpoints where the topic shifts, at the cost of extra processing time.
Chunk size is a genuine trade-off, not a setting to copy from a tutorial. Small chunks retrieve with precision but lose surrounding context, so an answer can be technically accurate and still miss the caveat in the next paragraph. Large chunks preserve context but dilute the embedding, since a vector for a whole page is a blurred average of everything on it, and blurred averages retrieve poorly against a specific question. Most teams land between a few hundred and a couple of thousand tokens, with a modest overlap between consecutive chunks so a fact split across a boundary can still be found.
Embeddings and the vector store
The embedding model converts each chunk, and later each query, into a vector that captures its meaning well enough that similar ideas sit near each other in that space. A general-purpose embedding model is a reasonable default, but domain-heavy content, legal contracts, clinical notes, dense engineering specifications, often retrieves noticeably better from a model tuned on similar text, because generic embeddings blur the specific vocabulary that actually distinguishes one document from another.
Whatever model you pick, treat changing it later as a significant event, not a config tweak. Every chunk in the store was embedded with a particular model's understanding of language, and swapping models means every vector needs recomputing, since a query embedded by a new model has no reliable relationship to vectors produced by the old one.
The vector store itself needs to do more than return nearest neighbours quickly. Metadata filtering lets a query be restricted to a tenant, a document type or a date range before similarity search runs, which matters as much for correctness as for speed. Many production systems also run hybrid search, combining vector similarity with keyword search such as BM25, because pure vector search is surprisingly weak on exact terms: product codes, error messages, names, the kind of string a keyword index finds instantly and similarity search can quietly miss.
Retrieval and reranking
Naive retrieval takes the user's question, embeds it, and pulls back the handful of chunks with the closest vectors. That works reasonably well for questions phrased the way the source documents are written, and noticeably less well for anything phrased differently, which is most real questions. Query rewriting, having a model restate the question more like the source material, and multi-query retrieval, generating several variants and merging their results, both help close that gap.
Reranking adds a second pass after initial retrieval. Rather than trusting vector similarity alone, a cross-encoder model looks at the query and each candidate chunk together and scores how relevant it genuinely is, a slower but far more accurate judgement than comparing two vectors in isolation. The usual pattern is to retrieve a wider net, thirty or fifty candidates, then rerank down to the handful that go to the generator, trading a little latency for a meaningful jump in precision.
How many candidates to retrieve, and how many to keep after reranking, is worth revisiting as the corpus grows. A setting that worked well against a thousand documents can start missing relevant material once the store holds a hundred thousand, because the same top-k now represents a much thinner slice of everything that exists.
Assembling context and generating the answer
By the time chunks reach the generator, they still need arranging into something the model can use well. Context windows are finite, so every chunk included is budget spent, and stuffing in every marginally relevant result usually produces a worse answer than a tightly curated few, because the model has to work harder to find the signal among the padding. Ordering matters too: many models weight the start and end of a prompt more heavily than the middle, so burying the most relevant chunk in the centre of a long context can cost accuracy.
Grounding is the other half of this stage. A well-built generation step asks the model to answer only from the supplied context, to cite which chunk supported which claim, and, critically, to say when the retrieved material does not actually answer the question rather than filling the gap with something plausible. That last behaviour is the one most demos skip, and the one most production incidents come back to, because a confident answer with no evidence behind it is indistinguishable from a correct one until someone acts on it.
Where RAG architectures break in production
The most common failure is silent staleness: the vector store keeps answering from documents that were correct when ingested and have since been superseded, because nothing in the architecture flags that a source changed. Unlike a crashed pipeline, this produces no error, just a confident answer built on outdated ground, which is exactly why it tends to surface in front of a customer rather than in a test.
Permission leakage is the second one that catches teams out. A retriever built without access control in mind will happily pull a chunk from a restricted document because it matched the query well, regardless of whether the person asking is allowed to see it. Any RAG system built over content with real access boundaries needs those boundaries enforced at retrieval time, not left to the generator to somehow respect.
The rest is more mundane but no less costly: latency and cost climbing as the corpus grows and nobody revisits retrieval settings fit for an earlier, smaller version of it; chunking decisions that quietly degrade as new document types get added; and no evaluation set, so a change that improves one type of question and worsens another goes unnoticed until users complain.
Getting the design decisions right
Orchestration frameworks such as LangChain and LlamaIndex earn their keep early, when the priority is standing up a working pipeline quickly and the default components are good enough. As a system matures, teams often replace pieces of that framework with custom code for the stages that matter most, usually chunking and retrieval, while keeping the framework for parts that genuinely benefit from not being reinvented, such as connectors and prompt templating.
What determines whether any of this holds up is treating RAG as an architecture designed deliberately, with an evaluation set, monitoring on retrieval quality, and a plan for what happens when source data changes, rather than a library installed and a prompt tuned until the demo looks convincing. That is the gap between a system that survives its first messy dataset and one that needs rebuilding six months in. It is also the substance of our search and retrieval work: designing ingestion, chunking and retrieval around the data you actually have, not the tidy dataset a demo was built on.
Frequently asked questions
What is RAG architecture?
RAG architecture is the arrangement of components, ingestion, chunking, embeddings, a vector store, retrieval, reranking and generation, that lets a language model answer from your own data instead of relying only on what it learned during training. Each stage feeds the next, so the quality of the final answer depends on every step in the chain, not just on the model doing the generating.
What is the difference between RAG architecture and fine-tuning?
Fine-tuning changes the model's weights by training it further on your data, which is expensive to update and does not tell you where an answer came from. RAG architecture keeps the model unchanged and instead retrieves relevant material at query time, which is far cheaper to keep current and lets you trace an answer back to the source document that supported it.
How do you choose a chunk size in a RAG pipeline?
There is no fixed correct size, because it is a trade-off between precision and context. Smaller chunks retrieve more precisely but can lose the surrounding detail a question needs, while larger chunks keep more context but produce blurrier embeddings that retrieve less accurately. Most pipelines settle somewhere between a few hundred and a couple of thousand tokens, with a small overlap between chunks, then adjust based on how retrieval actually performs against real questions.
Why does RAG architecture need a reranker?
Initial vector retrieval ranks candidates by similarity alone, which is fast but not always accurate, since a chunk can sit near a query in vector space without actually answering it well. A reranker looks at the query and each candidate together and scores genuine relevance, so retrieving a wider set of candidates and reranking them down to a handful typically produces noticeably better answers than trusting the first pass alone.
Why does a RAG pipeline that worked in testing break in production?
It usually comes down to assumptions that held for a small, clean test set failing to hold at scale: source documents change and the index goes stale, new document types arrive that do not fit the original chunking approach, or access permissions were never enforced at retrieval time. None of these show up as errors, which is why they tend to surface as wrong answers in front of real users rather than as failures caught in testing.
Want a straight view of where AI can help your business first? Talk to us and start with a clear picture instead of a vendor demo.