RAG consulting: what it involves and what to expect from a consultant

Your team has seen a retrieval demo. It answered questions with a small set of documents and seemed to be a complete product. However, moving from that demo to a full system requires a significant amount of work. A complete system must process all of your content, obey access control rules, and stay accurate as your documents are updated. A consultant oversees this transition. This guide outlines the specific tasks involved in that process and what happens during each development stage.

RAG consulting offers technical assistance to design, build, and evaluate retrieval-augmented generation systems that use your specific data. A consultant reviews the documents and access permissions that the system uses. They decide how to split and embed content and develop evaluation methods to verify retrieval accuracy. The consultant also links the system to your models and tools before delivering a finished product for your team to manage. The primary benefit of this service is the technical decision-making required to move a system from a demonstration phase into a production environment.

Shipshape Data builds data infrastructure and artificial intelligence applications. Retrieval is an important part of this process. This guide is meant for individuals responsible for selecting a consultant, giving them project requirements, and assessing their performance.

An explanation of RAG consulting

Retrieval-augmented generation links a language model to your specific content. When a user asks a question, the system finds relevant passages and supplies them to the model. The model then generates an answer based on that material. Our guide to how RAG works for teams details this process. RAG consulting is a service that implements this process for an organisation's data.

A consultant has four main responsibilities. First, they must understand the content by identifying its location, format, accuracy, and access permissions. Second, they design the retrieval pipeline, which includes methods for splitting documents into sections and selecting relevant passages for specific queries. Third, they evaluate the system using a set of actual questions to ensure quality is measured with data rather than subjective opinions. Fourth, they integrate the system into existing workflows and supply the documentation, monitoring, and update procedures needed for long-term operation.

Vector databases, embedding models, and orchestration frameworks are established products. The consulting process involves selecting the appropriate tools for your data, connecting them correctly, and verifying the final system.

Consultant or product: how to decide

Many products now include assistants that answer questions using documents stored in your current platform. A product is a suitable first step if your documents are kept in a single, well-governed system with correct permissions. This option works well if the questions are general and users do not need to access the answers within a different workflow. The requirements are a subscription fee and a few days of configuration.

Consulting is useful when one or more of the following conditions are met.

• Your content is located in multiple systems, such as a document store, a ticketing tool, and a wiki. Answering a typical question requires information from more than one of these sources.
• The systems have different permission levels. Some content is sensitive, so providing an incorrect answer to an unauthorised person creates a compliance issue.
• This domain is specialised. Product specifications, contracts, engineering documents, and regulated content require chunking and retrieval methods that are adjusted to their specific writing styles.
• The answers are used in a process with a defined quality standard, such as support, sales, or operations, and someone will be responsible for meeting that standard.
• You have already tested a product pilot on your data, but the results were insufficient to support a full release.

The final scenario is the most common reason clients approach RAG consulting companies. The pilot phase has shown that the concept is viable and that the primary issues reside within the data. This is the point where the consulting engagement begins.

The engagement, stage by stage

Projects vary in duration, but the sequence of stages remains constant. If a consultant skips the first two stages, they will only create a demonstration. You already have a demonstration.

Corpus and permissions

The first stage involves an audit of the content the system will use to generate answers. The consultant lists all sources, samples the documents, and verifies the parsing process. Documents such as PDFs with scanned pages, tables, and diagrams often lose information during this step. These errors often go unnoticed until the system provides an incorrect answer. The audit also records who is allowed to see each source and the specific format of those permissions. The retrieval system must be able to enforce these rules. This stage often requires changes to the project plan. It typically identifies content that is outdated, duplicated, or impossible to use.

Chunking and embeddings

Documents are divided into chunks. Each chunk is turned into an embedding, a numerical representation of its meaning used for similarity searches. Our guide to what a vector database is explains the mechanics of that search process. Retrieval quality depends on chunk size, the amount of overlap between chunks, and whether headings and metadata are included with each piece. The consultant tests various settings against actual questions rather than using default values. Our guide to RAG architecture covers these components and the related design decisions.

Retrieval quality and the evaluation set

The consultant creates an evaluation set before showing any answers to a user. This set consists of real questions paired with the passages that answer them and the correct answers when they are available. Retrieval performance is measured using this data. The primary metric is whether the model includes the correct passage in the results for each query. This set is used consistently as the pipeline changes to show measurable improvements.

The model cannot provide the correct answer if the results do not include the relevant passage. You must measure retrieval quality first, as all other performance metrics depend on that figure.

Grounding and answer behaviour

With retrieval functioning, the model is then instructed to base its answers solely on the retrieved passages, to include citations, and to indicate when the answer cannot be found within the provided content. This is the step where the generation of false information is managed. It is also where the style, length, and format of responses are adjusted to suit the intended application, such as a support reply, an internal search result, or a report summary.

Integration

A retrieval system is deployed in the environments where people are presently working. The consultant links it to tools such as chat applications, support desks, intranets, or the product itself, and decides how the model will access your systems. When the assistant needs up-to-date data besides documents, the Model Context Protocol (MCP) provides a controlled way to grant access to tools and data. Our MCP connections service addresses this aspect.

Cost control and handover

The final stage defines the operating costs and the daily routine. The consultant computes the cost per query using your expected volume. They choose particular models and caching settings to remain within your budget. They also supply documentation describing how to update the index when documents change, who is responsible for quality monitoring, and the actions to take if performance drops. The handover contains the evaluation set so your team can apply the same methods to test future changes.

What to expect week by week

The initial weeks are devoted to the corpus audit and the evaluation set. The pipeline is then constructed and tuned in short cycles, each concluding with a test against that set. Integration and access control occur after retrieval quality reaches the agreed level. The final phase of the project involves operating the system together with your team before the handover. A consultant should deliver a scoped estimate after the audit is completed, not before it begins.

How you know retrieval is good

Retrieval quality is a metric rather than a subjective feeling. Three specific measurements account for most of the relevant performance data.

• Retrieval hit rate is the percentage of questions in the evaluation set for which the results sent to the model contain the passage that holds the answer.
• Answer faithfulness indicates whether the retrieved passages support the generated answer. A person reviews a sample of the data, while a second model assesses the remaining content using the same criteria.
• Refusal behaviour indicates whether the system declares it does not know the answer when the information is absent from the content. A system that does not refuse is offering guesses.

We log these numbers before and after each change to chunking, embeddings, or prompts. We keep using these same measures in production on a sample of live questions. This data set also safeguards your organisation after the engagement ends. When a document changes or a model is upgraded, your team can rerun the tests to determine if performance has changed. A consultant who cannot present these numbers for your specific data has not completed their work.

A demonstration shows a model's performance using ten selected documents. The engagement shows how that model performs using ten thousand of your own documents.

Governance and access control

A retrieval system can access more content than any single user. Governance is the process that stops the system from sharing content with unauthorised people. This process comprises four parts.

Permissions are verified when information is requested. Each text segment contains the access rules from the original document, and the system displays results only to users authorised to view them. Simply filtering results after they are retrieved is insufficient, because information a user is not permitted to see can still affect the results they do see.

Sensitive information comprises personal data, contract terms, and any details subject to regulations. We identify these items during the audit. We then exclude them, redact them upon ingestion, or restrict access via permissions.

The system keeps an audit trail. Each answer records the specific passages used to create it and the identity of the person who asked the question. This lets users investigate incorrect answers and allows auditors to review the process used to generate a response.

You must decide on residency and retention policies before indexing the first document. These settings are hard to change later. You need to determine where the index is stored, which model provider receives the queries, and how long the system retains logs.

A consultant should raise these points during the first week of the project without waiting for a request.

What drives the cost

We do not publish prices. Any consultant who gives a fixed quote before reviewing your content is making an estimate without sufficient data. The cost of a RAG consulting engagement depends on the following factors.

• The number of sources and the variety of formats. Each source requires a connector, and each format requires parsing that has been verified.
• The condition of the content. Documents that are duplicated, outdated, or poorly structured need time to clean or remove.
• The permission model. You must reconcile permissions that differ across systems before you can enforce them.
• The quality standard and the methods for measuring it. A support assistant evaluated by its resolution rate needs a larger set of evaluation data and more adjustment than an internal search tool.
• The integration surface. A single chat interface is less complex than a support desk, an intranet, and a product combined.
• Languages. For each additional language you must create specific evaluation questions. You might also need to choose different embedding models for each language.
• Operating costs consist of monthly fees for model calls, hosting, and the vector store. These costs rise with the number of queries and the length of the generated answers.

The final item is often omitted from proposals. You ought to ask for it as a separate line item reflecting your expected volume.

Questions to ask a RAG consultant

These questions separate a consultant who has operated retrieval systems in production from one who has only built demonstrations.

• What information did you find in our content that led you to modify your strategy? A thorough audit invariably leads to changes.
• How will we know retrieval is correct before anyone sees an answer? Listen for an evaluation set derived from our own questions.
• How are our document permissions enforced during a query? Listen for filtering during the retrieval process rather than after it.
• What does the system do when the answer is not in our documents? It should say so, and you should be able to see how often this happens.
• What is the cost of a query at our projected volume, and which factors affect this price? You should ask for the exact figure and the underlying assumptions.
• Which parts are custom and which are products we could replace? You should retain control over the design so you can swap the vector database or the model.
• Who updates the index when a document changes, and how quickly? Outdated content is the main cause of incorrect answers after a system is launched.
• What do you hand over, and who runs it after you leave? The answer should contain the evaluation set, the monitoring tools, and the name of a specific person on our side who will assume responsibility.

From demo to production system

A demo delivers answers using a small, selected set of documents. A production system must respond to unpredictable questions using all available content, while respecting user permissions and keeping a specific cost per query. It also must keep functioning as the content is updated. The distinction between these stages includes the audit, the evaluation set, the governance, and the operating routine described earlier. These steps are not hard to carry out, yet they are often ignored.

At Slimstock, the website assistant we built answers from the company's own content around the clock in 11 languages. The Slimstock case study details how we built it.

If you are choosing between a product and a custom engagement, or if a pilot project halted because of data issues, our search and retrieval service covers all the work described here. This spans from the initial corpus audit to the final handover. Talk to us and we will clarify what your content can support before you build anything.

Frequently asked questions