A logistics company applied fine-tuning to an open model with eighteen months of dispatcher notes. Their aim was to align the model with their particular communication style, which is short and direct. The project succeeded. Six weeks afterwards, the base model provider launched a new version and discontinued the previous one. The team had not kept the training data in a format that would let them repeat the process. Consequently, they lost the specific tone they had created. Reconstructing the model demanded redoing the whole project with a folder of poorly labelled exports. The employee who originally built the system had already departed the company.
Fine-tuning involves training a pre-trained model on specific examples to modify its behaviour for a given task, tone, or format. This approach differs from prompting and retrieval, which alter a model's behaviour without changing its underlying weights. Fine-tuning works well for narrow, repetitive tasks but is less suited for filling knowledge gaps. It demands ongoing maintenance since base models are updated over time.
Fine-tuning takes a language-trained model and trains it further using your own examples. This moves the model's weights towards your particular tasks, vocabulary, and answer style. It is not the sole method for adapting a model, and it often is not the first option you should attempt. Prompting alters behaviour by supplying instructions and examples when you pose a question. Retrieval augmentation modifies behaviour by providing the model with facts absent from its training data. Fine-tuning changes the model itself.
Shipshape Data constructs data and AI systems. Fine-tuning is a frequent request from our clients, yet we seldom recommend it. Although the process is effective, the outcomes most people desire are typically obtainable via less costly methods. This guide details the specific changes fine-tuning makes to a model and how the process differs from prompting and retrieval. It also outlines the required data formats, the costs beyond the initial training, and the long-term maintenance needed to keep these systems operating.
What changes when you fine-tune, and what stays the same
A pre-trained model arrives with a large amount of general capability embedded in its weights: grammar, world knowledge, reasoning patterns, and a rough sense of how humans phrase things. Fine-tuning takes that model and continues training it on a smaller, focused set of examples specific to your task, adjusting the weights so the model's default behaviour shifts towards what those examples demonstrate. Provide it with a few thousand support tickets answered the way your best agent would, and the model begins producing answers in that shape without prompting.
The base model keeps all the information it already possessed that your examples did not address.
Fine-tuning does not erase the existing model to start over. It makes small adjustments. This makes the process effective for establishing narrow, repeatable behaviours but unreliable for teaching a model new facts. A few thousand examples of a specific phrasing pattern give a strong and consistent signal. Conversely, a few thousand mentions of a fact scattered across varied text give a weak signal that the model may not retain.
The distinction that trips people up
The most common misunderstanding concerns the difference between changing how a model responds and changing what it knows. Fine-tuning is effective for the first goal. If you fine-tune a model on the approved phrasing of a compliance team, the model will use that phrasing consistently. It is not an effective tool for the second goal. A model fine-tuned on a product catalogue will not know that catalogue reliably. Instead, it learns a style of speaking about products that resembles your own. It will still confidently invent a SKU number if asked about one it did not see often enough to memorise. If your objective is to provide facts rather than style, fine-tuning addresses a different problem than the one you are facing.
Fine-tuning, prompting, and retrieval compared
People often talk about these three approaches as if they were competing options, but that description is inaccurate. They occupy different positions on a spectrum that is defined by the amount of effort required and the level of control needed. For most tasks, you should move through them sequentially rather than beginning with the most complex tool.
When prompting is enough
Prompting means giving instructions when you make a request, such as a system message, examples, or a description of how the output should appear. Changing prompts, using them, and maintaining them only involves editing the prompt text itself, and these actions do not create any costs. For many tasks, prompting can reach nearly the desired result. If a model is close to the correct answer but has problems with tone, formatting, or understanding particular situations, that is a prompting issue, not a flaw in the model's training. Solving this with model training would require significant resources when a simple set of instructions could fix it. Our guide to prompt engineering outlines the techniques to try before moving on to more complex solutions.
When retrieval augmentation is enough
Retrieval-augmented generation, or RAG, tackles a specific problem. It supplies the model with facts by inserting them into the context window when a query is issued. Fine-tuning does not reliably teach a model your return policy, current pricing, or the contents of a recently uploaded document. Facts stored in model weights are limited to the training date. Those facts are mixed with all other training data. Retrieval keeps information external and up to date. It links information to a particular source. You can update a policy instantly without retraining the model. Our article on retrieval-augmented generation describes how to build that pipeline.
When neither will do
Fine-tuning is useful when you need a consistent behaviour that is hard to reproduce in every prompt. It is suitable for applying a specific house style to a large volume of documents, maintaining a rigid output format for downstream parsing, or using domain jargon that would otherwise consume too much of the context window. It also addresses situations where a long system prompt with multiple examples exceeds your latency or cost limits. If your situation does not involve these particular needs, you should probably use prompting or retrieval methods instead of fine-tuning.
The data question nobody likes answering honestly
Every discussion about fine-tuning starts in the same way. A person expresses confidence that a large amount of data is available.
Data is usually available. Whether this data is suitable for fine-tuning is a separate issue, and the answer is typically that it is not ready yet.
Characteristics of high-quality training data
You must provide examples that match the exact format you expect the model to generate. Each example should pair an input with your preferred output. You should include enough variety to cover the different cases you encounter. A few hundred examples may be sufficient to change the tone of a narrow task. To achieve consistent behavioural changes across diverse inputs, you generally need several thousand or tens of thousands of examples. The required volume depends on how much the target behaviour differs from the default behaviour of the base model.
Common points of failure
Existing data is rarely the data you actually need. Support transcripts often contain incorrect answers that agents had to override. Historical documents were written to various standards over time, which gives the model inconsistent signals about what constitutes a good response. Existing labels were often applied by different people using different judgement calls. This causes the model to learn organisational disagreements as if they were a single rule. Cleaning this data properly is the primary task rather than a secondary one. This process is similar to standard data quality work, but it is applied to a training set instead of a warehouse. You should expect the fine-tuning project to reveal every data gap that your organisation has previously ignored.
If you lack sufficient real examples, you can use synthetic data to create plausible variations of rare cases. We use this method. However, it does not replace a thorough analysis of real data. If you train a model primarily on generated examples of what you believe edge cases look like, the model will reflect your assumptions instead of reality.
The cost that shows up on the invoice, and the one that doesn't
The training run is typically the smallest expense in the process. It is also the only cost that people include in their initial estimates.
What actually gets budgeted
Calculate the cost of the training run, any platform fees from managed services, and the engineering time required to prepare data and start the process. These costs are often low for many tasks. This low initial cost is why projects are approved quickly, though unexpected expenses often appear later.
What doesn't
If you do not serve a fine-tuned model through the same managed endpoint used for training, you will face continuous hosting costs. Evaluating the model requires a dedicated project. You must build a test set to measure actual improvements instead of judging the results based on a few examples. This process requires significant time and a clear definition of what constitutes success for your specific task. You must repeat every one of these steps whenever the underlying base model is updated, which is a requirement that many people overlook.
The maintenance bill you signed up for without noticing
Most discussions about fine-tuning do not include this section. However, this information determines if a project remains successful one year after completion.
Base models continue to develop independently
A fine-tuned model is a fixed version of a specific base model. Providers release new base models on their own schedules and eventually retire older versions. When a base model is deprecated, you must either recreate your fine-tune using the new version or continue using an outdated model that lacks recent improvements. Recreating a fine-tune requires significant work. Because different base models respond to training data in different ways, a process that worked for one version may produce unexpected results on another. You must verify the performance of the new model through testing rather than assuming it will function the same way.
Your inputs are also changing
Your own inputs change too. The products change, the policy wording changes, the way customers phrase complaints changes as new channels open. A fine-tune trained on last year's tickets slowly drifts out of step with this year's, in the same quiet way a forecasting model drifts once the world it was trained on moves on. Catching that needs the same discipline as monitoring any model in production: tracking real outputs against expectations over time, not trusting that a model which was good at launch stays good.
You do not finish your work on a fine-tuned model after you deploy it. You are now responsible for maintaining it, similar to how you manage software that relies on an external dependency with an unpredictable update schedule.
Who actually owns it afterwards
We must determine who is responsible for monitoring when a fine-tuned model becomes outdated, retraining it, evaluating the update, and deploying it without disrupting existing dependencies. In many projects that fail without notice, no one is assigned these tasks. This occurs because the project plan stops at the initial deployment and the schedule for retraining is never documented or added to a calendar.
Catastrophic forgetting and other types of model failure
Fine-tuning adds specific behaviours to a model, but it can also remove existing ones. This removal is usually unintentional. If you train a model too intensely on a narrow set of examples, it may lose the general capabilities it previously possessed. Researchers refer to this failure as catastrophic forgetting. The model becomes highly proficient at the training task but performs significantly worse on related tasks that were not included in the testing process.
A specific version of this problem occurs when a model overfits to the specific details of a training set instead of the intended pattern. If every training example ends with the same phrase, the model treats that phrase as a rule rather than a coincidence. It will then add that phrase to answers where it does not belong. Small training sets that do not represent the broader data are particularly susceptible to this issue. These sets do not contain enough information for the model to distinguish between meaningful patterns and accidental formatting. Finding the correct balance requires the same approach used to tune other model settings. A training process that runs for too long results in overfitting, while a process that is too short results in underfitting. You find the optimal point through testing rather than guessing.
When fine-tuning is the correct choice
Fine-tuning is necessary in specific situations where other methods are less effective. A classifier that must sort inbound documents into forty categories specific to your business will perform better with direct training. This is more efficient than using a prompt to describe those unknown categories during every call. Fine-tuning is also useful for models that must produce a rigid structured format at high volume. Because the format is integrated into the model, there is less risk that the output will deviate and cause errors in downstream systems. Finally, fine-tuning is a practical choice for large-scale tasks. Reducing a long prompt filled with examples to a shorter one for a fine-tuned model results in significant cost savings on every call.
These cases all involve narrow and stable behaviours where users have already attempted to use prompts. They reached a point where adding more instructions did not solve the problem. When you reach that limit, the effort required to perform and maintain fine-tuning becomes a practical choice.
Where to start
Do not begin with the training process. Instead, document exactly what the model is failing to do. Determine if the issue is a lack of knowledge, a failure to follow instructions, or a specific behavioural pattern. Most problems that appear to be behavioural are actually caused by missing information or poor instructions. You can often resolve these issues with better prompting or a retrieval layer rather than using more complex methods.
If you find that the performance gap is behaviour-specific and stays consistent, gather real examples of the problem. Recognise the complexity of these examples before you begin building. You need to determine who will be responsible for retraining the model prior to deployment. Do not wait for a support ticket to indicate that a provider update altered the model's tone. Fine-tuning works well for specific tasks, yet it demands a long-term maintenance commitment. If you are uncertain whether your task needs fine-tuning, or if your model is drifting without clear ownership, talk to us.
If you wish to integrate this into your own systems rather than a demo, you can use our AI development services.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG pulls relevant information when a question is posed and feeds it to the model, enabling updates to the source material without retraining the model. Fine-tuning changes the model's internal parameters, making it better suited for consistent behaviour or style rather than for altering facts. When the problem is a lack of knowledge, RAG is usually the preferred solution.
What is fine-tuning in AI?
Fine-tuning in AI means taking a pre-trained model and training it further on a small, specific set of examples. This process adjusts the model's default behaviour to align with the style of those examples. It changes how the model responds instead of expanding its knowledge base. Therefore, fine-tuning works well for adjusting tone and format, but it is not a reliable method for teaching new facts. For most tasks, using prompting or retrieval methods is preferable before turning to fine-tuning.
What kind of data does fine-tuning need?
Good training data includes examples that match the format you expect the model to generate, and it must contain sufficient examples with enough variation to cover the situations the model will face. A few hundred examples can tweak the model's tone for a specific, limited task, whereas larger changes to the model's behaviour typically need several thousand examples. When teams closely examine their data, they often discover it is less organised and consistent than they first believed.
What ongoing maintenance does a fine-tuned model need?
Fine-tuned models need oversight because providers retire base models on their own schedules. You must monitor these models as your inputs, products, and customer language evolve, just like any other production model. If you do not designate an owner to handle retraining, fine-tuned models will eventually lose effectiveness.
What is catastrophic forgetting in fine-tuning?
Catastrophic forgetting happens when fine-tuning concentrates a model on a specific set of examples, causing it to lose its previous general capabilities. The model gets better at the specific task but performs worse on related tasks that were not tested. A similar problem is overfitting, which occurs when small or unrepresentative datasets cause the model to treat coincidental patterns as fixed rules.