A model that performed well in its demonstration starts to give unexpected results in the first week of production. Training processes expected to finish in one afternoon run for three days without achieving a basic performance level. Two data scientists use the same dataset, produce results that differ by ten per cent, and neither can explain the discrepancy.
Many of these problems stem from hyperparameters that were not configured correctly.
Hyperparameters are the settings you choose before a model starts training. These comprise the learning rate, batch size, or the maximum depth of a tree. The model cannot modify these values after the training process begins. If you select the correct settings, a standard algorithm will work effectively. If you select the wrong settings, a sophisticated algorithm will perform worse than a simpler one.
Hyperparameters are the settings you select before a machine learning model begins training. The learning rate, the batch size, the number of layers in a network, and the maximum depth of a decision tree. Consider them as the dials on an oven. You set the temperature and the timer before the tray goes in, and the model cannot change them while it bakes. Get them roughly right and an ordinary algorithm will behave properly. Get them wrong and even a good algorithm on clean data will be outperformed by something much simpler.
Shipshape Data constructs data and AI systems. Hyperparameter tuning is a practical task that decides whether a project succeeds or fails. This guide describes what these settings are and why they significantly affect performance. It covers three effective tuning methods and the specific parameters employed in common algorithms. It also provides questions that non-technical leaders can use to monitor a project's progress. You do not need a mathematics degree to grasp these concepts.
Parameters and hyperparameters differ
These two terms appear similar and are often confused. Parameters are the values that a model learns on its own. Examples include the weights in a neural network or the split points in a decision tree. These values are adjusted automatically during training and are not set manually. Hyperparameters are the settings that control the training process itself.
A learning rate of 0.001 is a hyperparameter you select. The specific weight of 0.42 that the network assigns to a connection is a parameter it calculates on its own. You control the learning rate directly. You control the weight only indirectly by selecting effective hyperparameters and letting the training process run. This process involves adjusting the conditions under which the model determines its own answers rather than choosing the answers yourself.
Why hyperparameters move the numbers so much
These settings determine whether a model generates reliable predictions or inaccurate results that waste resources. If you choose poor settings, the model will either learn too slowly, increasing costs, or fail to learn the correct patterns. A well-tuned configuration often yields 95% accuracy compared to 85% for a poorly configured one. This difference has practical consequences. It leads to fewer errors in customer applications, reduces the need for manual review, and increases confidence among stakeholders.
Two ways it goes wrong: overfitting and underfitting
Overfitting occurs when model settings cause the system to memorise training data instead of identifying patterns that apply to new information. The model performs well on its training data but fails when a user provides unfamiliar input. Underfitting happens when settings are too restrictive for the model to identify the actual relationships within the data. This results in poor performance during both development and production. Both issues waste engineering resources and reduce stakeholder confidence. This loss of trust is often difficult to recover.
A practical method for visualising the two primary controls.
The learning rate determines the size of the step the model takes at each update. If the rate is too high, the model will overshoot the optimal solution repeatedly and fail to converge. If the rate is too low, the model moves slowly, consuming more time than planned or causing it to stop at a suboptimal point. The batch size is the number of examples the model processes before updating its parameters. Larger batches finish training more quickly but require more GPU memory and can lead to lower final accuracy. Smaller batches cause more frequent updates and typically produce better generalisation, though they increase total training time.
AI projects often fail to move from testing to production because of incorrect hyperparameter settings.
Three ways to tune, from brute force to clever
You need a systematic method because manually testing values without a plan causes significant time loss. The three approaches below cover most practical scenarios. You should choose an approach based on the number of settings you are tuning, the duration of a single training run, and your available computing resources.
Start with sensible defaults and iterate
Start your search by following the documentation for your specific algorithm. A learning rate of 0.001 is a standard starting point for many neural networks. Gradient boosting models typically begin with approximately 100 trees. Using default settings lets you create a functional model quickly and establishes a baseline for comparison. Test this model against a separate validation set to determine its current performance level. After establishing this baseline, adjust one setting at a time. Observe how each change affects accuracy, training time, and associated costs. A significant benefit of the tuning process is establishing a specific performance metric that you can try to improve.
Grid search: exhaustive and expensive
Grid search tests every combination of parameters in a list. If you provide learning rates of 0.001, 0.01, and 0.1 together with batch sizes of 32, 64, and 128, the system trains all nine models and identifies the best one. This method finds the optimal result within the specific range you define. However, the number of combinations increases quickly. Adding a third parameter with several options results in training dozens or hundreds of models. This process is difficult for large projects. Grid search is useful when you have only a few settings to adjust and sufficient computing power. It works best with parallel infrastructure so that the models train simultaneously rather than one after another.
Randomised search: more ground for the money
Randomised search selects combinations at random from defined ranges instead of testing every possible option. This method is often more effective than exhaustive searching. Fifty random trials frequently find a high-performing configuration faster than one hundred exhaustive trials. This happens because only a few settings typically influence the results significantly. Random sampling focuses effort on these important variables rather than repeating tests on settings that do not change the outcome. You should set the search ranges based on your existing knowledge of effective values and let the process explore those areas.
Bayesian optimisation: learning from each trial
Bayesian optimisation uses previous results to determine the next settings to test. The system creates a model of how different parameters influence performance. Each trial provides more data to improve this model, which makes later selections more accurate than random sampling. This method is useful when training runs are expensive or time-consuming because it reduces the number of unsuccessful configurations. The tuning process functions as its own learning system. AWS and Google Cloud provide managed services for this process to simplify the initial setup.
Important settings for each algorithm
Different algorithms provide different parameters. Knowing which settings significantly affect performance prevents you from wasting time on adjustments that have little impact. The following parameters are the most important for the models you will likely use.
Neural networks and deep learning
Neural networks have many configuration options. This makes them versatile, but it also makes them difficult to set up correctly. You should adjust the learning rate first. These values are typically between 0.0001 and 0.1. A common starting point is 0.001. If the value is below 0.0001, the network trains too slowly. If it is above 0.1, the training process often becomes unstable. You should choose the batch size next. Sizes of 32 or 64 provide frequent updates and usually help the model generalise well. Sizes of 256 or 512 allow for faster training, but they require more GPU memory and can result in lower final accuracy.
The number of layers and the neurons in each layer determine the capacity of the network. Increased depth allows the model to identify complex patterns, but it also increases the risk of overfitting on small datasets. Starting with two or three hidden layers is appropriate for simple problems. You should only add more layers if the task requires it, which is common in image or language processing. Dropout is the primary method for preventing overfitting. This technique deactivates a random percentage of neurons during training to prevent the network from relying too much on specific paths. Rates between 0.2 and 0.5 are effective for most situations. Higher rates provide stronger regularisation but result in slower convergence.
Gradient boosting and models based on trees
Tree-based methods such as XGBoost and LightGBM use different parameters, and they are often more effective for tabular business problems. The learning rate, which is sometimes called eta, functions similarly to other models but typically ranges from 0.01 to 0.3. Lower values require a higher number of trees to reach the same result. The number of estimators determines how many trees are built in sequence. You can start with 100 trees, though complex problems might require 1000 or more. You should monitor validation performance to ensure you stop before the model begins to memorise the data.
Maximum depth limits the number of splits in each tree. Shallow trees with a depth of three to five train fast and generalise well. Deeper trees with a depth of ten to twenty capture more complex patterns but tend to memorise the training data. Minimum child weight defines the smallest amount of data required in a new leaf. Increasing this value to between 1 and 10 prevents the model from creating leaves that only describe a few examples. One benefit is that gradient boosting models often achieve production-ready results by tuning only three to five settings. This makes them easier to use than deep learning for many business tasks.
Support vector machines
Support vector machines function by establishing a boundary between different classes. The C parameter balances the smoothness of this boundary against the accuracy of training point classification. Low values such as 0.1 create a wider margin and allow for more errors, while high values like 10 or 100 fit the training data closely and increase the risk of overfitting. The kernel type determines the shape of the boundary. A linear kernel is used for data that can be separated cleanly, while a radial basis function kernel is used for more complex patterns. When using an RBF kernel, the gamma parameter dictates the range of influence for a single data point. Small values like 0.01 result in a broad influence, whereas large values like 1.0 restrict influence to a local area. Because C and gamma interact, you should tune them together as a pair instead of adjusting them individually.
What changes once it's a real project
Deploying systems in a business setting creates problems that academic research often does not address.
Projects involve budgets, deadlines, and stakeholders who require functional systems rather than experimental models. The goal of finding a theoretically optimal model is limited by compute costs, delivery dates, and long-term maintenance requirements. These practical constraints dictate how you approach the tuning process.
Time and money establish the limits
Most enterprise AI projects allow weeks rather than months to build and validate a model. You cannot run an exhaustive grid search using hundreds of GPU hours when the cloud budget is limited to a few thousand pounds. Randomised search and Bayesian optimisation are therefore the standard choices rather than optional features. An effective team identifies which settings are most important for their specific problem, focuses their tuning efforts on those areas, and uses documented defaults for other parameters. Early stopping is also useful. You should end training runs that show poor results in the first few epochs instead of allowing them to finish.
Accuracy is not the only expense
Running a model in production incurs a cost for every prediction it generates. A deeper network may provide a small increase in accuracy while being three times more expensive to operate at scale. This expense continues to accumulate throughout the life of the model. In some cases, a lightweight model with 93% accuracy is more practical than a heavyweight model with 96% accuracy when considering the total annual inference costs. Model size and speed are parameters that must be optimised alongside accuracy rather than being treated as secondary concerns. Research papers often ignore these financial calculations, but finance departments prioritise them.
Tuning continues after the launch
Production data reveals patterns that were not present in the training set. Hyperparameters chosen at launch become less effective as real-world conditions change. You should monitor production performance and set alerts for when accuracy falls below your established threshold. When this happens, you must retune the model and retrain it using more recent data. The settings that perform best on a validation set during development are often different from the settings required for live traffic. You can only identify these differences by continuous observation. Configuration is a repetitive task rather than a single step completed at the beginning of a project.
Questions a non-technical leader should ask
You do not need to understand the mathematics to manage this work effectively. Most AI projects fail because of poor management rather than a lack of technical skill. You can determine if your team is disciplined or merely busy by asking specific questions. You should set clear boundaries, request evidence, and require defined inputs and measurable outputs as you would for any other engineering project.
Set the compute budget before you begin tuning
Set a defined budget for cloud computing expenses during experiments. Ask the team to forecast these expenses beforehand, using their planned methods and dataset sizes. If the team goes over the budget, demand a written justification for the excess. This procedure prompts the team to assess whether further experiments are needed. This approach stops teams from employing costly grid searches when cheaper randomised searches will do.
Insist on a final test on data nobody has touched
Make the team prove performance on a held-out test set that stayed sealed throughout tuning. Models tuned against a validation set often quietly overfit to it, then disappoint in production. Keep a final slice, 10 to 20% of the data, that the chosen configuration is measured against exactly once. That single discipline catches most overfitting before it reaches a customer.
Ask about the effects of data shifts
Customer behaviour evolves, making hyperparameters derived from old patterns less effective. Ask the team how they will detect performance drops in production and what their retraining timetable will be. A vague response indicates a major risk. Fixing this issue now costs less than waiting for a service incident.
Where to start
Follow three primary practices. First, begin with documented default settings and a benchmark. Second, apply randomised search or Bayesian optimisation rather than grid search, except when the problem is very small. Third, refrain from deploying a model until it has been tested on never-seen data. Most costly failures we are hired to resolve stem from omitting one of these steps, not from complex technical issues.
Transitioning an AI project from experiment to production tool demands specific decisions on data preparation, model selection, and hyperparameters. These decisions need to be made within real operational constraints, not ideal conditions. That is the service we offer. If you seek an objective assessment of where your current method works and where it loses time or accuracy, talk to us. We deliver a direct assessment rather than an estimate.
Many AI projects fail as teams find it hard to apply these concepts effectively. Our AI consultancy assists organisations in bridging this gap and deploying these tools.
Frequently asked questions
What are hyperparameters?
Hyperparameters are the settings selected before a model begins training, like learning rate, batch size, number of layers, or tree depth. The model cannot modify them after training starts, whereas parameters are adjusted automatically during training. Setting them approximately correctly often determines whether a model performs well in production or silently loses to a simpler method.
What is the difference between parameters and hyperparameters?
Parameters are the values a model develops independently during training. Examples are neural network weights or split points in a decision tree. These values adjust automatically as training proceeds. Hyperparameters are the external settings you set before training to control how the model learns. The learning rate is an example of a hyperparameter. You directly control hyperparameters, while you influence parameters only indirectly by choosing settings and letting training determine the final values.
How do you grid search hyperparameters?
Grid search trains on every combination of the values you supply. For instance, it will test three learning rates against three batch sizes and return the best-performing combination. This method is exhaustive within your defined parameters, yet the required runs grow rapidly. Adding a third parameter with multiple options can lead to training dozens or hundreds of models. Apply this approach when you have only a few settings to tune and enough computing resources.
How do you choose hyperparameters?
Start with the documented defaults for the algorithm, such as a learning rate around 0.001 for many neural networks, and use that as a benchmark to beat. From there, randomised search or Bayesian optimisation are the pragmatic defaults for most enterprise projects, since randomised search often reaches around 90% of grid search's result for a fraction of the compute. Bayesian optimisation is worth the extra setup when a single training run is slow or costly.
How do you tune hyperparameters in deep learning?
Set the learning rate first. This value typically lies between 0.0001 and 0.1, with 0.001 as a common starting point. Select the batch size next. Batch sizes of 32 or 64 give more frequent updates and usually generalise well. Batch sizes of 256 or 512 enable faster training but need more GPU memory. Dropout is the main technique for preventing overfitting and is usually set between 0.2 and 0.5. It disables a random fraction of neurons during training to stop the network from relying too much on specific paths.