A model that sailed through its demo starts making odd calls the week it goes live. Training that should take an afternoon runs for three days and still lands short of a simple baseline. Two data scientists run the same dataset and get results ten points apart, and neither can say why. Underneath a lot of these moments sits the same quiet culprit: the hyperparameters nobody stopped to set properly.
Hyperparameters are the settings you choose before a machine learning model starts training. The learning rate, the batch size, how many layers a network has, how deep a decision tree can grow. Think of them as the dials on an oven. You set the temperature and the timer before the tray goes in, and the model cannot reach out and change them while it bakes. Get them roughly right and an ordinary algorithm behaves. Get them wrong and even a good algorithm on clean data will lose to something far simpler.
We build data and AI systems at Shipshape Data, and hyperparameter tuning is one of those jobs that sounds academic until it quietly decides whether a project ships. This guide covers what these settings are, why they move the numbers so much, the three tuning methods worth knowing, the parameters that matter for the algorithms you are most likely to meet, and the questions a non-technical leader can ask to keep the work honest. No maths degree required.
Parameters and hyperparameters are not the same thing
The two words look almost identical and the distinction trips people up constantly, so it is worth thirty seconds. Parameters are what the model learns on its own. The weights in a neural network, the split points in a decision tree, these get adjusted automatically as training runs, and you never set them by hand. Hyperparameters are the settings that sit above that process and shape how the learning happens in the first place.
A learning rate of 0.001 is a hyperparameter you pick. The specific weight of 0.42 that the network settles on for one connection is a parameter it worked out for itself. You control the first directly and the second only indirectly, by choosing sensible hyperparameters and letting training do the rest. That is really the whole game. You are not tuning the model's answers, you are tuning the conditions under which it finds them.
Why hyperparameters move the numbers so much
Your choices here decide whether a model produces predictions you can rely on or an expensive pile of nearly-right. Set them badly and the model either learns too slowly, burning compute and budget, or learns the wrong lesson entirely. The distance between a well-tuned configuration and a careless one is routinely the distance between 85% and 95% accuracy in production, and that gap is not abstract. It is fewer wrong calls in a customer-facing app, less manual review to catch the errors, and more trust from the people who have to sign the thing off.
Two ways it goes wrong: overfitting and underfitting
Overfitting is when your settings let the model memorise the training data instead of learning patterns that carry over. It looks superb on the data it has already seen and falls apart the moment a real customer types something slightly unfamiliar. Underfitting is the opposite failure: settings so cautious the model never captures the real relationships in the data, so it is mediocre everywhere, in development and in production alike. Both waste engineering time, and both chip away at stakeholder confidence, which is usually the harder thing to win back.
A useful way to picture the two big dials. The learning rate is how large a step the model takes each time it updates. Too big and it strides straight past the good solution, over and over, never settling. Too small and it inches along, taking far longer than the budget allows or getting stuck somewhere mediocre. Batch size is how many examples the model looks at before each update. Bigger batches train faster but need more GPU memory and can, oddly, land on slightly worse final accuracy. Smaller batches update more often and often generalise a touch better, at the cost of time.
Poor hyperparameter configuration is one of the quiet reasons AI pilots stall in the lab and never reach production.
Three ways to tune, from brute force to clever
You need a method, because poking at values by hand and hoping is how weeks disappear. The three approaches below cover almost everything you will do in practice. Which one fits depends on how many settings you are tuning, how long a single training run takes, and how much compute you can spend.
Start with sensible defaults and iterate
Before any search, start where the algorithm's own documentation tells you to. A learning rate of 0.001 is a reasonable opening bid for many neural networks; gradient boosting models often start around 100 trees. Defaults give you a working model fast and, more importantly, a benchmark. Run it against a held-out validation set so you know what current performance looks like, then change one setting at a time and watch what each move does to accuracy, training time and cost. Half the value of tuning is simply having a number to beat.
Grid search: exhaustive and expensive
Grid search tries every combination you list. You give it, say, learning rates of 0.001, 0.01 and 0.1 and batch sizes of 32, 64 and 128, and it dutifully trains all nine models and reports the best. Within the box you have drawn, it will find the winner, and that thoroughness is comforting. The problem is the box grows fast. Add a third setting with a few options each and you are training dozens or hundreds of models, which gets painful for anything large. Grid search earns its place when you have only a couple of settings to tune and compute to spare, ideally with parallel infrastructure so the runs happen at once rather than in a long queue.
Randomised search: more ground for the money
Randomised search samples combinations at random from ranges you define, rather than marching through every option. It sounds worse and is usually better. Fifty random trials will often land near the best configuration faster than a hundred exhaustive ones, because most of the time only two or three settings really matter, and random sampling spreads its effort across them instead of re-testing the ones that barely move the needle. Set the ranges using whatever prior knowledge you have about where good values tend to sit, and let it explore.
Bayesian optimisation: learning from each trial
Bayesian optimisation uses the results so far to decide what to try next. It builds a rough model of how the settings affect performance, and every trial sharpens that model, so later choices are better informed than random ones. When a single training run is slow or costly, this pays for itself quickly, because it wastes fewer runs on configurations that were never going to work. The tuning process becomes a small learning system of its own. AWS and Google Cloud both offer managed services that run this for you, which takes most of the setup pain out of it.
The settings that matter, by algorithm
Different algorithms hand you different dials, and knowing which ones actually move performance saves you from fiddling with settings that barely register. Here are the ones worth your attention for the models you are most likely to be running.
Neural networks and deep learning
Neural networks give you the most to configure, which is both their strength and the reason they are fiddly. Learning rate is the one to get right first. Values usually sit between 0.0001 and 0.1, with 0.001 a common starting point; below 0.0001 the network crawls, above 0.1 training often goes unstable and thrashes. Batch size comes next: 32 or 64 give frequent updates and tend to generalise well, while 256 or 512 train faster but ask for more GPU memory and sometimes cost you a little accuracy at the end.
The number of layers and the neurons in each set the network's capacity. More depth can capture more intricate patterns and also overfits more readily on a small dataset, so two or three hidden layers is a sane start for a straightforward problem, scaling up only when the task genuinely needs it, image or language work being the usual reason. Dropout is your main guard against overfitting here: it switches off a random share of neurons during training so the network cannot lean too hard on any one path. Rates between 0.2 and 0.5 cover most cases, higher for stronger regularisation at the cost of slower convergence.
Gradient boosting and tree-based models
Tree-based methods like XGBoost and LightGBM run on a different set of dials, and for a lot of tabular business problems they are the better tool anyway. Learning rate, sometimes called eta here, plays a similar role but likes a different range, usually 0.01 to 0.3, with lower values needing more trees to get there. The number of estimators sets how many trees get built in sequence: 100 is a fair start, and complex problems may want 1000 or more, watching validation performance so you stop before the model starts memorising.
Maximum depth caps how many splits each tree makes. Shallow trees of depth three to five train quickly and generalise well; deeper trees of ten to twenty catch subtler structure but drift toward memorising the training set. Minimum child weight sets the smallest amount of data allowed in a new leaf, and raising it, somewhere between 1 and 10, stops the model carving out leaves that describe a mere handful of examples. One quiet advantage: gradient boosting models often reach production-ready performance with just three to five settings tuned, which makes them far more approachable than deep learning for a lot of enterprise work.
Support vector machines
Support vector machines turn on how they draw the boundary between classes. The C parameter trades a smooth boundary against classifying every training point correctly: low values like 0.1 give a wider margin and tolerate more errors, high values like 10 or 100 fit the training data more tightly and risk overfitting. Kernel type sets the shape of the boundary, a linear kernel for cleanly separable data, a radial basis function kernel for messier patterns. With an RBF kernel, gamma controls how far a single example's influence reaches: small values like 0.01 spread it broadly, large values like 1.0 keep it tightly local. C and gamma interact, so they are a natural pair to tune together rather than one after the other.
What changes once it is a real project
Enterprise deployments run into constraints a research paper never mentions. Budgets, deadlines, and stakeholders who quite reasonably want a working system rather than a beautiful experiment. The pursuit of the theoretically optimal model bumps up against compute cost, ship dates, and whatever it will take to keep the thing running afterwards. That changes how you tune.
Time and money set the boundaries
Most enterprise AI work gets weeks, not months, for building and validating a model. You are not going to run an exhaustive grid search over hundreds of GPU hours when the cloud budget tops out at a few thousand pounds. So randomised search and Bayesian optimisation stop being nice-to-haves and become the sensible defaults. A good team learns which settings matter most for the problem in front of them, spends the tuning effort there, and accepts documented defaults for the rest. Early stopping helps too: kill the training runs that look hopeless in the first few epochs rather than letting them finish out of politeness.
Accuracy is not the only cost
A model in production keeps costing money every time it makes a prediction. A deeper network might be a couple of points more accurate and also three times more expensive to run at scale, and that expense accrues quietly for as long as the model lives. Sometimes 93% from a lightweight model beats 96% from a heavyweight once you count the inference bill across a year. Model size and speed become settings you tune alongside accuracy, not afterthoughts. This is the economic arithmetic the papers skip and the finance team never does.
Tuning does not stop at launch
Production data shows you patterns the training set never did, so hyperparameters that were right at launch drift out of date as the world moves. Monitoring should track performance in production and flag when accuracy slips below the line you have agreed. Then you retune and retrain on fresher data. The settings that win on a validation set in development are frequently not the ones you want serving live traffic, and the only way to know is to keep watching. Treat configuration as an ongoing job, not a box ticked once at the start.
Questions a non-technical leader should ask
You do not need the maths to keep this work on track. Most AI projects that fail do so through weak management of the technical work, not through a shortage of technical skill, and a few pointed questions will tell you quickly whether your team is being disciplined or just busy. Set the boundaries, ask for the evidence, and hold people to defined inputs and measurable outputs the way you would on any other engineering job.
Agree the compute budget before tuning starts
Put a number on what the team can spend on cloud compute during experiments, and ask them to estimate it up front from their planned approach and dataset size. If they blow past it, ask for the reasoning in writing, which quietly forces the question of whether the next batch of experiments is actually worth it. This one habit stops a team reaching for an expensive grid search when a cheaper randomised one would have done the job.
Insist on a final test on data nobody has touched
Make the team prove performance on a held-out test set that stayed sealed throughout tuning. Models tuned against a validation set often quietly overfit to it, then disappoint in production. Keep a final slice, 10 to 20% of the data, that the chosen configuration is measured against exactly once. That single discipline catches most overfitting before it reaches a customer.
Ask what happens when the data shifts
Customer behaviour changes, and hyperparameters that fit last year's patterns slowly stop fitting. Ask how the team will spot performance degrading in production, and what the retraining schedule looks like when it does. If the answer is vague, that is the gap, and it is far cheaper to find it now than during an incident.
Where to start
If you take three things from this: begin with documented defaults and a benchmark, reach for randomised search or Bayesian optimisation before grid search unless the problem is genuinely tiny, and never let a model ship without a final test on data it has never seen. Most of the expensive failures we get called in to fix trace back to one of those three being skipped, not to anything exotic.
Getting an AI project from a promising experiment to something that earns its keep in production takes judgement about data preparation, model choice and, yes, hyperparameters, applied under real constraints rather than ideal ones. That is the work we do. If you would like a clear read on where your current approach is solid and where it is quietly leaking time or accuracy, talk to us and start with a straight assessment rather than a guess.