A team spent three weeks tuning a churn model. Better boosting parameters, deeper trees, a wider search over learning rates, nothing moved the accuracy more than a point. Then someone finally read the column list properly and found "days_since_last_contact", a field that got written by the same support process that logged a cancellation. The model had not learned who was about to leave. It had learned to read a field that was, in effect, already reporting the answer. Once that one column came out, accuracy dropped nine points on the leaderboard and rose sharply in production, which tells you which of those two numbers had ever been real.
Feature engineering is the work of turning raw data into the inputs a model actually learns from: selecting which fields deserve a place, transforming them into a shape a model can use, encoding categories as numbers, deciding what to do about the gaps, and checking that nothing in the input set is quietly reporting the future. None of it is glamorous. Most of it happens before anyone opens a notebook to choose an algorithm, and most of the eventual model's performance was decided right there.
We build data and AI systems at Shipshape Data, and the pattern above is one of the more common ways a project stalls. A client arrives wanting to talk about model architecture, and the actual gap turns out to sit two steps earlier, in what the model was ever given to look at. This guide covers what a feature is and why it matters more than the model, how to select and transform raw fields, how to encode categorical variables without creating a mess bigger than the one you started with, how to handle missing values honestly, and how leakage sneaks into a dataset and quietly wrecks the evaluation.
Why feature quality usually beats model choice
Swap a logistic regression for a gradient-boosted tree and you might gain a few points of accuracy, if the features are already good. Swap a set of weak, leaky, badly encoded features for a clean, well-chosen set and the gain is often much larger, on any model you try. That asymmetry is the whole argument for spending time here rather than rushing to the modelling step.
The reason is not mysterious. A model can only find patterns that exist in what it is shown. Feed it a column that mixes signal with noise, or a category flattened into an arbitrary number, and you have hidden the pattern before the algorithm gets a turn. No amount of tuning recovers information never given a usable shape.
The part that does not show up in a demo
This is also why feature work is unevenly distributed across a project. A working prototype in a notebook can look strong on a clean sample and fall over on the live feed, because the notebook quietly assumed fields would always be populated and categories would stay within a known set. Production data does not respect any of that. The features that survive contact with a real pipeline are usually simpler than the ones a first pass produces, and simpler by design.
What counts as a feature, and what does not
A feature is an input variable, in the shape a model consumes, that is meant to carry information relevant to the thing you are predicting. Raw data becomes a feature once someone has decided it matters and put it into a form a model can read: a number, a category, a vector. Not every column in your source system deserves that treatment, and treating every available field as a feature is a reliable way to drown the useful ones in noise.
Raw fields are not features yet
A customer record might hold forty columns. A handful, order history, tenure, support contacts, probably say something about whether that customer churns. Most of the rest, an internal record ID, an unstandardised notes field, a timestamp from an unrelated batch job, say nothing useful and can actively mislead a model that goes looking for patterns anywhere it is pointed. The job starts with an unglamorous question: what, out of everything available, is actually related to the outcome you are trying to predict?
Answering that takes some domain knowledge, not just statistics. A data scientist working alone can compute correlations all day and still miss that a particular field is populated by a process that only runs for customers who already cancelled. Somebody who knows the business, or at least knows how the data was produced, catches that in a five-minute conversation. Bring them in early rather than after the model has already learned the wrong lesson.
Selecting the features that matter
More features are not automatically better. Every additional column gives a model another way to find a coincidental pattern in a limited training sample, and coincidental patterns are exactly what causes a model to perform brilliantly in testing and unreliably everywhere else. That failure mode has a name, and it is worth reading our guide to overfitting if a model's training score looks better than it has any right to be.
Filter, wrapper and embedded methods
Filter methods score each feature against the target on its own, using something like correlation or mutual information, and drop the ones that score badly. Fast, easy to explain, and blind to how features interact with each other, so a field that is only useful in combination with another gets discarded by mistake. Wrapper methods try subsets of features against the actual model and keep whichever combination performs best, catching interactions but getting expensive fast as the feature count grows. Embedded methods build selection into training itself, Lasso regression zeroing out weak coefficients as it fits, a tree ranking features by how much they reduce error when it splits on them. Most teams reach for embedded methods first, because the selection comes nearly free as part of a step they were doing anyway.
Correlation is not the same as usefulness
A feature can correlate strongly with the target and still be useless, because it is a proxy for something you should not be predicting on, or because the correlation only held in the training window. A feature can also correlate weakly on its own while being genuinely valuable in combination with another. Selection is judgement applied with statistics, not statistics applied instead of judgement. Start with what a domain expert would expect to matter, check that against the data, and be suspicious of anything that looks too strong to be true. It usually is.
Transforming raw values into something a model can learn from
Selection decides what stays in. Transformation decides what shape it takes once it is in. Most raw fields need reshaping before a model can extract anything useful from them.
Scaling
A feature measured in the tens of thousands sitting next to one measured between zero and one will dominate any model sensitive to magnitude, distance-based methods especially. Standardisation rescales a feature to zero mean and unit variance. Min-max scaling squeezes everything into a fixed range, usually zero to one. Tree-based models mostly do not care about scale, which is one reason they are a forgiving place to start, but almost everything else does.
Binning and date decomposition
A continuous value sometimes carries more signal once it is grouped: age turned into bands, income turned into brackets, because the relationship with the target is not smooth, it is a step function, and forcing a linear model to fit a step function with a single continuous input wastes its capacity. Dates are their own small trap. A raw timestamp is nearly useless to most models. Decompose it, day of week, month, whether it is a weekend, and you have handed the model variables it can actually use instead of a large number with no learnable structure.
Interaction terms
Sometimes the useful signal is not in any single field but in how two combine: price per square foot rather than price and floor area held separately, spend per visit rather than spend and visit count as two unrelated columns. Building these by hand takes domain knowledge, and it is often the single highest-value thing a subject expert contributes, because no selection algorithm will invent a ratio nobody told it to consider.
Encoding categorical variables without making a mess
Models do arithmetic. Categories are not numbers, and turning "London", "Manchester" and "Bristol" into 1, 2 and 3 tells a model that Bristol is three times London, which is nonsense the model has no way of knowing is nonsense. Encoding is the set of techniques for representing categories numerically without inventing a false order.
One-hot, ordinal and target encoding
One-hot encoding gives each category its own binary column: a City_London column, a City_Manchester column, and so on, each one flagging whether a given row belongs to that category. It is the safe default for categories with no natural order and a modest number of values. Ordinal encoding assigns an integer where an order genuinely exists, small, medium, large, and only there, because using it on a field with no real order brings back the same false-distance problem one-hot was built to avoid. Target encoding replaces a category with the average outcome for that category, London customers get the average churn rate for London customers, which handles high-cardinality fields gracefully but needs care, because it leaks a little of the target straight into the feature.
High cardinality is the recurring headache
A categorical field with three values is trivial. One with fifty thousand product codes is a genuine design decision: one-hot encode it and you have added fifty thousand mostly-empty columns. The usual moves are grouping rare categories into an "other" bucket, hashing into a fixed number of buckets, or reaching for embeddings if the field is central enough to justify the extra complexity. None of these is free. Grouping loses detail, hashing risks collisions, embeddings need enough data to learn from. Pick the one that matches how much the field actually matters, not the one that looks most sophisticated on a slide.
Handling missing values honestly
Missing data is not an edge case. It is close to the default state of most real datasets, and pretending otherwise produces a model that either crashes on a live feed or, worse, runs quietly and confidently on garbage.
Deletion, imputation and the missingness signal
Dropping rows with missing values is the simplest option and the one that scales worst, because if enough fields have a few gaps each, the rows that survive deletion can be a small and biased slice of the data. Dropping a column entirely makes sense when the field is mostly empty, and at that point it is a data quality issue rather than a modelling one. Imputation fills the gap instead: mean or median for numeric fields, the most frequent value or an "unknown" category for categorical ones, or a model trained to predict the missing value from everything else, more accurate and considerably more work.
The detail that gets missed most often: the fact that a value is missing can itself be informative. A customer who never filled in their income field might behave differently from one who did, for reasons that have nothing to do with income and everything to do with why they skipped the form. Adding a binary "was this missing" flag alongside the imputed value keeps that signal instead of quietly erasing it during imputation. It is a small addition and it is one of the more reliably useful ones.
Consistency between training and production
Whatever rule you choose has to run identically on new data as it did on training data, using statistics computed on the training set alone rather than recalculated fresh each time. A median imputed from this month's incoming batch is not the same number as the median computed during training, and a model quietly fed a shifting imputation value will drift in ways that are hard to trace back to the actual cause. Write the rule down, apply it as a fixed pipeline step, and do not let it recompute itself on live data.
Leakage: the feature that already knows the answer
Leakage is when a feature contains information that would not actually be available at the moment of prediction, usually because it was created by, or shortly after, the event you are trying to predict. It is the single most common reason a model looks excellent in testing and disappoints in production, and it is often invisible until someone goes looking for it specifically.
A leaked feature does not help a model predict the outcome. It lets the model read the outcome and hand it back to you dressed as a forecast.
How it gets in
The churn example at the top of this guide is a common shape: a field populated by the same process that records the outcome, so it is essentially the answer wearing a different column name. Another is target encoding computed across the whole dataset rather than only the training fold, so a category's average outcome, including the test rows, ends up baked into a feature the model is evaluated on. Time is a third source: a feature summarising "total purchases this year" computed with today's data, applied retroactively to a row from six months ago, has quietly used information the model would not have had at that point.
How to catch it
Suspicion is the best early tool. A feature that improves accuracy by an unusually large margin deserves a specific question: could this have existed at prediction time, with nothing filled in retroactively? A strict time-based split, training on data before a cutoff and testing only on data after it, catches leakage that a random split hides, because a random split lets future information sit right next to the past it is contaminating. It is also worth walking through, field by field, exactly when each one was actually recorded relative to the event you are predicting. Slow, but it is the check that saves you from shipping a model built on information the real world would never have handed over in advance.
Where to start
Start with the fields a domain expert would expect to matter, not the full column list. Check each one for how it was actually produced and when, not just what it correlates with. Handle missing values with a rule you can repeat identically in production, and keep a flag for the fact that something was missing rather than only the filled-in value. Encode categories in a way that matches how many distinct values the field actually has, and be honest with yourself about whether a big accuracy jump is a genuine finding or a leak you have not found yet.
None of this is exciting work, and most of a model's eventual reliability was decided here rather than in whatever architecture ends up in the final report. If your team is trying to get from a promising notebook to something a business can trust in production, or you are not sure whether the features feeding a model are as clean as the dashboard suggests, we build the foundations that make that answer knowable. Good feature pipelines depend on the data underneath them being trustworthy in the first place, which is usually where our data quality work starts, and once a feature set is stable it is worth keeping it consistent across every model that reuses it through a proper feature store rather than rebuilding it by hand each time. If you want a straight read on where your own feature pipeline stands, talk to us.