Feature Engineering

Feature engineering is the process of creating, transforming, and selecting the input variables, or features, that a machine learning model learns from, using knowledge of the domain and the data. A raw timestamp becomes the hour of day and the day of week, a list of card transactions becomes a customer's spending over the last 30 days, and a trip's distance and duration become its speed. Well-chosen features let a simple model capture patterns it would otherwise miss, and they matter most for tabular and time-series data, where a model can only use the columns it is given. Deep learning models learn their own features from raw images, audio, and text, which moved feature engineering in those fields toward preparing data and building features from model outputs.

Common Techniques#

• Scaling and normalization: Standardizing features to zero mean and unit variance keeps features in large units from dominating distance-based models and gradient-based training, and a log transform compresses skewed values such as income or prices.
• Categorical encoding: One-hot encoding turns a category such as a color into binary columns, and target encoding replaces each category with the average target for that category, computed on separate folds so the encoding does not reveal each row's own label.
• Ratios and interactions: Debt-to-income ratio, price per square foot, and body mass index combine two raw values into the quantity that actually drives the outcome.
• Date and time features: Hour, weekday, month, and holiday flags expose cycles, and sine and cosine encodings of the hour place 23:00 next to 00:00.
• Lags and rolling windows: Previous values, rolling means, and counts over recent windows give a model the recent history of a sensor, account, or product.
• Binning: Grouping continuous values into ranges, such as age bands, captures step-like effects and limits the influence of extreme values.
• Dimensionality reduction with PCA: Principal component analysis compresses many correlated features into a few uncorrelated ones.
• Feature selection: Filter methods rank features by correlation or mutual information with the target, wrapper methods test feature subsets with a model, and embedded methods such as L1 regularization drop features during training.

A ratio feature shows why this works. A depth-limited decision tree can only split on one feature at a time, so it approximates the rule "average speed above 90 km/h" poorly from distance and duration alone, and exactly once speed is computed for it:

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

# 1,000 trips: raw distance (km) and duration (h); label = average speed above 90 km/h
rng = np.random.default_rng(0)
distance = rng.uniform(10, 300, 1000)
hours = rng.uniform(0.5, 4.0, 1000)
y = (distance / hours > 90).astype(int)

raw = np.column_stack([distance, hours])
engineered = (distance / hours).reshape(-1, 1)  # domain knowledge: speed = distance / time

tree = DecisionTreeClassifier(max_depth=2, random_state=0)
for name, X in [("Raw distance and duration", raw), ("Engineered speed", engineered)]:
    print(f"{name}: accuracy {cross_val_score(tree, X, y, cv=5).mean():.3f}")

With the same depth-2 tree, cross-validated accuracy rises from 0.861 on raw distance and duration to 1.000 on the engineered speed feature, because the diagonal boundary in distance-duration space becomes a single threshold on speed.

Validating Features without Leakage#

A feature is only valid if its value is known at the moment of prediction. A churn model that uses "number of support calls in the month after cancellation", or a target encoding computed on the full dataset, scores well in testing and fails in production, a form of data leakage. Every transform with learned parameters, such as scaler means or category encodings, is therefore fit on the training folds only, as scikit-learn pipelines do automatically, and each new feature is kept only if it improves the cross-validated score. Production systems must also compute each feature exactly as training did, which feature stores handle by serving the same feature definitions to both.

Feature Engineering in Computer Vision#

Classical computer vision relied on hand-designed features such as edge maps, SIFT keypoint descriptors, and histograms of oriented gradients, which classifiers such as support vector machines then labeled. Since AlexNet won the ImageNet challenge in 2012, convolutional neural networks have learned their own feature maps from pixels, and engineering effort shifted to the training data, through data augmentation, and to the outputs of vision models. The Ultralytics SpeedEstimator builds a speed feature from YOLO26 object tracking output: once a track has five positions, it converts the pixel displacement between the first and last to meters with the meter_per_pixel scale and divides by the elapsed time to report km/h.

Applications#

• Credit risk: Lenders derive debt-to-income ratio, credit utilization, and months since the last missed payment from raw account records, features that predict default better than raw balances.
• Predictive maintenance: Rolling averages, rates of change, and frequency-band energy computed from vibration and temperature streams reveal developing wear before any raw reading crosses an alarm limit.
• Fraud detection: Velocity features such as the number of card transactions in the past hour, and the distance between a purchase and the cardholder's usual location, separate fraud from normal spending.
• Demand forecasting: Retail models combine sales lags, rolling averages, price changes, promotion flags, and holiday indicators for each store and product.

Feature Engineering vs. Related Terms#

• Feature extraction: Computes features automatically with a fixed algorithm or a learned model, such as PCA or the backbone of a neural network. The two terms are often used interchangeably, with feature engineering covering the wider, domain-guided process.
• Data preprocessing: Cleans and formats data by fixing errors, filling missing values, and converting types, the step that comes before features are designed.
• Embeddings: Dense vectors that a model learns during training to represent words, images, or users, which replace hand-built features such as word counts in most language and vision systems.