Linear Regression

Linear regression is a supervised learning method that predicts a continuous value, such as a price, a temperature, or next month's demand, as a weighted sum of input features plus a constant. With one feature it fits a straight line, ŷ = w·x + b; with several it fits a plane or hyperplane, ŷ = w₁x₁ + w₂x₂ + … + wₙxₙ + b. Adrien-Marie Legendre published the least-squares method used to fit it in 1805, and Carl Friedrich Gauss published his own account in 1809. It is still the usual first model for numeric predictive modeling because it trains in milliseconds, works with little data, and each weight states how much the prediction changes per unit of its feature.

How Linear Regression Works#

Training chooses the weights and intercept that make predictions closest to the true targets in the training data. Ordinary least squares measures closeness as the sum of squared residuals, the differences between targets and predictions, which is equivalent to minimizing mean squared error (MSE), the standard loss function for regression. Squared error makes the loss a smooth bowl with a single minimum, so the best weights have a closed-form solution, the normal equation w = (XᵀX)⁻¹Xᵀy, which libraries compute with stable matrix factorizations such as QR or SVD instead of an explicit inverse. On datasets too large for that, gradient descent or stochastic gradient descent reaches the same weights iteratively, and it converges fastest when features are standardized to similar scales.

Each weight is the change in the prediction for a one-unit increase in its feature with the other features held fixed, which makes the model easy to audit. The scikit-learn LinearRegression estimator fits ordinary least squares, and this example generates house prices from known coefficients and checks how closely the fit recovers them:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

# 500 synthetic homes: price = $50,000 + $150 per sq ft - $2,000 per year of age + noise
rng = np.random.default_rng(0)
X = np.column_stack([rng.uniform(800, 3000, 500), rng.uniform(0, 50, 500)])
y = 50_000 + 150 * X[:, 0] - 2_000 * X[:, 1] + rng.normal(0, 20_000, 500)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# Fit by ordinary least squares
model = LinearRegression()
model.fit(X_train, y_train)

print(f"Per sq ft: {model.coef_[0]:.1f}, per year: {model.coef_[1]:.0f}, intercept: {model.intercept_:.0f}")
print(f"Test R^2: {r2_score(y_test, model.predict(X_test)):.3f}")

The fit recovers $147.7 per square foot against the true $150 and −$1,992 per year of age against the true −$2,000, and it explains 96.2% of the variance in held-out prices (R² = 0.962). R² is the share of the target's variance the model explains, and root mean squared error (RMSE) reports the typical error in the target's own units.

Assumptions and Diagnostics#

Linear regression assumes the average target is a weighted sum of the inputs, that errors are independent, and that their spread stays roughly constant across the range of predictions; confidence intervals and p-values for the weights also assume roughly normal errors. The inputs can be any fixed transformation of the raw features, so adding squared terms, logarithms, or products of features through feature engineering still gives a linear regression, one that can follow a curve. Plotting residuals against predictions is the standard check: a curved pattern points to a missing nonlinear term, and a fan shape means errors grow with the prediction, which a log transform of the target often fixes. Squared error also lets a few outliers pull the fit toward them, and robust losses such as the Huber loss reduce their influence.

When features are strongly correlated with each other, small changes in the data swing their weights widely. Ridge regression adds a penalty on squared weights that shrinks them, lasso adds a penalty on absolute weights that sets some exactly to zero and so selects features, and elastic net combines both. These forms of regularization accept a little bias in exchange for lower variance, the bias-variance tradeoff, and limit overfitting when there are many features relative to samples.

Linear Regression in Deep Learning#

A neural network that predicts numbers usually ends in a linear layer, so its last step is a linear regression on features that earlier layers learned, and a network with no hidden layers trained on MSE is exactly linear regression. Object detection models use this for box coordinates. In YOLO26, each box branch of the detection head ends in a 1×1 convolution that maps the features at each grid location linearly to four distances from that location to the edges of a bounding box, and training penalizes those distances with an L1 loss alongside a CIoU loss on box overlap.

Applications#

• Forecasting: Retailers and utilities regress sales or electricity load on price, promotions, temperature, and calendar indicators, and a fitted trend line is a common baseline for more complex forecasting models.
• Pricing and valuation: Hedonic pricing models regress house prices on size, age, and location, and each weight gives the market value of one more unit of that attribute.
• Effect estimation: Economists and health researchers regress an outcome, such as wages or blood pressure, on a factor of interest plus control variables to estimate how the outcome changes with that factor while the others are held fixed.
• Calibration curves: Laboratories regress instrument readings on the known concentrations of reference samples, then invert the fitted line to convert readings of new samples into concentrations.

Linear Regression vs. Related Methods#

• Logistic regression: Solves classification by passing the same weighted sum through a sigmoid function to output a probability between 0 and 1, and trains with log loss instead of squared error.
• Polynomial regression: Fits powers of a feature, such as x and x², and is linear regression on those expanded features.
• Decision trees and boosted trees such as XGBoost: Split the feature space into regions and learn nonlinear effects and interactions without hand-made features, but predict a constant within each region, so they cannot extrapolate a trend beyond the training range.
• Correlation: Pearson's correlation coefficient r measures the strength of a linear relationship symmetrically, and the slope of a simple linear regression of y on x equals r multiplied by the ratio of the standard deviations of y and x.