Logistic Regression

Logistic regression is a supervised learning method for classification that estimates the probability that an input belongs to a class by passing a weighted sum of its features through the logistic, or sigmoid, function. The name contains "regression" because the model regresses the log-odds of the class on the features, but it is used to classify: a spam filter, a credit model, or a screening test turns the probability into a yes-or-no decision with a threshold. Joseph Berkson developed it as a general statistical model starting in 1944, when he coined the term "logit", and it remains the standard baseline for classifying tabular data because it trains in seconds, outputs probabilities, and its weights read directly as effects on the odds.

How Logistic Regression Works#

The model computes a score, z = w·x + b, exactly as linear regression does, then maps it to a probability, p = 1 / (1 + e⁻ᶻ). The sigmoid squeezes any score into the range 0 to 1, returns 0.5 at z = 0, and approaches 1 or 0 as z grows large in either direction. Inverting it shows what the score means: z equals the log-odds, log(p / (1 − p)), so the model assumes the log-odds of the positive class change linearly with each feature, and its decision boundary is a straight line, plane, or hyperplane. A one-unit increase in a feature multiplies the odds by eʷ for that feature's weight w, which is why medical and credit studies report logistic regression results as odds ratios.

Training sets the weights by maximum likelihood, which is the same as minimizing log loss, also called binary cross-entropy: −[y·log(p) + (1 − y)·log(1 − p)], averaged over the training data. The loss is convex, so it has a single minimum, but no closed-form solution, so solvers such as L-BFGS, Newton's method, or gradient descent reach it iteratively. Without a penalty, classes that a hyperplane separates perfectly push the weights toward infinity, so libraries add regularization by default: scikit-learn's LogisticRegression applies an L2 penalty with C = 1.0, and smaller values of C regularize more strongly. Because log loss rewards accurate probabilities, a well-specified, suitably regularized logistic regression tends to return well-calibrated probabilities, the property that scores such as the Brier score measure.

Decision Thresholds#

A threshold turns each probability into a label, and 0.5 is only the default. Moving it trades precision against recall: a cancer screening model lowers it to miss fewer tumors, and a spam filter raises it to avoid hiding legitimate mail. The ROC curve plots this trade-off across every threshold, and the area under it measures how well the model ranks positives above negatives. This example trains logistic regression on the Wisconsin breast cancer dataset bundled with scikit-learn and compares two thresholds:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y  # make malignant (originally 0) the positive class
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)

# Standardize the 30 features, then fit weights by minimizing log loss
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)

# Predicted probability of malignancy, turned into labels at two thresholds
proba = model.predict_proba(X_test)[:, 1]
for threshold in (0.5, 0.1):
    pred = (proba >= threshold).astype(int)
    print(
        f"Threshold {threshold}: precision {precision_score(y_test, pred):.3f}, recall {recall_score(y_test, pred):.3f}"
    )

At the default threshold of 0.5, every tumor the model flags is malignant (precision 1.000), and it misses 2 of the 53 malignant tumors in the test set (recall 0.962). At 0.1 it catches all 53 (recall 1.000) and also flags 11 of the 90 benign tumors (precision 0.828).

Multiclass and Neural Network Classifiers#

For more than two mutually exclusive classes, multinomial logistic regression computes one score per class and converts the scores into probabilities that sum to 1 with the softmax function, trained with cross-entropy loss; one-vs-rest instead trains a separate binary model per class. Multi-label problems, where one input can carry several labels at once, use an independent sigmoid per class.

The output layer of most classification neural networks is one of these models applied to features learned by earlier layers, as in image classification networks that end in a linear layer and a softmax. The YOLO26 detection head ends each classification branch in a 1×1 convolution that outputs one logit per class at every grid location, and a sigmoid turns each logit into an independent class score trained with binary cross-entropy, so every class score is a logistic regression on the features at that location. The conf argument of predict mode, 0.25 by default, is the decision threshold applied to those scores.

Applications#

• Clinical risk scores: Models estimate the probability of a disease, a complication, or a hospital readmission from age, blood pressure, and lab values, and the odds ratios show clinicians how much each factor raises the risk.
• Credit scoring: Lenders build credit scorecards on logistic regression because each feature's contribution to the score can be explained to applicants and regulators.
• Ad click prediction: Google's 2013 paper on its production click-through system described logistic regression trained online with the FTRL-Proximal algorithm, which keeps the model sparse, to estimate the probability that a user clicks an ad.
• Churn prediction: Subscription businesses estimate the probability that each customer cancels in the next period and target retention offers at the highest-risk accounts.

Logistic Regression vs. Related Methods#

• Linear regression on 0/1 labels: Fitting squared error to class labels produces predictions below 0 and above 1, and points far from the boundary, even correctly classified ones, drag the fit, which the sigmoid and log loss avoid.
• Support vector machines (SVM): A linear SVM also learns a hyperplane, but it minimizes hinge loss to maximize the margin and outputs distances from the boundary instead of probabilities.
• Naive Bayes: A generative classifier that models how features arise from each class, and forms a classic pair with logistic regression, its discriminative counterpart. Ng and Jordan (2001) showed that naive Bayes approaches its higher asymptotic error with fewer training examples, while logistic regression reaches a lower error once data is plentiful.
• Perceptron: Applies a step function to the same weighted sum, outputs only 0 or 1, and updates its weights only when it misclassifies a training example.