Support Vector Machine (SVM)

A support vector machine (SVM) is a supervised learning algorithm that classifies data by finding the hyperplane that separates two classes with the widest possible margin, the gap between the boundary and the nearest training points of each class. Those nearest points, the support vectors, are the only training examples that determine the boundary. Kernel functions let SVMs learn curved boundaries, and variants handle regression and outlier detection. Vladimir Vapnik and colleagues developed the method at AT&T Bell Labs, with the kernel version in 1992 and the soft-margin formulation used today published by Cortes and Vapnik in 1995. Before deep learning, SVMs were among the most accurate classifiers for text, images, and biological data, and they remain strong on small to medium datasets with many features.

How SVMs Work#

A linear SVM predicts the class from the sign of w·x + b, so its decision boundary is the hyperplane w·x + b = 0, and with the weights scaled so that the closest points satisfy |w·x + b| = 1, the margin has a width of 2/‖w‖. Training maximizes the margin by minimizing ‖w‖² while requiring every training point to fall on the correct side of it. Real classes overlap, so the soft-margin SVM lets points cross the margin at a cost set by the regularization parameter C. Its hinge loss charges nothing for points classified correctly beyond the margin and a penalty that grows linearly with the violation for the rest. A small C tolerates more violations and gives a wider, smoother margin, and a large C fits the training data more tightly at a higher risk of overfitting. In the trained model only points on the margin or on its wrong side carry nonzero weight; these are the support vectors, and moving any other point without crossing the margin leaves the boundary unchanged.

The Kernel Trick#

When no hyperplane separates the classes, mapping the data into more dimensions can make them separable: two concentric rings in 2D become separable by a flat plane once a third feature, x₁² + x₂², measures each point's distance from the center. In its dual form, SVM training and prediction use the data only through dot products between pairs of points, so a kernel function k(x, x′) can return the dot product in the mapped space directly, without ever computing the mapped coordinates. This is the kernel trick. The common kernels are linear, polynomial, and the radial basis function (RBF), exp(−γ‖x − x′‖²), where γ sets how far each support vector's influence reaches. This example fits scikit-learn's SVC with a linear and an RBF kernel on two concentric rings:

from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

# Two concentric rings: no straight line separates the classes
X, y = make_circles(n_samples=500, factor=0.4, noise=0.1, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

for kernel in ("linear", "rbf"):
    clf = SVC(kernel=kernel, C=1.0)
    clf.fit(X_train, y_train)
    print(f"{kernel}: test accuracy {clf.score(X_test, y_test):.2f}, support vectors {clf.n_support_.sum()}")

The linear kernel scores 0.43 on held-out points, worse than guessing, and keeps 360 of the 375 training points as support vectors. The RBF kernel separates the rings perfectly (1.00) with only 42 support vectors.

Choosing a Kernel and Hyperparameters#

Distances and dot products depend on feature scale, so SVM inputs are standardized first. The practical guide from the authors of LIBSVM, the library behind scikit-learn's SVC, recommends scaling the features, starting with the RBF kernel, and choosing C and γ by grid search with cross-validation. When features far outnumber samples, as with text, a linear kernel performs about as well and trains much faster. Kernel SVM training time grows at least quadratically with the number of samples, so beyond tens of thousands of samples linear SVMs or approximate kernel features replace the exact kernel model.

SVMs in Computer Vision#

Classical computer vision paired hand-designed features with linear SVMs. Dalal and Triggs' 2005 pedestrian detector scored histograms of oriented gradients (HOG) with a linear SVM, and the 2014 R-CNN object detector classified CNN features of each region proposal with one linear SVM per class until Fast R-CNN replaced them in 2015 with a softmax layer trained together with the network. Modern detectors learn the features and the classifier together, and SVMs survive as fast classifiers on fixed features. A linear SVM trained on the vectors that YOLO26 returns from model.embed(), one embedding per image, is a linear probe that adds a new image category without retraining the network.

Applications#

• Text classification: Word-count vectors have tens of thousands of sparse dimensions, where linear SVMs separate topics, spam, and sentiment accurately, an advantage Thorsten Joachims demonstrated for text categorization in 1998.
• Bioinformatics: Gene-expression studies classify tumors from thousands of measured genes and only dozens or hundreds of patients, a setting where the margin limits overfitting, and string kernels compare protein sequences directly.
• Anomaly detection: A one-class SVM learns a boundary around normal data only and flags points outside it, which suits fraud and machine monitoring where faults are too rare to label.
• Regression: Support vector regression fits a function inside a tube of width ε, ignores errors smaller than ε, and keeps as support vectors only the points on or outside the tube.

SVM vs. Related Algorithms#

• K-nearest neighbors (KNN): Stores the whole training set and compares each new point with its neighbors at prediction time, while an SVM keeps only its support vectors.
• Decision trees and random forests: Split on one feature at a time, need no feature scaling, and handle mixed numeric and categorical columns, where SVMs need numeric, scaled inputs.
• Neural networks: Learn the feature mapping from data instead of fixing it with a kernel, which lets them scale to millions of images where kernel SVMs become too slow.
• Logistic regression: Also learns a linear boundary, but minimizes log loss over every training point and outputs probabilities. An SVM outputs signed distances from the boundary, and scikit-learn's SVC produces probabilities only with probability=True, which fits Platt scaling through an internal 5-fold cross-validation.