Naive Bayes

Naive Bayes is a family of probabilistic classifiers that apply Bayes' theorem under the "naive" assumption that features are independent of each other once the class is known. That assumption lets the model learn each feature's distribution within each class separately, so training is a single counting pass over the training data and prediction combines a handful of stored probabilities. Mehran Sahami and colleagues applied it to junk email in 1998, Bayesian spam filters built on it spread in the early 2000s, and it remains a fast, strong baseline in supervised learning for text and other very high-dimensional data.

How Naive Bayes Works#

Bayes' theorem gives the probability of a class C given features x₁ … xₙ as P(C | x) = P(C)·P(x | C) / P(x). Estimating P(x | C) for every combination of feature values would take an enormous dataset, so naive Bayes assumes conditional independence and factors it into P(x₁ | C)·P(x₂ | C)·…·P(xₙ | C). The classifier predicts the class with the largest prior P(C) times that product, and drops P(x), which is the same for every class. Implementations add log-probabilities instead of multiplying probabilities, because the product of thousands of small numbers underflows floating-point arithmetic.

Counting creates one failure mode: a word that never appeared with a class in training gets a probability of zero, which zeroes out that class for any document containing it, whatever the other evidence says. Additive smoothing fixes this by adding a pseudo-count α to every count; α = 1 is Laplace smoothing and the default in scikit-learn.

Types of Naive Bayes Classifiers#

The variants differ in the distribution they assume for each feature within a class. Multinomial naive Bayes models counts, such as how often each word appears in a document, and is the standard choice for text. Bernoulli naive Bayes models binary presence or absence and also penalizes the absence of words typical of a class. Gaussian naive Bayes fits a normal distribution with its own mean and variance to each continuous feature in each class. Complement naive Bayes estimates each class's word statistics from all the other classes, which suits imbalanced text datasets, and categorical naive Bayes handles features that take a fixed set of category values.

This example trains scikit-learn's MultinomialNB on word counts from six labeled messages and scores two new ones:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

# Tiny labeled corpus: 1 = spam, 0 = legitimate
texts = [
    "win a free prize now",
    "free cash bonus click now",
    "claim your free reward today",
    "meeting moved to monday",
    "lunch with the team today",
    "project review notes attached",
]
labels = [1, 1, 1, 0, 0, 0]

# Count words, then learn class priors and per-class word probabilities (alpha=1 is Laplace smoothing)
vectorizer = CountVectorizer()
model = MultinomialNB(alpha=1.0)
model.fit(vectorizer.fit_transform(texts), labels)

new = vectorizer.transform(["free prize today", "notes from the monday meeting"])
print(f"P(spam): {model.predict_proba(new)[:, 1].round(3)}")

The first message gets a spam probability of 0.881 because "free" and "prize" appear only in spam, and the second gets 0.053 because "notes", "monday", and "meeting" appear only in legitimate mail. Without smoothing, "free" would have zero probability under the legitimate class, and any legitimate message containing it would be scored as certain spam.

Strengths and Limitations#

Training and prediction take time linear in the number of samples and features, so naive Bayes handles millions of documents with very low inference latency, and scikit-learn's implementations update from new batches through partial_fit for online learning. It also needs little data: Ng and Jordan (2001) showed that naive Bayes approaches its asymptotic error with fewer training examples than logistic regression, which reaches a lower error once data is plentiful. Its class predictions are often accurate even when features are correlated, because only the ranking of the classes needs to be right.

The probabilities themselves are poorly calibrated. Correlated features count the same evidence several times, which pushes scores toward 0 or 1, so applications that need trustworthy confidence values recalibrate them on held-out data. A bag of independent words also cannot represent word order or interactions, so "not good" looks like a mix of a negation and a positive word, which models with attention, such as transformers, read in context.

Applications#

• Spam filtering: Mail filters score each message from the words, sender patterns, and links it contains, and update word statistics as users mark messages as spam.
• Sentiment analysis: Reviews and support tickets are labeled positive or negative from the words they use, a common first model before fine-tuning a language model.
• Document routing: News articles, legal filings, and support requests are sorted into topics or queues, often with complement naive Bayes when some categories are much rarer than others.
• Natural language processing (NLP) baselines: Language identification and topic tagging use naive Bayes as a quick reference point that any more expensive model must beat.

Naive Bayes in Computer Vision#

Neighboring pixels are strongly correlated, so treating raw pixels as independent features discards the edges and textures that define objects, which convolutional neural networks capture by processing neighborhoods together. Before deep learning, the bag-of-visual-words approach applied naive Bayes to histograms of local descriptors, such as SIFT, grouped into visual words with k-means clustering, the visual counterpart of word counts. Counts of the object classes that YOLO26 finds through object detection form the same kind of count vector, so a multinomial naive Bayes model can label a scene as a street, a kitchen, or an office from its detections alone.

Naive Bayes vs. Related Models#

• Bayesian networks: Naive Bayes is the simplest Bayesian network, with the class as the only parent of every feature and no edges between features. General Bayesian networks add those edges to model dependencies.
• Bayesian inference: The "Bayes" in naive Bayes refers to Bayes' theorem. Standard naive Bayes estimates each probability as a smoothed frequency, a single point estimate, while fully Bayesian methods keep a distribution over every parameter.
• Support vector machines (SVM): Learn a discriminative boundary directly at a higher training cost. On sentiment classification, Wang and Manning (2012) found naive Bayes more accurate than SVMs on short snippets and SVMs more accurate on full-length reviews.