Hopfield Networks

Hopfield networks are recurrent neural networks that store patterns and recover them from incomplete or corrupted inputs. They act as associative memory: part of a pattern serves as the cue for retrieving the whole. A stored memory behaves like a valley in an energy landscape, and the network's changing state settles toward a nearby valley.

John J. Hopfield introduced the model in 1982 through his work on collective computation in neural networks. "Hopfield network," "Hopfield net," and "Hopfield model" are common synonyms. The architecture belongs to the recurrent neural network family because feedback connections repeatedly update its internal state.

How Hopfield Networks Store and Retrieve Patterns#

A classical Hopfield network contains interconnected neurons whose states are commonly represented as −1 or +1. Its model weights are symmetric: the connection from neuron A to neuron B equals the connection from B to A. Standard configurations set self-connections to zero. Hebbian learning stores patterns by strengthening connections between neurons whose states agree across the stored examples. Computationally, this adds each pattern's outer product—a matrix of pairwise products—to the weight matrix.

During retrieval, each neuron sums its weighted inputs and applies a threshold activation function. Asynchronous updating changes one neuron at a time. With symmetric weights, zero self-connections, and unchanged states at exact threshold ties, each state change lowers an energy function until the network reaches a stable configuration. This configuration is an attractor; the starting states that converge to it form its basin of attraction. Simultaneously updating all neurons can produce oscillations between configurations.

Storage and retrieval are separate operations. Storage changes the connections; retrieval changes neuron states while keeping those connections fixed. This makes retrieval a form of state optimization, rather than another round of backpropagation.

Pattern Recovery in Python#

The following PyTorch example stores one binary pattern using the Hebbian rule, flips two entries, and retrieves the memory through asynchronous updates.

import torch

memory = torch.tensor([1.0, -1.0, 1.0, -1.0, 1.0, -1.0, 1.0, -1.0])
weights = torch.outer(memory, memory) / memory.numel()
weights.fill_diagonal_(0)

state = memory.clone()
state[[1, 6]] *= -1
initial_errors = int((state != memory).sum())

while True:
    previous = state.clone()
    for i in range(state.numel()):
        field = weights[i] @ state
        # Preserve the current state when the weighted input is zero.
        if field != 0:
            state[i] = field.sign()
    if torch.equal(state, previous):
        break

print(f"Bit errors: {initial_errors} -> {int((state != memory).sum())}")

The output is Bit errors: 2 -> 0: recurrent updates reconstruct the stored pattern from its corrupted cue using the standard associative-memory dynamics.

Continuous Networks, Attention, and Applications#

Continuous Hopfield networks use graded neuron outputs, such as a sigmoid, and evolve toward equilibrium. Hopfield established this formulation in 1984. Modern Hopfield networks extend associative memory with different energy functions and higher storage capacities; their continuous-state variants operate on real-valued vectors.

Hubert Ramsauer and colleagues connected modern Hopfield retrieval to attention mechanisms in 2020. A query is compared with stored vectors, softmax converts similarity scores into weights, and a weighted combination retrieves associated information. This gives the attention operation used in a transformer an associative-memory interpretation.

For image representations, Ultralytics YOLO26 exposes feature extraction through model.embed(), returning image embeddings. These vectors provide the real-valued representation used by modern associative-memory layers.

Established applications include recovering stored visual patterns, pooling large collections of features, and constrained optimization. Hopfield and David Tank applied an analog network to the traveling-salesman problem in 1985, encoding route quality and constraints into the energy landscape. Modern Hopfield pooling also supports immune-repertoire classification, aggregating information across hundreds of thousands of biological sequences through multiple-instance learning.

Related Concepts and Storage Limits#

An autoencoder learns an encoder–decoder mapping that reconstructs inputs; a classical Hopfield network reconstructs patterns by settling into attractors. A memory bank stores representations for later access, while Hopfield dynamics specify how a cue retrieves an associated state.

Classical networks develop retrieval errors when memories interfere. Correlated patterns create cross-talk, and spurious attractors can return mixtures of stored memories. For large classical Hebbian networks storing random, approximately uncorrelated binary patterns, the familiar capacity estimate is about 0.14 times the neuron count when allowing a small fraction of retrieval errors—roughly 140 patterns for 1,000 neurons.