Neural Network (NN)

A neural network (NN), also called an artificial neural network (ANN), is a machine learning (ML) model built from layers of simple connected units called neurons. Each neuron multiplies its inputs by weights, adds a bias, and passes the sum through a nonlinear activation function. The network learns by adjusting those weights and biases until its outputs match examples in the training data, so its behavior is learned from data. The design was loosely inspired by biological neurons, but modern networks are best understood as large, trainable mathematical functions. Networks with many layers are the subject of deep learning (DL) and power computer vision, speech recognition, and large language models (LLMs).

How a Neural Network Computes#

Neurons are organized into layers. The input layer receives the data, such as pixel values or sensor readings, one or more hidden layers transform it, and the output layer produces the prediction, such as class scores or a predicted value. In a fully connected layer, every neuron receives every output of the previous layer, so the whole layer is a matrix multiplication followed by an element-wise activation. Passing data through all layers in order is the forward pass. The activation is what makes depth useful: without it, stacked layers collapse into one linear function. With it, a network with even one wide enough hidden layer can approximate any continuous function to arbitrary accuracy, a result known as the universal approximation theorem.

The PyTorch snippet below computes one neuron by hand and then a whole layer of four neurons at once:

import torch
from torch import nn

torch.manual_seed(1)
x = torch.tensor([0.5, -1.0, 2.0])  # one example with three input features

# A layer of four neurons: each has three weights and one bias
layer = nn.Linear(in_features=3, out_features=4)

# Neuron 0 by hand: weighted sum of inputs plus bias, then a ReLU activation
z = (layer.weight[0] * x).sum() + layer.bias[0]
print(f"Neuron 0 by hand: {torch.relu(z).item():.4f}")

# The layer computes all four neurons at once as one matrix multiplication
outputs = torch.relu(layer(x))
print(f"All four neurons: {[round(v, 4) for v in outputs.tolist()]}")

It prints Neuron 0 by hand: 0.3906 and All four neurons: [0.3906, 1.1465, 0.0, 0.0]. The hand calculation matches the first neuron of the layer, and the ReLU activation sets the two neurons with negative weighted sums to zero. The layer stores 16 learnable numbers, 12 weights and 4 biases, and real networks stack such layers into millions or billions of parameters.

How Neural Networks Learn#

Training starts from random weights and repeats a simple loop. A batch of labeled examples flows forward through the network, a loss function measures how far the predictions are from the labels, backpropagation applies the chain rule to compute how much each weight contributed to that error, and an optimizer such as stochastic gradient descent (SGD) or Adam nudges every weight in the direction that lowers the loss. One full pass over the training set is an epoch, and training usually stops when the loss on held-out validation data stops improving, because a network that keeps fitting the training set past that point starts overfitting.

Training from scratch usually takes large labeled datasets, and transfer learning cuts that requirement by starting from weights pretrained on a large dataset. Their millions of learned weights are also hard to interpret, which drives research in explainable AI (XAI).

Main Types of Neural Networks#

The way neurons connect, called the architecture, decides what data a network handles well:

• Feedforward networks: Fully connected layers that pass data in one direction, also called multilayer perceptrons (MLPs). They suit tabular data and appear as building blocks inside larger models.
• Convolutional neural networks (CNNs): Slide small shared filters across an image with convolution, the standard design for vision. Residual connections let them grow to hundreds of layers.
• Recurrent neural networks (RNNs): Process sequences step by step while carrying a hidden state, with gated versions such as the LSTM and GRU for longer memory.
• Transformers: Use self-attention to relate every element of a sequence to every other in parallel. They power LLMs and vision transformers.
• Spiking neural networks (SNNs): Communicate with discrete spikes over time, closer to biological neurons, and run efficiently on neuromorphic chips.

Most current models combine these designs. YOLO26 is a convolutional network with self-attention blocks on its lowest-resolution feature maps, and its nano detection model uses about 2.4 million parameters to locate and classify objects in a 640×640 image in a single forward pass.

Neural Networks vs. Related Terms#

• Deep learning: A neural network with one or two hidden layers is shallow, and deep learning is the practice of training networks with many layers, which learn their own hierarchy of features from raw data. Every deep learning model is a neural network.
• Machine learning: Neural networks are one family of machine learning models, alongside decision trees, support vector machines, and gradient-boosted trees, which still compete strongly on tabular data.
• Biological neural networks: Real neurons communicate with timed electrical spikes and adapt through chemical processes that artificial neurons do not model. Spiking neural networks reproduce more of that behavior than standard networks do.

A Short History#

Warren McCulloch and Walter Pitts described a mathematical model of the neuron in 1943, a unit that fires when the sum of its binary inputs crosses a threshold. Frank Rosenblatt's perceptron of 1958 learned the weights of such units from examples, but in 1969 Marvin Minsky and Seymour Papert showed that a single-layer perceptron cannot compute even the XOR function, and interest faded. Multilayer networks solve XOR, and in 1986 David Rumelhart, Geoffrey Hinton, and Ronald Williams showed that backpropagation trains their hidden layers to learn useful internal representations. Deep networks took over computer vision in 2012, when AlexNet, trained on two GPUs, won the ImageNet challenge with a top-5 error of 15.3%, against 26.2% for the next-best entry.