Neural Architecture Search (NAS)

Neural architecture search (NAS) is the branch of automated machine learning (AutoML) that designs the structure of a neural network automatically. Instead of an engineer choosing convolution and other layer types, depths, widths, and connections by hand, a search algorithm proposes candidate architectures, estimates how well each performs on the task, and uses the results to propose better ones. The objective is usually accuracy, often combined with a cost such as inference latency, parameter count, or FLOPs on target hardware. NAS produced several widely used computer vision backbones, including EfficientNet and MobileNetV3.

How NAS Works#

A survey in the Journal of Machine Learning Research sorts NAS methods along three dimensions:

1. Search space: The set of architectures that can be built, such as which operation types, kernel sizes, and skip connections are allowed and how they may be wired. Many methods search for a small repeating cell and stack copies of it, which shrinks the space and lets a cell found on a small dataset transfer to a large one.
2. Search strategy: The algorithm that chooses which candidates to try, from reinforcement learning controllers and evolution to gradient-based methods and plain random sampling.
3. Performance estimation strategy: How a candidate is scored without full training, using fewer training epochs, smaller images or datasets, predictions of final accuracy, or weights shared among candidates.

From Thousands of GPU-Days to Hours#

Early NAS was costly because each candidate was trained to convergence. In Zoph and Le's 2016 study, a recurrent controller trained with reinforcement learning generated the networks and the search ran on 800 GPUs at once, and the NASNet follow-up used 500 GPUs for four days to find a cell on CIFAR-10 that transferred to ImageNet. AmoebaNet reached similar results with regularized evolution at a cost of about 3,150 GPU-days. ENAS cut the search to under 16 hours on a single GPU by sharing model weights among all candidates, which it treats as subgraphs of one large network, and DARTS relaxed the discrete choice of operations into continuous weights so the architecture itself could be optimized by gradient descent in a few GPU-days. Benchmarking also tempered the field: random search with early stopping performed at least as well as ENAS on the Penn Treebank and CIFAR-10 benchmarks.

Hardware-Aware NAS and NAS-Built Models#

NAS has had its largest practical impact on efficient models for edge AI, where accuracy must be balanced against speed on a specific device. MnasNet put latency measured on real phones into the search objective and reached 75.2% ImageNet top-1 accuracy at 78 ms on a Pixel phone, 1.8 times faster than MobileNetV2 with 0.5% higher accuracy. EfficientNet-B0 came from a similar search and was then scaled into the B1-B7 family by compound scaling, and MobileNetV3 combined hardware-aware NAS with the NetAdapt algorithm to tune it for mobile CPUs. For object detection, Deci's YOLO-NAS models, found with its AutoNAC search engine and designed for low accuracy loss after INT8 quantization, run in the Ultralytics package through the NAS class for inference, validation, and export, though not for training. NVIDIA has since acquired Deci, and the models are no longer updated by their original team.

A Minimal NAS Example#

The following example runs the simplest search strategy, random sampling, over a small search space of multilayer perceptrons on scikit-learn's handwritten digits. Each candidate is trained for only 30 epochs as a cheap performance estimate and scored on validation data:

import random
import warnings

from sklearn.datasets import load_digits
from sklearn.exceptions import ConvergenceWarning
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

warnings.simplefilter("ignore", ConvergenceWarning)  # short training runs are the cheap estimate
X, y = load_digits(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X / 16, y, random_state=0)

# Search space: 1-3 hidden layers, each 8-128 units wide. Search strategy: 12 random samples
random.seed(0)
space = [8, 16, 32, 64, 128]
candidates = {tuple(random.choices(space, k=random.randint(1, 3))) for _ in range(12)}

# Performance estimation: train each candidate for only 30 epochs, then score it on validation data
scores = {}
for layers in candidates:
    model = MLPClassifier(hidden_layer_sizes=layers, max_iter=30, random_state=0)
    scores[layers] = model.fit(X_train, y_train).score(X_val, y_val)

worst, best = min(scores, key=scores.get), max(scores, key=scores.get)
print(f"{len(scores)} architectures: worst {worst} {scores[worst]:.3f}, best {best} {scores[best]:.3f}")

The 12 samples yield 10 distinct architectures, and their validation accuracy ranges from 0.398 for two layers of 8 units to 0.978 for layers of 128, 128, and 16 units, so the architecture choice alone moves accuracy by almost 60 points. A full NAS run would retrain the winner to convergence and report its score on a separate test set.

NAS vs. Related Terms#

• Hyperparameter tuning: Optimizes training settings, such as the learning rate, for a fixed architecture. NAS treats the architecture itself as the variable, and the two are often run together.
• Model pruning and knowledge distillation: Shrink or imitate an existing network after training, while NAS searches for an efficient design before training.
• Transfer learning: Reuses the learned weights of a pretrained model. NAS decides the structure those weights fill, and NAS-found networks such as EfficientNet are themselves popular pretrained backbones.