Facial Recognition

Facial recognition is a biometric technology that identifies or verifies a person by comparing an image of their face with stored faces. Modern systems use deep learning to convert each face into an embedding, a vector of numbers often called a faceprint, and decide whether two faces belong to the same person by how similar their embeddings are. It unlocks phones, speeds passengers through airport gates, and verifies customers opening bank accounts online, and it is also one of the most regulated uses of computer vision because a face can be captured without a person's knowledge.

How Facial Recognition Works#

A recognition system runs four stages on every image:

1. Detection: A face detector finds each face and returns its bounding box and a few facial landmarks.
2. Alignment: The face is rotated, scaled, and cropped so the eyes, nose, and mouth land in fixed positions, which removes most of the variation caused by head pose and distance.
3. Encoding with a CNN: The network maps the aligned crop to a fixed-length embedding.
4. Matching by cosine similarity: The new embedding is compared with enrolled embeddings, and a threshold decides whether it is a match. Large deployments store enrolled faceprints in a vector database for fast search.

Face detection is an object detection problem, and an Ultralytics YOLO26 model fine-tuned on a face dataset such as WIDER FACE can run it in real time on edge cameras and pass the face boxes to the later stages. Before deep learning, systems relied on hand-designed features, from distances between facial landmarks to Eigenfaces and local binary patterns; learned embeddings replaced them because they stay far more stable across lighting, expression, and age. FaceNet in 2015 trained compact 128-dimensional embeddings directly with a triplet loss, and margin-based losses such as ArcFace now push embeddings of different people farther apart. The matching stage serves two different questions:

Verification (1:1) answers whether a face belongs to a claimed identity, as when a phone checks its owner. Identification (1:N) searches a whole gallery, as when an access-control system checks a face against every enrolled employee. OpenCV Zoo models run the full pipeline in a few lines; this code enrolls the faces in an image, then identifies the faces in a mirrored copy against that gallery.

import urllib.request

import cv2

# Download OpenCV Zoo's face detector (YuNet), face recognizer (SFace), and a test image
zoo = "https://github.com/opencv/opencv_zoo/raw/main/models"
urllib.request.urlretrieve(f"{zoo}/face_detection_yunet/face_detection_yunet_2023mar.onnx", "yunet.onnx")
urllib.request.urlretrieve(f"{zoo}/face_recognition_sface/face_recognition_sface_2021dec.onnx", "sface.onnx")
urllib.request.urlretrieve("https://ultralytics.com/images/bus.jpg", "bus.jpg")
img = cv2.imread("bus.jpg")
detector = cv2.FaceDetectorYN.create("yunet.onnx", "", (img.shape[1], img.shape[0]))
recognizer = cv2.FaceRecognizerSF.create("sface.onnx", "")

# Enroll each detected face as an aligned 128-D embedding
gallery = [recognizer.feature(recognizer.alignCrop(img, face)) for face in detector.detect(img)[1]]

# Identify the faces of a mirrored copy against the gallery (SFace match: cosine >= 0.363)
mirror = cv2.flip(img, 1)
for face in detector.detect(mirror)[1]:
    probe = recognizer.feature(recognizer.alignCrop(mirror, face))
    print([round(recognizer.match(probe, g, cv2.FaceRecognizerSF_FR_COSINE), 2) for g in gallery])

It prints [0.87, 0.06] and [0.09, 0.89]: each mirrored face scores far above the 0.363 threshold against its own enrolled embedding and far below it against the other person's.

Facial Recognition vs. Related Terms#

• Face detection: Finds where faces are in an image without determining whose they are; it is the first stage of recognition and also drives camera autofocus and face blurring.
• Facial analysis and affective computing: Estimate attributes such as expression, head pose, or age without establishing identity.
• Object re-identification (Re-ID): Matches people across cameras by whole-body appearance such as clothing, which works from behind and at low resolution but only over short time spans.

Real-World Applications#

• Device unlocking: Phones verify their owner in a fraction of a second; Apple's Face ID projects more than 30,000 infrared dots to build a depth map that a printed photo cannot imitate.
• Travel and border control: Automated e-gates compare a live photo of the traveler with the photo stored on the chip of their e-passport.
• Identity verification: Banks and online services compare a selfie with the photo on a government ID during account opening, combined with liveness checks.
• Photo organization: Photo apps cluster faces across a personal library so users can find every picture of a family member.

Accuracy, Bias, and Spoofing#

Every system trades off two errors: a false match accepts the wrong person, and a false non-match rejects the right one, and the similarity threshold sets the balance. The NIST Face Recognition Technology Evaluation (FRTE), formerly FRVT, benchmarks commercial algorithms on millions of images, and its 2019 study of 189 algorithms from 99 developers found demographic differences in false positive rates for the majority of them, a central example of algorithmic bias. Recognition can also be fooled by photos, screen replays, and masks, so production systems add presentation attack detection, often called liveness detection, using depth sensors, infrared, or analysis of subtle motion.

Privacy and Regulation#

Faceprints are biometric data, so laws treat them as highly sensitive personal data. Under the GDPR, biometric data used to identify a person is a special category that needs an explicit legal basis, and Illinois's Biometric Information Privacy Act requires written consent before a company collects a faceprint. The EU AI Act goes further: since February 2, 2025, it has banned building facial recognition databases by untargeted scraping of images from the internet or CCTV, as well as real-time remote biometric identification in public spaces for law enforcement outside a few narrowly defined cases.