Neuromorphic Vision

Neuromorphic vision is an approach to computer vision that senses and processes images the way the biological retina and brain do. Its core device, the event camera or dynamic vision sensor (DVS), has pixels that work independently and each report a brightness change the moment it happens, instead of capturing full frames at a fixed rate. The output is a sparse stream of events with microsecond timing, high dynamic range, and almost no motion blur, and a static scene produces almost no data. The field grew out of the neuromorphic engineering of Carver Mead's group at Caltech in the late 1980s, whose analog chips mimicked neural circuits, and spiking neural networks are its natural processing counterpart, computing with the same kind of sparse, timed events.

How Event Cameras Work#

Each pixel stores the logarithm of the brightness it saw at its last event. When the current log brightness differs from that reference by more than a contrast threshold, the pixel emits an event that encodes its location, a timestamp, and a polarity, ON for brighter or OFF for darker, and then resets its reference. Working in log space makes the pixel respond to relative contrast, so the same sensor handles deep shadow and direct sunlight, with a dynamic range of about 140 dB against about 60 dB for frame cameras.

The same rule can be applied to ordinary video to see how sparse the output is:

import cv2
import numpy as np

# Read a clip of a person exercising in front of a fixed camera
cap = cv2.VideoCapture("https://github.com/ultralytics/assets/releases/download/v0.0.0/solution_ci_pose_demo.mp4")
ok, frame = cap.read()
reference = np.log1p(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32))
threshold = 0.2  # contrast threshold on log brightness
events, samples = 0, 0

while True:
    ok, frame = cap.read()
    if not ok:
        break
    brightness = np.log1p(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32))
    # A pixel fires an ON or OFF event only when its brightness changed by more than the threshold
    fired = np.abs(brightness - reference) >= threshold
    reference[fired] = brightness[fired]
    events += fired.sum()
    samples += fired.size

print(f"{events:,} events vs {samples:,} pixel values in frames ({100 * events / samples:.2f}%)")

It prints 233,492 events vs 13,593,600 pixel values in frames (1.72%): with a fixed camera, only the exercising person changes from frame to frame, so the event stream carries 1.72% of the values that full frames would. Simulators such as v2e turn videos into synthetic event training data the same way, optionally upsampling them first with frame interpolation and adding per-pixel threshold mismatch and noise.

Event Cameras vs. Frame Cameras#

Event cameras suit fast motion, scenes with extreme lighting, and always-on edge AI sensing where little changes, and their bandwidth follows the activity in the scene. They record no absolute brightness or color, a still camera watching a still scene reports nothing, and a moving camera floods the output with events from every edge. Hybrid sensors such as iniVation's DAVIS also capture standard frames from the same pixels, and many systems combine an event camera and a frame camera through sensor fusion. Standard models such as convolutional neural networks (CNNs) expect dense images, so pipelines either convert events into a dense representation or process them natively:

• Event frames: Counting ON and OFF events per pixel over a time window gives a two-channel image that any CNN can read.
• Time surfaces and voxel grids: Storing each pixel's latest timestamp, or binning events into several time slices, keeps timing information that plain counts discard.
• Spiking networks on neuromorphic chips: Processors such as Intel's Loihi 2 consume events directly as spikes and spend energy only when events arrive.
• Intensity reconstruction: Networks such as E2VID rebuild a grayscale video from the event stream, so frame-based models can run on it.

Ultralytics YOLO reads the input channel count from the channels key of a dataset YAML, as in the 10-channel COCO8-Multispectral dataset, so a YOLO26 object detection model can train directly on two-channel event frames of ON and OFF counts.

Real-World Applications#

• Automotive perception: A 2024 Nature study paired a 20 FPS color camera with an event camera and detected objects on the road with the latency of a 5,000 FPS camera at the bandwidth of a 45 FPS camera.
• Drones and robotics: An event-camera quadrotor from the University of Zurich dodged obstacles thrown at it at relative speeds up to 10 meters per second, with an overall latency of 3.5 milliseconds.
• Industrial monitoring: Event cameras count fast-moving parts, measure machine vibration, and track high-speed processes, and industrial camera makers such as IDS sell event-based models built on Prophesee sensors.