Frame Interpolation

Frame interpolation, also called video frame interpolation (VFI), is the computer vision task of synthesizing new frames between two existing frames of a video. Inserting one frame between every pair doubles the frame rate, turning 30 FPS footage into 60 FPS, and inserting several creates slow motion from ordinary footage. It raises a video's temporal resolution, the counterpart in time of super-resolution, and it only fills gaps between existing frames, unlike video prediction, which extrapolates frames after the last one.

How Frame Interpolation Works#

Averaging two frames produces ghosting, a double image of everything that moved. Flow-based methods instead estimate optical flow, the motion of every pixel between the two frames, move each frame partway along that motion to the target time, and blend the two results. Occlusions are the hard part: a background region uncovered by a walking person exists in only one of the frames, so deep learning models such as Super SloMo also predict visibility maps that decide which frame each output pixel comes from.

OpenCV's Farneback optical flow is enough to show the effect on real footage. The snippet holds out the middle of three frames, rebuilds it from its neighbors, and scores both methods with peak signal-to-noise ratio (PSNR):

import cv2
import numpy as np

# Read three consecutive frames of people walking and hold out the middle one
cap = cv2.VideoCapture("https://github.com/ultralytics/assets/releases/download/v0.0.0/decelera_portrait_min.mov")
f0, f1, f2 = (cap.read()[1] for _ in range(3))
g0, g2 = (cv2.cvtColor(f, cv2.COLOR_BGR2GRAY) for f in (f0, f2))

# Estimate dense optical flow in both directions between the outer frames
flow02 = cv2.calcOpticalFlowFarneback(g0, g2, None, 0.5, 3, 15, 3, 5, 1.2, 0)
flow20 = cv2.calcOpticalFlowFarneback(g2, g0, None, 0.5, 3, 15, 3, 5, 1.2, 0)

# Warp each outer frame halfway along its flow, then average the two warps
h, w = g0.shape
grid = np.dstack(np.meshgrid(np.arange(w), np.arange(h))).astype(np.float32)
warp0 = cv2.remap(f0, grid - 0.5 * flow02, None, cv2.INTER_LINEAR)
warp2 = cv2.remap(f2, grid - 0.5 * flow20, None, cv2.INTER_LINEAR)
middle = cv2.addWeighted(warp0, 0.5, warp2, 0.5, 0)

# Score both against the real middle frame: flow interpolation vs. plain blending
blend = cv2.addWeighted(f0, 0.5, f2, 0.5, 0)
print(f"Blend: {cv2.PSNR(blend, f1):.1f} dB, flow interpolation: {cv2.PSNR(middle, f1):.1f} dB")

It prints Blend: 22.3 dB, flow interpolation: 29.5 dB: moving each frame halfway along its motion rebuilds the held-out frame 7.2 dB more accurately than averaging. Benchmarks score models the same way: the Vimeo-90K triplet set holds 73,171 three-frame clips whose middle frame a model must predict, measured by PSNR and SSIM.

Learned Interpolation Methods#

• Flow-based networks: Super SloMo (2018) predicts flow and visibility for any time between two frames, and RIFE (2022) estimates the flow at the intermediate time directly, running 4 to 27 times faster than Super SloMo and DAIN.
• Kernel-based networks: SepConv (2017) predicts separable convolution kernels for every output pixel and applies them to patches of the two input frames, combining motion estimation and blending in one step.
• Large-motion models: Google's FILM (2022) shares weights across the scales of a feature pyramid, so a large motion at a fine scale is handled like the small motions it learns at coarser scales, and it trains from frame triplets alone.
• Generative models: Diffusion-based interpolators such as LDMVFI (2024) generate the middle frame by denoising a latent representation conditioned on the two inputs, instead of warping their pixels.
• Event-based interpolation: Time Lens (2021) adds the brightness changes an event camera records between two frames, which reveal the true motion during the gap.

Real-World Applications#

• Game frame generation: NVIDIA DLSS Multi Frame Generation inserts up to five generated frames for every rendered frame on GeForce RTX 50 Series GPUs; each generated frame needs the next rendered frame, which adds latency.
• Video editing: Editors create slow motion from standard footage and convert 24 or 30 FPS video for 60 Hz and 120 Hz displays, the same motion interpolation that TVs apply as motion smoothing.
• Animation: Interpolation models generate in-between drawings for hand-drawn keyframes, and video generators can produce whole clips that start and end on two given frames.

Object tracking applies the same idea to positions instead of pixels. When an object hidden for several frames reappears, the OC-SORT tracker in Ultralytics track mode (tracker="ocsort.yaml") linearly interpolates virtual boxes for the missed frames and replays them through its Kalman filter, repairing the velocity estimate it uses to match the object to the next YOLO26 detection.