Visual Servoing

Visual servoing is a feedback-control technique that uses camera images to continuously adjust a robot's movement toward a desired position or view. The robot observes visual features, measures their difference from a target configuration, and corrects its motion as new images arrive. This connects computer vision, which interprets images, with robotics, which turns those measurements into physical action. Visual servo control is a synonym. J. Hill and W. T. Park introduced the term in 1979 with Real Time Control of a Robot with a Mobile Camera.

How Visual Servoing Works#

A visual servoing system compares current observations with a desired image or relative pose. Feature extraction supplies measurements such as keypoints, line locations, or object contours. A controller converts the resulting error into motion commands. Movement changes the camera's observations, closing the feedback loop and allowing repeated corrections as the robot approaches its goal.

An interaction matrix, also called an image Jacobian, describes how camera motion changes visual measurements. Its pseudoinverse maps feature error into a corrective velocity; the controller gain sets the strength of that correction. Camera mounting defines two common arrangements: eye-in-hand places the camera on the moving tool, while eye-to-hand uses an external camera observing the workspace.

Ultralytics YOLO supplies perception measurements for this loop: YOLO26 produces bounding boxes through object detection, and its ROS integration passes camera-derived detections and depth information between robot software components. A downstream controller uses those measurements to calculate movement.

Image-Based and Position-Based Visual Servoing#

The principal methods differ in the representation of the control error:

• Image-based visual servoing (IBVS): Minimizes differences between current and desired image features, such as point coordinates. Point-feature controllers use camera geometry and depth estimates to relate image motion to physical movement.
• Position-based visual servoing (PBVS): Uses pose estimation to recover three-dimensional position and orientation, then reduces the difference from the desired pose. Camera calibration and geometric reconstruction affect its accuracy.
• Hybrid visual servoing: Combines image-space and three-dimensional measurements. The established 2.5D visual servoing approach combines image features with partial pose information.

Object tracking maintains feature or object correspondence across frames; visual servoing uses those observations to control motion. Visual SLAM estimates camera motion and builds a map. These perception capabilities support different parts of a robot's navigation and control system.

Applications and Practical Constraints#

• Robotic Grasping: During close-range grasping, an RGB-D camera can cross its minimum depth-sensing distance and lose valid range measurements. Closed-loop visual grasping uses IBVS to continue correcting the final approach with RGB feature measurements, including when the object moves.
• Mobile Robot Positioning: The ViSP Pioneer demonstration controls a camera-equipped Pioneer P3-DX robot relative to a visual target. Target observations determine corrective robot movement through a position-based controller.

Timing and visibility directly affect control. Image-processing delay introduces tracking lag and can destabilize the loop. Inference latency therefore matters alongside detection accuracy in real-time inference. Occlusion interrupts feature correspondence, while poorly conditioned feature geometry makes velocity estimates sensitive to image noise. Sensor fusion combines visual measurements with information such as joint encoders to maintain useful state estimates during motion.

A Minimal IBVS Simulation#

This Python simulation applies the standard pseudoinverse control law to a camera translating sideways while observing one stationary point. Install NumPy with pip install numpy.

import numpy as np

point = np.array([0.0, 0.0, 2.0])
camera = np.array([-0.4, 0.2, 0.0])
desired = np.zeros(2)
gain, dt, focal_pixels = 1.0, 0.05, 800.0
initial_error = np.linalg.norm((point - camera)[:2] / point[2])

for _ in range(200):
    relative = point - camera
    observed = relative[:2] / relative[2]
    error = observed - desired
    # Interaction matrix for sideways camera translation.
    interaction = -np.eye(2) / relative[2]
    velocity = -gain * np.linalg.pinv(interaction) @ error
    camera[:2] += velocity * dt

final_error = np.linalg.norm((point - camera)[:2] / point[2])
print(f"Pixel error: {initial_error * focal_pixels:.2f} -> {final_error * focal_pixels:.2f}")

The image error falls from approximately 178.89 pixels to 0.01 pixels as repeated corrections center the point. For point features, the translational interaction terms scale inversely with depth: doubling the point's depth halves the image displacement produced by the same sideways camera translation.