Merged Reality

Merged reality is a synonym for mixed reality (MR): an experience in which virtual objects appear inside the user's real surroundings and interact with them, resting on real tables, disappearing behind real people, and bouncing off real walls. Intel introduced the name in 2016 with Project Alloy, an untethered headset prototype whose RealSense depth cameras brought the user's hands and room into virtual experiences. The underlying idea is older: Paul Milgram and Fumio Kishino's 1994 taxonomy of mixed reality displays placed real and virtual environments at the two ends of a reality-virtuality continuum and called everything in between mixed reality. What separates merged reality from a simple overlay is environmental understanding, so it depends on computer vision and spatial computing to know where every surface and person is.

Merged Reality vs. AR and VR#

• Augmented reality (AR) adds digital content to a view of the real world, from text labels on a phone screen to 3D models in glasses. Mixed reality is AR whose content is registered to and interacts with the real scene, and the two terms overlap heavily in practice.
• Virtual reality (VR) replaces the real world entirely. Augmented virtuality, the side of the continuum Project Alloy aimed at, starts from a virtual world and brings real elements such as the user's hands into it.
• Extended reality (XR) is the umbrella term covering AR, MR, and VR.

Headsets reach the real world in one of two ways. Optical see-through devices such as Microsoft HoloLens 2 add light to the user's direct view through transparent displays, so virtual objects cannot be darker than what lies behind them. Video passthrough devices such as Meta Quest 3 and Apple Vision Pro film the room with outward-facing cameras and show the composited result on opaque displays, which lets real objects fully hide virtual ones at the cost of camera latency.

How Merged Reality Works#

Every displayed frame, at 72 to 120 frames per second on current headsets, combines the device's pose, a model of the scene, and rendered virtual content, so the perception models need real-time inference on the headset's own edge AI processors. Visual SLAM tracks the headset, depth sensors or depth estimation measure the distance to each real surface for a 3D reconstruction of the room, semantic segmentation labels floors, walls, and tables, object detection finds furniture and people, and pose estimation tracks hand keypoints for gestures. Occlusion is the step users notice most when it fails: a virtual object that floats in front of a person standing closer to the camera breaks the illusion at once, and Apple's ARKit has segmented people for occlusion since 2019.

Segmentation masks give a simple version of that occlusion test. This example uses an Ultralytics YOLO26 instance segmentation model to find people in a street photo and keeps their pixels in front of a virtual panel placed between them and the bus:

import cv2
import numpy as np
from ultralytics import YOLO

model = YOLO("yolo26n-seg.pt")
result = model("https://ultralytics.com/images/bus.jpg")[0]
camera = result.orig_img

# Render a virtual object: a green panel standing between the people and the bus
virtual = camera.copy()
cv2.rectangle(virtual, (50, 450), (760, 650), (0, 200, 0), thickness=-1)

# People are closer to the camera than the panel, so their pixels must hide it
occluder = np.zeros(camera.shape[:2], np.uint8)
for cls, polygon in zip(result.boxes.cls, result.masks.xy):
    if result.names[int(cls)] == "person":
        cv2.fillPoly(occluder, [polygon.astype(np.int32)], 255)

merged = np.where(occluder[..., None] > 0, camera, virtual)
cv2.imwrite("merged_reality.jpg", merged)

In the saved image, the panel covers the bus but passes behind every detected pedestrian, because the camera pixels win wherever a person mask lies. Production systems make the same decision per pixel with metric depth, which also handles tables, walls, and other surfaces that no class list covers.

Real-World Applications#

• Industrial training and maintenance: Technicians see step-by-step instructions, torque values, and a remote expert's annotations anchored to the machine in front of them, and trainees practice procedures on virtual equipment placed in a real workshop.
• Design and architecture: Teams review full-scale virtual car interiors or building models in a real room, walking around them and comparing them with physical parts.
• Gaming: Mixed reality games on Quest 3 scan the player's room so that virtual characters climb out of real walls and balls bounce off real furniture.
• Surgery: Augmedics' xvision Spine System, cleared by the US Food and Drug Administration in 2019, overlays a patient's CT-based anatomy and the planned screw trajectory in the surgeon's headset during spine procedures.