Observability

Observability is the ability to understand what is happening inside a software system from the telemetry it emits, mainly logs, metrics, and traces, well enough to answer new questions about its behavior without deploying new code. The term comes from control theory, where Rudolf Kálmán defined a system as observable in 1960 if its internal state can be determined from its outputs. Software teams adopted it in the 2010s, when systems split into many services and failures stopped matching the dashboards built in advance. For machine learning systems, observability also covers the data and predictions: what inputs a model receives, what it outputs, how confident it is, and how both change over time.

Logs, Metrics, and Traces#

Three kinds of telemetry, often called the pillars of observability, answer different questions:

• Metrics: Numeric time series such as request rate, error rate, latency percentiles, memory, and GPU utilization, cheap to store and suited to dashboards and alerts. Google's site reliability engineering book names four golden signals to measure for every service: latency, traffic, errors, and saturation.
• Logs: Timestamped records of individual events, most useful when structured as JSON with fields such as request ID, model version, and input size, so they can be filtered and counted.
• Traces: The path of one request through every service it touches, recorded as a tree of spans, each with a name, start time, duration, and attributes, which shows where the request spent its time.

OpenTelemetry, formed in 2019 by merging the OpenTracing and OpenCensus projects, provides vendor-neutral APIs, SDKs, and a wire protocol (OTLP) for all three signals, and backends such as Prometheus, Grafana, and Jaeger store and display them. This example traces one image classification request with the OpenTelemetry Python SDK, recording a parent span for the request and a child span for each stage:

import torch
import torchvision
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

# Keep finished spans in memory; a production service exports them over OTLP instead
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("classifier")
model = torchvision.models.resnet18().eval()
frame = torch.rand(3, 1080, 1920)

# Trace one request as a parent span with a child span for each stage
with tracer.start_as_current_span("predict", attributes={"model.name": "resnet18", "image.height": 1080}):
    with tracer.start_as_current_span("preprocess"):
        batch = torchvision.transforms.functional.resize(frame, [224, 224]).unsqueeze(0)
    with tracer.start_as_current_span("inference"), torch.inference_mode():
        logits = model(batch)

spans = exporter.get_finished_spans()
names = {span.context.span_id: span.name for span in spans}
for span in spans:
    parent = names[span.parent.span_id] if span.parent else "-"
    print(f"{span.name:<11} {(span.end_time - span.start_time) / 1e6:5.1f} ms  parent: {parent}")

On an Apple M5 Pro CPU, it prints a preprocess span of 1.0 to 1.4 ms and an inference span of 7 to 9 ms, both with predict as their parent, and a predict span of 8 to 10 ms, so the trace attributes most of the request's time to the model. When spans are exported over OTLP, every service that handles a request adds spans under the same trace ID, and a backend such as Jaeger draws them as one timeline.

Observability vs. Monitoring#

Model monitoring watches for known failure modes with predefined metrics, dashboards, and alert thresholds, such as an error rate above 1% or inference latency above a target. Observability is the broader property that makes it possible to investigate failures nobody anticipated, by slicing detailed telemetry after the fact by customer, model version, camera, or any other attribute. The Google SRE book asks monitoring to answer two questions, what is broken and why, and an investigation moves from the first answer to the second across all three signals:

Explainable AI is a separate field that explains why a model produced a particular output from its input features or internal activations, and its attributions can be logged as one more signal.

Observability for Machine Learning#

Machine learning systems often fail silently: a computer vision model can answer every request with HTTP 200 while its predictions degrade because the camera was moved, the lighting changed, or a new product appeared. ML observability therefore adds telemetry about data and predictions to the usual service signals. Teams record input statistics such as image size and brightness to detect data drift, track prediction distributions such as class counts, empty results, and mean confidence, and compute accuracy by segment once ground truth labels arrive, often days later. During training, callbacks send loss curves, metrics, and system usage for every run to experiment tracking tools such as TensorBoard and MLflow. For AI agents and large language model applications, traces capture each prompt, retrieval step, tool call, and response, and the OpenTelemetry GenAI semantic conventions standardize attributes such as gen_ai.usage.input_tokens so that token counts and costs line up across tools.

Every dedicated endpoint on Ultralytics Platform has deployment monitoring built in: request count, error rate, and p95 latency cards, a Logs tab with severity levels and an Errors filter for error and warning entries, and prediction statistics that chart top classes, mean confidence over time, predictions per image, and model inference time separately from request latency. A falling confidence curve or a shift in top classes shows up in those charts even while every request succeeds.