Serverless Computing

Serverless computing is a cloud execution model in which the provider runs code only when an event calls for it, starts and scales the underlying servers automatically, shuts them down to zero when traffic stops, and bills for requests and execution time instead of reserved machines. Servers still exist, but developers never provision, patch, or size them. The model became mainstream with AWS Lambda, announced in November 2014, and now spans function platforms such as Lambda, Azure Functions, and Cloud Run functions, along with serverless containers such as Google Cloud Run and Azure Container Apps. UC Berkeley researchers described its strengths and limits in the 2019 paper "Cloud Programming Simplified: A Berkeley View on Serverless Computing". For machine learning, serverless suits model serving with bursty or unpredictable traffic, where paying for an idle GPU server between bursts would cost more than the work itself.

How Serverless Execution Works#

Code is packaged as a function or a container image and attached to triggers: an HTTP request, a file landing in object storage, a message on a queue, or a schedule. When an event arrives and a warm instance is free, the platform runs the handler on it at once. Otherwise it performs a cold start: it creates an isolated sandbox, such as one of the Firecracker microVMs that AWS built for Lambda, loads the code, starts the language runtime, and runs the initialization code outside the handler before handling the event. Instances are stateless and disposable, so anything that must outlive a request goes to external storage or a database.

Cold Starts and Machine Learning Models#

Machine learning makes cold starts expensive, because initialization imports a framework such as PyTorch and loads model weights that can take hundreds of megabytes. The standard pattern loads the model once per instance, outside the handler or on first use, so that every warm invocation reuses it. This handler follows that pattern with a ResNet-50 classifier from torchvision:

import time

import torch
import torchvision

model = None  # module state survives between invocations while the instance stays warm


def handler(event, context):
    """Classify one image tensor, building the model only on a cold start."""
    global model
    cold_start = model is None
    if cold_start:
        model = torchvision.models.resnet50().eval()  # built once per instance
    with torch.inference_mode():
        logits = model(event["image"])
    return {"cold_start": cold_start, "class_id": int(logits.argmax())}


for _ in range(3):
    start = time.perf_counter()
    response = handler({"image": torch.rand(1, 3, 224, 224)}, context=None)
    print(response["cold_start"], f"{(time.perf_counter() - start) * 1000:.0f} ms")

On an Apple M5 Pro CPU, the first call builds the network and takes roughly 0.2 seconds, while the two warm calls that reuse it take about 15 ms each. Teams shrink the remaining cold start with smaller container images, compact or quantized models, and lighter runtimes such as ONNX Runtime, or remove it by keeping a minimum number of warm instances, such as Lambda provisioned concurrency or Cloud Run minimum instances, which bills for idle time again.

Serverless for AI Workloads#

Serverless fits event-driven computer vision pipelines. An upload to object storage can trigger a function that validates and resizes the image during data preprocessing before it joins a training dataset, and in a security alarm system, a motion sensor can send a single frame to an object detection function that alerts only when it finds a person. Platform limits decide the rest: AWS Lambda runs each invocation for at most 15 minutes with up to 10,240 MB of memory and no GPU, which fits CPU inference with compact models and rules out long training runs, let alone distributed training across many GPUs. GPU serverless has filled that gap: Cloud Run attaches NVIDIA L4 GPUs to containers that still scale to zero, and platforms such as Modal bill GPU time by the second.

Ultralytics Platform gives the same trade-off to YOLO deployments: an endpoint at the default size of 1 vCPU and 2 GiB is free and scales to zero when idle, and starting from idle can take up to about a minute, while custom sizes keep one instance warm and bill for its uptime.

Serverless vs. Related Models#

• Infrastructure as a Service: Virtual machines that run, and bill, around the clock whether or not requests arrive, with the customer managing the operating system and scaling. Long training jobs usually run here.
• Kubernetes: Orchestrates containers on clusters the team operates. Knative adds request-driven scaling to zero on top of it, which KServe uses to serve models.
• Edge computing: Runs code on or near the devices that produce data. Serverless runs it in a cloud region, and some edge platforms combine the two by running functions in many small locations near users.