Kubernetes

Kubernetes, often shortened to K8s, is an open-source system that runs containerized applications across a cluster of machines. Users declare the desired state of a workload, such as which container image to run, how many copies to keep, and how much CPU, memory, and GPU capacity each copy needs, and Kubernetes continuously starts, restarts, moves, and scales containers to match it. Google announced the project in June 2014, drawing on its internal Borg cluster manager, and with version 1.0 in July 2015 offered it as the seed project of the newly formed Cloud Native Computing Foundation. The name is Greek for helmsman, and K8s replaces the eight letters between K and s. In machine learning operations (MLOps), Kubernetes is the common layer that runs distributed training jobs, batch inference, and model serving on shared hardware.

Architecture and the Control Loop#

A cluster has a control plane and worker nodes. The control plane holds the API server that every client talks to, the etcd store of cluster state, the scheduler that assigns new pods to nodes, and controllers that each watch one kind of object. Every worker node, whether an on-premises server or a cloud virtual machine, runs a kubelet agent and a container runtime such as containerd or CRI-O. The smallest unit Kubernetes schedules is a pod: one or more containers that share a network address and storage volumes.

Workloads are declared as objects, usually written in YAML and applied with kubectl. A Deployment keeps a set number of identical pods running and replaces them gradually when the image changes, a Service gives those pods one stable address and spreads requests across them, and a Job runs pods until a task completes. Controllers compare the declared state with what is actually running and act on any difference:

The same loop makes Kubernetes self-healing. When a node fails, its pods drop out of the running state, and the controller creates replacements on healthy nodes. A pod left Pending for lack of capacity is also the signal that cluster autoscalers use to add nodes.

Kubernetes for Machine Learning#

GPUs reach pods through device plugins, which advertise them as a countable resource such as nvidia.com/gpu, and through Dynamic Resource Allocation, stable since Kubernetes 1.34, which lets a pod request devices by attributes such as GPU model or memory. NVIDIA's GPU Operator installs drivers and plugins on GPU nodes and can split one GPU into Multi-Instance GPU slices or time-share it between pods. OpenAI reported in 2021 that it had scaled single Kubernetes clusters to 7,500 nodes to train models such as GPT-3 and CLIP.

Open-source projects add ML-specific layers on top:

• Kubeflow: Runs distributed PyTorch training, ML pipelines, and notebooks on a cluster.
• Kueue: Queues batch and training jobs and enforces team quotas, so expensive GPUs stay busy without one team taking them all.
• KServe: Serves models behind autoscaling endpoints, including scale-to-zero for idle models.
• Horizontal Pod Autoscaler: Adds or removes inference replicas based on CPU use or custom metrics such as queue length, which holds latency steady as traffic changes.

A Job is the simplest way to run a one-off GPU task such as batch inference. The official Ultralytics Docker image already contains the yolo command and YOLO26 nano weights, so this Job only sets a command and asks for one GPU:

apiVersion: batch/v1
kind: Job
metadata:
  name: yolo26-predict
spec:
  backoffLimit: 2 # retry a failed pod at most twice
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: yolo
          image: ultralytics/ultralytics:latest
          command: ["yolo", "predict", "model=yolo26n.pt", "source=https://ultralytics.com/images/bus.jpg"]
          resources:
            limits:
              nvidia.com/gpu: 1 # schedule only onto a node with a free GPU

kubectl apply -f job.yaml submits it, the scheduler places the pod on a node with a free GPU, and kubectl logs job/yolo26-predict prints the object detection results from predict mode. A training Job follows the same pattern and adds a persistent volume for checkpoints, plus an in-memory volume at /dev/shm for PyTorch data loader workers.

Kubernetes vs. Docker and Serverless#

• Docker: Builds images and runs containers on one machine. Kubernetes places containers across many machines, and since version 1.24 it runs Docker-built images through containerd or CRI-O instead of Docker Engine.
• Serverless computing: Services such as Google Cloud Run and AWS Lambda hide the cluster entirely and bill per request, and Knative brings the same scale-to-zero model to Kubernetes clusters.
• Managed Kubernetes: Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS) run the control plane for their customers, and GKE Autopilot also manages the worker nodes.