Big Data

Big data is data whose size, arrival rate, or mix of formats exceeds what a single machine and conventional database tools can store and process, so it is spread across clusters of machines that work on it in parallel. Analyst Doug Laney of META Group described the challenge in 2001 as three Vs: volume, the amount of data; velocity, the speed at which it arrives and must be handled; and variety, the mix of structured tables, logs, text, images, audio, and video. Later definitions add veracity, how trustworthy the data is, and value. For machine learning, big data is both the raw material and an engineering problem: models improve with more training data, and the pipelines that collect, clean, and serve that data take much of the effort of building them.

Big Data in AI and Computer Vision#

Deep learning grew alongside datasets too large for one disk. ImageNet, introduced in 2009, grew to more than 14 million labeled images, and the 1.28 million training images of its 1,000-class challenge trained the convolutional neural network AlexNet that won the 2012 competition. Meta's SA-1B dataset, released in 2023 with the Segment Anything Model, holds 11 million images and 1.1 billion segmentation masks, and Meta pretrained Llama 3 on more than 15 trillion text tokens. Scaling laws describe why such collections pay off: loss falls predictably as data, parameters, and compute grow together.

In computer vision, volume comes from fleets of cameras and sensors. A single autonomous vehicle records camera, radar, and lidar streams for every hour it drives, and factories and stores add thousands of continuous video feeds. Velocity matters when results are needed as data arrives, as in real-time inference on traffic or production-line video. Variety covers images and video together with annotations, sensor metadata, and logs. More data helps only when its labels are accurate, so large vision datasets depend on data cleaning, deduplication, and active learning to pick which new images are worth labeling.

How Big Data Is Stored and Processed#

Big data systems scale out: they spread data and computation over many commodity machines in place of one larger server. Google's 2004 MapReduce paper described the model that open-source Apache Hadoop made widespread: split the input into shards, run a map function on every shard in parallel on the machine that stores it, and reduce the partial results into one answer. Apache Spark, started at UC Berkeley in 2009, keeps intermediate results in memory, which makes iterative jobs such as model training far faster than disk-based MapReduce. Raw files usually sit in cloud object storage organized as a data lake, columnar formats such as Apache Parquet let a query read only the columns it needs, and streaming platforms such as Apache Kafka carry high-velocity events.

This example runs the map and reduce steps on three shards of image labels with Python's standard library:

from collections import Counter
from multiprocessing import Pool

# Three shards of image labels; a cluster stores each shard on a different machine
shards = [
    ["car", "person", "car", "bus"],
    ["person", "person", "bicycle"],
    ["car", "person", "dog", "car"],
]


def count_labels(shard):
    """Map step: count the labels in one shard."""
    return Counter(shard)


if __name__ == "__main__":
    with Pool(processes=3) as pool:
        partial_counts = pool.map(count_labels, shards)  # one worker process per shard

    # Reduce step: merge the partial counts into one total
    totals = sum(partial_counts, Counter())
    print(totals.most_common())

Each shard is counted in its own process, and the reduce step merges the partial counts into [('car', 4), ('person', 4), ('bus', 1), ('bicycle', 1), ('dog', 1)]. Spark and Hadoop run the same two steps across thousands of machines, sending the map code to the machines that hold each shard.

Training Models on Big Data#

A training run reads big data in batches. Data loaders decode a few dozen to a few hundred images at a time from local disk or object storage across several worker processes, so the full dataset never has to fit in memory, and distributed training splits each batch across many GPUs. For object detection, Google's Open Images V7 detection set is a 561 GB download with 1.74 million training images across 601 classes, far more than most machines can hold in RAM. When asked to cache a dataset in memory with cache="ram", Ultralytics YOLO training estimates the decoded size from 30 random images and caches only if twice that estimate fits in available RAM; larger datasets are read and decoded from disk every epoch.

Big Data vs. Related Terms#

• Data mining: The search for patterns, anomalies, and associations in data of any size, often run on big data platforms.
• Data analytics: Querying, summarizing, and visualizing data to support decisions; big data analytics is the part that needs distributed engines.
• Data-centric AI: Improving models by improving the quality and coverage of their data, which matters most when a dataset is too large to inspect by hand.