Data Lake

A data lake is a central repository that stores data in its raw, native format, from database exports and JSON logs to images, video, audio, and documents, usually in low-cost cloud object storage, and applies structure only when the data is read. James Dixon, then chief technology officer of Pentaho, coined the term in 2010 to contrast a data mart, which he compared to cleaned and packaged bottled water, with a large body of data in its natural state that many users can draw from. Because nothing has to be modeled up front, a lake can keep everything an organization collects, which suits machine learning: training data for tomorrow's model is often raw data whose use nobody knew about when it was stored.

How a Data Lake Works#

A lake separates storage from compute. Files sit in object storage such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage, and engines such as Spark, Trino, or a Python process read them directly, so each team scales its own compute without copying data. Tabular data is usually kept in open columnar formats such as Apache Parquet, while images and video stay as ordinary files, often next to JSON or Parquet manifests that hold their metadata and labels.

Most lakes are organized in zones by refinement, which Databricks calls a medallion architecture: a raw (bronze) zone keeps data exactly as it arrived, a cleaned (silver) zone holds validated and deduplicated data, and a curated (gold) zone holds tables and datasets ready for reporting or training. A metadata catalog such as the AWS Glue Data Catalog, Hive Metastore, or Unity Catalog records what each dataset contains, who owns it, and who may read it. A lake without that catalog and data cleaning degrades into a "data swamp": files nobody can find, trust, or safely delete.

Schema-on-Read#

A lake applies a schema when data is read, which is called schema-on-read. Writers can land data as soon as it exists, even when two sources disagree on fields, and each reader decides which columns and types it needs. This example writes two raw JSON files with different fields, then reads both with one schema using Polars:

import json
from pathlib import Path

import polars as pl

# Raw zone: two cameras write detection events as JSON lines, each with its own fields
Path("lake/raw").mkdir(parents=True, exist_ok=True)
Path("lake/raw/dock-1.jsonl").write_text(json.dumps({"label": "forklift", "conf": 0.91}))
Path("lake/raw/dock-2.jsonl").write_text(json.dumps({"label": "person", "conf": 0.62, "lux": 120}))

# Schema-on-read: this reader chooses the columns and types when it queries the files
schema = {"label": pl.String, "conf": pl.Float32, "lux": pl.Int32}
events = pl.scan_ndjson("lake/raw/*.jsonl", schema=schema, include_file_paths="source")
print(events.collect())

The result has two rows with columns label, conf, lux, and source. The first camera never recorded lux, so its value comes back as null, and source records which raw file each row came from. A second job could read the same files with a different schema without rewriting them.

Data Lake vs. Data Warehouse vs. Lakehouse#

• Data warehouse: Stores cleaned, structured tables in a fixed schema that data must match when it is loaded (schema-on-write), usually after an extract, transform, load (ETL) step. Warehouses such as Snowflake, BigQuery, and Redshift answer SQL and business intelligence queries quickly, with less flexibility for images, text, and other unstructured data.
• Data lake: Stores any format at low cost and transforms data after loading it (ELT), which favors exploration, data mining, and machine learning. Plain lake files lack transactions, so concurrent writers can leave readers with half-written results.
• Lakehouse: Adds warehouse features to lake storage through open table formats such as Apache Iceberg, Delta Lake, and Apache Hudi, which provide ACID transactions, schema enforcement, and time travel to earlier versions of a table. Databricks researchers described the architecture in a 2021 paper.

Data Lakes in Machine Learning#

For computer vision teams, the lake holds every camera frame, its capture metadata, its annotations, and the predictions deployed models make on it. Autonomous vehicle programs upload drive logs from their fleets and mine them for rare scenes, such as a pedestrian at night in rain, to label and add to training sets. In healthcare, de-identified scans and reports collected in one lake supply the data for medical image analysis models. Production predictions that flow back into the lake for review and relabeling close a data flywheel, and table snapshots or tools such as DVC pin the exact data version each model was trained on. Feature stores and vector databases are often built on top of a lake, serving features or embeddings derived from its raw files.

Ultralytics Platform can turn a folder in Amazon S3, Google Cloud Storage, or Azure Blob Storage into a training dataset without uploading a copy: it indexes up to 500,000 objects per import in place with read-only credentials, parses YOLO .txt label files next to the images, and assigns train, val, and test splits from folder names. Each indexed image is pinned to its object version, so Platform fails closed on any image whose file is overwritten in the lake.