OpenLineage

What it is

OpenLineage is an open framework for collecting and analysing data lineage, hosted under the Linux Foundation. Lineage answers two questions every governance review asks: where did this data come from, and what depends on it? OpenLineage defines a standard API for lineage events, so different tools can report lineage in one format instead of each inventing its own.

Core model

• Job. A process that reads and writes data, such as a nightly pipeline step or a training script.
• Run. One execution of a job, with its start, end and status.
• Dataset. An input or output, such as a table, a file or a topic.
• Facets. Extra metadata attached to any of these, such as the schema, the data source, row counts or data quality results.

Integrations exist for common tools such as Apache Airflow, Apache Spark and dbt, which send events to a compatible backend. Marquez is the reference backend that stores and displays them.

Why it matters for AI governance

To answer "which training data went into this model version?" or "which models are affected if this source table is wrong?", you need lineage. The EU AI Act's data governance and record-keeping requirements and ISO/IEC 42001's data controls both assume you can trace data. Lineage turns that from a spreadsheet someone forgot to update into something recorded automatically.

Is it worth your time?

For the technical path, yes, at the level of understanding the model and running one integration. You will meet lineage in data catalogues and platforms whether or not they use OpenLineage underneath. For the policy path, read the overview so you know what automated lineage can show an auditor, and ask for it.

Prerequisites

Comfort with data pipelines: what a scheduler does, and reading and writing tables. Without that, start with a data engineering course first.

Free or paid?

Free and open source.