Great Expectations (GX Core)
An open source Python framework for data quality testing: declare Expectations about your data, run them as validations in pipelines and publish readable Data Docs as evidence that data was checked.
Tool
AI & Machine Learning
Great Expectations
A day to set up a first validation
Intermediate
Free
What it is
Great Expectations, now called GX Core for the open source library, is a Python framework for testing data. Instead of hoping a dataset is clean, you write down what you expect of it and check every batch against those expectations. It is widely used in data engineering and machine learning pipelines.
Key ideas
- Expectation. One verifiable claim about data, such as "this column is never null", "values are between 0 and 120" or "this column's values are unique".
- Expectation Suite. A named set of expectations for one dataset.
- Validation and Checkpoint. Running a suite against a batch of data and acting on the result, for example stopping a pipeline when a check fails.
- Data Docs. Human-readable reports of what was checked and what passed, generated from the results.
Why it matters for AI governance
Most AI failures start in the data. The EU AI Act asks providers of high-risk systems for data governance practices covering relevance, representativeness, errors and completeness, and ISO/IEC 42001 has controls on data for AI systems. Automated data quality checks are how engineers implement that, and Data Docs are the kind of evidence an auditor or governance reviewer can read without running code. If you are on the policy side, learning what a validation report shows helps you ask for the right evidence.
How to use it
Follow the getting started guide with a CSV you know well. Write five expectations covering nulls, ranges, allowed categories, uniqueness and row counts, run them, then break the data on purpose and watch the checks fail. Keep the generated report as a portfolio artefact.
Alternatives
dbt tests cover similar ground if your data already lives in a warehouse managed by dbt, and pandas-based checks are fine for small projects. The point is not the tool but that checks run automatically and leave a record.
Free or paid?
GX Core is free and open source. The company also sells a hosted product, GX Cloud, which you do not need for learning.
Ready to start?
Opens on Great Expectations in a new tab.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.