Evidently Documentation

What Evidently does

Evidently is an open source library for evaluating, testing and monitoring ML and LLM systems, from experiments to production. It works with tabular and text data and ships more than 100 built-in metrics, from data drift detection to LLM judges. You can use it for one-off checks in a notebook, as tests in a pipeline or CI, or as a full monitoring service with a dashboard.

The pieces to learn

• Reports: compute a set of metrics, usually from a preset such as the data drift preset, comparing current data with reference data. View them in Python or save them as HTML or JSON.
• Tests: add pass or fail conditions to a report, or let Evidently generate them from the reference data, and use the result to fail a pipeline step or a CI job.
• Datasets and descriptors: for text and LLM outputs, descriptors score each row, for example length, sentiment, whether it contains certain words, or an LLM-as-judge verdict.
• Monitoring UI: a dashboard that tracks metrics and test results over time. You can self-host the open source version.

A note on versions

The API changed substantially from version 0.7: reports are now built from evidently.presets and run as report.run(current, reference). Many blog posts and courses were written for the older API, so check that a tutorial matches the version you install and prefer the official docs.

How long it takes

A first drift report on your own data takes an evening. Turning it into scheduled tests with thresholds you trust, and a dashboard, is a weekend.

Free or paid?

The library and the self-hosted monitoring UI are free and open source under the Apache 2.0 licence. Evidently Cloud is the company's hosted platform with extra features such as alerting and user management, and its README says it has a free tier; check the official site for current plans.

How it fits an MLOps workflow

Evidently covers the monitoring stage. Run a weekly report on logged predictions against the training data, fail the pipeline when drift tests fail, and trigger a gated retrain. For LLM features, the same library scores samples of live outputs.

Common questions

Which drift method should I use? The presets choose defaults by column type; you can set methods such as PSI explicitly. Tune thresholds on past data before alerting on them.

What can I do after it? Detect data drift and data quality problems automatically, gate pipelines on tests, and show model health over time.

Best for: anyone with a model in production, or a portfolio project, that needs monitoring beyond uptime.