DeepEval

What DeepEval is

DeepEval is an open source framework for testing LLM applications, described in its README as similar to pytest but specialised for unit testing LLM apps. You write test cases with an input, the actual output and optionally an expected output or the tools that should be called, choose metrics, and assert that each case passes a threshold. Tests run with deepeval test run and fit into CI like any other test suite.

Install it with pip install -U deepeval. A TypeScript package, deepeval on npm, integrates with Vitest.

Metrics for agents

Most eval tools grew out of RAG and chat. DeepEval has a set of agentic metrics listed in its README:

• Task Completion: did the agent accomplish its goal?
• Tool Correctness: were the right tools called with the right arguments?
• Argument Correctness: were tool arguments valid for the task?
• Step Efficiency, Plan Quality and Plan Adherence: did the agent take unnecessary steps or ignore its own plan?
• Goal Accuracy and Tool Use.

It also offers G-Eval, a research-backed LLM-as-a-judge metric for custom criteria, a deterministic graph-based judge builder, and RAG metrics such as faithfulness and contextual recall. The @observe() decorator traces the steps of your agent so metrics can score individual components, not just the final answer.

Is it worth it?

If you build agents in Python and want evals that live next to your unit tests, yes. The pytest-like workflow lowers the barrier to writing the first twenty cases, which is the step most teams skip. Treat model-judged metrics with care: compare a sample of their verdicts with your own before trusting a threshold, and prefer exact code checks wherever they are possible.

Prerequisites and time

Python and pytest basics, an agent or LLM app to test, and an API key for whichever model you use as the judge. Setting up a first suite takes an afternoon; building a useful eval set is ongoing work.

Free or paid?

The framework is free and open source under the Apache 2.0 licence, and metrics can use any model you choose. Confident AI, the company behind it, offers an optional cloud platform for sharing reports, regression tracking and production monitoring, with a free tier and paid plans.

What you can do after it

Write agent evals that check tool choice, arguments and task completion, run them on every pull request, and trace which step of a failing run caused the problem.

Common questions

DeepEval or promptfoo? Both are open source. DeepEval feels like pytest and is Python first; promptfoo is configured in YAML and is strong at red teaming. Many teams use one for regression tests and the other for security testing.