DeepEval

EvaluationApache-2.0

DeepEval is a Python framework that scores LLM outputs and agent runs against test cases, including metrics for tool use and MCP servers. It is made by Confident AI, and test reports can be sent to the Confident AI platform.

The README lists metrics for MCP task completion and MCP use, TypeScript integrations for the Vercel AI SDK and Mastra, and a command line used to run test files and log in to the platform.

Pinout

  • MCP server: not checked yet
  • HTTP API: not checked yet
  • command line: supported, verified
  • Python SDK: not checked yet
  • TypeScript SDK: supported, verified
  • OpenAPI description: not checked yet
Where each pin comes from
InterfaceStatusSource we readRead on
MCP serverNot checked yet——
HTTP APINot checked yet——
Command lineSupported, verifiedREADME, "run test_ai_agent.py in the CLI"
Python SDKNot checked yet——
TypeScript SDKSupported, verifiedGetting started, Installation (npm install --save-dev deepeval)
OpenAPI descriptionNot checked yet——

Facts

Facts about DeepEval
LicenseApache-2.0. GitHub API, repository license field, read .
Main languagePython (as detected by GitHub). GitHub API, repository record, read .
Publisherconfident-ai (GitHub organization). GitHub API, repository record, read .
Latest GitHub releasepython-v4.2.4, published 2026-09-22. GitHub API, latest release, read .
MCP metricsMCP Task Completion, MCP Use, Multi-Turn MCP Use. README in the repository, read .
Hosted optionConfident AI platform for shared test reports. README in the repository, read .
Last commit. GitHub API, latest commit on the default branch of confident-ai/deepeval, read .

Limits

  • Cloud test reports need an account and an API key on the Confident AI platform, entered through the command line, as described in the README.

Strengths

  • The README documents metrics written for agents that call MCP servers.

Alternatives to DeepEval

All evaluation tools for agents
  1. Open-core platform to trace, test and evaluate agents and coding assistants.

    Evaluation
  2. Framework for evaluating, testing and monitoring ML and LLM systems.

    Evaluation
  3. Readymade evaluators that score an agent's trajectory of steps.

    Evaluation
  4. Observability and evaluation platform for tracing and improving AI agents.

    Evaluation

Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check

Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.