Evaluation tools for agents

Tools that test agents: replaying saved runs, scoring answers and traces against test cases, and comparing two versions of an agent on the same set. We write the input formats and interfaces we could read in each tool's docs, its license and its last commit. We do not run the tools ourselves and we do not score them, so there is no grade on this page, only dated facts. On this page, tools are sorted by the date of their last check, newest first. There are no votes and no rank, and a dashed pin means we have not looked yet.Read more
  • Supported, verified
  • Not supported
  • Not checked yet
Must have:

Showing 11 of 11 tools.

Evaluation tools and their verified pins, sorted by last check
ToolMCPAPICLIPyTSOASLicenseLast commitChecked
AgentEvalsnot checked yetnot checked yetnot checked yetsupported, verifiedsupported, verifiednot checked yetMIT
Braintrustsupported, verifiednot checked yetsupported, verifiedsupported, verifiedsupported, verifiednot checked yetApache-2.0 (SDK repositories)
OpenEvalsnot checked yetnot checked yetnot checked yetsupported, verifiedsupported, verifiednot checked yetMIT
Ragasnot checked yetnot checked yetsupported, verifiednot checked yetnot checked yetnot checked yetApache-2.0
LangWatchnot checked yetnot checked yetnot checked yetnot checked yetsupported, verifiednot checked yetApache-2.0
Evidentlynot checked yetnot checked yetnot checked yetsupported, verifiednot checked yetnot checked yetApache-2.0
DeepEvalnot checked yetnot checked yetsupported, verifiednot checked yetsupported, verifiednot checked yetApache-2.0
Giskardnot checked yetnot checked yetnot checked yetsupported, verifiednot checked yetnot checked yetApache-2.0
TruLensnot checked yetnot checked yetnot checked yetsupported, verifiednot checked yetnot checked yetMIT
Inspectnot checked yetnot checked yetsupported, verifiednot checked yetnot checked yetnot checked yetMIT
Promptfoosupported, verifiednot checked yetsupported, verifiednot checked yetsupported, verifiednot checked yetMIT

How to choose

  1. Interface. MCP if your agent already speaks it. HTTP API if you call from a queue or a worker. A Python or TypeScript SDK if the tool runs inside your own code.
  2. License. Self-hosting needs a license that allows it. Source-available is not open source.
  3. Activity. Check the last commit date. A tool with no commit for months may still work, but nobody is fixing it.

Nearby