Evaluation tools for agents
Tools that test agents: replaying saved runs, scoring answers and traces against test cases, and comparing two versions of an agent on the same set. We write the input formats and interfaces we could read in each tool's docs, its license and its last commit. We do not run the tools ourselves and we do not score them, so there is no grade on this page, only dated facts. On this page, tools are sorted by the date of their last check, newest first. There are no votes and no rank, and a dashed pin means we have not looked yet.Read more
- Supported, verified
- Not supported
- Not checked yet
Must have:
Showing 11 of 11 tools.
| Tool | MCP | API | CLI | Py | TS | OAS | License | Last commit | Checked |
|---|---|---|---|---|---|---|---|---|---|
AgentEvals | not checked yet | not checked yet | not checked yet | supported, verified | supported, verified | not checked yet | MIT | ||
Braintrust | supported, verified | not checked yet | supported, verified | supported, verified | supported, verified | not checked yet | Apache-2.0 (SDK repositories) | ||
OpenEvals | not checked yet | not checked yet | not checked yet | supported, verified | supported, verified | not checked yet | MIT | ||
Ragas | not checked yet | not checked yet | supported, verified | not checked yet | not checked yet | not checked yet | Apache-2.0 | ||
LangWatch | not checked yet | not checked yet | not checked yet | not checked yet | supported, verified | not checked yet | Apache-2.0 | ||
Evidently | not checked yet | not checked yet | not checked yet | supported, verified | not checked yet | not checked yet | Apache-2.0 | ||
DeepEval | not checked yet | not checked yet | supported, verified | not checked yet | supported, verified | not checked yet | Apache-2.0 | ||
Giskard | not checked yet | not checked yet | not checked yet | supported, verified | not checked yet | not checked yet | Apache-2.0 | ||
TruLens | not checked yet | not checked yet | not checked yet | supported, verified | not checked yet | not checked yet | MIT | ||
Inspect | not checked yet | not checked yet | supported, verified | not checked yet | not checked yet | not checked yet | MIT | ||
Promptfoo | supported, verified | not checked yet | supported, verified | not checked yet | supported, verified | not checked yet | MIT |
How to choose
- Interface. MCP if your agent already speaks it. HTTP API if you call from a queue or a worker. A Python or TypeScript SDK if the tool runs inside your own code.
- License. Self-hosting needs a license that allows it. Source-available is not open source.
- Activity. Check the last commit date. A tool with no commit for months may still work, but nobody is fixing it.










