AgentEvals
EvaluationMIT
AgentEvals is a LangChain package of readymade evaluators for agent trajectories, meaning the intermediate steps an agent takes. It offers trajectory match evaluators with strict, unordered, subset and superset modes, LLM-as-judge scoring of a trajectory, graph trajectory evaluators and async support in Python. Developers install it with pip or npm, pass OpenAI-style messages, and receive a key, a score and a comment, optionally logging results through LangSmith or extracting trajectories from LangGraph threads.
Its README describes a companion package, openevals, for more general evaluation.

Pinout
- MCP server: not checked yet
- HTTP API: not checked yet
- command line: not checked yet
- Python SDK: supported, verified
- TypeScript SDK: supported, verified
- OpenAPI description: not checked yet
| Interface | Status | Source we read | Read on |
|---|---|---|---|
| MCP server | Not checked yet | — | — |
| HTTP API | Not checked yet | — | — |
| Command line | Not checked yet | — | — |
| Python SDK | Supported, verified | GitHub README, langchain-ai/agentevals, Install | |
| TypeScript SDK | Supported, verified | GitHub README, langchain-ai/agentevals, Install | |
| OpenAPI description | Not checked yet | — | — |
Facts
| Purpose | Readymade evaluators for agent trajectories. GitHub README, langchain-ai/agentevals, read . |
|---|---|
| Install | pip install agentevals for Python; npm install agentevals @langchain/core for TypeScript. GitHub README, langchain-ai/agentevals, Install, read . |
| Trajectory match | Strict, unordered, subset and superset modes with configurable tool-argument matching. GitHub README, langchain-ai/agentevals, Evaluator types, read . |
| LLM-as-judge | Scores a trajectory with an LLM, optionally against a reference. GitHub README, langchain-ai/agentevals, Evaluator types, read . |
| Graph trajectories | LLM-as-judge and strict-match evaluators for graph-based agents. GitHub README, langchain-ai/agentevals, Evaluator types, read . |
| Integrations | LangSmith through its pytest, Vitest or Jest runners and evaluate function; LangGraph trajectory extraction. GitHub README, langchain-ai/agentevals, Integrations, read . |
| License | MIT (permission text and 2025 LangChain copyright read in the LICENSE file; the file does not name the license). Repository LICENSE file, read . |
| Last commit | . GitHub API, latest commit on the default branch of langchain-ai/agentevals, read . |
Limits
- The README's quickstart needs an OpenAI API key.
- Graph trajectory evaluators are aimed at graph-based agents only.
- The README says it is intended as a conceptual starting point for agent evals.
Strengths
- Python and TypeScript packages are documented.
- Four trajectory match modes are listed.
- Results can be logged through LangSmith.
Alternatives to AgentEvals
All evaluation tools for agents
Readymade evaluators for LLM applications in Python and TypeScript.

Python toolkit for metrics and test-set generation for LLM applications.

Open-core platform to trace, test and evaluate agents and coding assistants.

Framework for evaluating, testing and monitoring ML and LLM systems.
Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check
Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.