AgentEvals

EvaluationMIT

AgentEvals is a LangChain package of readymade evaluators for agent trajectories, meaning the intermediate steps an agent takes. It offers trajectory match evaluators with strict, unordered, subset and superset modes, LLM-as-judge scoring of a trajectory, graph trajectory evaluators and async support in Python. Developers install it with pip or npm, pass OpenAI-style messages, and receive a key, a score and a comment, optionally logging results through LangSmith or extracting trajectories from LangGraph threads.

Its README describes a companion package, openevals, for more general evaluation.

Pinout

  • MCP server: not checked yet
  • HTTP API: not checked yet
  • command line: not checked yet
  • Python SDK: supported, verified
  • TypeScript SDK: supported, verified
  • OpenAPI description: not checked yet
Where each pin comes from
InterfaceStatusSource we readRead on
MCP serverNot checked yet——
HTTP APINot checked yet——
Command lineNot checked yet——
Python SDKSupported, verifiedGitHub README, langchain-ai/agentevals, Install
TypeScript SDKSupported, verifiedGitHub README, langchain-ai/agentevals, Install
OpenAPI descriptionNot checked yet——

Facts

Facts about AgentEvals
PurposeReadymade evaluators for agent trajectories. GitHub README, langchain-ai/agentevals, read .
Installpip install agentevals for Python; npm install agentevals @langchain/core for TypeScript. GitHub README, langchain-ai/agentevals, Install, read .
Trajectory matchStrict, unordered, subset and superset modes with configurable tool-argument matching. GitHub README, langchain-ai/agentevals, Evaluator types, read .
LLM-as-judgeScores a trajectory with an LLM, optionally against a reference. GitHub README, langchain-ai/agentevals, Evaluator types, read .
Graph trajectoriesLLM-as-judge and strict-match evaluators for graph-based agents. GitHub README, langchain-ai/agentevals, Evaluator types, read .
IntegrationsLangSmith through its pytest, Vitest or Jest runners and evaluate function; LangGraph trajectory extraction. GitHub README, langchain-ai/agentevals, Integrations, read .
LicenseMIT (permission text and 2025 LangChain copyright read in the LICENSE file; the file does not name the license). Repository LICENSE file, read .
Last commit. GitHub API, latest commit on the default branch of langchain-ai/agentevals, read .

Limits

  • The README's quickstart needs an OpenAI API key.
  • Graph trajectory evaluators are aimed at graph-based agents only.
  • The README says it is intended as a conceptual starting point for agent evals.

Strengths

  • Python and TypeScript packages are documented.
  • Four trajectory match modes are listed.
  • Results can be logged through LangSmith.

Alternatives to AgentEvals

All evaluation tools for agents
  1. Readymade evaluators for LLM applications in Python and TypeScript.

    Evaluation
  2. Python toolkit for metrics and test-set generation for LLM applications.

    Evaluation
  3. Open-core platform to trace, test and evaluate agents and coding assistants.

    Evaluation
  4. Framework for evaluating, testing and monitoring ML and LLM systems.

    Evaluation

Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check

Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.