OpenEvals

EvaluationMIT

OpenEvals is a package of readymade evaluators from LangChain for testing LLM applications, available in Python and TypeScript. It includes LLM-as-judge evaluators with prebuilt prompts, structured extraction checks, code checks, agent trajectory matching and similarity measures, and it supports custom evaluators and multiturn simulation. Developers use it as a starting point for writing evals, running them on their own or logging results through LangSmith runners. LLM-as-judge evaluators need an LLM client.

The README points to a sibling package, agentevals, for agent-specific evaluation.

Pinout

  • MCP server: not checked yet
  • HTTP API: not checked yet
  • command line: not checked yet
  • Python SDK: supported, verified
  • TypeScript SDK: supported, verified
  • OpenAPI description: not checked yet
Where each pin comes from
InterfaceStatusSource we readRead on
MCP serverNot checked yet——
HTTP APINot checked yet——
Command lineNot checked yet——
Python SDKSupported, verifiedREADME, langchain-ai/openevals, Installation
TypeScript SDKSupported, verifiedREADME, langchain-ai/openevals, Installation
OpenAPI descriptionNot checked yet——

Facts

Facts about OpenEvals
PurposeEvaluators with many prebuilt prompts for LLM apps, described as a starting point for writing evals. README, langchain-ai/openevals, read .
Evaluator familiesLLM-as-judge, prebuilt prompts, extraction and tool calls, code, sandboxed code, agent trajectory, and exact match, Levenshtein and embedding similarity. README, langchain-ai/openevals, Evaluator types, read .
IntegrationsLangChain, the OpenAI client, E2B for sandboxed code, LangSmith runners and LangGraph for multiturn simulation. README, langchain-ai/openevals, Integrations, read .
Judge requirementLLM-as-judge evaluators need an LLM client; the package includes LangChain's OpenAI integration by default. README, langchain-ai/openevals, read .
Agent evaluationThe README points agent-specific evaluation to agentevals. README, langchain-ai/openevals, read .
LicenseMIT (permission text and 2025 LangChain copyright read in the LICENSE file; the file does not name the license). Repository LICENSE file, read .
Last commit. GitHub API, latest commit on the default branch of langchain-ai/openevals, read .

Limits

  • LLM-as-judge evaluators need an LLM client and an API key.
  • The README says agent-specific evaluation belongs in a separate package.

Strengths

  • Both Python and TypeScript packages exist.
  • Several evaluator families are listed.
  • The package integrates with LangSmith runners.

Alternatives to OpenEvals

All evaluation tools for agents
  1. Readymade evaluators that score an agent's trajectory of steps.

    Evaluation
  2. Python toolkit for metrics and test-set generation for LLM applications.

    Evaluation
  3. Open-core platform to trace, test and evaluate agents and coding assistants.

    Evaluation
  4. Framework for evaluating, testing and monitoring ML and LLM systems.

    Evaluation

Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check

Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.