OpenEvals
EvaluationMIT
OpenEvals is a package of readymade evaluators from LangChain for testing LLM applications, available in Python and TypeScript. It includes LLM-as-judge evaluators with prebuilt prompts, structured extraction checks, code checks, agent trajectory matching and similarity measures, and it supports custom evaluators and multiturn simulation. Developers use it as a starting point for writing evals, running them on their own or logging results through LangSmith runners. LLM-as-judge evaluators need an LLM client.
The README points to a sibling package, agentevals, for agent-specific evaluation.

Pinout
- MCP server: not checked yet
- HTTP API: not checked yet
- command line: not checked yet
- Python SDK: supported, verified
- TypeScript SDK: supported, verified
- OpenAPI description: not checked yet
| Interface | Status | Source we read | Read on |
|---|---|---|---|
| MCP server | Not checked yet | — | — |
| HTTP API | Not checked yet | — | — |
| Command line | Not checked yet | — | — |
| Python SDK | Supported, verified | README, langchain-ai/openevals, Installation | |
| TypeScript SDK | Supported, verified | README, langchain-ai/openevals, Installation | |
| OpenAPI description | Not checked yet | — | — |
Facts
| Purpose | Evaluators with many prebuilt prompts for LLM apps, described as a starting point for writing evals. README, langchain-ai/openevals, read . |
|---|---|
| Evaluator families | LLM-as-judge, prebuilt prompts, extraction and tool calls, code, sandboxed code, agent trajectory, and exact match, Levenshtein and embedding similarity. README, langchain-ai/openevals, Evaluator types, read . |
| Integrations | LangChain, the OpenAI client, E2B for sandboxed code, LangSmith runners and LangGraph for multiturn simulation. README, langchain-ai/openevals, Integrations, read . |
| Judge requirement | LLM-as-judge evaluators need an LLM client; the package includes LangChain's OpenAI integration by default. README, langchain-ai/openevals, read . |
| Agent evaluation | The README points agent-specific evaluation to agentevals. README, langchain-ai/openevals, read . |
| License | MIT (permission text and 2025 LangChain copyright read in the LICENSE file; the file does not name the license). Repository LICENSE file, read . |
| Last commit | . GitHub API, latest commit on the default branch of langchain-ai/openevals, read . |
Limits
- LLM-as-judge evaluators need an LLM client and an API key.
- The README says agent-specific evaluation belongs in a separate package.
Strengths
- Both Python and TypeScript packages exist.
- Several evaluator families are listed.
- The package integrates with LangSmith runners.
Alternatives to OpenEvals
All evaluation tools for agents
Readymade evaluators that score an agent's trajectory of steps.

Python toolkit for metrics and test-set generation for LLM applications.

Open-core platform to trace, test and evaluate agents and coding assistants.

Framework for evaluating, testing and monitoring ML and LLM systems.
Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check
Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.