Evaluation & testing
Open-source Python framework for testing LLM applications: it runs metric-based scoring on test cases and on captured agent trajectories, including individual LLM calls, tool use, retrieval and sub-agent handoffs. It executes locally in developer or CI environments and provides no access control or audit records itself.
open_source · generally available · Research snapshot 2026-09-06
Visit the official product source ↗Evaluation & testing
Useful conversation with: AI engineer, QA lead.
Show me a CI test run that scores my agent's full trajectory, including whether the right tools were called with the right arguments.
Support labels reflect the supplied research. Documentation and vendor claims are not independent product tests. “Not established” means the researcher did not find support; it does not prove a capability is absent.
Documented by provider
DeepEval supports end-to-end and component-level evaluation, scoring complete agent trajectories across decisions and actions as well as individual steps such as LLM calls, tool use, retrieval and sub-agent handoffs.
Limit: The repository README does not establish accuracy of the bundled metrics or any governance controls.
Documented by provider
Evaluations run locally in the user's own environment, with an optional connection to the Confident AI cloud for centralized test reports.
Limit: Self-hosting of the cloud reporting component is not documented; no RBAC, SSO, retention or audit-log features are described.
Documented by provider
Documented evaluation integrations include OpenAI Agents, LangChain, LangGraph, CrewAI, Pydantic AI, LlamaIndex, Google ADK, AWS AgentCore and Mastra, plus MCP-related task-completion metrics.
Limit: Integration depth per framework is not described; presence in a list does not establish maintained parity.
Listing does not imply partnership, supplier status, a working DutyGraph integration, or a compliance certification.
Suggest a correction