eval-framework

Eval framework. Define correct, test against it, get results.

Find what your AI agent gets wrong — before you have a rubric. Qualitative eval for PMs.

Grades agent completion outputs across weighted multi-criteria rubrics, computes inter-rater agreement variance, and flags inconsistent outlier score distributions | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

Grades agent completion outputs across weighted multi-criteria rubrics, computes inter-rater agreement variance, and flags inconsistent outlier score distributions | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

RAG context precision and context recall evaluator measuring retrieved chunk relevance and rank quality

RAG context precision and context recall evaluator measuring retrieved chunk relevance and rank quality

One-stop CLI for running structured skill evals across all agent harnesses

A web-based interactive demo for the GuessArena evaluation framework

4-model parallel planning workflow with eval framework — Claude, Gemini, Codex, GLM-5 · OpenClaw ecosystem

Evaluate and trace tool-calling LLM agents, with one guarantee: a tool failure can never look like an empty result.

Curated AI agent evaluation skills from Microsoft's Eval Guide — plan, generate, run, and interpret eval suites for Copilot Studio agents

Self-hosted evaluation framework for tool-calling LLM agents. Define tasks in YAML with expected tool calls and scoring rubrics, run against Claude or GPT-4, get scored reports via CLI and FastAPI dashboard. Lightweight, local-first, no lock-in.

🚀 基于Java的开源AI自动化评测框架 / An open source AI automation evaluation framework based on Java

CLI tool for defining and running LLM evaluation suites against any OpenAI-compatible endpoint. YAML test cases, multi-metric scoring (exact match, regex, LLM judge), and rich terminal reports. Built for fast feedback loops in CI/CD.

CLI tool for evaluating and comparing LLM prompts. Define test cases in YAML, run against OpenAI and Anthropic models, score outputs, and generate HTML reports. A lightweight, local-first alternative to cloud eval platforms.

Open-source evaluation framework for LLM agents. Run head-to-head A/B tests, score with LLM-as-judge rubrics, and visualize results in a Streamlit dashboard. Model-agnostic, self-hosted, zero external infrastructure.

Evaluate RAG pipelines end-to-end. Auto-generates Q&A benchmarks from your documents, runs them through your retrieval system, and scores answers on faithfulness, relevance, and context recall using LLM-as-judge. Local vector store, pluggable LLM backend.

Self-hosted evaluation harness for AI agents using rubric-based LLM-as-judge scoring plus deterministic checks. Web UI for test case management, rubric definition, trace inspection, and side-by-side agent comparison.

Python tool for extracting validated, schema-defined JSON from documents (PDF, HTML, Markdown, plaintext) using LLMs, with source grounding and a built-in eval harness. Works with Anthropic Claude or OpenAI.

Open-source evaluation framework for MCP servers powered by Claude. Auto-discovers tools, generates test scenarios, runs LLM-as-judge scoring to assess correctness and safety, audits for security issues, and visualizes traces in a web dashboard.