chatbot-testing

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.

Conversation tests for AI agents: a simulated user talks to your agent through its API, a judge checks every claim, and convy reports pass rate, tokens, time and what changed between builds.

EvalBot — local-first chatbot security & quality evaluation (FastAPI + Next.js). Evaluate chatbot answers against your own docs & guidelines with ML/NLP + AI-judge scoring. Apache-2.0.

pytest lab for testing LLMs: RAG eval, red teaming, guardrails, drift monitoring — 14 modules, 382 tests, zero API calls needed

An automated approach for exploring and testing conversational agents using large language models. TRACER discovers chatbot functionalities, generates user profiles, and creates comprehensive test suites for conversational AI systems.

Automated acceptance tests for HTTP and n8n chatbots: catch broken answers in CI.

A Python library to connect and interact with chatbots.

QA framework for testing conversational AI systems (LLM agents, chatbots, voice assistants) with workflow validation and regression checks

An open-source framework for robust, LLM-powered testing and tracing of conversational AI applications.

Give your AI agent its own Linux computer on your Mac — and watch everything it does. 25 MCP tools, live noVNC view, pause/handoff, approval guardrail, audit log. BYOK or local Ollama.

A framework for testing LLM-based chatbots in regulated industries (telco, banking, insurance). Covers hallucination detection, prompt injection resistance, response quality scoring and regression testing.

Bilingual portfolio project for evaluating chatbot helpfulness, accuracy, tone, safety, and instruction following.

Layered evaluation of two LLM features, a RAG support assistant and ticket triage: rules, reference checks, an LLM judge validated against human labels, safety cases, stability and a regression gate in CI, replayed from real recorded calls.

Automated generation of questions and answers from documents.

Modular, extensible QA framework for evaluating AI chatbots — built for CI/CD pipelines, model comparison, and continuous quality monitoring.

Automated QA for AI chatbots: simulated multi-turn conversations, hard checks and an LLM judge that catch regressions after prompt changes.

A dataset-driven LLM evaluation harness for testing behavioral reliability across OpenAI and Claude. Uses invariant-based assertions, suite organization, and CLI filtering for multi-model comparison.

Quality auditor for AI chatbots. Analyzes your conversation logs to show where the bot is underperforming.

Lightweight evaluation lab for LLM chatbots — quality metrics, regression tests, and model comparison harness in pytest.