benchmark-integrity

A kernel-userland protocol enforcing information-theoretic bounds on AI adaptivity leakage, benchmark gaming, and capability spillover.

Your AI “improved.” Did it get smarter—or memorize the test? Find out before you trust the score.

Model-harness co-training: does RL against a programmatic-tool-calling harness reverse the bash-over-PTC ordering, and does the winner survive a harness swap?

Behavioral evaluation framework for sentience-, emotion-, and welfare-related AI claims, with anti-sandbagging analysis.

Reproducible audit of the computational claims in arXiv:2604.27092. Reproduces the released accuracies to within 5e-4 and finds that both benchmarks place repeated measurements of the same physical pair in training and test sets.

Horizon-Eval: evaluation-integrity framework for portable long-horizon agent benchmarks, with QA gates, trajectory auditing, replayable run bundles, and safety-gap analysis.

Scanner for evaluator trust boundaries — the places your eval pipeline trusts things it shouldn't. Born from published research; 100+ defect reports across 40+ orgs. Free, MIT, forever.

Selected projects, runnable checks, and a published paper.