Add eval, alert and audit authoring skills; policy author gets a testing reference

[chhhee10]

Summary

Three new authoring skills for FailproofAI Cloud, plus a testing reference for failproofai-policy-author. Brainstorming (failproofai-eval-brainstorm) decides what to watch; these cover writing it so it measures, fires or sweeps the right thing, grounded in the org's own data rather than guesses.

Per skill

failproofai-eval-author (new, mirror): write a hosted evaluation once you know what to measure.

• Choose code vs jev vs judge with a three-question dividing line; the one-expression Python against the session surface; ground every payload key and compared value in the org's real event shapes; repair from a dry-run error or backtest verdict (broken / all_skipped / unfounded / constant / uniform / healthy).
• Why: a guessed payload key does not fail. It scores every session the same and looks like it works. Grounding is the step that prevents that.
• The references are also, verbatim, the instructions the Cloud's own evaluation drafter runs on. SKILL.md says so and explains how to read them by hand.
• Files: SKILL.md, references/{choosing-the-mechanism,output-contract,python-surface,grounding,writing-the-code,worked-examples,repair,cli}.md

failproofai-alert-author (new, mirror): an alert that fires when it should and stays quiet otherwise.

• Pick the trigger kind from what the intent must filter (metric rules cannot filter by agent, so use a query). Set the threshold from the alert's own per-check p95/p99 over 7 and 30 days, not a round number. Replay the rule, then repair it once if it is noisy (more than about 1 fire a day) or silent.
• Files: SKILL.md, references/{triggers,noise,cli}.md

failproofai-audit-author (new, mirror): a scheduled sweep at a sensible scope and cost.

• Map the intent to the deterministic signals (secret, PII, injection and denial patterns; completion, loop and drift lenses). Check that the scope has traffic and how often each signal fires. Choose cadence, window, sensitivity and failures per run. Say when a policy, alert or evaluation is the better tool.
• Files: SKILL.md, references/{signals,form,wrong-tool,cli}.md

failproofai-policy-author (maintained here):

• SKILL.md gains a 3-line pointer in Test both directions to the new references/testing.md. That file covers where the must-fire and must-allow lists come from, how to read a history replay (for a guard, a fire on a successful call is the point, not a false hit), and how to repair in place.
• One wording fix: a customer-specific script name in a Jev probe example is now a neutral wa-send.sh.
• Applied on top of main's current version (Jev end to end: the policy author and publish skills, for failproofai 1.0.9 #24, failproofai 1.0.9). No other content changed.

failproofai (umbrella): its routing table and references/skill-directory.md now list the three new skills. Its description is unchanged (996 chars, no room).

Public vs upstream

Readers here are coding agents with fp, not the in-dashboard assistant. So each mirrored skill:

• carries a public-only references/cli.md, the same grounding and testing done with fp list … and fp query run (ClickHouse SQL over events), plus the dashboard forms (/<org>/eval-authoring/new, /<org>/alerts/new, /<org>/audits/new, which show the drafted evidence) or fp alerts create / fp audits create with a form-to-flag table;
• leaves out the upstream-only files: the assistant's tool map (references/dashboard.md) and the drafters' own prompt contracts (references/compose.md);
• has no source-tree paths or assistant tool names in any shared file. Upstream now enforces this with a test over every mirrored file.

Every other file is byte-identical to upstream (FailproofAI/agenteye feat/agent-skills @ 24206eff). failproofai-eval-brainstorm did not change.

Evidence: the authoring bench (upstream, in-dashboard drafter)

One fresh build, demo org, one run per kind:

kind	cases	baseline pass	final pass	quality (baseline → final)	median / max draft s
policy	24	20/24	23/24	golden positives blocked 39/43 → 42/43; golden false fires 1/68 → 0/68	44.1 / 89.7 → 16.3 / 37.7
eval	17	14/17	17/17	mean AUC good-vs-bad 0.917 → 1.00; constant-output drafts 1 → 0	8.5 / 19.1 → 15.8 / 26.3
alert	8	2/8	7/8	trigger right 0.75 → 1.00; fire rate within bound 0.25 → 0.875; draft's own 7-day count = independent replay 0/8 → 8/8	28.5 / 240 → 8.2 / 15.3
audit	9	0/9	9/9	signal recall 0 → 1.00; preview present and sane 0/9 → 9/9	22.6 / 34.6 → 14.1 / 34.1

Limits of the bench: the eval AUC of 1.00 is a ceiling, because each case's cohorts come from one predicate the draft can mirror. It proves grounding, not judgement. All 17 eval drafts were code, and jev/judge were not scored. The alert miss is by design: the user named a threshold, so it is kept and a quieter one is offered. The audit "sane" check only tests shape. The bench measures the drafter that runs on these files, not a coding agent reading them.

Source of truth

• failproofai-eval-author, failproofai-alert-author, failproofai-audit-author: synced from FailproofAI/agenteye → agent/skills/<name>/ (private). Do not hand-edit here. README rows and CONTRIBUTING say so, and CONTRIBUTING notes which files stay upstream.
• failproofai-policy-author: maintained here. Agenteye re-vendors it byte-identically from this branch.

Test plan

• python3 scripts/validate-skills.py: 9 skills, 0 errors, 0 warnings
• Descriptions measured by parsing the YAML: eval-author 813, alert-author 817, audit-author 841, policy-author 975 (all < 1024)
• Mirrored files diff -r identical to upstream except the listed upstream-only files
• Scan of all new/changed files: no source paths, no assistant tool names, no key-shaped strings, no tenant identifiers
• Upstream agent suite: npm run build && npm test, 1130 pass / 0 fail / 1 skipped. Includes the eval-drafter prompt fixture (byte-identical, unchanged) and the policy skill section selection (all headings found, within budget)
• Reviewer: install one new skill (npx skills add FailproofAI/skills#feat/authoring-skills --skill failproofai-alert-author -a claude-code) and check that cli.md's SQL runs against a real org with fp query run

🤖 Generated with Claude Code