Skip to content

feat(sanjose): add Jev demo, model guides and verified charts - #52

Draft
glebedel wants to merge 5 commits into
mainfrom
codex/sanjose-demo-review
Draft

glebedel wants to merge 5 commits into
mainfrom
codex/sanjose-demo-review

Conversation

@glebedel

Copy link
Copy Markdown
Contributor

The 16-slide San José deck now includes the completed Jev comparison, healthy classifier results and a clearer Jira → Notion → Slack attack. The example also exposes Jev through the actual Vercel AI SDK search loop, with concise implementation and model-training guides. Follow-up to #51.

Changes

  • Slide 7 names MetaTool ToolE and labels the published BGE result Custom Embeddings model: 93.8 nDCG@5, alongside Jev 84.2 and BM25 38.5. Small StackOne marks identify the custom model and classifier.
  • Slide 13 separates the Sonnet/Qwen bars and percentage labels. Healthy Defender 0.8.2 retries replace the missing classifier results: Sonnet 24/72 and Qwen 57/72. Combined defense remains explicitly unmeasured.
  • The chain explains that the agent reads an attacker-edited Jira instruction and a Notion destination before attempting the Slack post. All writes are intercepted. Closing links point to Guillaume’s X/LinkedIn, StackOne and the talk site.
  • Add optional --search-provider jev, metadata-only search, and a six-tool staff lookup fixture. StackOne’s hosted search remains the default. Jev validates every candidate response and records resolved model, requests, tokens and elapsed time.
  • Add source-pinned attack fixtures and retain all 33 selected rehearsal attempts, including refusals. Per-turn provider token usage is recorded separately from response bytes.
  • Shorten the main README and add diagrams for discovery, code mode and defense. Training guides explain BGE embeddings, MiniLM classifier heads/calibration, and MiniCPM5-1B reviewer fine-tuning/serving. Slide 13 notes separate the spoken cue from backup detail.

Validation

  • TypeScript, Biome and 31 tests pass. Knip, Madge and duplicate-code checks pass.
  • Publication test and all nine deck builds pass. Documentation links and staged credential checks pass.
  • All 16 local slides reviewed at 1280 × 720. Email reveal, notes, images and closing links checked; no browser warnings/errors. Signed commit verified.
  • Jev’s benchmark includes all 94,000 valid query/tool scores. Live validation of the new six-tool CLI path is pending credential access.

Measurement limits

The 93.8 result is published, while Jev and BM25 were reproduced. Ranking quality is not task accuracy or latency. The selected complex attacks largely bypassed the classifier; the separately collected bars do not establish a reliable improvement. The Playground hybrid cascade and CLI combined policy differ, and neither has a matched combined Sonnet/Qwen result here. The live reviewer checkpoint still needs verification.

Keep this draft until the new Jev CLI rehearsal is recorded. Complete a timed stage rehearsal and review the exported PDF before upload. No merge or deployment is included.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
stackone-talks 3267ecf Sep 22 2026, 08:49 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant