AI Adoption Lab

Status: local demonstration and learning lab. Four bounded business workflows show how typed inputs, retrieval, approval gates and evaluation fit together. The default model is deterministic and fake, so the core examples can be studied without a paid model account.

This is an independent project, not a system deployed at or endorsed by a trading firm. The legacy Python namespace jump_ai_adoption_lab is retained for compatibility with the existing commands; it is not an employment or affiliation claim.

Start with architecture, approval handling, and evaluation design. The test suite and example data are learning evidence, not validation for real financial, legal or hiring decisions.

A Python lab for exploring and demonstrating internal AI adoption - agents, evals, RAG, and observability.

What This Is

A hands-on training environment covering realistic AI adoption workflows for Finance, Legal, HR, and Support. It demonstrates:

• Workflow-first AI agents with typed inputs/outputs
• RAG with citations - keyword retrieval, chunking, citation validation
• Evaluation harness - golden datasets, scorers, failure taxonomy, release gates
• Model routing - cost/quality/risk-based model selection
• Human-in-the-loop approval - draft-only defaults, audit logs
• KPI dashboards - adoption, quality, cost, latency monitoring
• Observability - trace logging, event streams
• Python exercises - 14 coding exercises with solutions

Quickstart

# Clone and enter
cd agent-training

# Set up virtual environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest

# Demo commands
python -m jump_ai_adoption_lab.cli demo legal
python -m jump_ai_adoption_lab.cli demo finance
python -m jump_ai_adoption_lab.cli demo hr
python -m jump_ai_adoption_lab.cli demo support

# Evaluations
python -m jump_ai_adoption_lab.cli evals run --tier smoke
python -m jump_ai_adoption_lab.cli evals run --tier regression
python -m jump_ai_adoption_lab.cli evals estimate-cost

# Metrics
python -m jump_ai_adoption_lab.cli metrics generate-demo-data
python -m jump_ai_adoption_lab.cli metrics summarize

# Exercises
python -m jump_ai_adoption_lab.cli exercises list
python -m jump_ai_adoption_lab.cli exercises run lru_cache

# Dashboard
streamlit run src/jump_ai_adoption_lab/dashboards/app.py

Repo Structure

├── README.md
├── pyproject.toml
├── Makefile
├── .env.example
├── .gitignore
├── data/
│   ├── synthetic/
│   ├── evals/
│   └── traces/
├── docs/
│   ├── interview_playbook.md
│   ├── glossary.md
│   ├── architecture.md
│   ├── evals_and_model_selection.md
│   ├── adoption_operating_model.md
│   ├── security_and_governance.md
│   ├── demo_script.md
│   └── math_practice.md
├── src/jump_ai_adoption_lab/
│   ├── schemas.py          # Pydantic models for all domain types
│   ├── config.py           # Environment-based configuration
│   ├── cli.py              # Typer CLI with rich output
│   ├── llm/
│   │   ├── base.py         # Abstract LLM client interface
│   │   ├── fake.py         # Deterministic fake for testing
│   │   ├── model_router.py # Risk/cost-based model selection
│   │   └── providers.py    # Optional Anthropic/OpenAI adapters
│   ├── rag/
│   │   ├── chunking.py     # Word-based document chunking
│   │   ├── retrieval.py    # TF-IDF keyword retrieval
│   │   ├── citations.py    # Citation validation
│   │   └── corpus.py       # Synthetic document corpus
│   ├── agents/
│   │   ├── base.py         # Abstract agent interface
│   │   ├── approval.py     # Human-in-the-loop approval gate
│   │   ├── legal_contract_review.py
│   │   ├── finance_invoice_exception.py
│   │   ├── hr_feedback_intelligence.py
│   │   └── support_triage.py
│   ├── evals/
│   │   ├── runner.py       # Eval orchestration with tiers
│   │   ├── cases.py        # Golden dataset (25+ cases)
│   │   ├── scorers.py      # Exact match, rubric, safety, etc.
│   │   ├── reports.py      # Report generation and release gates
│   │   └── failure_taxonomy.py
│   ├── metrics/
│   │   ├── adoption.py     # WAU, activation, retention
│   │   ├── quality.py      # Success rate, acceptance, FPR/FNR
│   │   ├── cost.py         # Cost per task, cost by model
│   │   ├── latency.py      # P50/P95/P99 percentiles
│   │   ├── retrieval.py    # Precision@k, recall@k
│   │   └── dashboard_data.py
│   ├── observability/
│   │   ├── tracing.py      # Trace logger
│   │   ├── events.py       # Workflow event logging
│   │   └── store.py        # JSONL storage backend
│   ├── dashboards/
│   │   └── app.py          # Streamlit KPI dashboard
│   └── exercises/          # 14 Python coding exercises
│       ├── lru_cache.py
│       ├── rate_limiter.py
│       ├── retry_backoff.py
│       ├── log_parser.py
│       ├── document_chunker.py
│       ├── keyword_retriever.py
│       ├── precision_recall_at_k.py
│       ├── invoice_matcher.py
│       ├── dag_scheduler.py
│       ├── eval_runner_mini.py
│       ├── model_router_mini.py
│       ├── redaction.py
│       ├── latency_percentiles.py
│       └── math_questions.py
└── tests/                  # Run locally to verify the current result

Key Design Principles

1. Start with workflows, not models - AI is useful when it accelerates a specific, bounded business workflow
2. Agents are bounded systems - typed inputs, typed outputs, guardrails, traces
3. Safe defaults - read-only, draft mode, human approval for irreversible actions
4. Evals before shipping - golden datasets, release gates, regression suites
5. Model routing - right model for the right task at the right cost
6. Measure everything - adoption, quality, cost, latency, incidents
7. Weekly KPI ownership - review metrics, identify bottlenecks, improve continuously

Agents

Agent	Domain	Guardrail
Legal Contract Review	Vendor contracts, clause analysis	Never approves/signs contracts
Finance Invoice Exception	PO matching, discrepancy detection	Never releases payment
HR Feedback Intelligence	Interview feedback analysis	Never ranks/accepts/rejects candidates
Support Triage	Ticket classification, runbook retrieval	Never changes production systems

Evaluation Tiers

Tier	Purpose	Scope
smoke	Pre-commit sanity	~10 cases, fast
regression	Pre-merge coverage	~50 cases
release	Pre-deploy full suite	~200 cases + safety
audit	Human review sampling	Production traces

Dashboard

The Streamlit dashboard shows:

• Executive overview (completions, quality, cost, incidents)
• Adoption metrics (activation, retention, team usage)
• Quality (eval pass rate, acceptance rate, FPR/FNR)
• Cost & latency (per model, per workflow, percentiles)
• Workflow drilldown
• Trace explorer

No Paid Model Account Required

Everything runs locally with a deterministic fake LLM. Optional real LLM integration via ANTHROPIC_API_KEY or OPENAI_API_KEY environment variables.

How to Extend

• Add a new agent: subclass BaseAgent, add eval cases, wire into CLI
• Add eval cases: edit evals/cases.py, add to the golden dataset
• Add a new workflow type: extend WorkflowType enum in schemas
• Add a new metric: add to the appropriate module in metrics/
• Add a document to RAG: add to rag/corpus.py

Documentation

• docs/interview_playbook.md - Talking points, 30/60/90 plan, questions
• docs/glossary.md - Plain English definitions of all key terms
• docs/architecture.md - System design, data flow, security boundaries
• docs/evals_and_model_selection.md - How to eval, compare, upgrade models
• docs/adoption_operating_model.md - Weekly cadence, pilot selection, scaling
• docs/security_and_governance.md - Permissions, PII, audit, prompt injection
• docs/demo_script.md - 10-minute walkthrough script
• docs/math_practice.md - 25 probability/statistics questions with solutions