[amar-python]

What

M6 of the ART plan: approval and a transactional staged import (BR-32, BR-33, BR-34; ADR-0002; decisions 4 and 5). Tracker: #110.

Stacked on #226 (M5). The base is the M5 branch, so this diff shows only M6. Merge #226 first and GitHub will retarget this to main.

For dynamic-mode uploads the flow is now stage → review → submit → approve or reject → import:

• Stage (POST /api/csv/uploads): stores the original and parsed rows; nothing is imported. Duplicate-row flags against earlier versions (M5) are computed here, so the reviewer sees them before approving.
• Submit / approve / reject (POST /api/csv/uploads/{id}/submit, /approval): a rejection needs a reason, and a rejected upload can be reviewed again and resubmitted. Approving needs every row that isn't excluded to validate.
• Self-approval (decision 5): a signed-in member can't approve data they uploaded or submitted. A tenant with one user turns it on with accounts.set_self_approval or scripts/create_tenant_admin.py --allow-self-approval. A request with no session has no identity to compare, so the rule doesn't apply to it.
• Approval snapshots the review. It records the newest row decision and cell edit at that moment; the import applies exactly those, and review is refused once approved. What was approved is what is imported.
• Import (POST /api/csv/uploads/{id}/import, Idempotency-Key header): one transaction COPYs the approved rows into a temporary staging table, promotes them into csv_<slug>_v<n> with _source_row_id and _import_id, registers the file, writes the imports record and the audit row, and commits. Any failure rolls back all of it. A repeat with the same key returns the original result; another key for an imported upload is a 409.
• Reconciliation is stored per import (rows in, excluded, edited, duplicate, imported) and the database checks rows in = excluded + duplicate + imported.
• Migration 0011: submissions, approvals, imports (insert-only, and in APPEND_ONLY so the runtime role can't change them), the staged status, uploads.types_provenance (Dynamic mode: confirm inferred types before table creation, with a provenance flag #195), and tenants.allow_self_approval.

Decisions to check

• Dynamic mode only. T&E mode and the direct POST /api/csv/upload are unchanged, so they still import without approval, with row-level isolation. VCRM keeps BR-32, BR-33 and BR-34 at Partial for that reason. Moving T&E over needs a stored target table, DB-constraint handling and T&E review typing; I'd do it as its own piece.
• The audit gap is closed for the new flow only. The tracker says M6 closes the post-commit audit gap; the direct upload path still writes its audit row afterwards. README, the audit and originals docstrings, and ADR-0002 now say exactly that.
• Validation is in Python before COPY, not in SQL after it (ADR-0002 said SQL). It uses the same cast_value contract as the loaders and the frontend, and a failing row stops approval, so nothing invalid reaches staging. Recorded in an implementation note on the ADR.
• UNIQUE (upload_id) on imports, stricter than the ADR's (upload_id, approval_id): an upload is imported once, ever.
• Failed imports leave no record, by design: the import record is written inside the transaction, so it exists only if the data does.
• Two staged uploads of the same bytes are allowed to coexist; whichever imports second gets a 409 and stays approved but unimportable.
• New roles: submit_import (uploader, tenant admin), approve_import (reviewer, tenant admin), run_import (uploader, reviewer, tenant admin). docs/ROLES.md and the role-matrix test cover them.

Not in this PR

• Frontend. No UI for staging, submitting, approving or importing yet; the API is complete and tested.
• The M6 branch history contains one wip: commit from before I merged main; squash on merge.

Verification

Full suite (everything except the browser job) against PostgreSQL 16, built as CI builds it: all 525 tests pass. scripts/lint.sh clean, bandit shows only the existing api/db.py notes, docs/TEST_CASES.md regenerated.

New: tests/system/test_staged_import.py (24 tests) covers the tracker's list: reject then resubmit; failure injected at three steps of the import, each followed by a successful same-key retry; a retry after success; concurrent imports with different and with the same key. It also covers reconciliation and lineage, data fidelity through COPY (quotes, \N, newlines, unicode, NULLs, typed columns), self-approval, and the workflow tables being insert-only. Five migration tests cover 0011, and the staged import runs as the restricted runtime role. Breaking the self-approval rule, or committing before the import finishes, makes the relevant tests fail.

#110 stays open until this merges.

---

Generated by Claude Code