Skip to content

ART M5: versioned datasets, per-tenant dedup, cross-version duplicate flags (#109) - #226

Merged
amar-python merged 8 commits into
mainfrom
claude/issue-109-dedup-versioned-datasets
Nov 10, 2026
Merged

ART M5: versioned datasets, per-tenant dedup, cross-version duplicate flags (#109)#226
amar-python merged 8 commits into
mainfrom
claude/issue-109-dedup-versioned-datasets

Conversation

@amar-python

@amar-python amar-python commented Nov 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What

M5 of the ART plan: deduplication and naming (BR-28..31, decisions 2 and 3). Tracker: #109.

  • Versioned datasets (BR-30). A repeated filename becomes the next version, stored in csv_<slug>_v<n>. Nothing is overwritten and the overwrite flag is ignored. Same-name uploads are serialised per dataset (advisory lock), so concurrent uploads get distinct versions.
  • Per-tenant exact-file dedup (BR-28). Every request dedups within its own registry: a tenant's, or the default registry for a request with no session. overwrite no longer bypasses it.
  • Cross-version duplicate rows (BR-29). Rows already in an earlier version are loaded but flagged (row_flags, insert-only) and shown on the review page. The user decides per row with the existing M4 include/exclude decision; M6 applies it.
  • Schema check (BR-31). A new version's columns are compared with the previous version's (added, removed, retyped) and the result is returned. The previous table is never altered.
  • Migration 0010. (Numbered 0010 because main took 0009 for Dynamic mode: confirm inferred types before table creation, with a provenance flag #195.) Adds dataset/version (backfilled, oldest first), replaces UNIQUE (file_name) with UNIQUE (dataset, version) plus a partial unique index for non-dynamic modes, creates row_flags. Downgrade refuses while versions share a file name.
  • T&E loader. Its file-name matching is now limited to non-dynamic registry entries. Without this, a T&E upload named like a dataset would have deleted its versions.

Decisions to check

  • Table names are csv_<slug>_v<n>, not the plan's bare <slug>_v<n>. The prefix keeps the existing csv_ guards working and stops a filename such as uploads.csv colliding with a registry table.
  • BR-29's configurable rule (key columns per dataset) is not built; only the default exact whole-row match. VCRM keeps BR-29 at Partial.
  • A T&E upload with content identical to a dynamic dataset still replaces that entry and drops its table (permission-gated, RV-S5: upload overwrite and mode-switch bypass delete-permission gate #175). Left alone as out of scope; VCRM keeps BR-31 at Partial for this.
  • Deleting the newest version frees its number, so the next upload reuses it.

Not in this PR

  • Frontend. The overwrite prompt can now only appear for exact-content duplicates, where retrying with overwrite returns the same duplicate. The UI has no display for flagged rows or schemaCheck yet.
  • README, STARTER_GUIDE, API_INTEGRATION and QUICKSTART still describe csv_<hash> tables and filename dedup. docs: fix STARTER_GUIDE walkthrough accuracy (#165, #166) #206 owns those files, so I left them.

Verification

After merging current main: the full suite (everything except the browser job) passes against PostgreSQL 16 (495 tests). Offline evals 23/23, scripts/lint.sh clean, docs/TEST_CASES.md regenerated. Earlier, before the merge, scripts/test_report.py --strict gave 418 passed, 0 failed, 0 skipped.

New: tests/test_versioned_datasets.py (22 tests) and four migration tests. With the duplicate-row lookup disabled, four of them fail. Seven older tests that pinned overwrite or single-name behaviour were updated to the new behaviour.

Also includes a fix found while reviewing my own work: row_flags was missing from APPEND_ONLY, so the runtime role still held UPDATE and DELETE on it. It is covered in the runtime-role test.

Closes nothing yet: #109 stays open until this merges.

https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq

Generated by Claude Code

claude added 5 commits November 9, 2026 15:08
@claude
feat: versioned datasets, per-tenant file dedup, cross-version duplic…
0ed50a1 
…ate flags (M5, #109)

BR-28..31, decisions 2 and 3.

- A repeated filename is the next version of its dataset, stored in
  csv_<slug>_v<n>. Nothing is overwritten; the overwrite flag is ignored.
  Same-name uploads are serialised per dataset with an advisory lock so
  concurrent uploads get distinct versions.
- Exact-file dedup stays per registry (per tenant, or the default registry
  for a request with no session); overwrite no longer bypasses it.
- Rows already in an earlier version are loaded but flagged (row_flags,
  insert-only) and shown on the review page; the user decides per row with
  the existing include/exclude decision.
- A new version's columns are compared with the previous version's and the
  difference is reported. The previous table is never altered.
- T&E loads match registry entries by file name only among non-dynamic
  entries, so they can't delete dataset versions.
- Migration 0009 adds dataset/version (backfilled), replaces
  UNIQUE(file_name) with UNIQUE(dataset, version) plus a partial unique
  index for non-dynamic modes, and creates row_flags.

Tests: tests/test_versioned_datasets.py (22), migration tests for 0009;
tests that pinned the old overwrite behaviour now pin versioning.
Full suite: 418 passed, 0 failed, 0 skipped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
@claude
fix: grant the runtime role insert-only access to row_flags (M5, #109)
27f83bb 
row_flags was insert-only by trigger but missing from APPEND_ONLY, so the
runtime role still held UPDATE and DELETE grants on it. The trigger blocked
the change anyway; this restores the privilege layer the other insert-only
tables have, and covers it in the runtime-role test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
@claude
wip: M6 approval and staged import (not yet rebased on current main)
42b143c 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
@claude
Merge main into the M5 branch (#109)
2d890c2 
Main moved on: #195 took migration 0009 (types_provenance), the tests were
reorganised into category folders (#167), and the loader, routes, db and
tests were edited.

- Renumbered this branch's migration 0009 -> 0010 so it chains after #195's.
- Registry insert carries both types_provenance (#195) and dataset/version (M5).
- APPEND_ONLY keeps row_flags and takes main's alembic_version_te.
- tests/test_versioned_datasets.py moved to tests/system/ (it needs PostgreSQL).
- Migration tests keep both sets; mine are renumbered. Head checks use
  get_current_head(), as main now does.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
@claude
feat: approval and transactional staged import (M6, #110; BR-32, BR-3…
76094c8 
…3, BR-34; ADR-0002)

For dynamic-mode uploads: stage, review, submit, approve or reject, then import
in one transaction.

- POST /api/csv/uploads stores the original and parsed rows; nothing is imported.
- Submit, then a reviewer approves or rejects (a rejection needs a reason; a
  rejected upload can be reviewed again and resubmitted). A signed-in member
  can't approve data they uploaded or submitted unless the tenant allows it.
- Approval snapshots the review (newest decision and edit ids), and review is
  refused once approved, so what was approved is what is imported.
- Import (Idempotency-Key header): COPY into a temporary staging table, promote
  into csv_<slug>_v<n> with _source_row_id and _import_id, register the file, write
  the imports record and the audit row, commit. Any failure rolls all of it back.
  A repeat with the same key returns the original; another key is a 409.
- Reconciliation is stored per import and checked by the database:
  rows_in = excluded + duplicate + imported.
- Migration 0011: submissions, approvals, imports (insert-only), the staged status,
  uploads.types_provenance, and tenants.allow_self_approval (off by default).
- The direct /api/csv/upload path and T&E mode are unchanged.

Merges the updated M5 branch (which carries current main): migration renumbered
0010 -> 0011, tests moved to tests/system/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
@amar-python amar-python mentioned this pull request Nov 9, 2026
Merged
@claude
fix: let browsers send Idempotency-Key to the import endpoint (M6, #110)
2f9315b 
The CORS allow-list lacked Idempotency-Key, so a browser's preflight for
POST /api/csv/uploads/{id}/import was refused with 400 and the import could
only be called by scripts. Adds the header and a preflight test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
This was referenced Nov 10, 2026
Merged
Merged
@amar-python
Merge pull request #227 from amar-python/claude/issue-110-approval-st…
dac5dd4 
…aged-import

ART M6: approval and transactional staged import (#110)
@amar-python
amar-python marked this pull request as ready for review November 10, 2026 03:15
@amar-python Claude

amar-python commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

The accessibility (Playwright + axe-core) check failed on this head with one scenario, upload_mapping_dialog: a new color-contrast violation on 13 muted-text elements inside the column-mapping dialog.

This isn't caused by this PR. It changes no frontend files (0 of 29), and the accessibility job stubs the API, so backend changes can't alter what it scans. The same scenario has passed on this branch's earlier heads and on the stacked PRs' heads, and failed once before, on #228's first attempt, then passed on retry. The 13 elements are all text-muted-foreground text, which fits axe scanning while the dialog's fade-in is still mid-transition, but I haven't confirmed that. I found no fix for it on main.

I'll re-run the job once when the rest of this run finishes (GitHub won't re-run a single job while the run is in progress). If it fails again, I'll treat it as real and dig in.

Generated by Claude Code

@claude
Wait for animations before axe scans to stop flaky contrast failures
715469c 
A dialog still fading in is blended with the page behind it, so axe
reported false color-contrast violations in the column mapping dialog.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhKR6dS4KmR9nHSMJr
@amar-python
amar-python merged commit a1e02cd into main Nov 10, 2026
4 checks passed
amar-python pushed a commit that referenced this pull request Nov 10, 2026
@claude
Merge main into the M7 branch (after #226)
baa52f4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@amar-python @claude