Skip to content

feat(db): Alembic migrations and registry UNIQUE constraints (#84, #81) - #87

Merged
amar-python merged 1 commit into
mainfrom
fix/issue-81-84-alembic-registry-unique
Oct 26, 2026
Merged

amar-python merged 1 commit into
mainfrom
fix/issue-81-84-alembic-registry-unique

Conversation

@amar-python

Copy link
Copy Markdown
Owner

Fixes #84 and #81. Part of ART Milestone 1 (BR-23, BR-28).

#84: Alembic migrations

  • Adds alembic.ini and alembic/. api/db.py bootstrap() already ran alembic upgrade head whenever alembic.ini existed. It never did, so every start took the fallback CREATE TABLE IF NOT EXISTS path. It now migrates on startup.
  • 0001 baseline mirrors the fallback schema exactly, with IF NOT EXISTS. Databases created by the old fallback adopt it with no change.
  • The version table lives in the uploads schema, so each schema carries its own migration state. That's ready for schema-per-tenant (ART decision 6).

#81: registry uniqueness

  • 0002 adds UNIQUE (file_hash) and UNIQUE (file_name) on csv_files. dynamic_loader.py relied on these to settle concurrent identical uploads, but they never existed. The fallback schema gets the same constraints.
  • If a database already holds duplicates (possible because of the old race), the upgrade stops and lists them rather than deleting anything. The API then won't start until they're cleaned up. This is deliberate: it's loud, and no data is lost.
  • Dynamic upload losing a race: now returns duplicate_file. Before, it surfaced as a generic DB error from the concurrent CREATE TABLE.
  • T&E load losing a race: now returns a structured error and rolls back its rows. With the new constraint it would otherwise have been an uncaught 500.
  • UNIQUE (file_name) matches today's semantics. ART decision 2 (versioned datasets) revisits it in M5.

Evidence

Local PostgreSQL 16, replicating the CI integration job step for step:

Check Result
scripts/test_report.py --strict 247 collected, 247 passed, 0 failed, 0 skipped (239 before + 8 new)
evals/runner.py --tiers p,i,s,e 26/26
pytest tests/test_migrations.py 8 passed; with the fix removed: 7 failed, 1 passed
scripts/lint.sh (now also covering api/db.py, the migrations and the new test) flake8 clean, bandit clean

The lost-race tests are deterministic, not timing-based. A competing transaction holds an uncommitted registry row, the loader blocks on the UNIQUE index, then the competitor commits.

Deployment note: after this merges, the next API start creates csv_uploads.alembic_version and adds the two constraints. On your local te_mgmt_dev, if startup fails with "Cannot add UNIQUE", the error lists the duplicate rows to remove.


Generated by Claude Code

#84: add alembic.ini and alembic/ with a baseline (0001) matching the
fallback bootstrap schema, using IF NOT EXISTS so existing databases
adopt it. bootstrap() now finds alembic.ini and runs upgrade head on
start. The version table lives in the uploads schema.

#81: migration 0002 adds UNIQUE (file_hash) and UNIQUE (file_name) to
csv_files, the constraints dynamic_loader's race handling relied on but
which never existed. If duplicates already exist, the upgrade stops and
lists them instead of deleting anything. The fallback schema gains the
same constraints.

Loaders: a dynamic upload that loses the race now returns duplicate_file
(previously a generic DB error), and a T&E load that loses it returns a
structured error and rolls back its rows instead of raising a 500.

Tests (tests/test_migrations.py, integration): upgrade/downgrade on a
scratch schema, adoption of a fallback-created schema, refusal on
existing duplicates, DB-level uniqueness, and deterministic lost-race
tests for both loaders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YPoXzBSj89ZHgw7nfTmbB
@amar-python amar-python added this to the ART extension milestone Oct 26, 2026 — with Claude
@amar-python amar-python added bug Something isn't working art ART framework extension work labels Oct 26, 2026 — with Claude
@amar-python
amar-python merged commit da18cfd into main Oct 26, 2026
3 checks passed
amar-python pushed a commit that referenced this pull request Oct 26, 2026
Brings in #87, #90 and #86. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YPoXzBSj89ZHgw7nfTmbB
@amar-python
amar-python deleted the fix/issue-81-84-alembic-registry-unique branch October 27, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

art ART framework extension work bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

F4: no Alembic migrations; bootstrap always uses the fallback schema

2 participants