Skip to content

fix(db): make directories migrations idempotent — journal-vs-schema drift boot-crashes the hub #270

Description

@aarontrowbridge

Important

Problem — Booting a current-fork server binary against a long-lived production database can crash-loop on duplicate column name: directories (observed on the erlich hub, 2026-08-30: a rebuilt binary exited ~1s after start, 29 systemd restarts, hub down). The database has both the directories columns AND complete migration-journal records — yet the runner sometimes replays an ADD COLUMN migration anyway, and the migration SQL itself is not idempotent, so any journal-vs-schema drift (e.g. a column added by a binary lineage that didn't journal it) becomes an unrecoverable boot crash instead of a self-heal.

Approach — Make schema-adding migrations tolerate reality: guard each ADD COLUMN with a column-existence check (skip + let the journal record it — the journal then reconciles to the actual schema on next boot). Apply to the two directories migrations first (both implicated in the incident); the same guard pattern is available for the other ADD COLUMN migrations as follow-up.

Scope — in: guarded directories migrations (session + project), boot verified against a production-DB copy · out: rewriting the migration runner's concurrency model (a separate suspected race — two subsystems migrating one file — filed as follow-up investigation with strace artifacts).

Acceptance Criteria

  • Booting against a copy of a drifted production DB (column present, journal row present) succeeds; before the fix it crashes with duplicate column name.
  • Booting against a copy with the journal row removed but the column present also succeeds and re-records the row (journal self-heal).
  • A genuinely fresh database still bootstraps and records all migrations (no behavior change on the happy path).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    hitlNeeds a human decision/review

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions