Skip to content

Records append is not rollback-safe; read_records accepts any JSON object #206

Closed

Description

@guijacquemet

Problem

  1. run/description_file.py _append_records appends straight to the live morphobricks.records.jsonl (O_APPEND, then write, flush, fsync). If any of these raises after bytes were written, publish_description restores only morphobricks.toml. The records file then keeps a torn line, or a full record naming a SHA-256 of a description that was rolled back. The existing failure test (tests/test_description_file.py, test_a_failed_append_puts_the_previous_file_back) replaces _append_records with a function that raises before any byte is written, so it never saw this.
  2. read_records accepts any JSON object, for example {}. Spec docs/experiment-v1.md section 4.6 (Spec main 964f097) fixes three record types (reconfirmation, image_count_changed, removed), each with a fixed key set, and says a line of one type carries no key of another.

Reproduced on main (1c54f4e)

  • A binary write that fails after half of the bytes, or after all of them, or a failing fsync, leaves the records file changed while the TOML is put back.
  • read_records returns ({},) for a file holding {}.

Fix

  • Build the new records file (old bytes plus new lines) in full in a synced temporary file and replace the live file with one atomic replace; on any failure restore both the description and the records bytes.
  • read_records refuses a line that is not one of the three types with exactly that type's keys.
  • Define the reader rule for a record whose SHA does not match the published file: the hashes are history, never compared with the file now published (later publications change the file without a record).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions