PP-12: staging and resume, crash-consistent publication, OME-Zarr inventory fingerprint (#86, #90)

AGENTS.md

@@ -95,7 +95,9 @@ src/morphobricks_preprocessing/
   axes.py           AxisMapping (role per dimension), prefill, check_mapping, review_axes (one mapping
                     applied to every file, missing axes as singletons only when confirmed)
   axis_review.py    review_axes and check_mapping (moved from axes.py in PP-4)
-  fingerprint.py    Fingerprint (size, mtime) and RowKey: when a file's rows are reused (D5);
+  inventory.py      source_signature/source_fingerprint: a file's size and mtime, an OME-Zarr
+                    folder's total size, newest mtime and inventory digest (issue #86; PP-12)
+  fingerprint.py    Fingerprint (size, mtime, inventory) and RowKey: when a file's rows are reused (D5);
                     confirm_axes/unchanged_confirmation (a confirmation of the files as they are)
   recorded_sizes.py RecordedSize: pixel sizes by role with unit and source (manifest v2):
                     header, stated or pixels (no value)
@@ -110,6 +112,7 @@ src/morphobricks_preprocessing/
   unrecorded.py     unrecorded_files: images and masks the run neither recorded nor excluded, by
                     preprocessing's discovery and pairing; RunInputs.open raises
                     UnrecordedFilesError for them (PP-8b, contract §9.1)
+  run_inputs_text.py the texts RunInputs refuses a run with (moved from run_inputs.py in PP-12)
   mask_record.py    the manifest's masks entry with fingerprints and roles (manifest v2)
   axis_check.py     the axis check (contract §2): header_axis_issue, axis_disagreements,
                     likely_axis_cause, check_header_axes, ConfirmedAxes (a confirmation covers
@@ -133,7 +136,11 @@ src/morphobricks_preprocessing/
                     describe_masks, recorded_masks; PP-10d), result.py (RunResult),
                     entries.py (manifest entries, new/changed/unchanged by RowKey),
                     reuse.py (which earlier rows are reused), rows.py (a file's rows and their
-                    pixel checks), publish.py (staged outputs, published together)
+                    pixel checks), publish.py (write_report), generation.py (the outputs
+                    as one generation: .publish, the manifest's replacement as the commit,
+                    finish_publication, published_path/read_stable for readers; PP-12),
+                    staging.py (each file's rows staged in .staging), resume.py (resume or
+                    read each file between fingerprints, source_changed_during_run)
   report.py         plain-language summaries shared by CLI and notebook, group statistics tables
   mask_text.py      the masks' texts (mask_summary, mask_report_summary) and the notebook's
                     tables of the mask step (mask_type_rows, mask_relationship_rows,
@@ -165,8 +172,8 @@ src/morphobricks_preprocessing/
                     under the confirmed roles, blocks written in place (BigTIFF when large)
   conversion_record.py the copy tree's .morphobricks-conversion.json (version 2: fingerprints,
                     copy SHA-256, roles, sizes, channel names, bits), review_conversion (the
-                    run's conversion_source_changed and copy_changed_after_conversion
-                    warnings), changed_copies
+                    run's conversion_source_changed warning and copy_changed_after_conversion
+                    error, PP-12), changed_copies
   content_hash.py   content_sha256: a copy's SHA-256 (file, or OME-Zarr folder without labels)
   conversion_check.py conversion_problem: RunInputs.open's check of a copy tree (record SHA-256,
                     version, each recorded copy's SHA-256, read once at open)
@@ -427,17 +434,23 @@ functions the tests use.
     stays 1); flags are JSON booleans only and the versions are strings (`"1"`, `"4"`);
   - channel names and significant bits (OME only, information only);
   - the CSV's SHA-256 and package versions (including `morphobricks-normalization`).
-- **A file is unchanged** when its size, mtime, labels and axis layout match and the table
+- **A file is unchanged** when its size, mtime (an OME-Zarr image: also its inventory digest,
+  `inventory_sha256`, issue #86), labels and axis layout match and the table
   still holds all its rows. Unchanged rows are reused; new and changed files are read;
   deleted, renamed or excluded files lose their rows.
 - **Full rebuild** happens in `mode="full"` (PP-7; `force_rebuild` was removed), when there
   is no readable manifest, when
   the dtype, bin count, histogram settings (`RunSettings.histogram`) or axis mapping
   changed, or when the CSV's hash no longer matches.
   Changing only the memory limit or the exclusions recomputes nothing.
-- **Writes:** the CSV is written first, then the manifest, each via a temporary file and
-  replace. Nothing is written if any error is found. An interrupted run is caught next
-  time by the hash check.
+- **Writes (PP-12, contract §9.4):** each image's rows are staged in `.staging` as soon
+  as it is read (an interrupted run resumes from the staged rows whose key still matches;
+  anything else is discarded and reported). The outputs are one generation, written into
+  `.publish` and committed by the manifest's `os.replace` (the one atomic step), then moved
+  into place; the next holder of the lock finishes or discards an interrupted publication.
+  Every image is fingerprinted when its header is read, before and after its pixels are
+  read, and before the commit (`source_changed_during_run`, issue #90). Nothing is
+  published if any error is found.
 - **Exclusions:** the user may exclude images explicitly (`exclude=[...]`, e.g. damaged
   files). Excluded images are not read, are reported (`excluded_images` warning), are
   recorded in the manifest, and lose their rows.

docs/development.md

@@ -87,11 +87,13 @@ is pinned to the release in `RELEASE_TAG` (plan §12 question 58);
 - Floats are written at full precision (`repr`); files are UTF-8.
 - Outputs are written to a temporary file and replace the previous version only after a
   complete successful write.
-- Experiment runs stage both the CSV and manifest (and any OME-Zarr copies) before
-  publishing either, back up earlier outputs, and restore their previous state if
-  publication raises. Preserve
-  backups if recovery fails. The CSV hash check still detects forced termination between
-  replacements; run only one writer per output folder.
+- Experiment runs publish their outputs as one generation (contract §9.4, PP-12): the
+  whole generation is written into `.publish`, then the manifest's `os.replace` commits
+  it, then the other files are moved in. Whatever takes the output folder's lock first
+  finishes or discards an interrupted publication (`run.generation.finish_publication`);
+  readers go through the manifest (`published_path`, `read_stable`). Each image's rows are
+  staged in `.staging` as soon as it is read, so an interrupted run resumes. Run only one
+  writer per output folder (the lock).
 
 ## Tests
 

docs/plan.md

@@ -568,6 +568,40 @@ API, CLI). The notebook and CLI only call the API.
   `invalid_physical_calibration`, a stated one is refused, and an infinite size is never
   the same as another; `check` and `design` exit with status 1 whenever they print an
   error.
+- **PP-12 (staging and resume)** *(implemented, PR #132; design v0.4 §1, §8; issues #86,
+  #90; question 69; contract §9.4)*:
+  - Move only, first: `source_fingerprint` and `LABELS_GROUP` from `manifest` to
+    `inventory` (io; `manifest` re-exports `source_fingerprint`).
+  - #86: an OME-Zarr image's fingerprint adds the SHA-256 of its inventory (each file's
+    relative path, size and `mtime_ns`, sorted by path; still metadata only):
+    `Fingerprint.inventory`, `FileEntry.inventory_sha256` (OME-Zarr only), kept for label
+    images and copies too; `check_sources` compares it; a record without it counts as
+    changed. A folder inside the image that cannot be listed raises instead of being
+    skipped.
+  - Staging (`run.staging`) and resume (`run.resume`): each image's rows staged in
+    `.staging` as soon as it is read; the next run reuses them only when the staging key
+    (row key, pixel type, schema version, the versions of preprocessing and normalization)
+    and the fingerprint still match, and reports `resumed` and `staged_discarded` (reason
+    per file); a stopped run keeps `.staging`, a published run removes it; `full` reuses
+    none.
+  - Crash-consistent publication (`run.generation`, owner decision of 2 November 2026,
+    question 69): the generation in `.publish` with a publication record, committed by
+    the manifest's `os.replace`, then moved in; the lock finishes or discards an
+    interrupted publication; `RunInputs` reads by SHA-256 (`published_path`) and again if
+    the manifest changed (`read_stable`). The backup-and-restore protocol is gone.
+  - #90: headers bound to fingerprints in the run (`bind_headers=True`), the entry records
+    that fingerprint, the image is fingerprinted after its pixels are read (before its rows
+    are staged) and every image before the commit: `source_changed_during_run` (error).
+  - Owner decision: `copy_changed_after_conversion` is an error that stops the run before
+    it writes (§17).
+  - Not in this PR: a v0.4.0rc3 run on OME-Zarr images must be run again (its entries
+    have no inventory: recomputed, and `RunInputs` refuses them).
+  - **Follow-ups (review of PR #132):** (a) masks are fingerprinted when the manifest is
+    written, as before, not bound to the fingerprint taken with their headers during the
+    run (owner and coordinator: a follow-up, not a blocker); (b) a run killed while an
+    atomic write's temporary file is open leaves that hidden `.<name>.<random>.tmp` file in
+    the output folder (on main too, for the provisional report); the lock could remove
+    them.
 - **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release
   candidates, in the release PR before the tag is pushed:
   - set `version` in `pyproject.toml`;
@@ -1681,3 +1715,33 @@ Raised during review; to be decided before the milestones that need them:
     contradicted roles. Images without confirmed roles that the header names are compared
     by their header, as before. The notebook's axis card still shows only the axis review's
     own issues; the corrected warnings appear in the run's result. Contract §11.*
+69. How does a run resume after an interruption, and how is its publication made
+    crash-consistent (PP-12, design v0.4 §1)? *Decided (owner, 2 November 2026; contract
+    §9.4):* (1) **Resume:** each image's rows are staged as soon as it is read; staged rows
+    are reused only when their row key, the pixel type, the schema version and the
+    versions of the two packages that compute rows are unchanged and the image still has
+    its fingerprint; anything else is discarded and reported, never reused. The staging
+    folder is kept until a run publishes (design v0.4 §1 says it is removed "at the start
+    of the next run"; keeping it until publication means a second interruption loses
+    nothing; accepted by the owner and coordinator on 2 November 2026 on the condition,
+    checked in review, that every run re-validates every staged row at its start). (2)
+    **Publication:** the owner asked that any reader see the old or the new generation,
+    never a mix, after a kill between any two steps. Chosen: manifest-last commit, the
+    simplest design that keeps the documented file names as plain files on every system:
+    the whole generation in `.publish`, a publication record, then one `os.replace` of
+    `run_manifest.json` (the commit), then the moves; the next holder of the lock finishes
+    or discards it; readers through the manifest read each file by its SHA-256 (from
+    `.publish` while unfinished); a plain reader of `plane_histograms.csv` always reads one
+    whole table (the old one until moved). Rejected: a generation folder with an atomic
+    symbolic link, which would make every reader consistent but needs symbolic links
+    (Windows without Developer Mode, Google Drive in Colab and copied or zipped output
+    folders do not keep them); generation-named files with the manifest as pointer, which
+    changes the documented `plane_histograms.csv` path or doubles the disk space. A plain
+    reader that opens several files without the manifest can pair two generations during
+    or after an interrupted move; it is told to read through `RunInputs`. (3) **#90:**
+    headers bound to fingerprints, each image fingerprinted after its pixels are read and
+    every image before the commit; a change is `source_changed_during_run` and nothing is
+    published. (4) **#86:** the OME-Zarr inventory digest (each file's path, size and
+    time), metadata only as design v0.4 §1 decides; content hashing stays the owner's
+    call. (5) **`copy_changed_after_conversion`** is an error that stops the run before it
+    writes (owner, 2 November 2026): the bricks would refuse the run anyway.

docs/user_guide.md

@@ -768,8 +768,10 @@ record, **.morphobricks-conversion.json** in the folder of copies, lists every i
 its original's and its copy's size and modification time, the copy's SHA-256 (a checksum of
 its content), the roles, the pixel sizes, the channel names and the significant bits; the
 run records the record's SHA-256 and warns when an original changes after the conversion
-(`conversion_source_changed`) or a copy is changed (`copy_changed_after_conversion`, also a
-change that keeps the copy's size and time). Checking the copies reads each of them whole,
+(`conversion_source_changed`). A copy that changed (also a change that keeps the copy's size
+and time), went missing or was added since the conversion stops the run before anything but
+the report is written (`copy_changed_after_conversion`, an error): later bricks would refuse
+such a run, so convert the originals again into a new folder and run on that one. Checking the copies reads each of them whole,
 so `check` and `run` on a folder of copies, and opening their run, take about one read of
 the copies longer. An original that changes while `convert` reads its metadata is refused
 (`source_changed_during_check`); convert again once nothing writes into the experiment.
@@ -870,11 +872,28 @@ The default output folder is next to the experiment folder:
 When masks are given, `run_manifest.json` also lists every image's masks (`masks`), with
 each mask's axes and size (for example `TYX` and 50 x 512 x 512 for a mask per time point).
 
-The table and manifest are both prepared before the earlier files are replaced. If a
-replacement fails, the earlier files are restored. This needs temporary disk space for
-the new outputs and copies of the earlier pair. If restoration also fails (for example,
-another program locks an output file), the error names a backup folder: close programs
-using the output files and restore those backups before running again.
+**Interrupted runs resume.** As soon as an image is read, its rows are kept in the
+hidden folder `.staging` in the output folder. If the run is interrupted (a crash, a
+stopped kernel, a lost Colab session, a computer switched off), run again with the same
+settings: the images already read are not read again, as long as they are unchanged (same
+size, time and, for OME-Zarr, the same files) and the settings that change rows are the
+same. The report lists them as `resumed`, and lists any rows it did not reuse with the
+reason (`staged_discarded`). A `full` run reuses none. After a killed run, remove its lock
+first (`--remove-stale-lock`). Once a run has written its results, `.staging` is deleted.
+
+**Results are switched in one step.** The new table, design, manifest and report are first
+written into the hidden folder `.publish`; replacing `run_manifest.json` then makes them
+the current results in one step, and the other files are moved into place. A run stopped
+before that step leaves the earlier results exactly as they were; one stopped after it is
+finished by the next run (or `check`, `masks`, `design`, `calibrate`). Later bricks always
+read one whole set of results, old or new. If a file cannot be moved into place (for
+example because another program has the table open on Windows), the run says so: close
+that program and run again. This needs temporary disk space for the new outputs.
+
+**Images that change during the run.** If an image changes or disappears while the run
+checks or reads it (an acquisition or a copy still writing into the experiment), the run
+stops with `source_changed_during_run` and publishes nothing. Wait until nothing writes to
+the experiment, then run again: the images already read resume.
 
 Every output file gets the permissions a file you save yourself would get (your umask;
 usually readable by everyone, writable by you), so colleagues can open the results in a
@@ -971,7 +990,8 @@ continues.
 | `not_enough_space` | Error | `convert`: the copies would not fit on the drive. Free space or choose another drive. |
 | `conversion_record_unreadable` | Error | The conversion record of a folder of named-axis copies (**.morphobricks-conversion.json**) cannot be read, or is of version 1 (made by v0.4.0rc3 or earlier, without the copies' SHA-256), so `run` stops and `convert` refuses the folder. Convert again into a new folder. |
 | `conversion_source_changed` | Warning | An original changed or went missing after its named-axis copy was made; the copies hold the images as converted. If the originals were corrected, convert again into a new folder. |
-| `copy_changed_after_conversion` | Warning | An image of the folder of named-axis copies changed (its size, time or SHA-256), went missing or was added after the conversion, so the conversion record does not describe it. The run goes on, but the bricks (`RunInputs`) refuse a run on such copies: convert the originals again into a new folder. |
+| `copy_changed_after_conversion` | Error | An image of the folder of named-axis copies changed (its size, time or SHA-256), went missing or was added after the conversion, so the conversion record does not describe it, and the bricks (`RunInputs`) would refuse a run on it. The run stops before it writes anything but the report: convert the originals again into a new folder, and run preprocessing on that one (same output folder). |
+| `source_changed_during_run` | Error | An image changed or went missing while the run was checking or reading it, so its rows would not describe it. Nothing is published. Wait until nothing writes to the experiment (an acquisition or a copy still running), then run again: the images already read resume from their staged rows. |
 | `design_unreadable` | Error | The `experiment_design.csv` an earlier run wrote cannot be read. Correct it, or delete it to confirm the design again. |
 | `no_usable_planes` | Error | No plane has finite pixel values. Check the images. |
 | `tiff_outside_leaf` | Error | Images sit in a folder that also holds subfolders, so they have no condition or repeat of their own. A folder holds either images or subfolders, never both. Move the images into a folder of their own, or out of the experiment folder. |

pyproject.toml

@@ -179,10 +179,12 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.image_type" = "pure"
 "morphobricks_preprocessing.intensity" = "io"
 "morphobricks_preprocessing.manifest" = "io"
+"morphobricks_preprocessing.inventory" = "io"
 "morphobricks_preprocessing.fingerprint" = "io"
 "morphobricks_preprocessing.narrow_ranges" = "pure"
 "morphobricks_preprocessing.recorded_sizes" = "pure"
 "morphobricks_preprocessing.run_inputs" = "io"
+"morphobricks_preprocessing.run_inputs_text" = "pure"
 "morphobricks_preprocessing.unrecorded" = "io"
 "morphobricks_preprocessing.testing" = "orchestration"
 "morphobricks_preprocessing.mask_scan" = "io"
@@ -223,6 +225,9 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.run.steps" = "orchestration"
 "morphobricks_preprocessing.run.preflight" = "io"
 "morphobricks_preprocessing.run.publish" = "io"
+"morphobricks_preprocessing.run.generation" = "io"
+"morphobricks_preprocessing.run.staging" = "io"
+"morphobricks_preprocessing.run.resume" = "io"
 "morphobricks_preprocessing.run.result" = "pure"
 "morphobricks_preprocessing.run.reuse" = "io"
 "morphobricks_preprocessing.run.rows" = "io"