PP-11c: conversion integrity (copy SHA-256, metadata epochs, exact CLI sizes)

AGENTS.md

@@ -155,13 +155,21 @@ src/morphobricks_preprocessing/
                     swap, question 59) and OME metadata of a named-axis copy
   ome_tiff_copy.py  write_named_copy: one image as uncompressed OME-TIFF, planes from iter_planes
                     under the confirmed roles, blocks written in place (BigTIFF when large)
-  conversion_record.py the copy tree's .morphobricks-conversion.json (fingerprints, roles, sizes),
-                    review_conversion (the run's conversion_source_changed and
-                    copy_changed_after_conversion warnings)
-  conversion.py     convert_experiment/plan_conversion: the copy tree under its lock, .partial
-                    files moved into place together (any failure removes what it wrote),
-                    planes checked against the memory limit first, unchanged copies reused,
-                    record last; conversion_issues.py its messages (pure)
+  conversion_record.py the copy tree's .morphobricks-conversion.json (version 2: fingerprints,
+                    copy SHA-256, roles, sizes, channel names, bits), review_conversion (the
+                    run's conversion_source_changed and copy_changed_after_conversion
+                    warnings), changed_copies
+  content_hash.py   content_sha256: a copy's SHA-256 (file, or OME-Zarr folder without labels)
+  conversion_check.py conversion_problem: RunInputs.open's check of a copy tree (record SHA-256,
+                    version, each recorded copy's SHA-256, read once at open)
+  conversion.py     convert_experiment: the copy tree under its lock, record last;
+                    conversion_issues.py its messages (pure)
+  conversion_plan.py plan_conversion: headers checked with the confirmed roles, each bound to
+                    its image's fingerprint (source_changed_during_check), planes checked
+                    against the memory limit first, PlannedCopy per image
+  conversion_reuse.py compare: unchanged copies reused, anything else there refused
+  conversion_write.py write_all: .partial files moved into place together (any failure
+                    removes what it wrote)
   zarr_input.py     OME-Zarr input: open_ngff (OME-NGFF 0.4/0.5 metadata -> level-0 array, axis
                     letters and physical sizes from the scale and axis units),
                     ngff_label_names/open_ngff_labels (label images, for later bricks),

docs/contract.md

@@ -195,10 +195,16 @@ masks_root=None)` (design v0.4 §6). It refuses (`RunInputsError`) a version 1 m
 `plane_histograms.csv`, `experiment_design.csv` or `calibration.csv` whose SHA-256 is not the
 manifest's (also a `calibration.csv` added since the run), the conversion record of a run
 on named-axis copies whose SHA-256 is not the manifest's `conversion_record_sha256` (changed
-or gone since the run; §17), an image or mask changed or gone since the run, and a mask recorded without axes (a v0.4.0rc1 run wrote `axes: ""` for a
+or gone since the run; §17), a conversion record of version 1 (no copy SHA-256: convert
+again), a copy of a recorded image whose fingerprint or SHA-256 is not the record's (or that
+the record does not list; PP-11c, issue #87), an image or mask changed or gone since the run, and a mask recorded without axes (a v0.4.0rc1 run wrote `axes: ""` for a
 mask whose header puts X before Y; the message says to save the mask again with
 `numpy.swapaxes` and run again, never "changed or missing"); every read checks the file's
-fingerprint again.
+fingerprint again. The copies' SHA-256 is checked once, when the run is opened, for every
+image the run recorded (`conversion_check`): each copy is read whole then, so opening a
+run on named-axis copies costs one read of those copies (the excluded images' copies are
+not read). Checking on every read instead would read a copy whole for each of its planes.
+Reads after `open` check the fingerprint, as for any image.
 
 It also refuses an experiment holding files the run neither recorded nor excluded
 (`UnrecordedFilesError`, a `RunInputsError`, PP-8b): images that preprocessing's own
@@ -508,12 +514,12 @@ exclude=, missing_as_singleton=, memory_limit=, progress=)` (`conversion`), the
 | Rule | Detail |
 |---|---|
 | What is converted | Every image with an axis error (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`, §2) or unnamed axes (`axes_not_named`) under the mapping given; it is never taken from the headers. Every other image is copied byte for byte: its header names its axes as confirmed, it is an OME-Zarr image (with its label images), or it is excluded (`exclude`: never read). So the copy tree is the whole experiment. The command and the API are offered for any experiment; an experiment without these images is copied as it is. |
-| Checks first | The experiment is checked with the mapping as a run would check it (folders, headers, `review_axes`), from the headers only. The axis errors are what the conversion corrects; any other error stops it before anything is written, and so do images of a type OME-TIFF cannot hold (`cannot_convert_pixel_type`: `int64`, `uint64`; pixel values are never changed to fit) and images to convert whose planes cannot be read within the memory limit, not even in blocks (`check_planes`: `plane_over_memory_limit`, `unsupported_layout`; `planes_read_in_blocks` is a warning). `memory_limit` must be a whole number of bytes above 0 and `missing_as_singleton` True or False (`ValueError` before anything is read). The axis errors and `axes_not_named` are not reported: each converted image's record entry gives its code. |
+| Checks first | The experiment is checked with the mapping as a run would check it (folders, headers, `review_axes`), from the headers only. The axis errors are what the conversion corrects; any other error stops it before anything is written, and so do images of a type OME-TIFF cannot hold (`cannot_convert_pixel_type`: `int64`, `uint64`; pixel values are never changed to fit) and images to convert whose planes cannot be read within the memory limit, not even in blocks (`check_planes`: `plane_over_memory_limit`, `unsupported_layout`; `planes_read_in_blocks` is a warning). `memory_limit` must be a whole number of bytes above 0 and `missing_as_singleton` True or False (`ValueError` before anything is read); the CLI's `--memory-limit` (`run`, `convert`) is parsed exactly (`cli_run.parse_size`, no floating point): a size that is not a whole number of bytes above 0 (`1.5`, `0.5B`) is refused while the command line is read, before anything is read (`1.5GiB` is 1610612736 bytes; issue #80). One metadata epoch (issue #96): each image is fingerprinted before and after its header is read (`preflight(bind_headers=True)`); an image whose two fingerprints differ (or that went missing) is `source_changed_during_check` and nothing is written. The copy's axes, sizes, channel names, significant bits and pixel type come from that header (the header is not read again for them), and the fingerprint taken with it is the one recorded as the source's and checked before and after the copy is written. A plane the reader gives in another pixel type than that header's is refused (`conversion_failed`), never cast. The axis errors and `axes_not_named` are not reported: each converted image's record entry gives its code. |
 | Copy tree | A folder the user names (`--to`), outside the experiment and not holding it (`copy_folder_overlaps_experiment`; a file is `copy_folder_not_a_folder`): new or empty (else `copy_folder_not_empty`, nothing overwritten), or the copy tree of an earlier conversion of the same experiment. The originals are only read. Each image keeps its relative path and file name (an OME-TIFF may end in `.tif`), so `source_relative_path`, `plane_id` and the FOV key are the originals'. |
 | Copies | Uncompressed OME-TIFF, BigTIFF when the pixels and 1 KiB per page (for its directory) would take more than 4 GiB less 32 MiB. Axes: every axis with a role in the source's order, then Y and X (`ZCYX`); an axis without a role (length 1) or missing (`missing_as_singleton`) is not added. Each plane is the plane the run reads under the confirmed roles (`iter_planes`), bit for bit: same pixel type (in native byte order), no rescaling, NaN and infinities kept; a Y/X swap and X before Y are written rows (Y) then columns (X). Planes over the memory limit are read in blocks, as the run reads them, and each block is written to its place: no plane is held whole when the run would not hold it. The copy's metadata names every axis, so `check` and `run` read it with `axes_from: "header"`. |
 | Pixel sizes | They follow the roles; a relabelled axis gets an unknown scale, never a guessed one (`copy_axes`). A size the source header gives is written only for the role the header names it for, and only when the user confirmed that role for that same axis; otherwise it is left out and the record says why. One exception (owner decision of 2 November 2026, plan §12 question 59): a Y or X relabelled as the header's other one of Y and X (a Y/X swap, X before Y) keeps the header's size when the header gives X and Y exactly equal, same value and unit, since nothing can be exchanged; otherwise both are unknown in the copy (state them in `calibration.csv`, §9.3, if known). A Y/X swap therefore never exchanges the sizes of Y and X. A z-step is never written as a time step. A plain stack's Y and X confirmed Y before X keep the header's sizes. Units are written as OME symbols (`µm`, `s`, ...). Channel names are written when the C axis keeps its role and every channel is named; significant bits as they are. |
 | Safe | The copy tree's lock (`.morphobricks-preprocessing.lock`, §9.2's rule; `--remove-stale-lock`) is held throughout; a second conversion is `output_locked`. Each copy is written as `<name>.partial`, the source's fingerprint checked before and after; once every copy is written they are moved into place, then the record is written (atomic, with the file-mode rule of §9). `not_enough_space` is refused before writing. Any failure or interruption removes the `.partial` files of the conversion, the copies already moved into place (each where nothing was) and the folders it made, so the folder is left as it was; a read or write failure is `conversion_failed`, anything else is raised again; `.partial` files of an interrupted conversion are written again. New files get the mode a plain `open()` gives. |
-| Repeat | A copy already in the tree is reused when its source (fingerprint and axes), roles, copy axes, pixel sizes (and unknown ones) and the copy's fingerprint are as recorded; a new image is added; when every copy is reused the record is kept as it was (same SHA-256). Anything else (a source changed, other roles, a copy changed, an image no longer in the experiment, another experiment's copy tree, a file at a new image's place that the record does not list) is `copy_differs`: nothing is overwritten. A conversion record that cannot be read is `conversion_record_unreadable`. |
-| Conversion record | `.morphobricks-conversion.json` in the copy tree's root (a dot-file, so discovery skips it), record version 1: `source_root`, `mapping`, `missing_as_singleton`, `written_at`, `versions`, and per image `relative_path`, `action` (`converted` or `copied`), `reason` (the axis code, `excluded`, or null), `source` and `copy` fingerprints (`size`, `mtime_ns`), `source_axes`, `roles`, `copy_axes`, `pixel_sizes` (by role, as §9) and `unknown_sizes` (role: why). `read_conversion_record(folder)` reads it with its SHA-256. |
-| The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and when a copy changed, went missing or was added since (`copy_changed_after_conversion`); an unreadable record stops the run (`conversion_record_unreadable`). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`). |
+| Repeat | A copy already in the tree is reused when its source (fingerprint and axes), roles, copy axes, pixel sizes, unknown sizes (roles and reasons), channel names, significant bits, the copy's fingerprint and the copy's SHA-256 (read whole) are as recorded; a new image is added; when every copy is reused the record is kept as it was (same SHA-256). Anything else (a source changed, other roles, a copy changed, an image no longer in the experiment, another experiment's copy tree, a file at a new image's place that the record does not list) is `copy_differs`: nothing is overwritten. A conversion record that cannot be read, or of version 1, is `conversion_record_unreadable`. |
+| Conversion record | `.morphobricks-conversion.json` in the copy tree's root (a dot-file, so discovery skips it), record version 2: `source_root`, `mapping`, `missing_as_singleton`, `written_at`, `versions`, and per image `relative_path`, `action` (`converted` or `copied`), `reason` (the axis code, `excluded`, or null), `source` and `copy` fingerprints (`size`, `mtime_ns`), `copy_sha256` (the copy's SHA-256, computed in one pass after it is written; an OME-Zarr copy's is the SHA-256 of its files' relative paths and SHA-256s in path order, its `labels` group left out as for fingerprints: `content_hash`), `source_axes`, `roles`, `copy_axes`, `pixel_sizes` (by role, as §9), `unknown_sizes` (role: why), `channel_names` (as the copy holds them; `[]` for none) and `significant_bits` (or null). Sources keep the fingerprint rule (design v0.4 §1); the copies' SHA-256 is the owner's decision of 2 November 2026 (plan §12 question 59 (16)). Version 2 (PP-11c) added `copy_sha256`, `channel_names` and `significant_bits`; a version 1 record (v0.4.0rc3 or earlier) is refused everywhere it is read (`convert`, the run's `review_conversion`, `RunInputs`) with a message to convert the originals again into a new folder: its copies cannot be checked. `read_conversion_record(folder)` reads it with its SHA-256. |
+| The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and when a copy changed (its fingerprint or its SHA-256: each copy is read whole, so `check` and `run` on a copy tree cost one more read of the copies), went missing or was added since (`copy_changed_after_conversion`; the run goes on, and its message says that `RunInputs` will refuse the run for a changed or added copy); an unreadable record stops the run (`conversion_record_unreadable`). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`), is of version 1, or whose recorded images' copies no longer have the recorded fingerprint and SHA-256 (§9.1). |
 | Notebook | Not yet: the axis-check card names the `convert` command; the notebook card is the next PR (PP-11b, plan §10). |

docs/plan.md

@@ -524,6 +524,14 @@ API, CLI). The notebook and CLI only call the API.
   for files with errors": a card offered after the axis check when images have an axis
   error or unnamed axes, calling `convert_experiment` with the confirmed roles and a
   folder the user names, showing the result and the next run on the copies.
+- **PP-11c (conversion integrity)** *(this PR; issues #80, #87, #88, #92, #96; question 59
+  (16)-(18))*: `conversion.py` split into `conversion_plan`, `conversion_reuse` and
+  `conversion_write` (move only); the copies' SHA-256 in the conversion record (version 2,
+  `content_hash`) verified at reuse, by `review_conversion` and by `RunInputs.open`
+  (`conversion_check`); channel names, significant bits and the unknown sizes' reasons
+  compared at reuse; each header bound to its image's fingerprint
+  (`preflight(bind_headers=True)`, `source_changed_during_check`); the copy's pixel type
+  from that header; `parse_size` exact.
 - **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release
   candidates, in the release PR before the tag is pushed:
   - set `version` in `pyproject.toml`;
@@ -1465,4 +1473,26 @@ Raised during review; to be decided before the milestones that need them:
     disappeared, as for `calibration_sha256` and the design's SHA-256.
     (15) **Notebook card**: the next PR, PP-11b (§10); the axis-check card names the
     command meanwhile.
+    (16) **The copies' SHA-256** *(owner decision of 2 November 2026, after an independent
+    review of 8db1f4a; implemented in PP-11c, issue #87)*: a fingerprint (size and time)
+    does not see a change that keeps both, so the record gives each copy's SHA-256,
+    computed in one pass after it is written; reuse, the run's `review_conversion` and
+    `RunInputs.open` verify it (`RunInputs` once at open, for the copies of every image the
+    run recorded: one read of those copies, documented in contract §9.1). Sources keep the
+    fingerprint rule (design v0.4 §1). The record is version 2; a version 1 record is
+    refused with a message to convert again (its copies cannot be checked). Reuse also
+    compares the channel names and significant bits (#88) and the reasons of the unknown
+    sizes (#92).
+    (17) **One metadata epoch** *(same review; PP-11c, issue #96)*: each image is
+    fingerprinted before and after the header read the copy's layout comes from, and is
+    refused (`source_changed_during_check`) when the two differ; that fingerprint is the
+    one recorded and checked before and after the copy is written. The copy's pixel type
+    also comes from that header (the writer no longer opens the header for it), and a
+    plane of another type is refused, never cast. Still read again, by design: the planes'
+    memory check (`check_planes`, the run's own check) and the reader that reads the pixels
+    (it opens the file and parses its header); both are bounded by the same fingerprint.
+    (18) **Exact memory sizes on the command line** *(same review; PP-11c, issue #80)*:
+    `--memory-limit` is parsed without floating point and must be a whole number of bytes
+    above 0 (`1.5GiB` is accepted, `1.5` and `0.5B` are refused before anything is
+    read).