Turn content hashing of the sources on by default

CHANGELOG.md

@@ -8,6 +8,16 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **Content hashing of the sources is on by default (#159; owner decision of 7 November 2026).**
+  A run records each image's content SHA-256 and reuses a row only for the same content, so an
+  image edited in place with its size and modification time kept is read again (manifest
+  version 3). Opt out with `RunMode(verify_content=False)`, `--no-verify-content` or by
+  unticking the notebook's box; the manifest then records `settings.verify_content: false`
+  (version 2 and 3 manifests now always write the key; one written before has none and reads
+  as not verified). An output folder keeps the choice of its earlier run, so an existing one
+  is not recomputed. Cost on 60 OME-TIFF images of 25 MB (1.5 GB), cold cache: a fresh run
+  15.9 s without and 19.8 s with hashing (+25 %); a rerun that reuses every row 5.4 s without
+  and 8.3 s with (+3 s). The standalone `masks` command is unchanged.
 - **Experiment-description publication (Stage 1, PR 5).** `description_file.read_description(path)`
   reads and validates a `morphobricks.toml` (a symbolic link is `R_FILE_SYMLINK`, never followed);
   `description_transition` compares a candidate with the published file and reports every `R_*`

docs/contract.md

@@ -199,11 +199,11 @@ interpretation.
 
 | Key | Meaning |
 |---|---|
-| `manifest_version`, `schema_version` | Manifest format and CSV format (`"4"`), written as JSON **strings**, not numbers. The manifest format is `"2"` (since v0.4.0) for a default run and `"3"` (v0.4.0rc5, release review) for a content-verification run (`verify_content`, below): version 3 is version 2 with `settings.verify_content` `true` and a `content_sha256` in every file entry (`manifest_versions`). The version, not only those keys, marks a strict run, because readers of v0.4.0rc4 and earlier ignore keys they do not know but refuse a version they do not know (`ManifestFormatError: Manifest version '3' is not supported`, tested by running v0.4.0rc4's code on a strict manifest): they never open a strict run as a fingerprint-only one. The reader and the writer refuse a version 3 manifest without `verify_content` or without a file's `content_sha256`, and a version 1 or 2 manifest that holds either (a strict manifest written as version 2, only by unreleased rc5 builds before this rule). A default run's manifest is version 2, byte for byte as before (tested against the code of the commit before the change). This version reads `"1"`, `"2"` and `"3"`; `RunInputs`, `restart` reuse and the recorded inputs treat 2 and 3 alike, and switching between a strict and a default run recomputes every row as before. A manifest whose `schema_version` is not the current one makes the next run recompute every row. |
+| `manifest_version`, `schema_version` | Manifest format and CSV format (`"4"`), written as JSON **strings**, not numbers. The manifest format is `"3"` (v0.4.0rc5, release review) for a content-verification run (`verify_content`, below), which every run is unless it opts out (#159), and `"2"` (since v0.4.0) for a run that opted out: version 3 is version 2 with `settings.verify_content` `true` and a `content_sha256` in every file entry (`manifest_versions`). The version, not only those keys, marks a strict run, because readers of v0.4.0rc4 and earlier ignore keys they do not know but refuse a version they do not know (`ManifestFormatError: Manifest version '3' is not supported`, tested by running v0.4.0rc4's code on a strict manifest): they never open a strict run as a fingerprint-only one. The reader and the writer refuse a version 3 manifest without `verify_content` or without a file's `content_sha256`, and a version 1 or 2 manifest that holds either (a strict manifest written as version 2, only by unreleased rc5 builds before this rule). A run that opted out writes version 2, as before except for `settings.verify_content: false` (tested against the code of the commit before version 3). This version reads `"1"`, `"2"` and `"3"`; `RunInputs`, `restart` reuse and the recorded inputs treat 2 and 3 alike, and switching between a strict and a default run recomputes every row as before. A manifest whose `schema_version` is not the current one makes the next run recompute every row. |
 | `written_at` | UTC time of writing, ISO 8601. A rerun whose manifest would differ from the earlier one only in `written_at` keeps the earlier time, so the manifest's bytes and SHA-256, which later bricks pin, stay the same (cross-brick test F10, 3 November 2026); the report still records the rerun. |
 | `experiment_root` | Absolute path of the experiment folder, for information. |
 | `dtype` | Pixel type of the experiment. |
-| `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `histogram` (`HistogramSettings.to_dict()` of the normalization brick: `library_version`, `histogram_range`, `robust_quantiles`, `min_central_bins`, §4; the default `minmax` range when absent, in manifests written before follow-up P6), `verify_content` (JSON boolean `true` when the run verified content, below; written only then, in a version 3 manifest, so a default run's `settings` keep the v0.4.0rc4 keys; false when absent). |
+| `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `histogram` (`HistogramSettings.to_dict()` of the normalization brick: `library_version`, `histogram_range`, `robust_quantiles`, `min_central_bins`, §4; the default `minmax` range when absent, in manifests written before follow-up P6), `verify_content` (JSON boolean, below: `true` when the run verified content, `false` when it opted out, written by every version 2 and 3 manifest so that an opt-out is on record (#159); a manifest written before that has no key and reads as false, not verified). |
 | `axis_mapping` | `axes` (file axis labels) and `roles` (`X`, `Y`, `Z`, `C`, `T` or null per axis). |
 | `labels` | One entry per leaf folder: `folder` (relative path, `.` for the root) and its three labels. |
 | `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1: its name without the extension, never edited; required in version 2, where an entry without it, with null, or with another value than `position_id(relative_path)` is refused naming the entry, by the reader and the writer, round-2 audit A-m5; null when absent in a version 1 manifest), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files, `mtime_ns` the time of its newest file, and `inventory_sha256` (v0.4.0, PP-12, issue #86; only for OME-Zarr images) the SHA-256 of its inventory: one line per file (its path relative to the image with `/`, a NUL, its size, a NUL, its `mtime_ns`, a newline), sorted by path, its root `labels` group left out; links to folders inside the image are followed, as the reader follows them, and links that reach one folder twice are refused. Adding, removing, renaming, resizing or touching any chunk or metadata file changes it; an OME-Zarr image recorded without it (v0.4.0rc3 and earlier) counts as changed (recomputed; `RunInputs` refuses it). Like a TIFF file's size and time it is metadata only (design v0.4 §1): a file rewritten with the same size and its time set back is not seen, only hashing the content would. **Version 2 (v0.4.0):** `roles` is required, with `axes_from` (required): `header` when the metadata names every axis and the roles agree, `confirmed` for a plain stack whose roles the user confirmed (§2); and `pixel_sizes`, one entry per role `X`, `Y`, `Z`, `T` with `value`, `unit` (OME-NGFF name) and `source`: `header` (the image's metadata) or `stated` (`calibration.csv`, §9.3) with a value, or `pixels` (stated as deliberately uncalibrated) with `value` and `unit` null; `value` and `source` null for an axis nothing gives a size (never guessed). A `header` or `stated` size without a value, or a value with no source or with `pixels`, is refused. The reader refuses a version 2 entry without `roles` or `axes_from`, and the writer refuses to write one. `content_sha256` (only in a `verify_content` run): the SHA-256 of the image's bytes, or for an OME-Zarr image of one line per file (its path relative to the image with `/`, a NUL, the file's SHA-256, a newline) over the files its inventory covers, in path order (`source_content.source_content_sha256`). |
@@ -218,7 +218,7 @@ interpretation.
 | `calibration_sha256` | SHA-256 of the output folder's `calibration.csv` as the run read it (§9.3); null when there was none (or absent in manifests written before PP-8). `RunInputs` checks it. |
 | Version 1 | A manifest written before v0.4.0 (`"1"`) is still read, for reporting, but no row of it is reused: its axes were never checked. The next run recomputes every row ("the earlier run was written before v0.4.0, without the axis check"). Bricks refuse it. |
 | Row reuse | A file's rows are reused only when its row key is the earlier run's (`fingerprint.RowKey`, one rule for "the same file, read the same way", audit D5): its fingerprint (`fingerprint.Fingerprint`: size and modification time, and for an OME-Zarr image the inventory digest), labels, axes, roles, `singleton_roles`, `axes_from` and the path it is read from (so moving or converting the experiment recomputes it), and the settings that change rows (bins, histogram range, robust quantiles, range-check limit). **The identity rule is the fingerprint, which is metadata** (design v0.4 §1): its known blind spot is a file rewritten with the same size whose modification time (for an OME-Zarr image, every changed file's) is set back to its earlier value; its earlier rows are then reused, its staged rows resumed, and `RunInputs` accepts it. |
-| Content verification (opt-in, owner 3 November 2026) | `RunMode(name, verify_content=True)` (`run_experiment(mode=...)`, CLI `--verify-content`, the notebook's "Verify each image's content" box) closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (false without one); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the default run is unchanged. A strict run writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). |
+| Content verification (on by default, owner 7 November 2026, #159; opt-in before) | Every run does it unless it opts out with `RunMode(name, verify_content=False)` (`run_experiment(mode=...)`, CLI `--no-verify-content`, the notebook's "Verify each image's content" box unticked); the opt-out is recorded as `settings.verify_content: false`. It closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (true without one: a first run, or a version 1 manifest); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the cost is in the pull request that made it the default (about 3 seconds per 1.5 GB read, measured at 1 GB/s of SHA-256). A run that verifies writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). |
 | Removed keys | `settings.convert_tiffs`, `settings.conversion` and `files[].ome_zarr_path` were written by v0.3 runs that converted TIFF images to OME-Zarr in the output folder (removed in v0.4.0, §10); a manifest that holds them is still read, and they are ignored. The copies such a run left in `<output>/ome_zarr` are not touched. |
 | Flags | `missing_as_singleton`, `verify_content`, `read_in_blocks`, `channel_names_trimmed` and `channels_from_rgb_samples` are JSON booleans (`true`/`false`); any other value (`"false"`, `0`) makes the manifest malformed, never read as true or false, and `write_manifest` refuses to write one (`ValueError`), so a written manifest always reads back. |
 | `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. |