Z projection 5/6: the run side (part of #176)

CHANGELOG.md

@@ -8,6 +8,22 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **The run side of the Z projection (part of #176).** A run on projected copies records a
+  `projection` summary in its manifest (method, slices requested, and for every image the
+  slices used, types, dtype rule, rounding and the source's content SHA-256, with the
+  conversion record it comes from), `RunInputs.projection` returns it (None for an unprojected
+  run, whose manifest is byte for byte as before), and the report says "max projections of
+  slices 1:4". 2-D masks next to projected images are paired as usual and the run warns
+  `masks_on_projected_images` (report and manifest); 3-D masks still stop the run
+  (`mask_planes_mismatch`). `RunInputs.open` refuses a manifest whose `projection` is not
+  what the conversion record gives, projected copies recorded without it, and a `projection`
+  without a record: nothing is read as a plain image.
+- **`projection_record_missing` (owner decision, 8 November 2026).** `check` and `run` read the
+  OME `Image/Description` header (the first page's description tag, no pixels) of every
+  analysed TIFF the conversion record does not list as `projected`. A description that begins
+  "MorphoBricks Z projection:" in a tree with no record, or whose record does not list the
+  image as projected, is an error naming the images and saying to rerun `convert --project`
+  or restore the record. Plain images are unaffected.
 - **`A_*` error classes and `RunInputs.section_digests()` (Stage 2 design PR 2, owner answer 2).**
   `ChannelNotFoundError` now has `code = "A_SCOPE_INCOMPLETE"`: one rule and one code for a channel
   no image has and for one some images lack; the message names the image and ends with the code.

docs/user_guide.md

@@ -929,7 +929,7 @@ they are. OME-Zarr images are not projected yet (`projection_zarr_not_supported`
 them). One folder holds one method and one slice range (`copy_differs` otherwise). The command prints the next commands with a
 new output folder (by default the copy folder's name with `_MorphoBricks_Preprocessing`):
 the projected copies are a different source, so the description starts again there (the
-design and the stated pixel sizes are entered again). The notebook card follows.
+design and the stated pixel sizes are entered again). A run on the copies records the projection in its manifest (`projection`: the method, the slices used, the source SHA-256 of every image) and the report says "max projections of slices 1:4". Masks next to projected images are used as they are, with the warning `masks_on_projected_images`: check that they were made on, or for, the projection. A run on projected copies reads the conversion record strictly: a record that is missing or does not match the run is refused, never read as a plain image. The notebook card follows.
 
 **In the notebook.** When step 2's axis check reports these images, a **Make named-axis
 copies (optional)** card appears under it, calling the same functions as the command:

pyproject.toml

@@ -175,9 +175,12 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.copy_axes" = "pure"
 "morphobricks_preprocessing.projection" = "pure"
 "morphobricks_preprocessing.projection_record" = "pure"
+"morphobricks_preprocessing.projection_run" = "pure"
+"morphobricks_preprocessing.run_inputs_projection" = "pure"
 "morphobricks_preprocessing.projection_issues" = "pure"
 "morphobricks_preprocessing.projection_layout" = "pure"
 "morphobricks_preprocessing.projection_masks" = "io"
+"morphobricks_preprocessing.projection_marker" = "io"
 "morphobricks_preprocessing.projection_plan" = "orchestration"
 "morphobricks_preprocessing.planned_copy" = "pure"
 "morphobricks_preprocessing.ome_tiff_copy" = "io"

src/morphobricks_preprocessing/conversion_check.py

@@ -10,20 +10,28 @@
 
 from __future__ import annotations
 
-from collections.abc import Iterable
+from collections.abc import Iterable, Mapping
 from pathlib import Path
 
 from .conversion_record import (
     RECORD_FILENAME,
+    ConversionRecord,
     ConversionRecordError,
     changed_copies,
     read_record,
 )
 from .manifest import file_sha256
+from .projection_run import run_projection
 from .validation import agree, plural
 
 
-def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | None:
+def conversion_problem(
+    root: Path,
+    expected: str | None,
+    paths: Iterable[str],
+    projection: Mapping | None = None,
+    masks: Iterable[tuple[str, str]] = (),
+) -> str | None:
     """Return why a run on named-axis copies cannot be used, or None when it can.
 
     Parameters
@@ -34,13 +42,20 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str |
         The record's SHA-256 the run recorded.
     paths
         The images the run recorded, relative to ``root``.
+    projection
+        The manifest's ``projection`` entry, or None. It must be exactly what the record
+        gives for ``paths`` and ``masks``: projected copies are never read as plain images.
+    masks
+        ``(image, mask path)`` of the masks the run recorded.
 
     Returns
     -------
     str or None
         The message for the user, or None.
     """
     path = root / RECORD_FILENAME
+    if expected is None:
+        return None if projection is None else _no_record(path, root)
     if not path.exists() or file_sha256(path) != expected:
         return (
             f"The conversion record {path.name} in {root} is not the one the run read "
@@ -57,11 +72,43 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str |
         )
     changed = changed_copies(root, found[0], paths) if found is not None else []
     if not changed:
-        return None
+        return _projection_problem(found[0], expected, list(paths), projection, masks)
     them = agree(changed, "it", "them")
     return (
         f"{plural(len(changed), 'named-axis copy', 'named-axis copies')} changed since the "
         f"conversion wrote {them} (or the conversion record does not list {them}): "
         f"{', '.join(changed[:5])}{' and more' if len(changed) > 5 else ''}. Convert the "
         "originals again into a new folder and run preprocessing on that one."
     )
+
+
+def _no_record(path: Path, root: Path) -> str:
+    """Say that a run recorded a projection but no conversion record to read it from."""
+    return (
+        f"The run recorded Z-projected images, but no conversion record {path.name} (a run "
+        f"records the record's SHA-256), so it is not known how the images in {root} were "
+        "made. Run preprocessing again on the projected copies."
+    )
+
+
+def _projection_problem(
+    record: ConversionRecord,
+    sha256: str,
+    paths: list[str],
+    recorded: Mapping | None,
+    masks: Iterable[tuple[str, str]],
+) -> str | None:
+    """Return why the manifest's projection is not what the record says, or None.
+
+    Projected copies recorded without their projection (or the reverse, or with other
+    slices, method, types or source SHA-256) are refused, never read as plain images.
+    """
+    expected = run_projection(record, sha256, paths, masks)
+    if (None if expected is None else expected.to_dict()) == recorded:
+        return None
+    what = "no projection" if recorded is None else "a projection that differs from the record"
+    return (
+        f"The run recorded {what}, but its conversion record says how these images were "
+        "made (Z-projected copies, or not): the manifest and the record disagree, so these "
+        "images are not read as plain images. Run preprocessing again on the projected copies."
+    )

src/morphobricks_preprocessing/conversion_record.py

@@ -32,7 +32,9 @@
 from .fingerprint import Fingerprint, current_fingerprint
 from .manifest import file_sha256
 from .projection import Projection
+from .projection_marker import marked_images, marker_issue
 from .projection_record import ProjectionEntry, projection_from_data, projection_to_data
+from .projection_run import RunProjection, masks_issue, run_projection
 from .recorded_sizes import RecordedSize, sizes_from_data, sizes_to_data
 from .validation import ValidationIssue, error, plural, warning
 
@@ -369,9 +371,15 @@ class ConversionReview:
     source_root: str | None
     issues: tuple[ValidationIssue, ...] = ()
     record: str = RECORD_FILENAME
+    projection: RunProjection | None = None
 
 
-def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> ConversionReview | None:
+def review_conversion(
+    root: str | os.PathLike,
+    images: Iterable[str],
+    analysed: Iterable[str] | None = None,
+    masks: Iterable[tuple[str, str]] = (),
+) -> ConversionReview | None:
     """Check a copy tree against its conversion record; None for an ordinary experiment.
 
     Parameters
@@ -380,18 +388,26 @@ def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> Convers
         The experiment folder of the run.
     images
         The images discovery found in it, relative to it.
+    analysed
+        The images the run analyses (not excluded); by default ``images``.
+    masks
+        ``(image, mask path)`` of every mask found, for ``masks_on_projected_images``.
 
     Returns
     -------
     ConversionReview or None
-        None when the folder holds no conversion record.
+        None when the folder holds no conversion record. For projected copies it carries
+        the :class:`.projection_run.RunProjection` and, with masks, the warning.
     """
+    images = list(images)
     try:
         found = read_record(root)
     except ConversionRecordError as exc:
         return ConversionReview(None, None, (_unreadable(exc),))
-    if found is None:
-        return None
+    analysed = images if analysed is None else list(analysed)
+    if found is None:  # a projected copy needs its record: refuse it, nothing is guessed
+        marked = marked_images(root, analysed)
+        return ConversionReview(None, None, (marker_issue(marked),)) if marked else None
     record, sha256 = found
     sources = Path(record.source_root)
     changed = [
@@ -402,7 +418,14 @@ def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> Convers
     listed = {e.relative_path for e in record.files}
     copies = sorted(set(changed_copies(root, record)) | (set(images) - listed))
     issues = [*_source_issue(changed, record.source_root), *_copy_issue(copies)]
-    return ConversionReview(sha256, record.source_root, tuple(issues))
+    done = {e.relative_path for e in record.files if e.action == PROJECTED}
+    marked = marked_images(root, [name for name in analysed if name not in done])
+    if marked:
+        issues.append(marker_issue(marked))
+    projection = run_projection(record, sha256, analysed, masks)
+    if projection is not None:
+        issues.extend(masks_issue(projection))
+    return ConversionReview(sha256, record.source_root, tuple(issues), projection=projection)
 
 
 def changed_copies(

src/morphobricks_preprocessing/discovery_report.py

@@ -147,6 +147,8 @@ def text(self) -> str:
             lines.append(f"  changed from {self.outcome.previous_root}")
         if data.get("conversion"):
             lines.append(f"  named-axis copies of {data['conversion']['source_root']}")
+            if "projection" in data["conversion"]:
+                lines.extend(_projection_lines(data["conversion"]["projection"]))
         lines.extend(_folder_lines(data))
         lines.extend(_image_lines(data))
         lines.extend(_mask_lines(data))
@@ -225,11 +227,26 @@ def _conversion(pre: Preflight) -> dict | None:
     """Return the conversion record of a tree of named-axis copies, or None (design §5)."""
     if pre.conversion is None:
         return None
-    return {
+    found = pre.conversion.projection
+    data = {
         "record": pre.conversion.record,
         "sha256": pre.conversion.sha256,
         "source_root": pre.conversion.source_root,
     }
+    if found is not None:  # the key is left out for copies that are not projected
+        data["projection"] = {**found.to_dict(), "summary": found.describe()}
+    return data
+
+
+def _projection_lines(projection: dict) -> list[str]:
+    """Return the report lines of projected copies: the method and what masks are paired."""
+    lines = [f"  {projection['summary']} of {projection['source_root']}"]
+    if projection["masks_paired"]:
+        lines.append(
+            f"  {len(projection['masks_paired'])} masks are paired with the {projection['method']} "
+            "projections (masks_on_projected_images; masks are never projected)"
+        )
+    return lines
 
 
 def _issue_dict(issue: ValidationIssue) -> dict:

src/morphobricks_preprocessing/manifest.py

@@ -1,23 +1,20 @@
 """The run manifest: what a run read, with which settings, and what it wrote.
 
-``run_manifest.json`` sits next to ``plane_histograms.csv``. It records the
-confirmed folder labels and axis mapping, the settings, the package versions,
-and each source file's fingerprint (size and modification time, and for an
-OME-Zarr folder its inventory). The next run compares these with the experiment
-folder to find new, changed and deleted files, and reads again only those
-(``docs/plan.md`` §7). The fingerprint is metadata: a file rewritten with the same
-size and its modification time set back is not seen as changed, unless the run
-verifies content (``verify_content``: each source's SHA-256 is recorded and must
-match). The format is described in ``docs/contract.md`` §9.
+``run_manifest.json`` sits next to ``plane_histograms.csv``. It records the confirmed folder
+labels and axis mapping, the settings, the package versions, and each source file's
+fingerprint (size and modification time, and for an OME-Zarr folder its inventory). The next
+run compares these with the experiment folder to find new, changed and deleted files, and
+reads again only those (``docs/plan.md`` §7). The fingerprint is metadata: a file rewritten
+with the same size and its modification time set back is not seen as changed, unless the run
+verifies content (``verify_content``: each source's SHA-256 is recorded and must match). The
+format is described in ``docs/contract.md`` §9.
 
 The manifest also records, for information only, each file's channel names and
-significant bits when its metadata provides them, the SHA-256 of the
-experiment design used, so later bricks can check that it was not changed, and
-the image type the user confirmed, which later bricks normalize by, and the
-limits of the intensity checks run on the table. It also records the histogram
-range the table was binned with, and the planes whose central intensity range
-was too narrow (follow-up P6), so a later run that keeps their rows still
-reports them.
+significant bits, the SHA-256 of the experiment design used (so later bricks can check it
+was not changed), the image type the user confirmed (which later bricks normalize by), the
+limits of the intensity checks, the histogram range the table was binned with, and the
+planes whose central intensity range was too narrow (follow-up P6), so a later run that
+keeps their rows still reports them. For projected copies it records the projection.
 """
 
 from __future__ import annotations
@@ -58,11 +55,14 @@
 from .manifest_versions import MANIFEST_VERSION as MANIFEST_VERSION
 from .manifest_versions import STRICT_MANIFEST_VERSION as STRICT_MANIFEST_VERSION
 from .narrow_ranges import NarrowRange
+from .projection_run import KEY as PROJECTION_KEY
+from .projection_run import recorded_projection
 from .recorded_sizes import RecordedSize, sizes_from_data, sizes_to_data
 from .versions import package_versions as package_versions  # re-exported (moved)
 
 MANIFEST_FILENAME = "run_manifest.json"
 AXES_FROM = ("header", "confirmed")
+_LABEL_KEYS = ("folder", "condition", "biological_repeat", "technical_repeat")
 """Where a file's roles come from (:attr:`FileEntry.axes_from`)."""
 
 
@@ -254,6 +254,8 @@ class RunManifest:
     experiment_description
         What a run with a published ``morphobricks.toml`` records about it
         (:mod:`.description_record`); None, and the key left out, for a run without one.
+    projection
+        The Z projection of the images (:mod:`.projection_run`); None (key left out) if none.
     schema_version
         Version of the CSV format.
     manifest_version
@@ -280,6 +282,7 @@ class RunManifest:
     conversion_record_sha256: str | None = None
     calibration_sha256: str | None = None
     experiment_description: dict | None = None
+    projection: dict | None = None
     schema_version: str = SCHEMA_VERSION
     manifest_version: str = MANIFEST_VERSION
 
@@ -339,12 +342,7 @@ def to_dict(self) -> dict:
             },
             "axis_mapping": {"axes": self.mapping.axes, "roles": list(self.mapping.roles)},
             "labels": [
-                {
-                    "folder": folder,
-                    "condition": labels[0],
-                    "biological_repeat": labels[1],
-                    "technical_repeat": labels[2],
-                }
+                dict(zip(_LABEL_KEYS, (folder, *labels), strict=True))
                 for folder, labels in self.leaves.items()
             ],
             "files": [_file_dict(f, strict=self.manifest_version != "1") for f in self.files],
@@ -365,6 +363,7 @@ def to_dict(self) -> dict:
             "versions": dict(self.versions),
         }
         optional = {"masks": self.masks, DESCRIPTION_KEY: self.experiment_description}
+        optional[PROJECTION_KEY] = self.projection
         data.update({key: value for key, value in optional.items() if value is not None})
         return data
 
@@ -425,6 +424,7 @@ def from_dict(cls, data: dict) -> RunManifest:
                 conversion_record_sha256=optional_text(data.get("conversion_record_sha256")),
                 calibration_sha256=optional_text(data.get("calibration_sha256")),
                 experiment_description=recorded_description(data.get(DESCRIPTION_KEY)),
+                projection=recorded_projection(data.get(PROJECTION_KEY)),
                 schema_version=str(data["schema_version"]),
                 manifest_version=str(version),
             )
@@ -471,15 +471,9 @@ def file_entry_with(
 ) -> FileEntry:
     """Describe a source image with a fingerprint already taken, as :func:`file_entry_for`.
 
-    Parameters
-    ----------
-    signature
-        ``(size, mtime_ns, inventory)`` (:func:`.inventory.source_signature`), for example
-        the one taken when its header was read.
-    relative_path, labels, mapping
-        As :func:`file_entry_for`.
-    **header
-        The keyword arguments of :func:`file_entry_for` (channel names, bits, ...).
+    ``signature`` is ``(size, mtime_ns, inventory)`` (:func:`.inventory.source_signature`), for
+    example the one taken when the header was read; ``relative_path``, ``labels``, ``mapping``
+    and the ``header`` keywords (channel names, bits, ...) are those of :func:`file_entry_for`.
 
     Returns
     -------