Described runs: pixel-type and dimension checks per unit (#239, narrowed)

AGENTS.md

@@ -153,6 +153,10 @@ src/morphobricks_preprocessing/
                     calibration_differs_from_header); calibration_io.py reads/writes
                     calibration.csv strictly; run/calibrate.py write_calibration (locked)
   pixel_size_checks.py the §11 pixel-size checks on resolved sizes (moved from metadata.py)
+  image_consistency.py dtype_issues (mixed_dtypes) and arrangement_issues (missing_axes,
+                    inconsistent_dimensions), moved from metadata.py; unit_consistency.py the same
+                    checks per unit of a described run (#239): unit_checked (notebook folder
+                    check, inside each unit) and declared_type_issues (the run, per image_type)
   narrow_ranges.py  NarrowRange (moved from manifest.py)
   sources.py        check_sources (moved from manifest.py in PP-4)
   manifest_fields.py a file entry's position_id (required in v2, the name's: A-m5) and roles,

docs/contract.md

@@ -69,7 +69,11 @@ channel order; an image without a C axis has one channel.
 Defined by the [normalization brick's contract][normalization contract] §1.2.1: integers
 int8–uint64 and float32/float64, everything else rejected, with their default and allowed
 bin counts. This brick adds: one dtype per experiment (`mixed_dtypes` error), and all
-planes of an experiment use the same bin count.
+planes of an experiment use the same bin count. In a described run (a published
+`morphobricks.toml`) the pixel type is compared among the units that declare the same
+`image_type`, and the notebook's folder check compares images inside each unit; units of
+different image types (RGB and fluorescence) are refused as `R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`
+(Stages 1-2), not as `mixed_dtypes`. An undescribed run compares every image of the experiment.
 
 ## 4. Histogram semantics
 

docs/user_guide.md

@@ -1128,7 +1128,7 @@ continues.
 | `ome_zarr_axes_mismatch` | Error | An OME-Zarr image declares an axis with several planes that the experiment has no role for (or has no X/Y), or the confirmed roles give one of its axes another role than its metadata declares (for example its depth `z` confirmed as time `T`); the message names each image with its declared axes and the confirmed roles. An OME-Zarr image is read only with the roles it declares. Check the axis roles, or process it separately. |
 | `multiple_series` | Error | A file or OME-Zarr folder holds several images (positions, scenes, series, wells or fields). Export them separately, or exclude it. |
 | `unsupported_dtype` | Error | Unsupported pixel type (e.g. float16). Convert, or exclude it. |
-| `mixed_dtypes` | Error | Images have different pixel types. Process them separately or convert. |
+| `mixed_dtypes` | Error | Images have different pixel types. Process them separately or convert. With a saved experiment description the images are compared within each unit (and, in the run, among units of the same image type): colour (RGB) and fluorescence units in one description are not reported here but stop the run with `R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`. |
 | `different_channel_counts` | Error | Images have different numbers of channels (an image without a channel dimension has one). Every image of an experiment must have the same channels: split the others into a separate experiment, or exclude them. If a dimension got the channel role by mistake, correct the axis roles. |
 | `inconsistent_dimensions`, `inconsistent_mapping` | Error | Images have their dimensions in a different order. Export them the same way or process separately; named-axis copies (`convert`) keep each image's order, so they do not solve this. |
 | `missing_axes` | Error | Some images lack a dimension. Confirm they have one plane along it (step 2, or `--missing-as-singleton` for `check` and `run`), or export them again. |

pyproject.toml

@@ -250,6 +250,7 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.folder_rules" = "io"
 "morphobricks_preprocessing.log_target" = "io"
 "morphobricks_preprocessing.write_target" = "io"
+"morphobricks_preprocessing.image_consistency" = "pure"
 "morphobricks_preprocessing.image_type" = "pure"
 "morphobricks_preprocessing.intensity" = "io"
 "morphobricks_preprocessing.manifest" = "io"
@@ -263,6 +264,7 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.recorded_sizes" = "pure"
 "morphobricks_preprocessing.run_inputs" = "io"
 "morphobricks_preprocessing.run_inputs_text" = "pure"
+"morphobricks_preprocessing.unit_consistency" = "pure"
 "morphobricks_preprocessing.unrecorded" = "io"
 "morphobricks_preprocessing.testing" = "orchestration"
 "morphobricks_preprocessing.testing_described" = "orchestration"

src/morphobricks_preprocessing/image_consistency.py

@@ -0,0 +1,99 @@
+"""The pixel-type and dimension-arrangement checks of a set of image headers.
+
+These are the folder check's ``mixed_dtypes`` and ``inconsistent_dimensions`` (or
+``missing_axes``) errors. :func:`.metadata.check_metadata` applies them to every image of an
+undescribed (v0.4) experiment; a described run applies them per image type of the
+description (:mod:`.unit_consistency`). The checks themselves are the same in both.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from .size_checks import CONVERT_KEEPS_ORDER, breakdown
+from .validation import ValidationIssue, error
+
+if TYPE_CHECKING:
+    from .metadata import TiffMetadata
+
+
+def dtype_issues(files: Sequence[TiffMetadata]) -> list[ValidationIssue]:
+    """Report images of more than one pixel type (``mixed_dtypes``).
+
+    Parameters
+    ----------
+    files
+        The readable image headers to compare.
+
+    Returns
+    -------
+    list of ValidationIssue
+        The error ``mixed_dtypes`` naming the files not of the most common type, or nothing.
+    """
+    if len({f.dtype for f in files}) < 2:
+        return []
+    summary, minority = breakdown({f.relative_path: f.dtype.name for f in files}, str)
+    return [
+        error(
+            "mixed_dtypes",
+            f"The images use different pixel types: {summary}. One experiment must "
+            "use a single pixel type so that its histograms can be combined. The "
+            "files listed are those not of the most common type.",
+            minority,
+            "Process each pixel type as a separate experiment, or convert the images "
+            "to one pixel type.",
+        )
+    ]
+
+
+def _is_subsequence(short: str, long: str) -> bool:
+    remaining = iter(long)
+    return all(char in remaining for char in short)
+
+
+def arrangement_issues(files: Sequence[TiffMetadata]) -> list[ValidationIssue]:
+    """Report images whose dimensions are arranged differently.
+
+    Parameters
+    ----------
+    files
+        The readable TIFF headers to compare.
+
+    Returns
+    -------
+    list of ValidationIssue
+        ``missing_axes`` when the only difference is one dimension that ``tifffile`` drops
+        (a single channel or z-slice), else ``inconsistent_dimensions``; nothing when every
+        image has the same arrangement.
+    """
+    arrangements = {f.axes for f in files}
+    if len(arrangements) < 2:
+        return []
+    summary, minority = breakdown({f.relative_path: f.axes for f in files}, str)
+    longest = max(arrangements, key=len)
+    if all(_is_subsequence(axes, longest) for axes in arrangements) and (
+        sum(len(axes) == len(longest) for axes in arrangements) == 1
+    ):
+        # The common case: one channel or one z-slice, which tifffile drops.
+        return [
+            error(
+                "missing_axes",
+                f"Some images lack dimensions that the others have: {summary}. "
+                "Processing is blocked until it is confirmed how the missing "
+                "dimensions should be read.",
+                minority,
+                "If the missing dimensions simply have one plane in these files (for "
+                "example a single channel), confirm this during axis review. Otherwise "
+                "export the images again with the same dimensions.",
+            )
+        ]
+    return [
+        error(
+            "inconsistent_dimensions",
+            f"The images have their dimensions arranged differently: {summary}. One "
+            "axis mapping cannot describe all of them.",
+            minority,
+            CONVERT_KEEPS_ORDER,
+        )
+    ]

src/morphobricks_preprocessing/metadata.py

@@ -28,11 +28,11 @@
 
 from .axis_names import OPEN_AXIS, named_letters
 from .discovery import DiscoveryResult
+from .image_consistency import arrangement_issues, dtype_issues
 from .ome_header import ome_details
 from .physical_sizes import PhysicalSizes, _tiff_sizes
 from .pixel_size_checks import pixel_size_issues
 from .size_checks import (
-    CONVERT_KEEPS_ORDER,
     breakdown,
     channel_count_issues,
     image_size_issues,
@@ -495,21 +495,8 @@ def check_metadata(files: Iterable[TiffMetadata]) -> list[ValidationIssue]:
             )
         )
 
-    if len({f.dtype for f in readable}) > 1:
-        summary, minority = _breakdown(readable, lambda f: f.dtype.name, str)
-        issues.append(
-            error(
-                "mixed_dtypes",
-                f"The images use different pixel types: {summary}. One experiment must "
-                "use a single pixel type so that its histograms can be combined. The "
-                "files listed are those not of the most common type.",
-                minority,
-                "Process each pixel type as a separate experiment, or convert the images "
-                "to one pixel type.",
-            )
-        )
-
-    issues.extend(_check_dimensions([f for f in readable if not f.is_ome_zarr]))
+    issues.extend(dtype_issues(readable))
+    issues.extend(arrangement_issues([f for f in readable if not f.is_ome_zarr]))
     issues.extend(_check_channel_names(readable))
     issues.extend(_check_counts(readable))
     issues.extend(dimension_issues(readable))
@@ -599,39 +586,6 @@ def _check_channel_names(files: list[TiffMetadata]) -> list[ValidationIssue]:
     ]
 
 
-def _check_dimensions(files: list[TiffMetadata]) -> list[ValidationIssue]:
-    arrangements = {f.axes for f in files}
-    if len(arrangements) > 1:
-        summary, minority = _breakdown(files, lambda f: f.axes, str)
-        longest = max(arrangements, key=len)
-        if all(_is_subsequence(axes, longest) for axes in arrangements) and (
-            sum(len(axes) == len(longest) for axes in arrangements) == 1
-        ):
-            # The common case: one channel or one z-slice, which tifffile drops.
-            return [
-                error(
-                    "missing_axes",
-                    f"Some images lack dimensions that the others have: {summary}. "
-                    "Processing is blocked until it is confirmed how the missing "
-                    "dimensions should be read.",
-                    minority,
-                    "If the missing dimensions simply have one plane in these files (for "
-                    "example a single channel), confirm this during axis review. Otherwise "
-                    "export the images again with the same dimensions.",
-                )
-            ]
-        return [
-            error(
-                "inconsistent_dimensions",
-                f"The images have their dimensions arranged differently: {summary}. One "
-                "axis mapping cannot describe all of them.",
-                minority,
-                CONVERT_KEEPS_ORDER,
-            )
-        ]
-    return []
-
-
 def header_channel_count(f: TiffMetadata) -> int:
     """Return the channel count the header suggests: the C axis, else the RGB samples."""
     return f.length("C") if "C" in f.axes else f.length("S")
@@ -693,8 +647,3 @@ def _breakdown(
 ) -> tuple[str, list[str]]:
     """Summarise files by ``key``, most common first (:func:`.size_checks.breakdown`)."""
     return breakdown({f.relative_path: key(f) for f in files}, describe)
-
-
-def _is_subsequence(short: str, long: str) -> bool:
-    remaining = iter(long)
-    return all(char in remaining for char in short)

src/morphobricks_preprocessing/notebook_folder.py

@@ -22,6 +22,7 @@
     labels_html,
 )
 from .run import Preflight, preflight
+from .unit_consistency import unit_checked
 from .validation import ValidationIssue, plural
 
 
@@ -208,6 +209,8 @@ def check_folder(self) -> Preflight | None:
                     masks_folder=masks,
                     mask_axes=mask_axes,
                 )
+                if self._describe:
+                    pre = unit_checked(pre)  # a described run checks inside each unit
                 self._note_edits()  # a check fills only what the user left as filled
                 self.missing_as_singleton.set(self._recorded_missing_as_singleton(), notify=False)
                 # The report reads images lacking some axes by that answer, as 'check

src/morphobricks_preprocessing/physical_check.py

@@ -187,7 +187,7 @@ def layout_of(meta: TiffMetadata, roles: Sequence[str | None] | None = None) ->
     return Layout(count, tuple(meta.raw_channel_names), meta.rgb_samples, axes)
 
 
-def _units(description: Description) -> dict[UnitKey, tuple[str, Unit]]:
+def declared_units(description: Description) -> dict[UnitKey, tuple[str, Unit]]:
     """Return the declared units by source folders, with the repeat ID they belong to."""
     found: dict[UnitKey, tuple[str, Unit]] = {}
     for repeat_id, repeat in description.repeats.items():
@@ -201,7 +201,10 @@ def _units(description: Description) -> dict[UnitKey, tuple[str, Unit]]:
     return found
 
 
-def _discovered(discovery: DiscoveryResult, excluded: Collection[str]) -> dict[UnitKey, list[str]]:
+def discovered_units(
+    discovery: DiscoveryResult, excluded: Collection[str]
+) -> dict[UnitKey, list[str]]:
+    """Return the images of each discovered unit (folder labels), without the excluded ones."""
     found: dict[UnitKey, list[str]] = {}
     for leaf in discovery.leaves:
         included = [image for image in leaf.images if image not in excluded]
@@ -379,8 +382,8 @@ def check_physical(
         checked.
     """
     metas = {m.relative_path: m for m in metadata}
-    declared = _units(description)
-    found = _discovered(discovery, set(excluded))
+    declared = declared_units(description)
+    found = discovered_units(discovery, set(excluded))
     issues = [PhysicalIssue(UNIT_NOT_FOUND, key) for key in declared if key not in found]
     issues += [PhysicalIssue(UNIT_NOT_DECLARED, key) for key in found if key not in declared]
     layouts: dict[UnitKey, Layout] = {}

src/morphobricks_preprocessing/run/description_run.py

@@ -30,6 +30,7 @@
 from ..description_run_text import heterogeneous_issue, physical_issue, reader_issue
 from ..mask_representation import NOT_CHECKED, file_status, read_checked
 from ..physical_check import PhysicalReport, check_physical
+from ..unit_consistency import declared_type_issues
 from ..validation import ValidationIssue, error
 from .description_file import RecordsFileError, read_records
 from .preflight import Preflight
@@ -81,8 +82,11 @@ def check_description(
 ) -> tuple[RunDescription, list[ValidationIssue]]:
     """Return the description with its physical report, and every refusal that applies.
 
-    The layouts are checked first and alone: with more than one layout the run stops with
-    ``R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`` and the folder checks add nothing. The physical
+    The layouts are checked first: with more than one layout the run stops with
+    ``R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`` and the physical checks do not run. The folder's
+    pixel-type and dimension checks apply per declared image type
+    (:func:`..unit_consistency.declared_type_issues`), so units of different image types
+    never produce ``mixed_dtypes`` or ``inconsistent_dimensions`` between them. The physical
     checks need the confirmed axis roles; without them they are not run (the run stops on
     the missing roles anyway) and the report is None, never a pass.
 
@@ -97,10 +101,11 @@ def check_description(
         None when no axis roles were confirmed.
     """
     layouts = declared_layouts(found.description)
+    units = declared_type_issues(found.description, pre)
     if is_heterogeneous(layouts):
-        return found, [heterogeneous_issue(layouts)]
+        return found, [heterogeneous_issue(layouts), *units]
     if roles is None:
-        return found, []
+        return found, units
     masks = pre.masks
     report = check_physical(
         found.description,
@@ -110,7 +115,7 @@ def check_description(
         roles={path: mapping.roles for path, mapping in roles.items()},
         mask_types=None if masks is None else masks.types,
     )
-    return replace(found, physical=report), [physical_issue(i) for i in report.issues]
+    return replace(found, physical=report), [*units, *(physical_issue(i) for i in report.issues)]
 
 
 GLOBAL_CHANNEL_CODES = frozenset({"channel_names_differ", "different_channel_counts"})
@@ -122,9 +127,25 @@ def check_description(
 and by ``R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`` for units with different channel lists."""
 
 
+GLOBAL_UNIT_CODES = frozenset({"mixed_dtypes", "inconsistent_dimensions"})
+"""The v0.4 folder checks that compare pixel types and dimensions across the whole experiment.
+
+A description declares each unit's ``image_type``; they are replaced by the same checks per
+declared image type (:func:`..unit_consistency.declared_type_issues`; the axis review checks the
+arrangement of dimensions). Units of different image types are then refused as a layout
+(``R_HETEROGENEOUS_LAYOUT_NOT_ENABLED``), not by these."""
+
+
 def without_global_channel_checks(issues: Iterable[ValidationIssue]) -> list[ValidationIssue]:
-    """Return ``issues`` without the experiment-wide channel checks (described runs only)."""
-    return [issue for issue in issues if issue.code not in GLOBAL_CHANNEL_CODES]
+    """Return ``issues`` without the experiment-wide checks a description replaces.
+
+    These are the channel checks (:data:`GLOBAL_CHANNEL_CODES`) and the pixel-type and
+    dimension checks of every image of the experiment (``mixed_dtypes``,
+    ``inconsistent_dimensions``), which :func:`check_description` makes per declared image
+    type. Described runs only.
+    """
+    dropped = GLOBAL_CHANNEL_CODES | GLOBAL_UNIT_CODES
+    return [issue for issue in issues if issue.code not in dropped]
 
 
 def checked_description(output: Path, pre: Preflight, checked: Checked) -> RunDescription | None: