PP-5: masks (stated axes, never converted, labels always used)

AGENTS.md

@@ -113,7 +113,8 @@ src/morphobricks_preprocessing/
                     pixel checks), publish.py (staged outputs, published together)
   report.py         plain-language summaries shared by CLI and notebook, group statistics tables
   cli.py, __main__  `morphobricks-preprocessing check|run|masks` (also `python -m morphobricks_preprocessing`)
-  masks.py          masks (Q32): find_masks pairs every image with its masks (masks folder with one
+  masks/           masks (Q32; a package since PP-5: model, files, header, layout, read,
+                    checks, labels): find_masks pairs every image with its masks (masks folder with one
                     folder per type mirroring the experiment, or <image>_<type>.tif in the
                     experiment's folders, mixed; OME-Zarr label images), header-only
                     checks (2D, image YX, integer/bool), MaskInventory (to_manifest), read_mask,

docs/user_guide.md

@@ -86,7 +86,9 @@ experiment/
   an error (`invalid_name`).
 - Keep masks out of the experiment folder (see Masks below). Images named like a mask of
   another image (`f1_mask.tif` beside `f1.tif`), or a folder named like a masks folder
-  holding such files, give a warning (`image_named_as_mask`, `mask_folder_in_experiment`).
+  holding such files, are reported: an image named as a mask is an error (`image_named_as_mask`:
+  rename it or exclude it), a masks folder inside the experiment a warning
+  (`mask_folder_in_experiment`).
 - Each image is one position (field of view), named after its file without the extension:
   `img_01.tif` and `img_01.ome.zarr` are both position `img_01`. The same name may be used
   in different folders (`tech_rep1/img_01.tif` and `tech_rep2/img_01.tif` are two
@@ -401,6 +403,7 @@ never writes the file. Other options:
 | `--outlier-limit N`, `--saturation-limit FRACTION`, `--min-images N` | Limits of the intensity checks printed after the run (default 3, 0.01 and 5) |
 | `--log FILE` | Write technical details to a file |
 | `--masks FOLDER` | The masks folder (§2). `check` lists the mask types found and checks every mask from its header; `run` stops on a mask error and records the masks in `run_manifest.json` |
+| `--mask-axes TYPE=ROLES` | The axes of a mask type whose files do not name them, one role (Y, X, Z or T) per axis as stored: `Cells=TYX` for plain stacks of time points, `Cells=YX` for one 2D mask shared by every plane of its image (repeatable; `check`, `run`, `masks`). Masks are never reordered. |
 
 After writing, `run` prints the intensity checks of the table, grouped by kind, each with
 the channel, the image or biological repeat and the values compared, and what to do once
@@ -554,7 +557,7 @@ continues.
 | `no_usable_planes` | Error | No plane has finite pixel values. Check the images. |
 | `tiff_outside_leaf` | Error | Images sit in a folder that also holds subfolders, so they have no condition or repeat of their own. A folder holds either images or subfolders, never both. Move the images into a folder of their own, or out of the experiment folder. |
 | `non_tiff_file`, `empty_folder` | Warning | Files or folders that are skipped. |
-| `image_named_as_mask` | Warning | An image is named like a mask of another image (`f1_mask.tif` beside `f1.tif`; the endings `_mask`, `_labels`, `_label`, `_seg`, `_segmentation`), so a mask would be analysed as an image. If it is a mask, move it into a masks folder outside the experiment; if it is an image, nothing to do. |
+| `image_named_as_mask` | Error | An image is named like a mask of another image (`f1_mask.tif` beside `f1.tif`; the endings `_mask`, `_masks`, `_cp_masks`, `_labels`, `_label`, `_seg`, `_segmentation`), and a mask is never analysed as an image. If it is a mask, move it into a masks folder outside the experiment, or exclude it; if it is an image, rename it. |
 | `mask_folder_in_experiment` | Warning | A folder inside the experiment is named like a masks folder (`masks`, `ctrl_masks`, `CellLabels`) and holds files named like masks of the images; they would be analysed as images. Move it out of the experiment and give it as the masks folder, or rename it if these are images. |
 | `planes_read_in_blocks` | Warning | Planes too large for the memory limit are read in blocks. The results are the same; nothing to do. Raise the limit to read them whole, which is faster. |
 | `different_sizes` | Warning | Images have different widths or heights. Usually expected; otherwise check the camera settings. |
@@ -580,18 +583,21 @@ continues.
 | `no_finite_pixels` | Warning | Some planes have a channel without finite pixels; those channels are left out of group statistics. |
 | `excluded_images`, `excluded_path_not_found` | Warning | Images excluded as requested, or an excluded path that matches no image. |
 | `masks_folder_not_found` | Error | The masks folder does not exist. Select the folder that holds one folder per kind of mask. |
+| `output_inside_masks` | Error | The output folder is inside the masks folder, where the run's files would be read as masks. Choose another output folder. |
 | `masks_inside_experiment` | Error | The masks folder is inside the experiment folder, where its folders would be read as conditions and repeats. Move it next to the experiment folder. |
 | `mask_names_differ_in_case` | Error | Two kinds of mask have names that differ only in upper/lower case. Rename one. |
 | `duplicate_mask` | Error | An image has two masks of one kind (two extensions, or a file and an OME-Zarr label image of the same name). Keep one. |
 | `unreadable_mask` | Error | A mask cannot be opened, or holds several images, or the `labels` group of an OME-Zarr image cannot be read (it must list its label images). Save the mask again as a TIFF label image, or repair the OME-Zarr labels. |
-| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). |
+| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) State the axes of its mask type once (`--mask-axes Cells=TYX`), or save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). |
+| `invalid_mask_axes`, `mask_axes_contradict_header` | Error | Axes stated for a mask type do not fit its masks (one role, Y, X, Z or T, per axis; Y and X once each), name a type that is not found, or contradict the axes a mask's metadata names. Correct the statement. |
+| `mask_layout` | Error | A mask is stored with its Y and X the other way round from its image as read (X before Y, or the image's height and width swapped). Masks are never transposed: save it again with `numpy.swapaxes(mask, -1, -2)`. |
 | `mask_planes_differ` | Error | The masks of one image have different numbers of depth slices or time points. Make every mask of an image with the image's planes. |
-| `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. |
+| `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse) unless its type is stated `YX` (`--mask-axes Cells=YX`: one mask shared by every plane), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. |
 | `mask_size_mismatch` | Error | A mask does not have the width and height of its image. Check that it was made from that image at full resolution. |
 | `mask_pixel_type` | Error | A mask's pixels are not whole numbers (for example decimals). Save it as an integer label image. |
 | `mask_missing` | Warning | Some images have no mask of a kind that other images have; their objects of that kind cannot be measured. Add the masks, or exclude the images. |
 | `ambiguous_mask` | Error | A mask's name could belong to two images (one image name starts another's) or fit both layouts. Rename the masks or the images so each names one image and one kind. |
-| `mask_without_image`, `mask_outside_type_folder` | Warning | Files in the masks folder that match no analysed image (check the folder and file names), or other files directly in the masks folder. They are not used. |
+| `mask_without_image`, `mask_outside_type_folder`, `mask_of_excluded_image` | Warning | Files in the masks folder that match no image of the experiment (check the folder and file names), other files directly in the masks folder, or masks of excluded images. They are not used. |
 | `images_not_in_manifest`, `images_missing`, `images_changed`, `position_id_not_recorded`, `identity_differs`, `axis_mapping_mismatch` | Error (other bricks) | A later brick checked its images against this brick's run (`check_sources`): an image the run did not analyse, no longer there, changed since, recorded with other labels or position (or by a version that did not record positions), or read with axes the run gave other roles. Run this brick again on the experiment as it is, or use the run's labels and axis roles. |
 | `zarr_not_installed` | Error | OME-Zarr images need the `zarr` extra. Install it, or exclude the OME-Zarr images. |
 
@@ -674,8 +680,9 @@ write_mask_report(report, "/data/exp1_MorphoBricks_Preprocessing/mask_relationsh
 ```
 
 `scan_masks(..., thresholds={"inside_object_share": 0.8})` changes a threshold (the others
-keep their defaults, `DEFAULT_THRESHOLDS`). `preflight(..., ome_zarr_labels=True)` uses the
-label images of OME-Zarr images without a masks folder; `read_mask` reads one mask's pixels
+keep their defaults, `DEFAULT_THRESHOLDS`). The label images of OME-Zarr images are always
+used, with or without a masks folder; `preflight(..., mask_axes={"Cells": "TYX"})` states
+the axes of a mask type whose files do not name them; `read_mask` reads one mask's pixels
 (a `ValueError` for a negative label or a label above 2,147,483,647, the int32 limit of
 every MorphoBricks brick, as in `piece_counts`, `pair_counts` and `scan_masks`; `check_labels`
 is that check alone);

pyproject.toml

@@ -125,6 +125,9 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing._io" = "io"
 "morphobricks_preprocessing.axes" = "pure"
 "morphobricks_preprocessing.axis_check" = "pure"
+"morphobricks_preprocessing.axis_names" = "pure"
+"morphobricks_preprocessing.image_names" = "pure"
+"morphobricks_preprocessing.mask_names" = "pure"
 "morphobricks_preprocessing.axis_review" = "pure"
 "morphobricks_preprocessing.mask_checks" = "pure"
 "morphobricks_preprocessing.mask_record" = "io"
@@ -141,6 +144,14 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.recorded_sizes" = "pure"
 "morphobricks_preprocessing.mask_scan" = "io"
 "morphobricks_preprocessing.masks" = "io"
+"morphobricks_preprocessing.masks.checks" = "io"
+"morphobricks_preprocessing.masks.files" = "io"
+"morphobricks_preprocessing.masks.header" = "io"
+"morphobricks_preprocessing.masks.labels" = "pure"
+"morphobricks_preprocessing.masks.layout" = "io"
+"morphobricks_preprocessing.masks.model" = "pure"
+"morphobricks_preprocessing.masks.read" = "io"
+"morphobricks_preprocessing.masks.stated" = "io"
 "morphobricks_preprocessing.metadata" = "io"
 "morphobricks_preprocessing.notebook_style" = "ui"
 "morphobricks_preprocessing.notebook_ui" = "ui"

src/morphobricks_preprocessing/__init__.py

@@ -27,6 +27,7 @@
     header_axis_issue,
     likely_axis_cause,
 )
+from .axis_names import OPEN_AXIS, PLACEHOLDER_AXES
 from .axis_review import check_mapping, review_axes
 from .dataset import (
     CONSISTENCY_CODES,
@@ -59,8 +60,6 @@
 )
 from .discovery import (
     FOV_KEY,
-    MASK_FOLDER_WORDS,
-    MASK_NAME_SUFFIXES,
     MISSING_TEXT,
     DiscoveryResult,
     IdentityError,
@@ -71,10 +70,8 @@
     check_plane_index,
     default_output_folder,
     discover_experiment,
-    images_named_as_masks,
-    mask_name_issues,
-    position_id,
 )
+from .image_names import position_id
 from .image_type import (
     FLUORESCENCE,
     IMAGE_TYPES,
@@ -105,6 +102,12 @@
     write_manifest,
 )
 from .mask_checks import mask_issues, mask_size_issues
+from .mask_names import (
+    MASK_FOLDER_WORDS,
+    MASK_NAME_SUFFIXES,
+    images_named_as_masks,
+    mask_name_issues,
+)
 from .mask_scan import (
     DEFAULT_THRESHOLDS,
     MASK_REPORT_FILENAME,
@@ -123,13 +126,12 @@
     MaskInventory,
     check_labels,
     find_masks,
+    mask_axes_option,
     mask_plane_issues,
     read_mask,
     read_mask_file,
 )
 from .metadata import (
-    OPEN_AXIS,
-    PLACEHOLDER_AXES,
     ImageMetadata,
     MetadataReport,
     TiffMetadata,
@@ -288,6 +290,7 @@
     "file_mapping_for",
     "file_sha256",
     "find_masks",
+    "mask_axes_option",
     "group_summaries",
     "has_errors",
     "header_axes",

src/morphobricks_preprocessing/axis_check.py

@@ -16,7 +16,8 @@
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING
 
-from .metadata import OPEN_AXIS, TiffMetadata, named_axes
+from .axis_names import OPEN_AXIS
+from .metadata import TiffMetadata, named_axes
 from .validation import ValidationIssue, error, plural, warning
 
 if TYPE_CHECKING:  # the mapping class lives in .axes, which imports this module

src/morphobricks_preprocessing/axis_names.py

@@ -0,0 +1,41 @@
+"""Which axes a file's metadata names, for images and masks alike (contract §16, D-d).
+
+One rule (plan-structure audit D3): a dimension is named when ``tifffile`` (or OME-NGFF)
+gives it a letter from the file's metadata. Left open (:data:`OPEN_AXIS`) are placeholder
+axes (:data:`PLACEHOLDER_AXES`), and, in a plain stack of pages without axis metadata, the
+``Y`` and ``X`` too: they only say where each page's rows and columns are stored.
+:func:`.metadata.named_axes` applies it to images, :mod:`.masks` to masks.
+"""
+
+from __future__ import annotations
+
+PLACEHOLDER_AXES = frozenset("IQ")
+"""``tifffile``'s letters for a dimension whose meaning the file does not give: ``I`` (a
+plain stack of pages) and ``Q`` (a "shaped" file without axes, or an OME-NGFF axis of
+another type). They are never metadata: the user gives such a dimension its role."""
+
+OPEN_AXIS = "?"
+"""Marks a dimension the metadata does not name in :func:`named_letters`."""
+
+
+def named_letters(axes: str, from_metadata: bool) -> str:
+    """Return ``axes`` with every dimension the metadata does not name as ``?``.
+
+    Parameters
+    ----------
+    axes
+        The axes ``tifffile`` (or OME-NGFF) reports, one letter per dimension.
+    from_metadata
+        Whether the axes come from axis metadata (OME, ImageJ, shaped with axes, OME-NGFF),
+        :func:`.metadata.series_axes_from_metadata`; False for a plain stack of pages.
+
+    Returns
+    -------
+    str
+        One character per dimension, e.g. ``"CYX"``, ``"?YX"`` or ``"???"``.
+    """
+    plain_stack = not from_metadata and bool(PLACEHOLDER_AXES & set(axes))
+    return "".join(
+        OPEN_AXIS if axis in PLACEHOLDER_AXES or (plain_stack and axis in "YX") else axis
+        for axis in axes
+    )

src/morphobricks_preprocessing/cli.py

@@ -23,6 +23,7 @@
 import re
 import sys
 from collections.abc import Sequence
+from dataclasses import replace
 from datetime import date
 from pathlib import Path
 
@@ -56,6 +57,7 @@
     scan_masks,
     write_mask_report,
 )
+from .masks import MaskInventory, mask_axes_option
 from .planes import DEFAULT_MEMORY_LIMIT
 from .report import (
     axis_check_summary,
@@ -69,7 +71,7 @@
     preflight_summary,
     run_summary,
 )
-from .run import preflight, run_experiment
+from .run import Preflight, preflight, run_experiment
 from .validation import has_errors
 
 _UNITS = {
@@ -233,6 +235,11 @@ def main(argv: Sequence[str] | None = None) -> int:
     args = build_parser().parse_args(argv)
     handler = _attach_log(args.log) if getattr(args, "log", None) else None
     try:
+        try:
+            args.mask_axes = mask_axes_option(args.mask_axes)
+        except ValueError as exc:
+            print(exc)
+            return 1
         return {"check": _check, "run": _run, "masks": _masks}[args.command](args)
     finally:
         if handler is not None:
@@ -241,6 +248,7 @@ def main(argv: Sequence[str] | None = None) -> int:
 
 
 def _common(parser: argparse.ArgumentParser) -> None:
+    """Add the experiment and the options every command shares."""
     parser.add_argument("experiment", help="the experiment folder")
     parser.add_argument(
         "--output",
@@ -260,10 +268,35 @@ def _common(parser: argparse.ArgumentParser) -> None:
         help="masks folder: one subfolder per mask type, each mirroring the experiment's "
         "folders, outside the experiment; the label images of OME-Zarr images are added",
     )
+    _mask_axes_option(parser)
+
+
+def _mask_axes_option(parser: argparse.ArgumentParser) -> None:
+    """Add ``--mask-axes TYPE=ROLES``, stated once per mask type (design v0.4 §3)."""
+    parser.add_argument(
+        "--mask-axes",
+        action="append",
+        default=[],
+        metavar="TYPE=ROLES",
+        help="the axes of a mask type whose files do not name them, one role (Y, X, Z or T) "
+        "per axis as stored: Cells=TYX for plain stacks of time points, Cells=YX for one 2D "
+        "mask shared by every plane of its image (repeatable)",
+    )
+
+
+def _preflight(args: argparse.Namespace) -> Preflight:
+    """Run :func:`preflight` with the options every command shares."""
+    return preflight(
+        args.experiment,
+        args.output,
+        exclude=args.exclude,
+        masks_folder=args.masks,
+        mask_axes=args.mask_axes,
+    )
 
 
 def _check(args: argparse.Namespace) -> int:
-    pre = preflight(args.experiment, args.output, exclude=args.exclude, masks_folder=args.masks)
+    pre = _preflight(args)
     print(preflight_summary(pre))
     if pre.discovery.leaves:
         print()
@@ -327,13 +360,8 @@ def _masks(args: argparse.Namespace) -> int:
     except ValueError as exc:
         print(exc)
         return 1
-    pre = preflight(
-        args.experiment,
-        args.output,
-        exclude=args.exclude,
-        masks_folder=args.masks,
-        ome_zarr_labels=True,
-    )
+    pre = _preflight(args)
+    pre = replace(pre, masks=pre.masks or MaskInventory(folder=None))  # none found
     print(mask_summary(pre.masks))
     blocking = [*pre.discovery.issues, *pre.exclusion_issues, *pre.masks.issues]
     if blocking:
@@ -365,7 +393,7 @@ def progress(done: int, total: int, path: str) -> None:
 
 
 def _run(args: argparse.Namespace) -> int:  # noqa: PLR0915
-    pre = preflight(args.experiment, args.output, exclude=args.exclude, masks_folder=args.masks)
+    pre = _preflight(args)
     axes = reference_axes(pre.metadata.files)
     if axes is None:
         print(format_issues(pre.issues) or "No readable images were found.")
@@ -385,10 +413,6 @@ def _run(args: argparse.Namespace) -> int:  # noqa: PLR0915
             saturation_limit=args.saturation_limit,
             min_images=args.min_images,
         )
-    except ValueError as exc:
-        print(exc)
-        return 1
-    try:
         histogram_settings = HistogramSettings(
             histogram_range=args.histogram_range, robust_quantiles=tuple(args.robust_quantiles)
         )
@@ -435,6 +459,7 @@ def progress(done: int, total: int, path: str) -> None:
         progress=progress,
         design=design,
         masks_folder=args.masks,
+        mask_axes=args.mask_axes,
         image_type=image_type,
         intensity_settings=intensity_settings,
         histogram_settings=histogram_settings,