diff --git a/AGENTS.md b/AGENTS.md index db2625ce..28d63b6e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,7 +113,8 @@ src/morphobricks_preprocessing/ pixel checks), publish.py (staged outputs, published together) report.py plain-language summaries shared by CLI and notebook, group statistics tables cli.py, __main__ `morphobricks-preprocessing check|run|masks` (also `python -m morphobricks_preprocessing`) - masks.py masks (Q32): find_masks pairs every image with its masks (masks folder with one + masks/ masks (Q32; a package since PP-5: model, files, header, layout, read, + checks, labels): find_masks pairs every image with its masks (masks folder with one folder per type mirroring the experiment, or _.tif in the experiment's folders, mixed; OME-Zarr label images), header-only checks (2D, image YX, integer/bool), MaskInventory (to_manifest), read_mask, diff --git a/docs/contract.md b/docs/contract.md index d477055c..ffa9f3db 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -30,7 +30,7 @@ whole experiment; any change to them goes through the normalization brick. | System files | Dot-files and dot-folders (`.DS_Store`, `._*`, `.ipynb_checkpoints`), `Thumbs.db`, `desktop.ini` and `__MACOSX` are ignored without a warning. | | Scope | A group is identified by its full label path, e.g. `(control, bio_rep1)`. | | Positions | Every image is exactly one position (field of view): files with several positions, scenes, series or tiles are refused (§2). Its `position_id` is the image name without its extension (`.tif`, `.tiff` or `.zarr`, any case, with a preceding `.ome`): `img_01.ome.tif` gives `img_01`; a name that is only an extension (`.tif`, `.ome.zarr`) is refused. `discovery.position_id` is the one function every MorphoBricks brick names positions with (cross-brick decision D85, 29 October 2026), so it also removes the other image extensions bricks read (`.png`, `.jpg`, `.jpeg`, `.bmp`, `.gif`, `.npy`; naming only: this brick still analyses TIFF and Zarr images only). A position is identified by `condition`, `biological_repeat`, `technical_repeat` and `position_id` (cross-brick decision D02, 26 October 2026), so equal names in two technical repeats are two positions. Two images in one leaf folder with the same `position_id`, ignoring case (`img_01.tif` and `img_01.ome.zarr`), are an error (`duplicate_position_id`): the brick does not guess which is which. | -| Masks in the experiment | Masks belong in a masks folder outside the experiment (§15); discovery warns about masks it would analyse as images (decision D-i, `mask_name_issues`, §16): `image_named_as_mask` for an image named after another image's `position_id`, then `_mask`, `_labels`, `_label`, `_seg` or `_segmentation` (any case, anywhere in the experiment: `f1_mask.tif` beside `f1.tif`; `A1_2.tif` and `f1_DNA.tif` are images), and `mask_folder_in_experiment` for a folder named like a masks folder (`masks`, `ctrl_masks`, `CellLabels`) holding images named as masks of the other images (`` or `_`). Warnings: such files may be images. | +| Masks in the experiment | Masks belong in a masks folder outside the experiment (§15); discovery screens for masks it would analyse as images (decision D-i, `mask_name_issues`, §16): `image_named_as_mask` (an error since v0.4.0, design v0.4 §3: rename the file or exclude it; an excluded file is not reported) for an image named after another image's `position_id`, then `_mask`, `_masks`, `_cp_masks`, `_labels`, `_label`, `_seg` or `_segmentation` (any case, anywhere in the experiment: `f1_mask.tif` beside `f1.tif`; `A1_2.tif` and `f1_DNA.tif` are images), and `mask_folder_in_experiment` for a folder named like a masks folder (`masks`, `ctrl_masks`, `CellLabels`) holding images named as masks of the other images (`` or `_`), a warning: such files may be images. | | Paths | `source_relative_path` is relative to the experiment root, with `/` separators. | ## 2. Axes @@ -121,8 +121,8 @@ method is an offset and a scale, `(pixels − offset) / scale`. | Severity | Situations | Effect | |---|---|---| -| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X, contradict its header or put X before Y (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | -| Warning | Images whose metadata does not name their axes, read with the confirmed roles (`axes_not_named`, §2); images named as masks of other images, or folders named like a masks folder holding them (`image_named_as_mask`, `mask_folder_in_experiment`, §1); intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported | +| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X, contradict its header or put X before Y (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment or inside the masks folder (`output_inside_masks`, §15); images named as masks of other images (`image_named_as_mask`, §1); no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named or stated Z/T (`mask_not_2d`), has stated axes that do not fit it or contradict the axes its metadata names (`invalid_mask_axes`, `mask_axes_contradict_header`), is stored with X before Y or with its image's Y and X sizes swapped (`mask_layout`; never transposed), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | +| Warning | Images whose metadata does not name their axes, read with the confirmed roles (`axes_not_named`, §2); folders named like a masks folder holding images named as masks (`mask_folder_in_experiment`, §1); intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no image, masks of excluded images, files outside the mask-type folders (§15) | Processing continues; the item is reported | Every message states what happened in plain language, lists the affected paths, and suggests an action. Technical details go to the diagnostic log. @@ -161,7 +161,7 @@ file's schema or statistical interpretation. | `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. **Version 2 (v0.4.0):** `roles` is required, with `axes_from` (required): `header` when the metadata names every axis and the roles agree, `confirmed` for a plain stack whose roles the user confirmed (§2); and `pixel_sizes`, one entry per role `X`, `Y`, `Z`, `T` with `value`, `unit` (OME-NGFF name) and `source` (`header` for now; `stated` and `pixels` with stated calibration), `value` and `source` null for an uncalibrated axis (never guessed). The reader refuses a version 2 entry without `roles` or `axes_from`, and the writer refuses to write one. | | `csv_sha256` | SHA-256 of the table written with this manifest. | | `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. | -| `masks` | The masks paired with the images (§15), when the run was given a masks folder: `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes other than Y and X, `YX` for a 2D mask (`[1, 512]` for a mask one pixel high), with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the manifest stays version 1, and the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes. **Version 2:** each mask entry also has `size` and `mtime_ns` (its fingerprint, as `source_fingerprint` gives it: the mask file, or the label image's folder), `roles` (one per axis of `axes`), `axes_from` (`header`) and `shared` (`false`; a 2-D mask stated as shared by every plane of its image comes with the mask-axis statements). | +| `masks` | The masks paired with the images (§15), when the run was given a masks folder or its OME-Zarr images have label images (always used): `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes other than Y and X, `YX` for a 2D mask (`[1, 512]` for a mask one pixel high), with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes unless `shared` is true. **Version 2:** `axes` are the mask's axes as stored (its header's letters: `IYX` for a plain stack), and each mask entry also has `size` and `mtime_ns` (its fingerprint, as `source_fingerprint` gives it: the mask file, or the label image's folder), `roles` (one per axis of `axes`, what each is read as: `TYX` for that plain stack stated `--mask-axes Cells=TYX`), `axes_from` (`header`, or `stated` when the axes were stated for its mask type) and `shared` (true only for a 2-D mask stated `YX` for its type: one plane shared by every plane of its image). A mask is read as stored, never reordered. | | `image_type` | `fluorescence` or `rgb`, the image type the user confirmed (§13); null when the run was given none, or absent in manifests written before it. Optional key: the manifest stays version 1. Changing it recomputes nothing. | | `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. | | `intensity_checks` | The limits of the intensity checks run on the table (§14): `IntensityCheckSettings.to_dict()` of the normalization brick (`library_version`, `outlier_limit`, `saturation_limit`, `min_images`); absent in manifests written before it. Optional key. Changing the limits recomputes nothing. | @@ -293,14 +293,14 @@ masks/ masks folder, outside the experiment folder | Rule | Detail | |---|---| -| Layout | Either one subfolder per mask type, each mirroring the experiment's folders (`//.../.tif`), or the experiment's folders with the type after the image name (`/.../_.tif`, as segmentation tools often write it, e.g. MorphoRelate's JUMP-Target-2 conversion); the two may be mixed. Any number of mask types. The label images of an OME-Zarr image (its OME-NGFF `labels` group, with the `zarr` extra) are masks too, of the type named by the label image; they are used whenever masks are asked for, with or without a masks folder. | -| Pairing | A mask belongs to the image in the same leaf folder with the same `position_id` (§1), ignoring upper/lower case; its file is a TIFF (`.tif`, `.tiff`, optionally `.ome`, any case). Excluded images have no masks. | +| Layout | Either one subfolder per mask type, each mirroring the experiment's folders (`//.../.tif`), or the experiment's folders with the type after the image name (`/.../_.tif`, as segmentation tools often write it, e.g. MorphoRelate's JUMP-Target-2 conversion); the two may be mixed. Any number of mask types. The label images of an OME-Zarr image (its OME-NGFF `labels` group, with the `zarr` extra) are masks too, of the type named by the label image; since v0.4.0 they are always used when present, with or without a masks folder. Each label image has its own fingerprint in the manifest, and the `labels` group is left out of its image's fingerprint (`source_fingerprint`), so adding or changing a label image does not recompute the image's rows. | +| Pairing | A mask belongs to the image in the same leaf folder with the same `position_id` (§1), ignoring upper/lower case; its file is a TIFF (`.tif`, `.tiff`, optionally `.ome`, any case). Excluded images have no masks: their masks are reported apart (`mask_of_excluded_image`, warning) and not used. | | Names | A mask type is named by its folder, by the suffix after the image name, or by its label image. A file that could belong to two images (an image name that starts another's, `a_b_cells.tif` beside images `a` and `a_b`) or fit both layouts is the error `ambiguous_mask`. Two names differing only in case are the error `mask_names_differ_in_case`; one image with two masks of one type (two extensions, or a file and a label image) is the error `duplicate_mask`. | -| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes other than Y and X are removed (Y and X are kept whatever their length, so a mask one pixel high or wide, or a single pixel, is a 2D `YX` mask; a TIFF whose series has no Y or X but is one 2D page, such as a 1D array `tifffile` saved as `{"shape": [N]}` and reads as `X`, is that page, `YX` of 1 row; plan §12 question 44), a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed. A mask has the Y and X sizes of its image (`mask_size_mismatch`; `check` and `preflight` compare with the Y and X the image's header names, and not for a plain stack, whose Y and X are not named; `run` compares with the image's Y and X under the confirmed roles, `mask_size_issues`, and leaves out the masks of an image with an axis error (§2), which is never read — before v0.4.0 `run` compared with the header's letters, so a swapped image gave a misleading size error) and integer or boolean pixels (`mask_pixel_type`; a mask holding a negative label or a label above 2,147,483,647, the int32 limit every brick shares (§16, decision D-b), raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`, as is an OME-Zarr image's `labels` group that cannot be read or does not list its label images (path `/labels`), so its mask types never disappear unreported. All five are errors. | -| Planes | A mask belongs to the image's planes (issue #36): its Z and T lengths (1 for an axis it lacks) must equal the image's under the confirmed axis roles, checked by `run` (`mask_planes_mismatch`, error; `mask_plane_issues`; not for an image with an axis error, §2). So a 2D mask belongs only to an image with one depth slice and one time point, and is refused for a Z-stack or a time-lapse, even as a projection (a decision for the owner if wanted later); a mask's plane at index `i` along Z or T belongs to the image's rows with `z = i` or `t = i`. `read_mask(..., z=, t=)` reads one plane and requires `z` and `t` for a mask with those axes. `check` and the `masks` command have no confirmed roles and do not compare with the image. | -| Folder | The masks folder must exist (`masks_folder_not_found`) and be outside the experiment folder (`masks_inside_experiment`), where its folders would be read as labels. Both errors. | -| Warnings | `mask_missing` (images without a mask of a type that other images have: their objects of that type cannot be measured), `mask_without_image` (files matching no analysed image), `mask_outside_type_folder` (files directly in the masks folder that are not TIFF masks). | -| Run | `run_experiment(..., masks_folder=...)` stops on a mask error and records the masks in the manifest's `masks` key (§9). Without masks, the run and its outputs are unchanged. | +| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes other than Y and X are removed (Y and X are kept whatever their length, so a mask one pixel high or wide, or a single pixel, is a 2D `YX` mask; a TIFF whose series has no Y or X but is one 2D page, such as a 1D array `tifffile` saved as `{"shape": [N]}` and reads as `X`, is that page, `YX` of 1 row; plan §12 question 44), a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed, unless the axes are stated for its mask type (`--mask-axes Cells=TYX`, `mask_axes={"Cells": "TYX"}`: one role, Y, X, Z or T, per axis of the header with length-1 axes other than Y and X left out; recorded as `axes_from: "stated"`; a statement that does not fit is `invalid_mask_axes`, one that contradicts an axis the metadata names is `mask_axes_contradict_header`, one for a type not found is `invalid_mask_axes`). Masks are never converted or reordered: a mask whose X is stored before its Y, or whose Y and X are its image's swapped, is `mask_layout`, with a `numpy.swapaxes` hint. A mask has the Y and X sizes of its image (`mask_size_mismatch`; `check` and `preflight` compare with the Y and X the image's header names, and not for a plain stack, whose Y and X are not named; `run` compares with the image's Y and X under the confirmed roles, `mask_size_issues`, and leaves out the masks of an image with an axis error (§2), which is never read — before v0.4.0 `run` compared with the header's letters, so a swapped image gave a misleading size error) and integer or boolean pixels (`mask_pixel_type`; a mask holding a negative label or a label above 2,147,483,647, the int32 limit every brick shares (§16, decision D-b), raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`, as is an OME-Zarr image's `labels` group that cannot be read or does not list its label images (path `/labels`), so its mask types never disappear unreported. All five are errors. | +| Planes | A mask belongs to the image's planes (issue #36): its Z and T lengths (1 for an axis it lacks) must equal the image's under the confirmed axis roles, checked by `run` (`mask_planes_mismatch`, error; `mask_plane_issues`; not for an image with an axis error, §2). So a 2D mask belongs only to an image with one depth slice and one time point, and is refused for a Z-stack or a time-lapse, unless its type is stated `YX` (`--mask-axes Cells=YX`): then it is one plane shared by every plane of its image (`shared: true` in the manifest), read for any `z` and `t`; a mask's plane at index `i` along Z or T belongs to the image's rows with `z = i` or `t = i`. `read_mask(..., z=, t=)` reads one plane and requires `z` and `t` for a mask with those axes. `check` and the `masks` command have no confirmed roles and do not compare with the image. | +| Folder | The masks folder must exist (`masks_folder_not_found`) and be outside the experiment folder (`masks_inside_experiment`), where its folders would be read as labels; the output folder may not be inside it (`output_inside_masks`), where the run's files would be read as masks. All errors. | +| Warnings | `mask_missing` (images without a mask of a type that other images have: their objects of that type cannot be measured), `mask_without_image` (files matching no image of the experiment), `mask_of_excluded_image` (masks of excluded images, not used), `mask_outside_type_folder` (files directly in the masks folder that are not TIFF masks). | +| Run | `run_experiment(..., masks_folder=..., mask_axes=...)` stops on a mask error and records the masks in the manifest's `masks` key (§9). Without masks, the run and its outputs are unchanged. | ### 15.1 The mask step (`mask_relationships.json`, report version 1) diff --git a/docs/user_guide.md b/docs/user_guide.md index 62ef677f..8c5529fd 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -86,7 +86,9 @@ experiment/ an error (`invalid_name`). - Keep masks out of the experiment folder (see Masks below). Images named like a mask of another image (`f1_mask.tif` beside `f1.tif`), or a folder named like a masks folder - holding such files, give a warning (`image_named_as_mask`, `mask_folder_in_experiment`). + holding such files, are reported: an image named as a mask is an error (`image_named_as_mask`: + rename it or exclude it), a masks folder inside the experiment a warning + (`mask_folder_in_experiment`). - Each image is one position (field of view), named after its file without the extension: `img_01.tif` and `img_01.ome.zarr` are both position `img_01`. The same name may be used in different folders (`tech_rep1/img_01.tif` and `tech_rep2/img_01.tif` are two @@ -401,6 +403,7 @@ never writes the file. Other options: | `--outlier-limit N`, `--saturation-limit FRACTION`, `--min-images N` | Limits of the intensity checks printed after the run (default 3, 0.01 and 5) | | `--log FILE` | Write technical details to a file | | `--masks FOLDER` | The masks folder (§2). `check` lists the mask types found and checks every mask from its header; `run` stops on a mask error and records the masks in `run_manifest.json` | +| `--mask-axes TYPE=ROLES` | The axes of a mask type whose files do not name them, one role (Y, X, Z or T) per axis as stored: `Cells=TYX` for plain stacks of time points, `Cells=YX` for one 2D mask shared by every plane of its image (repeatable; `check`, `run`, `masks`). Masks are never reordered. | After writing, `run` prints the intensity checks of the table, grouped by kind, each with the channel, the image or biological repeat and the values compared, and what to do once @@ -554,7 +557,7 @@ continues. | `no_usable_planes` | Error | No plane has finite pixel values. Check the images. | | `tiff_outside_leaf` | Error | Images sit in a folder that also holds subfolders, so they have no condition or repeat of their own. A folder holds either images or subfolders, never both. Move the images into a folder of their own, or out of the experiment folder. | | `non_tiff_file`, `empty_folder` | Warning | Files or folders that are skipped. | -| `image_named_as_mask` | Warning | An image is named like a mask of another image (`f1_mask.tif` beside `f1.tif`; the endings `_mask`, `_labels`, `_label`, `_seg`, `_segmentation`), so a mask would be analysed as an image. If it is a mask, move it into a masks folder outside the experiment; if it is an image, nothing to do. | +| `image_named_as_mask` | Error | An image is named like a mask of another image (`f1_mask.tif` beside `f1.tif`; the endings `_mask`, `_masks`, `_cp_masks`, `_labels`, `_label`, `_seg`, `_segmentation`), and a mask is never analysed as an image. If it is a mask, move it into a masks folder outside the experiment, or exclude it; if it is an image, rename it. | | `mask_folder_in_experiment` | Warning | A folder inside the experiment is named like a masks folder (`masks`, `ctrl_masks`, `CellLabels`) and holds files named like masks of the images; they would be analysed as images. Move it out of the experiment and give it as the masks folder, or rename it if these are images. | | `planes_read_in_blocks` | Warning | Planes too large for the memory limit are read in blocks. The results are the same; nothing to do. Raise the limit to read them whole, which is faster. | | `different_sizes` | Warning | Images have different widths or heights. Usually expected; otherwise check the camera settings. | @@ -580,18 +583,21 @@ continues. | `no_finite_pixels` | Warning | Some planes have a channel without finite pixels; those channels are left out of group statistics. | | `excluded_images`, `excluded_path_not_found` | Warning | Images excluded as requested, or an excluded path that matches no image. | | `masks_folder_not_found` | Error | The masks folder does not exist. Select the folder that holds one folder per kind of mask. | +| `output_inside_masks` | Error | The output folder is inside the masks folder, where the run's files would be read as masks. Choose another output folder. | | `masks_inside_experiment` | Error | The masks folder is inside the experiment folder, where its folders would be read as conditions and repeats. Move it next to the experiment folder. | | `mask_names_differ_in_case` | Error | Two kinds of mask have names that differ only in upper/lower case. Rename one. | | `duplicate_mask` | Error | An image has two masks of one kind (two extensions, or a file and an OME-Zarr label image of the same name). Keep one. | | `unreadable_mask` | Error | A mask cannot be opened, or holds several images, or the `labels` group of an OME-Zarr image cannot be read (it must list its label images). Save the mask again as a TIFF label image, or repair the OME-Zarr labels. | -| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). | +| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) State the axes of its mask type once (`--mask-axes Cells=TYX`), or save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). | +| `invalid_mask_axes`, `mask_axes_contradict_header` | Error | Axes stated for a mask type do not fit its masks (one role, Y, X, Z or T, per axis; Y and X once each), name a type that is not found, or contradict the axes a mask's metadata names. Correct the statement. | +| `mask_layout` | Error | A mask is stored with its Y and X the other way round from its image as read (X before Y, or the image's height and width swapped). Masks are never transposed: save it again with `numpy.swapaxes(mask, -1, -2)`. | | `mask_planes_differ` | Error | The masks of one image have different numbers of depth slices or time points. Make every mask of an image with the image's planes. | -| `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. | +| `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse) unless its type is stated `YX` (`--mask-axes Cells=YX`: one mask shared by every plane), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. | | `mask_size_mismatch` | Error | A mask does not have the width and height of its image. Check that it was made from that image at full resolution. | | `mask_pixel_type` | Error | A mask's pixels are not whole numbers (for example decimals). Save it as an integer label image. | | `mask_missing` | Warning | Some images have no mask of a kind that other images have; their objects of that kind cannot be measured. Add the masks, or exclude the images. | | `ambiguous_mask` | Error | A mask's name could belong to two images (one image name starts another's) or fit both layouts. Rename the masks or the images so each names one image and one kind. | -| `mask_without_image`, `mask_outside_type_folder` | Warning | Files in the masks folder that match no analysed image (check the folder and file names), or other files directly in the masks folder. They are not used. | +| `mask_without_image`, `mask_outside_type_folder`, `mask_of_excluded_image` | Warning | Files in the masks folder that match no image of the experiment (check the folder and file names), other files directly in the masks folder, or masks of excluded images. They are not used. | | `images_not_in_manifest`, `images_missing`, `images_changed`, `position_id_not_recorded`, `identity_differs`, `axis_mapping_mismatch` | Error (other bricks) | A later brick checked its images against this brick's run (`check_sources`): an image the run did not analyse, no longer there, changed since, recorded with other labels or position (or by a version that did not record positions), or read with axes the run gave other roles. Run this brick again on the experiment as it is, or use the run's labels and axis roles. | | `zarr_not_installed` | Error | OME-Zarr images need the `zarr` extra. Install it, or exclude the OME-Zarr images. | @@ -674,8 +680,9 @@ write_mask_report(report, "/data/exp1_MorphoBricks_Preprocessing/mask_relationsh ``` `scan_masks(..., thresholds={"inside_object_share": 0.8})` changes a threshold (the others -keep their defaults, `DEFAULT_THRESHOLDS`). `preflight(..., ome_zarr_labels=True)` uses the -label images of OME-Zarr images without a masks folder; `read_mask` reads one mask's pixels +keep their defaults, `DEFAULT_THRESHOLDS`). The label images of OME-Zarr images are always +used, with or without a masks folder; `preflight(..., mask_axes={"Cells": "TYX"})` states +the axes of a mask type whose files do not name them; `read_mask` reads one mask's pixels (a `ValueError` for a negative label or a label above 2,147,483,647, the int32 limit of every MorphoBricks brick, as in `piece_counts`, `pair_counts` and `scan_masks`; `check_labels` is that check alone); diff --git a/pyproject.toml b/pyproject.toml index f58c4e0a..ff64add9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -125,6 +125,9 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing._io" = "io" "morphobricks_preprocessing.axes" = "pure" "morphobricks_preprocessing.axis_check" = "pure" +"morphobricks_preprocessing.axis_names" = "pure" +"morphobricks_preprocessing.image_names" = "pure" +"morphobricks_preprocessing.mask_names" = "pure" "morphobricks_preprocessing.axis_review" = "pure" "morphobricks_preprocessing.mask_checks" = "pure" "morphobricks_preprocessing.mask_record" = "io" @@ -141,6 +144,14 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.recorded_sizes" = "pure" "morphobricks_preprocessing.mask_scan" = "io" "morphobricks_preprocessing.masks" = "io" +"morphobricks_preprocessing.masks.checks" = "io" +"morphobricks_preprocessing.masks.files" = "io" +"morphobricks_preprocessing.masks.header" = "io" +"morphobricks_preprocessing.masks.labels" = "pure" +"morphobricks_preprocessing.masks.layout" = "io" +"morphobricks_preprocessing.masks.model" = "pure" +"morphobricks_preprocessing.masks.read" = "io" +"morphobricks_preprocessing.masks.stated" = "io" "morphobricks_preprocessing.metadata" = "io" "morphobricks_preprocessing.notebook_style" = "ui" "morphobricks_preprocessing.notebook_ui" = "ui" diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 31dd877f..948c896a 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -27,6 +27,7 @@ header_axis_issue, likely_axis_cause, ) +from .axis_names import OPEN_AXIS, PLACEHOLDER_AXES from .axis_review import check_mapping, review_axes from .dataset import ( CONSISTENCY_CODES, @@ -59,8 +60,6 @@ ) from .discovery import ( FOV_KEY, - MASK_FOLDER_WORDS, - MASK_NAME_SUFFIXES, MISSING_TEXT, DiscoveryResult, IdentityError, @@ -71,10 +70,8 @@ check_plane_index, default_output_folder, discover_experiment, - images_named_as_masks, - mask_name_issues, - position_id, ) +from .image_names import position_id from .image_type import ( FLUORESCENCE, IMAGE_TYPES, @@ -105,6 +102,12 @@ write_manifest, ) from .mask_checks import mask_issues, mask_size_issues +from .mask_names import ( + MASK_FOLDER_WORDS, + MASK_NAME_SUFFIXES, + images_named_as_masks, + mask_name_issues, +) from .mask_scan import ( DEFAULT_THRESHOLDS, MASK_REPORT_FILENAME, @@ -123,13 +126,12 @@ MaskInventory, check_labels, find_masks, + mask_axes_option, mask_plane_issues, read_mask, read_mask_file, ) from .metadata import ( - OPEN_AXIS, - PLACEHOLDER_AXES, ImageMetadata, MetadataReport, TiffMetadata, @@ -288,6 +290,7 @@ "file_mapping_for", "file_sha256", "find_masks", + "mask_axes_option", "group_summaries", "has_errors", "header_axes", diff --git a/src/morphobricks_preprocessing/axis_check.py b/src/morphobricks_preprocessing/axis_check.py index f39d596b..c3eb9c38 100644 --- a/src/morphobricks_preprocessing/axis_check.py +++ b/src/morphobricks_preprocessing/axis_check.py @@ -16,7 +16,8 @@ from dataclasses import dataclass, field from typing import TYPE_CHECKING -from .metadata import OPEN_AXIS, TiffMetadata, named_axes +from .axis_names import OPEN_AXIS +from .metadata import TiffMetadata, named_axes from .validation import ValidationIssue, error, plural, warning if TYPE_CHECKING: # the mapping class lives in .axes, which imports this module diff --git a/src/morphobricks_preprocessing/axis_names.py b/src/morphobricks_preprocessing/axis_names.py new file mode 100644 index 00000000..e0916829 --- /dev/null +++ b/src/morphobricks_preprocessing/axis_names.py @@ -0,0 +1,41 @@ +"""Which axes a file's metadata names, for images and masks alike (contract §16, D-d). + +One rule (plan-structure audit D3): a dimension is named when ``tifffile`` (or OME-NGFF) +gives it a letter from the file's metadata. Left open (:data:`OPEN_AXIS`) are placeholder +axes (:data:`PLACEHOLDER_AXES`), and, in a plain stack of pages without axis metadata, the +``Y`` and ``X`` too: they only say where each page's rows and columns are stored. +:func:`.metadata.named_axes` applies it to images, :mod:`.masks` to masks. +""" + +from __future__ import annotations + +PLACEHOLDER_AXES = frozenset("IQ") +"""``tifffile``'s letters for a dimension whose meaning the file does not give: ``I`` (a +plain stack of pages) and ``Q`` (a "shaped" file without axes, or an OME-NGFF axis of +another type). They are never metadata: the user gives such a dimension its role.""" + +OPEN_AXIS = "?" +"""Marks a dimension the metadata does not name in :func:`named_letters`.""" + + +def named_letters(axes: str, from_metadata: bool) -> str: + """Return ``axes`` with every dimension the metadata does not name as ``?``. + + Parameters + ---------- + axes + The axes ``tifffile`` (or OME-NGFF) reports, one letter per dimension. + from_metadata + Whether the axes come from axis metadata (OME, ImageJ, shaped with axes, OME-NGFF), + :func:`.metadata.series_axes_from_metadata`; False for a plain stack of pages. + + Returns + ------- + str + One character per dimension, e.g. ``"CYX"``, ``"?YX"`` or ``"???"``. + """ + plain_stack = not from_metadata and bool(PLACEHOLDER_AXES & set(axes)) + return "".join( + OPEN_AXIS if axis in PLACEHOLDER_AXES or (plain_stack and axis in "YX") else axis + for axis in axes + ) diff --git a/src/morphobricks_preprocessing/cli.py b/src/morphobricks_preprocessing/cli.py index 1991abd8..0a4893a1 100644 --- a/src/morphobricks_preprocessing/cli.py +++ b/src/morphobricks_preprocessing/cli.py @@ -23,6 +23,7 @@ import re import sys from collections.abc import Sequence +from dataclasses import replace from datetime import date from pathlib import Path @@ -56,6 +57,7 @@ scan_masks, write_mask_report, ) +from .masks import MaskInventory, mask_axes_option from .planes import DEFAULT_MEMORY_LIMIT from .report import ( axis_check_summary, @@ -69,7 +71,7 @@ preflight_summary, run_summary, ) -from .run import preflight, run_experiment +from .run import Preflight, preflight, run_experiment from .validation import has_errors _UNITS = { @@ -233,6 +235,11 @@ def main(argv: Sequence[str] | None = None) -> int: args = build_parser().parse_args(argv) handler = _attach_log(args.log) if getattr(args, "log", None) else None try: + try: + args.mask_axes = mask_axes_option(args.mask_axes) + except ValueError as exc: + print(exc) + return 1 return {"check": _check, "run": _run, "masks": _masks}[args.command](args) finally: if handler is not None: @@ -241,6 +248,7 @@ def main(argv: Sequence[str] | None = None) -> int: def _common(parser: argparse.ArgumentParser) -> None: + """Add the experiment and the options every command shares.""" parser.add_argument("experiment", help="the experiment folder") parser.add_argument( "--output", @@ -260,10 +268,35 @@ def _common(parser: argparse.ArgumentParser) -> None: help="masks folder: one subfolder per mask type, each mirroring the experiment's " "folders, outside the experiment; the label images of OME-Zarr images are added", ) + _mask_axes_option(parser) + + +def _mask_axes_option(parser: argparse.ArgumentParser) -> None: + """Add ``--mask-axes TYPE=ROLES``, stated once per mask type (design v0.4 §3).""" + parser.add_argument( + "--mask-axes", + action="append", + default=[], + metavar="TYPE=ROLES", + help="the axes of a mask type whose files do not name them, one role (Y, X, Z or T) " + "per axis as stored: Cells=TYX for plain stacks of time points, Cells=YX for one 2D " + "mask shared by every plane of its image (repeatable)", + ) + + +def _preflight(args: argparse.Namespace) -> Preflight: + """Run :func:`preflight` with the options every command shares.""" + return preflight( + args.experiment, + args.output, + exclude=args.exclude, + masks_folder=args.masks, + mask_axes=args.mask_axes, + ) def _check(args: argparse.Namespace) -> int: - pre = preflight(args.experiment, args.output, exclude=args.exclude, masks_folder=args.masks) + pre = _preflight(args) print(preflight_summary(pre)) if pre.discovery.leaves: print() @@ -327,13 +360,8 @@ def _masks(args: argparse.Namespace) -> int: except ValueError as exc: print(exc) return 1 - pre = preflight( - args.experiment, - args.output, - exclude=args.exclude, - masks_folder=args.masks, - ome_zarr_labels=True, - ) + pre = _preflight(args) + pre = replace(pre, masks=pre.masks or MaskInventory(folder=None)) # none found print(mask_summary(pre.masks)) blocking = [*pre.discovery.issues, *pre.exclusion_issues, *pre.masks.issues] if blocking: @@ -365,7 +393,7 @@ def progress(done: int, total: int, path: str) -> None: def _run(args: argparse.Namespace) -> int: # noqa: PLR0915 - pre = preflight(args.experiment, args.output, exclude=args.exclude, masks_folder=args.masks) + pre = _preflight(args) axes = reference_axes(pre.metadata.files) if axes is None: print(format_issues(pre.issues) or "No readable images were found.") @@ -385,10 +413,6 @@ def _run(args: argparse.Namespace) -> int: # noqa: PLR0915 saturation_limit=args.saturation_limit, min_images=args.min_images, ) - except ValueError as exc: - print(exc) - return 1 - try: histogram_settings = HistogramSettings( histogram_range=args.histogram_range, robust_quantiles=tuple(args.robust_quantiles) ) @@ -435,6 +459,7 @@ def progress(done: int, total: int, path: str) -> None: progress=progress, design=design, masks_folder=args.masks, + mask_axes=args.mask_axes, image_type=image_type, intensity_settings=intensity_settings, histogram_settings=histogram_settings, diff --git a/src/morphobricks_preprocessing/discovery.py b/src/morphobricks_preprocessing/discovery.py index 36c8020f..508babd8 100644 --- a/src/morphobricks_preprocessing/discovery.py +++ b/src/morphobricks_preprocessing/discovery.py @@ -24,14 +24,15 @@ import logging import os -import re -from collections.abc import Iterable, Mapping, Sequence +from collections.abc import Iterable, Sequence from dataclasses import dataclass -from pathlib import Path, PurePosixPath +from pathlib import Path import numpy as np from morphobricks_normalization.records import LEVELS, SINGLE +from .image_names import position_id +from .mask_names import mask_name_issues from .validation import ValidationIssue, agree, error, has_errors, plural, warning from .zarr_input import is_zarr_name @@ -48,14 +49,6 @@ ROOT = "." -# Image name endings removed to give the position identifier, longest first, so -# that "img_01.ome.tif" and "img_01.ome.zarr" both give "img_01". -IMAGE_SUFFIXES = (".ome.tiff", ".ome.tif", ".ome.zarr", ".tiff", ".tif", ".zarr") -# The extensions position_id removes: the images this brick reads, then the other formats -# MorphoBricks bricks read (CellPatcher: PNG, JPEG, BMP, GIF, NumPy), so every brick names a -# position by this one function (MorphoBricks decision D85, 2026-10-29). Naming only: the -# images discovery accepts are still IMAGE_SUFFIXES. -NAME_SUFFIXES = (*IMAGE_SUFFIXES, ".png", ".jpeg", ".jpg", ".bmp", ".gif", ".npy") FOV_KEY = (*LEVELS, "position_id") """The columns that identify one position (field of view) in every MorphoBricks table: @@ -67,20 +60,6 @@ as a missing value (MorphoBricks decision D-e).""" -MASK_NAME_SUFFIXES = ("_mask", "_labels", "_label", "_seg", "_segmentation") -"""Image name endings that read as a mask of another image (``f1_mask.tif`` beside ``f1.tif``), -compared in any case: CellPatcher's former default mask suffixes (MorphoBricks decision D-i -moved this screen here from MorphoRelate).""" - -MASK_FOLDER_WORDS = frozenset( - {"mask", "masks", "label", "labels", "seg", "segmentation", "segmentations"} -) -"""Words that make a folder name read as a masks folder (``masks``, ``ctrl_masks``, -``CellLabels``), compared in any case after splitting the name into words.""" - -_WORDS = re.compile(r"[A-Z]+(?![a-z])|[A-Z]?[a-z]+|\d+") - - class IdentityError(ValueError): """Raised for a label, position identifier or plane index that cannot identify anything.""" @@ -230,48 +209,6 @@ def is_ome_zarr_folder(entry: os.DirEntry) -> bool: return entry.is_dir() and is_zarr_name(entry.name) -def position_id(image: str) -> str: - """Return the position identifier of an image: its name without the extension. - - Every image is exactly one position, because files holding several - positions, scenes or tiles are refused, so the file name identifies the - position within its technical repeat. The full position (field of view) key - is ``condition``, ``biological_repeat``, ``technical_repeat`` and this - identifier (contract §1). The extension is ``.tif``, ``.tiff`` or ``.zarr`` - (any case), together with a preceding ``.ome``, or one of the other image - formats MorphoBricks bricks read (``.png``, ``.jpg``, ``.jpeg``, ``.bmp``, - ``.gif``, ``.npy``; :data:`NAME_SUFFIXES`), so every brick names a position - with this one function (MorphoBricks D85). - - Parameters - ---------- - image - The image's path relative to the experiment root, or its name. - - Returns - ------- - str - For example ``"img_01"`` for ``"ctrl/rep1/img_01.ome.tif"``. - - Raises - ------ - ValueError - If the name is only an extension (``".tif"``, ``".ome.zarr"``): it has - no name to identify a position by. The longest extension decides, so - ``".ome.tif"`` is refused rather than read as ``".ome"``. - """ - name = image.replace("\\", "/").rsplit("/", 1)[-1] - lowered = name.lower() - for suffix in NAME_SUFFIXES: - if lowered.endswith(suffix): - if len(name) == len(suffix): - raise ValueError( - f"{image!r} is only an extension: it has no name to identify a position by" - ) - return name[: -len(suffix)] - return name - - def default_output_folder(experiment_root: str | os.PathLike) -> Path: """Return the default output folder, a sibling of the experiment folder. @@ -415,180 +352,6 @@ def discover_experiment( return DiscoveryResult(root=root, leaves=tuple(walker.leaves), issues=issues) -def mask_name_issues( - leaves: Iterable[LeafFolder], mask_names: Sequence[str] = () -) -> list[ValidationIssue]: - """Warn about masks in the experiment folder, which would be analysed as images. - - A mask is never a position, and masks belong in a masks folder outside the experiment - (contract §15). Two screens over every image of the experiment (all leaf folders - together), run by :func:`discover_experiment` (MorphoBricks decision D-i, moved from - MorphoRelate's ``project_image_names``): - - - ``image_named_as_mask``: an image named after another image's ``position_id``, then - one of :data:`MASK_NAME_SUFFIXES` (or ``_`` and one of ``mask_names``), in any case: - ``f1_mask.tif`` beside ``f1.tif``. The longest such position owns it, an image of the - same folder first. Any other name is an image: ``A1_2.tif`` beside ``A1.tif``, - ``f1_DNA.tif`` beside ``f1.tif``. - - ``mask_folder_in_experiment``: a folder named like a masks folder (a word of - :data:`MASK_FOLDER_WORDS` in its name: ``masks``, ``ctrl_masks``, ``CellLabels``) - holding images named as masks of the other images, with any name - (```` or ``_``). - - Both are warnings: such files may be images that only look like masks. - - Parameters - ---------- - leaves - The experiment's leaf folders (:attr:`DiscoveryResult.leaves`). - mask_names - Mask names the caller expects (``"cells"`` for ``f1_cells.tif``), screened besides - :data:`MASK_NAME_SUFFIXES`; :func:`discover_experiment` gives none. - - Returns - ------- - list of ValidationIssue - The warnings, or nothing. - """ - leaves = list(leaves) - images = [path for leaf in leaves for path in leaf.images] - stems = _stems(images) - suffixes = {suffix.lower() for suffix in MASK_NAME_SUFFIXES} - suffixes |= {f"_{name.lower()}" for name in mask_names} - named = images_named_as_masks(leaves, sorted(suffixes)) - by_folder: dict[str, list[tuple[str, str]]] = {} - for relative in images: - folder = mask_folder_part(relative) - if folder is None: - continue - stem = position_id(relative).lower() - owner = stems[stem][0] if stem in stems else _mask_owner(stem, stems, "", None) - if owner is not None: - by_folder.setdefault(folder, []).append((relative, owner)) - issues = [] - if named: - issues.append( - warning( - "image_named_as_mask", - f"{plural(len(named), 'image')} " - f"{agree(named, 'is', 'are')} named as a mask of another " - "image: that image's name followed by " - + ", ".join(sorted(suffixes)) - + f" ({_few([f'{file} (of {owner})' for file, owner in named])}). A mask is " - "never a position, so these files are analysed as images.", - [file for file, _ in named], - "If they are masks, move them into a masks folder outside the experiment " - "folder, laid out like the experiment (//" - "_.tif). If they are images, nothing needs to change.", - ) - ) - if by_folder: - found = [pair for pairs in by_folder.values() for pair in pairs] - issues.append( - warning( - "mask_folder_in_experiment", - f"{plural(len(by_folder), 'folder')} inside the experiment " - f"{'is' if len(by_folder) == 1 else 'are'} named like a masks folder and " - f"{'holds' if len(by_folder) == 1 else 'hold'} files named as masks of the " - f"images ({_few([f'{file} (of {owner})' for file, owner in found])}). They are " - "analysed as images, with the folder as a condition or repeat.", - sorted(by_folder), - "If they are masks, move the folder out of the experiment folder and give it as " - "the masks folder. If they are images, rename the folder.", - ) - ) - return issues - - -def images_named_as_masks( - leaves: Iterable[LeafFolder], suffixes: Sequence[str] = MASK_NAME_SUFFIXES -) -> list[tuple[str, str]]: - """Return the images named as a mask of another image, each with that image. - - The detection of :func:`mask_name_issues`'s ``image_named_as_mask``, for a brick that acts - on the files itself (MorphoRelate leaves out of its images those named with a mask name - the user declared): an image is named as a mask of another when its ``position_id`` is - that image's ``position_id`` followed by one of ``suffixes``, compared in lower case, - anywhere in the experiment. The longest such position owns it, an image of the same - folder first. Images inside a folder named like a masks folder (:func:`mask_folder_part`) - are left out, as owners too. - - Parameters - ---------- - leaves - The experiment's leaf folders (:attr:`DiscoveryResult.leaves`); a caller leaves out - images it has already set aside by giving leaves without them. - suffixes - The name endings, each with its ``_`` (``"_mask"``, ``"_cells"``); by default - :data:`MASK_NAME_SUFFIXES`. - - Returns - ------- - list of tuple - ``(image, owner)`` relative paths, in the leaves' order. - """ - images = [path for leaf in leaves for path in leaf.images] - regular = [path for path in images if mask_folder_part(path) is None] - stems = _stems(images) - wanted = {suffix.lower() for suffix in suffixes} - named = [] - for relative in regular: - path = PurePosixPath(relative) - owner = _mask_owner(position_id(path.name).lower(), stems, str(path.parent), wanted) - if owner is not None: - named.append((relative, owner)) - return named - - -def _stems(images: Sequence[str]) -> dict[str, list[str]]: - """Return the images outside masks folders by their ``position_id`` in lower case.""" - stems: dict[str, list[str]] = {} - for relative in images: - if mask_folder_part(relative) is None: - stems.setdefault(position_id(relative).lower(), []).append(relative) - return stems - - -def mask_folder_part(relative: str) -> str | None: - """Return the first folder of an image's path named like a masks folder, or None. - - For example ``"masks"`` for ``"masks/ctrl/rep1/f1.tif"``; a name counts when one of its - words (``CellLabels`` is ``Cell`` and ``Labels``) is in :data:`MASK_FOLDER_WORDS`. - """ - parts = PurePosixPath(relative).parts[:-1] - for index, part in enumerate(parts): - if {word.lower() for word in _WORDS.findall(part)} & MASK_FOLDER_WORDS: - return "/".join(parts[: index + 1]) - return None - - -def _mask_owner( - stem: str, stems: Mapping[str, list[str]], folder: str, suffixes: set[str] | None -) -> str | None: - """Return the image whose ``position_id`` is followed in ``stem`` by ``_`` and a suffix. - - Any suffix when ``suffixes`` is None. ``stem`` and the keys of ``stems`` are lower case. - The longest such position wins, an image of ``folder`` first; None when there is none. - """ - keys = [ - key - for key in stems - if key != stem - and stem.startswith(key + "_") - and (suffixes is None or stem[len(key) :] in suffixes) - ] - if not keys: - return None - owners = stems[max(keys, key=len)] - near = [owner for owner in owners if str(PurePosixPath(owner).parent) == folder] - return (near or owners)[0] - - -def _few(items: list[str], shown: int = 5) -> str: - more = f" and {len(items) - shown} more" if len(items) > shown else "" - return ", ".join(items[:shown]) + more - - class _Walker: """Collects leaf folders and problem paths during one walk.""" diff --git a/src/morphobricks_preprocessing/image_names.py b/src/morphobricks_preprocessing/image_names.py new file mode 100644 index 00000000..fe6c56f2 --- /dev/null +++ b/src/morphobricks_preprocessing/image_names.py @@ -0,0 +1,59 @@ +"""An image's position identifier, from its name (contract §1, MorphoBricks D02, D85). + +Moved from ``discovery.py`` (PP-5), so that the mask-name screen can use it. +""" + +from __future__ import annotations + +# Image name endings removed to give the position identifier, longest first, so +# that "img_01.ome.tif" and "img_01.ome.zarr" both give "img_01". +IMAGE_SUFFIXES = (".ome.tiff", ".ome.tif", ".ome.zarr", ".tiff", ".tif", ".zarr") + + +# The extensions position_id removes: the images this brick reads, then the other formats +# MorphoBricks bricks read (CellPatcher: PNG, JPEG, BMP, GIF, NumPy), so every brick names a +# position by this one function (MorphoBricks decision D85, 2026-10-29). Naming only: the +# images discovery accepts are still IMAGE_SUFFIXES. +NAME_SUFFIXES = (*IMAGE_SUFFIXES, ".png", ".jpeg", ".jpg", ".bmp", ".gif", ".npy") + + +def position_id(image: str) -> str: + """Return the position identifier of an image: its name without the extension. + + Every image is exactly one position, because files holding several + positions, scenes or tiles are refused, so the file name identifies the + position within its technical repeat. The full position (field of view) key + is ``condition``, ``biological_repeat``, ``technical_repeat`` and this + identifier (contract §1). The extension is ``.tif``, ``.tiff`` or ``.zarr`` + (any case), together with a preceding ``.ome``, or one of the other image + formats MorphoBricks bricks read (``.png``, ``.jpg``, ``.jpeg``, ``.bmp``, + ``.gif``, ``.npy``; :data:`NAME_SUFFIXES`), so every brick names a position + with this one function (MorphoBricks D85). + + Parameters + ---------- + image + The image's path relative to the experiment root, or its name. + + Returns + ------- + str + For example ``"img_01"`` for ``"ctrl/rep1/img_01.ome.tif"``. + + Raises + ------ + ValueError + If the name is only an extension (``".tif"``, ``".ome.zarr"``): it has + no name to identify a position by. The longest extension decides, so + ``".ome.tif"`` is refused rather than read as ``".ome"``. + """ + name = image.replace("\\", "/").rsplit("/", 1)[-1] + lowered = name.lower() + for suffix in NAME_SUFFIXES: + if lowered.endswith(suffix): + if len(name) == len(suffix): + raise ValueError( + f"{image!r} is only an extension: it has no name to identify a position by" + ) + return name[: -len(suffix)] + return name diff --git a/src/morphobricks_preprocessing/manifest.py b/src/morphobricks_preprocessing/manifest.py index 2da3dd0b..8b398f13 100644 --- a/src/morphobricks_preprocessing/manifest.py +++ b/src/morphobricks_preprocessing/manifest.py @@ -36,7 +36,7 @@ from ._io import write_text_atomic from .axes import AxisMapping -from .discovery import position_id +from .image_names import position_id from .narrow_ranges import NarrowRange from .recorded_sizes import RecordedSize, sizes_from_data, sizes_to_data @@ -47,6 +47,9 @@ LEGACY_MANIFEST_VERSION = "1" """Manifests written before v0.4.0: read for reporting, never trusted (no row is reused).""" +LABELS_GROUP = "labels" +"""An OME-Zarr image's group of label images (masks): outside the image's fingerprint.""" + class ManifestFormatError(ValueError): """Raised when a file is not a run manifest this version can read.""" @@ -118,7 +121,7 @@ class FileEntry: read in blocks. For information only: the rows are the same either way. position_id The image's position identifier, its name without the extension - (:func:`~morphobricks_preprocessing.discovery.position_id`); None in + (:func:`~morphobricks_preprocessing.image_names.position_id`); None in manifests written before it was recorded. roles The role of each of the file's axes (``"X"``, ``"Y"``, ``"Z"``, ``"C"``, @@ -400,14 +403,18 @@ def source_fingerprint(path: str | os.PathLike) -> tuple[int, int]: A TIFF file gives its own size and time. An OME-Zarr folder gives the total size of its files and the time of the newest one, so rewriting any chunk or - metadata file counts as a change. + metadata file counts as a change. Its ``labels`` group is left out: label + images are masks, fingerprinted one by one (design v0.4 §2), so adding or + changing one does not change the image. """ path = Path(path) if not path.is_dir(): stat = path.stat() return stat.st_size, stat.st_mtime_ns size, newest = 0, path.stat().st_mtime_ns - for folder, _, names in os.walk(path): + for folder, subfolders, names in os.walk(path): + if Path(folder) == path and LABELS_GROUP in subfolders: + subfolders.remove(LABELS_GROUP) for name in names: stat = (Path(folder) / name).stat() size += stat.st_size diff --git a/src/morphobricks_preprocessing/mask_checks.py b/src/morphobricks_preprocessing/mask_checks.py index 8cbc4992..d39d8f6b 100644 --- a/src/morphobricks_preprocessing/mask_checks.py +++ b/src/morphobricks_preprocessing/mask_checks.py @@ -12,7 +12,10 @@ from collections.abc import Iterable, Mapping from .axes import AxisMapping, AxisReview -from .masks import _TEXTS, MaskInventory, _yx, mask_plane_issues +from .masks import MaskInventory, mask_plane_issues +from .masks.checks import _TEXTS +from .masks.layout import _yx +from .masks.stated import size_code from .metadata import MetadataReport, TiffMetadata from .validation import ValidationIssue, error @@ -47,11 +50,12 @@ def mask_size_issues( Returns ------- list of ValidationIssue - The error ``mask_size_mismatch`` naming the masks that do not fit, or nothing. + The error ``mask_layout`` naming the masks stored with Y and X the other way round + (never transposed), and ``mask_size_mismatch`` naming the other masks that do not fit. """ headers = {meta.relative_path: meta for meta in images} skip = set(skip) - mismatched = [] + mismatched: dict[str, list[str]] = {} for mask in inventory.files: mapping = file_mappings.get(mask.image) meta = headers.get(mask.image) @@ -60,12 +64,17 @@ def mask_size_issues( y, x = mapping.index_of("Y"), mapping.index_of("X") if y is None or x is None: continue - if _yx(mask.axes, mask.shape) != (meta.shape[y], meta.shape[x]): - mismatched.append(mask.path) - if not mismatched: - return [] - message, action = _TEXTS["mask_size_mismatch"] - return [error("mask_size_mismatch", message, sorted(mismatched), action)] + mask_yx, image_yx = _yx(mask.axes, mask.shape), (meta.shape[y], meta.shape[x]) + if mask_yx != image_yx: + mismatched.setdefault(size_code(mask_yx, image_yx), []).append(mask.path) + return [ + error(code, _TEXTS[code][0], sorted(paths), _TEXTS[code][1]) + for code, paths in sorted(mismatched.items()) + ] + + +SIZE_CODES = frozenset({"mask_size_mismatch", "mask_layout"}) +"""The codes of a mask whose Y and X are not its image's, which the roles decide.""" def mask_issues( @@ -96,7 +105,7 @@ def mask_issues( """ skip = review.axis_error_files return [ - *(i for i in masks.issues if i.code != "mask_size_mismatch"), + *(i for i in masks.issues if i.code not in SIZE_CODES), *mask_size_issues(masks, metadata.files, review.file_mappings, skip=skip), *mask_plane_issues(masks, metadata.files, review.file_mappings, skip=skip), ] diff --git a/src/morphobricks_preprocessing/mask_names.py b/src/morphobricks_preprocessing/mask_names.py new file mode 100644 index 00000000..563790f5 --- /dev/null +++ b/src/morphobricks_preprocessing/mask_names.py @@ -0,0 +1,224 @@ +"""Files in the experiment named like masks: screened, and blocked as images (contract §1). + +Moved from ``discovery.py`` (PP-5). +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping, Sequence +from pathlib import PurePosixPath +from typing import TYPE_CHECKING + +from .image_names import position_id +from .validation import ValidationIssue, agree, error, plural, warning + +if TYPE_CHECKING: + from .discovery import LeafFolder + +MASK_NAME_SUFFIXES = ( + "_mask", + "_masks", + "_cp_masks", + "_labels", + "_label", + "_seg", + "_segmentation", +) +"""Image name endings that read as a mask of another image (``f1_mask.tif`` beside ``f1.tif``), +compared in any case: CellPatcher's former default mask suffixes (MorphoBricks decision D-i +moved this screen here from MorphoRelate), and Cellpose's ``_cp_masks`` and ``_masks`` +(design v0.4 §2).""" + + +MASK_FOLDER_WORDS = frozenset( + {"mask", "masks", "label", "labels", "seg", "segmentation", "segmentations"} +) +"""Words that make a folder name read as a masks folder (``masks``, ``ctrl_masks``, +``CellLabels``), compared in any case after splitting the name into words.""" + + +MASK_NAME_CODES = frozenset({"image_named_as_mask", "mask_folder_in_experiment"}) +"""The codes of :func:`mask_name_issues`.""" + +_WORDS = re.compile(r"[A-Z]+(?![a-z])|[A-Z]?[a-z]+|\d+") + + +def mask_name_issues( + leaves: Iterable[LeafFolder], mask_names: Sequence[str] = (), excluded: Iterable[str] = () +) -> list[ValidationIssue]: + """Refuse images named as masks, and warn about masks folders inside the experiment. + + A mask is never a position, and masks belong in a masks folder outside the experiment + (contract §15). Two screens over every image of the experiment (all leaf folders + together), run by :func:`discover_experiment` (MorphoBricks decision D-i, moved from + MorphoRelate's ``project_image_names``): + + - ``image_named_as_mask``: an image named after another image's ``position_id``, then + one of :data:`MASK_NAME_SUFFIXES` (or ``_`` and one of ``mask_names``), in any case: + ``f1_mask.tif`` beside ``f1.tif``. The longest such position owns it, an image of the + same folder first. Any other name is an image: ``A1_2.tif`` beside ``A1.tif``, + ``f1_DNA.tif`` beside ``f1.tif``. + - ``mask_folder_in_experiment``: a folder named like a masks folder (a word of + :data:`MASK_FOLDER_WORDS` in its name: ``masks``, ``ctrl_masks``, ``CellLabels``) + holding images named as masks of the other images, with any name + (```` or ``_``). + + ``image_named_as_mask`` is an error (design v0.4 §3): a mask is never analysed as an + image, so the file is renamed or excluded. ``mask_folder_in_experiment`` is a warning: + the folder may hold images that only look like masks. + + Parameters + ---------- + leaves + The experiment's leaf folders (:attr:`DiscoveryResult.leaves`). + mask_names + Mask names the caller expects (``"cells"`` for ``f1_cells.tif``), screened besides + :data:`MASK_NAME_SUFFIXES`; :func:`discover_experiment` gives none. + excluded + Images the user excluded: not reported (they still own the masks named after them). + + Returns + ------- + list of ValidationIssue + The issues, or nothing. + """ + excluded = set(excluded) + leaves = list(leaves) + images = [path for leaf in leaves for path in leaf.images] + stems = _stems(images) + suffixes = {suffix.lower() for suffix in MASK_NAME_SUFFIXES} + suffixes |= {f"_{name.lower()}" for name in mask_names} + named = images_named_as_masks(leaves, sorted(suffixes)) + named = [(file, owner) for file, owner in named if file not in excluded] + by_folder: dict[str, list[tuple[str, str]]] = {} + for relative in images: + folder = mask_folder_part(relative) + if folder is None or relative in excluded: + continue + stem = position_id(relative).lower() + owner = stems[stem][0] if stem in stems else _mask_owner(stem, stems, "", None) + if owner is not None: + by_folder.setdefault(folder, []).append((relative, owner)) + issues = [] + if named: + issues.append( + error( + "image_named_as_mask", + f"{plural(len(named), 'image')} " + f"{agree(named, 'is', 'are')} named as a mask of another " + "image: that image's name followed by " + + ", ".join(sorted(suffixes)) + + f" ({_few([f'{file} (of {owner})' for file, owner in named])}). A mask is " + "never a position, so it is never analysed as an image.", + [file for file, _ in named], + "If they are masks, move them into a masks folder outside the experiment " + "folder, laid out like the experiment (//" + "_.tif), or exclude them. If they are images, rename them.", + ) + ) + if by_folder: + found = [pair for pairs in by_folder.values() for pair in pairs] + issues.append( + warning( + "mask_folder_in_experiment", + f"{plural(len(by_folder), 'folder')} inside the experiment " + f"{'is' if len(by_folder) == 1 else 'are'} named like a masks folder and " + f"{'holds' if len(by_folder) == 1 else 'hold'} files named as masks of the " + f"images ({_few([f'{file} (of {owner})' for file, owner in found])}). They are " + "analysed as images, with the folder as a condition or repeat.", + sorted(by_folder), + "If they are masks, move the folder out of the experiment folder and give it as " + "the masks folder. If they are images, rename the folder.", + ) + ) + return issues + + +def images_named_as_masks( + leaves: Iterable[LeafFolder], suffixes: Sequence[str] = MASK_NAME_SUFFIXES +) -> list[tuple[str, str]]: + """Return the images named as a mask of another image, each with that image. + + The detection of :func:`mask_name_issues`'s ``image_named_as_mask``, for a brick that acts + on the files itself (MorphoRelate leaves out of its images those named with a mask name + the user declared): an image is named as a mask of another when its ``position_id`` is + that image's ``position_id`` followed by one of ``suffixes``, compared in lower case, + anywhere in the experiment. The longest such position owns it, an image of the same + folder first. Images inside a folder named like a masks folder (:func:`mask_folder_part`) + are left out, as owners too. + + Parameters + ---------- + leaves + The experiment's leaf folders (:attr:`DiscoveryResult.leaves`); a caller leaves out + images it has already set aside by giving leaves without them. + suffixes + The name endings, each with its ``_`` (``"_mask"``, ``"_cells"``); by default + :data:`MASK_NAME_SUFFIXES`. + + Returns + ------- + list of tuple + ``(image, owner)`` relative paths, in the leaves' order. + """ + images = [path for leaf in leaves for path in leaf.images] + regular = [path for path in images if mask_folder_part(path) is None] + stems = _stems(images) + wanted = {suffix.lower() for suffix in suffixes} + named = [] + for relative in regular: + path = PurePosixPath(relative) + owner = _mask_owner(position_id(path.name).lower(), stems, str(path.parent), wanted) + if owner is not None: + named.append((relative, owner)) + return named + + +def _stems(images: Sequence[str]) -> dict[str, list[str]]: + """Return the images outside masks folders by their ``position_id`` in lower case.""" + stems: dict[str, list[str]] = {} + for relative in images: + if mask_folder_part(relative) is None: + stems.setdefault(position_id(relative).lower(), []).append(relative) + return stems + + +def mask_folder_part(relative: str) -> str | None: + """Return the first folder of an image's path named like a masks folder, or None. + + For example ``"masks"`` for ``"masks/ctrl/rep1/f1.tif"``; a name counts when one of its + words (``CellLabels`` is ``Cell`` and ``Labels``) is in :data:`MASK_FOLDER_WORDS`. + """ + parts = PurePosixPath(relative).parts[:-1] + for index, part in enumerate(parts): + if {word.lower() for word in _WORDS.findall(part)} & MASK_FOLDER_WORDS: + return "/".join(parts[: index + 1]) + return None + + +def _mask_owner( + stem: str, stems: Mapping[str, list[str]], folder: str, suffixes: set[str] | None +) -> str | None: + """Return the image whose ``position_id`` is followed in ``stem`` by ``_`` and a suffix. + + Any suffix when ``suffixes`` is None. ``stem`` and the keys of ``stems`` are lower case. + The longest such position wins, an image of ``folder`` first; None when there is none. + """ + keys = [ + key + for key in stems + if key != stem + and stem.startswith(key + "_") + and (suffixes is None or stem[len(key) :] in suffixes) + ] + if not keys: + return None + owners = stems[max(keys, key=len)] + near = [owner for owner in owners if str(PurePosixPath(owner).parent) == folder] + return (near or owners)[0] + + +def _few(items: list[str], shown: int = 5) -> str: + more = f" and {len(items) - shown} more" if len(items) > shown else "" + return ", ".join(items[:shown]) + more diff --git a/src/morphobricks_preprocessing/mask_record.py b/src/morphobricks_preprocessing/mask_record.py index d4f93a7c..a66b11b5 100644 --- a/src/morphobricks_preprocessing/mask_record.py +++ b/src/morphobricks_preprocessing/mask_record.py @@ -3,8 +3,10 @@ Every mask entry adds to what :meth:`.masks.MaskInventory.to_manifest` gives its fingerprint (``size``, ``mtime_ns``), so a brick can tell a mask changed since the run, and how it is read: ``roles`` (one per axis of ``axes``), ``axes_from`` (``"header"``: the -mask's metadata names them) and ``shared`` (a 2-D mask stated as shared by every plane of -its image; never without a statement). +mask's metadata names them; ``"stated"``: the user stated them for its mask type) and +``shared`` (a 2-D mask stated as shared by every plane of its image; never without a +statement). A label image of an OME-Zarr image has its own fingerprint, left out of its +image's (:func:`.manifest.source_fingerprint`). """ from __future__ import annotations @@ -30,7 +32,9 @@ def mask_manifest(inventory: MaskInventory, experiment_root: str | os.PathLike) ------- dict :meth:`.masks.MaskInventory.to_manifest` with, per file, ``size``, ``mtime_ns``, - ``roles``, ``axes_from`` and ``shared``. + ``roles``, ``axes_from`` and ``shared``. ``axes`` are the mask's axes as stored + (its header's letters, ``"IYX"`` for a plain stack), ``roles`` what they are read as + (``"TYX"`` when stated), so a brick reads the mask as stored, never reordered. """ data = inventory.to_manifest() by_path = {mask.path: mask for mask in inventory.files} @@ -38,11 +42,12 @@ def mask_manifest(inventory: MaskInventory, experiment_root: str | os.PathLike) mask = by_path[entry["path"]] fingerprint = Fingerprint.of(_mask_path(mask, inventory.folder, Path(experiment_root))) entry.update( + axes=mask.header_axes or mask.axes, size=fingerprint.size, mtime_ns=fingerprint.mtime_ns, roles=mask.axes, - axes_from="header", - shared=False, + axes_from=mask.axes_from or "header", + shared=mask.shared, ) return data diff --git a/src/morphobricks_preprocessing/masks.py b/src/morphobricks_preprocessing/masks.py deleted file mode 100644 index 72260e92..00000000 --- a/src/morphobricks_preprocessing/masks.py +++ /dev/null @@ -1,878 +0,0 @@ -"""Find the masks of every image and check them from their headers (``docs/contract.md`` §15). - -Masks (segmentations: label images of cells, nuclei, ...) are kept in a masks folder outside -the experiment, one subfolder per mask type, each mirroring the experiment's folders:: - - /////.tif - -A mask belongs to the image with the same position (same leaf folder, same ``position_id``, -contract §1). The type may also follow the image name instead of being a folder:: - - ////_.tif - -and the two layouts may be mixed. Its file may be a TIFF (``.tif``, ``.tiff``, any case, with -an optional ``.ome``). The label images of an OME-Zarr image (its OME-NGFF ``labels`` group) -are masks too, of the type named by the label image, when the ``zarr`` extra is installed. - -Any number of mask types is accepted; each is named by its folder, by the suffix after the -image name, or by its label image, and two -names may not differ only in case. Only headers are read here, never pixels: a mask has the -image's Y and X sizes and integer or boolean pixels. Its other dimensions (length-1 axes -aside; Y and X are kept whatever their length, so a mask one pixel high is a 2D ``YX`` mask) -must be labelled depth (Z) or time (T) in its own metadata, never guessed, and every -mask of one image has the same depth slices and time points. Whether those are the image's -planes needs the confirmed axis roles: :func:`mask_plane_issues`, run by -:func:`.run.run_experiment` (issue #36). Reading the pixels and relating the masks to each -other is :mod:`.mask_scan`. -""" - -from __future__ import annotations - -import logging -import math -import os -from collections.abc import Iterable, Mapping -from dataclasses import dataclass -from pathlib import Path -from types import MappingProxyType - -import numpy as np -import tifffile - -from .axes import AxisMapping -from .discovery import DiscoveryResult, is_system_name, is_tiff_name, position_id -from .metadata import TiffMetadata, named_axes -from .validation import ValidationIssue, agree, error, has_errors, plural, warning -from .zarr_input import ( - NgffError, - ZarrUnavailableError, - is_zarr_name, - ngff_label_names, - open_ngff_labels, -) - -logger = logging.getLogger(__name__) - -FOLDER = "folder" -"""Source of a mask kept as a TIFF file in the masks folder.""" - -OME_ZARR = "ome-zarr" -"""Source of a mask kept as a label image inside an OME-Zarr image.""" - -PLANE_AXES = ("Z", "T") -"""The dimensions a mask may have besides Y and X: depth slices and time points.""" - -_YX = frozenset("YX") -"""The axes of a mask plane, kept by the header and the readers even when of length 1.""" - - -@dataclass(frozen=True) -class MaskFile: - """One mask of one image. - - Attributes - ---------- - image - The image's path relative to the experiment root. - mask_type - The mask type: its folder name in the masks folder, or the label image's name. - path - The mask's path: relative to the masks folder for a TIFF mask; for a label image of - an OME-Zarr image, ``/labels/`` relative to the experiment root. - source - :data:`FOLDER` or :data:`OME_ZARR`. - shape - Shape of the mask without its length-1 axes other than Y and X, ``(Y, X)`` for a 2D - mask (``(1, 512)`` for a mask one pixel high). - dtype - Pixel type, or None when the header could not be read. - axes - One letter per entry of ``shape``, from the mask's own metadata: ``"YX"`` for a 2D - mask, with ``Z`` and/or ``T`` for a mask with several depth slices or time points - (e.g. ``"TYX"``). Empty when the header could not be read. - """ - - image: str - mask_type: str - path: str - source: str - shape: tuple[int, ...] = () - dtype: np.dtype | None = None - axes: str = "" - - @property - def planes(self) -> dict[str, int]: - """Number of depth slices (``"Z"``) and time points (``"T"``); 1 for an axis it lacks.""" - return { - axis: self.shape[self.axes.index(axis)] if axis in self.axes else 1 - for axis in PLANE_AXES - } - - -@dataclass(frozen=True) -class MaskInventory: - """Every mask found for the images of an experiment. - - Attributes - ---------- - folder - The masks folder, as an absolute path; None when only OME-Zarr label images were used. - types - The mask types found, sorted. - files - One entry per mask, sorted by image then type. - issues - Errors and warnings about the masks. - """ - - folder: Path | None - types: tuple[str, ...] = () - files: tuple[MaskFile, ...] = () - issues: tuple[ValidationIssue, ...] = () - - @property - def ok(self) -> bool: - """True if no issue blocks using the masks.""" - return not has_errors(self.issues) - - @property - def images(self) -> tuple[str, ...]: - """Images with at least one mask, sorted.""" - return tuple(sorted({f.image for f in self.files})) - - def for_image(self, image: str) -> Mapping[str, MaskFile]: - """Return the masks of one image, by type (empty when it has none).""" - return MappingProxyType({f.mask_type: f for f in self.files if f.image == image}) - - def to_manifest(self) -> dict: - """Return the inventory as recorded in the run manifest (contract §9).""" - return { - "folder": None if self.folder is None else self.folder.as_posix(), - "types": list(self.types), - "files": [ - { - "image": f.image, - "type": f.mask_type, - "path": f.path, - "source": f.source, - "axes": f.axes, - "shape": list(f.shape), - } - for f in self.files - ], - } - - -def find_masks( - masks_folder: str | os.PathLike | None, - discovery: DiscoveryResult, - images: Iterable[TiffMetadata], - *, - ome_zarr_labels: bool = True, -) -> MaskInventory: - """Pair every image with its masks and check them from their headers, without pixels. - - Parameters - ---------- - masks_folder - The masks folder (contract §15), or None to use only the OME-Zarr images' own label - images. - discovery - The experiment's folders, which say where each image is. - images - Header information of the analysed images (excluded images left out); their Y and X - sizes are what each mask must match. - ome_zarr_labels - Also use the label images of OME-Zarr images (needs the ``zarr`` extra; skipped - silently without it). - - Returns - ------- - MaskInventory - The masks found and every issue about them. - """ - images = {meta.relative_path: meta for meta in images} - issues: list[ValidationIssue] = [] - files: list[MaskFile] = [] - folder = None - if masks_folder is not None: - folder = Path(masks_folder).resolve() - issues.extend(_folder_issues(folder, discovery.root)) - if not has_errors(issues): - found, extra = _folder_masks(folder, discovery, images) - files.extend(found) - issues.extend(extra) - unreadable: list[str] = [] - if ome_zarr_labels: - found, unreadable = _label_masks(discovery.root, images) - files.extend(found) - issues.extend(_name_issues(files)) - files.sort(key=lambda f: (f.image, f.mask_type)) - issues.extend(_duplicate_issues(files)) - issues.extend(_header_issues(files, images, folder, discovery.root, unreadable)) - types = tuple(sorted({f.mask_type for f in files})) - issues.extend(_missing_issues(types, files, images)) - return MaskInventory( - folder=folder, - types=types, - files=tuple(files), - issues=tuple(sorted(issues, key=lambda issue: (not issue.is_error, issue.code))), - ) - - -def read_mask( - mask: MaskFile, - masks_folder: Path | None, - experiment_root: Path, - *, - z: int | None = None, - t: int | None = None, -) -> np.ndarray: - """Read one plane of a mask as a 2D array. - - Parameters - ---------- - mask - The mask, as found by :func:`find_masks`. - masks_folder, experiment_root - Where its file is (:attr:`MaskInventory.folder` for a TIFF mask, the experiment for - a label image). - z, t - The depth slice and time point to read, zero-based, the image's ``z`` and ``t``. Each - is required when the mask has that axis (``mask.axes``), and may only be None or 0 - otherwise: a mask's planes are never guessed (issue #36). - - Raises - ------ - ValueError - If the mask's dimensions are not the ones its header gave (``mask.axes``), or ``z`` - or ``t`` is missing or outside the mask, or the plane holds a label outside 0 to - :data:`LARGEST_LABEL` (:func:`check_labels`). - """ - wanted = {"Z": z, "T": t} - for axis, index in wanted.items(): - length = mask.planes[axis] if mask.axes else 1 - if index is None and length > 1: - raise ValueError( - f"the mask {mask.path} has {length} planes along {axis}; say which one to read" - ) - if index is not None and not 0 <= index < length: - raise ValueError(f"the mask {mask.path} has no plane {axis}={index}") - if mask.source == FOLDER: - pixels, axes = _read_tiff_plane(Path(masks_folder) / mask.path, wanted) - else: - name = mask.path.rsplit("/labels/", 1)[1] - image = experiment_root / mask.path.rsplit("/labels/", 1)[0] - labels = open_ngff_labels(image, name) - selection = _plane_index(labels.axes, labels.array.shape, wanted) - pixels = np.asarray(labels.array[selection]) - axes = "".join(a for a, s in zip(labels.axes, selection, strict=True) if s == slice(None)) - if axes != "YX" or (mask.axes and pixels.shape != tuple(_yx(mask.axes, mask.shape))): - raise ValueError(f"the mask {mask.path} is not the 2D plane its header gave ({axes})") - check_labels(pixels, f"the mask {mask.path}") - return pixels - - -def read_mask_file( - path: str | os.PathLike, *, z: int | None = None, t: int | None = None -) -> np.ndarray: - """Read one plane of a mask file by its own header, without :func:`find_masks`. - - For bricks given a mask path directly (MorphoBricks D86): the mask is read as - :func:`read_mask` reads it, with the same rules. Its axes are those its metadata names - (length-1 axes aside, except Y and X: a mask one pixel high or wide is read as a 2D - array): Y and X, plus at most a depth axis Z and a time axis T; a - dimension the metadata does not name (a plain stack, ``IYX``) is refused, never taken - for depth, time or a channel. ``z`` and ``t`` select the plane and are required for a - mask with that axis. The labels are checked with :func:`check_labels`. - - Parameters - ---------- - path - A TIFF mask, or a label image of an OME-Zarr image, ``.zarr/labels/`` - (needs the ``zarr`` extra). - z, t - The depth slice and time point to read, zero-based. - - Returns - ------- - numpy.ndarray - The 2D plane, in the mask's own pixel type. - - Raises - ------ - ValueError - If the mask cannot be opened or holds several series, has a dimension its metadata - does not name as Z or T, is not of integers or booleans, ``z`` or ``t`` is missing - or outside the mask, or a label is outside 0 to :data:`LARGEST_LABEL`. - """ - path = Path(path).absolute() # the reader joins the mask's path to its folder - label_image = path.parent.name == "labels" and is_zarr_name(path.parent.parent.name) - mask = MaskFile("", "", path.as_posix(), OME_ZARR if label_image else FOLDER) - try: - axes, shape, dtype = _mask_header(mask, path.parent, path.parent) - except (OSError, ValueError, tifffile.TiffFileError, NgffError, ZarrUnavailableError) as exc: - raise ValueError(f"the mask {path} could not be opened: {exc}") from exc - if not _known_axes(axes): - raise ValueError( - f"the mask {path} has axes {axes!r} (shape {shape}): {_axes_problem(axes, shape)}" - ) - if dtype.kind not in "biu": - raise ValueError( - f"the mask {path} is of {dtype}, not integer labels or true/false; save the mask " - "as integers" - ) - return read_mask( - MaskFile("", "", mask.path, mask.source, shape, dtype, axes), - path.parent, - path.parent, - z=z, - t=t, - ) - - -def _read_tiff_plane(path: Path, wanted: Mapping[str, int | None]) -> tuple[np.ndarray, str]: - """Read the plane at ``wanted`` Z and T of a TIFF mask; return it and its axes. - - One page is read when the planes are stored one per page (the usual layout); otherwise - the whole mask is read and the plane taken from it. - """ - with tifffile.TiffFile(path) as tif: - series = tif.series[0] - axes, shape = _tiff_axes(series) - index = _plane_index(axes, shape, wanted) - kept = "".join(a for a, i in zip(axes, index, strict=True) if i == slice(None)) - pages = len(series.pages) - split = next( - (k for k in range(len(shape) + 1) if math.prod(shape[:k]) == pages), - None, - ) - if ( - split is not None - and all(isinstance(i, int) for i in index[:split]) - and all(n == 1 or a in "YX" for a, n in zip(axes[split:], shape[split:], strict=True)) - and pages > 1 - ): - key = int(np.ravel_multi_index(index[:split], shape[:split])) - pixels = tif.asarray(key=key, series=0).reshape(shape[split:]) - pixels = pixels[index[split:]] - else: - pixels = series.asarray().reshape(shape)[index] - return pixels, kept - - -def _tiff_axes(series: tifffile.TiffPageSeries) -> tuple[str, tuple[int, ...]]: - """Return the axes and shape of a TIFF mask's series, as ``tifffile`` gives them. - - One exception: a series without Y or X stored in one 2D page, for example a 1D array - written with ``tifffile``'s default metadata (``{"shape": [N]}``, axes ``X``), is that - page: ``YX`` with the page's shape, one row of N pixels. The page's rows and columns are - the TIFF's own (as for any plain 2D page, decision D-d), so nothing is guessed; any other - series without Y or X keeps its axes and is refused. - """ - axes, shape = str(series.axes), tuple(int(n) for n in series.shape) - if "Y" in axes and "X" in axes: - return axes, shape - page = series.keyframe - if len(series.pages) == 1 and page.axes == "YX" and math.prod(page.shape) == math.prod(shape): - return "YX", tuple(int(n) for n in page.shape) - return axes, shape - - -def _plane_index( - axes: str, shape: tuple[int, ...], wanted: Mapping[str, int | None] -) -> tuple[int | slice, ...]: - """Return the index of the plane at ``wanted`` Z and T: whole Y and X, 0 on other axes. - - Y and X are kept whatever their length, so a mask one pixel high (or wide) stays 2D; - every other length-1 axis is dropped (index 0). The header check (:func:`_mask_header`, - :func:`_known_axes`) has already refused any other axis longer than 1. - """ - index: list[int | slice] = [] - for a, n in zip(axes, shape, strict=True): - if a in _YX: - index.append(slice(None)) - elif a in wanted and n > 1: - index.append(wanted.get(a) or 0) - else: - index.append(0 if n == 1 else slice(None)) - return tuple(index) - - -def _yx(axes: str, shape: tuple[int, ...]) -> tuple[int, int]: - return shape[axes.index("Y")], shape[axes.index("X")] - - -LARGEST_LABEL = int(np.iinfo(np.int32).max) -"""The largest label a mask may hold, 2,147,483,647: MorphoBricks labels are int32 -(decision D-b, 30 October 2026), so every brick stores a label the same way.""" - - -def check_labels(pixels: np.ndarray, name: str = "the mask") -> None: - """Refuse a mask whose pixels are not labels from 0 to :data:`LARGEST_LABEL`. - - Label 0 is the background and every object has a positive label (contract §15). A - negative value is neither, so it is refused rather than read as an object (which would - also make the background one). A label above :data:`LARGEST_LABEL` cannot be stored as - an int32 label, the one label type of every MorphoBricks brick (decision D-b), so it is - refused rather than wrapped or widened. Pixels that are not whole numbers (floats) are - not labels. Boolean masks are labels 0 and 1. This is the one label check every brick - calls (MorphoBricks D86). - - Parameters - ---------- - pixels - The mask, or one plane of it. - name - The mask as the message names it, for example ``"the mask Cells/ctrl/a.tif"``. - - Raises - ------ - ValueError - Naming the mask and the offending value: a pixel below 0, a pixel above - :data:`LARGEST_LABEL`, or a pixel type other than integer or boolean. - """ - pixels = np.asarray(pixels) - if pixels.dtype.kind not in "biu": - raise ValueError( - f"{name} does not hold whole-number labels (pixel type {pixels.dtype}); save the " - "mask as integers (labels) or true/false." - ) - if pixels.dtype.kind == "b" or not pixels.size: - return - smallest = pixels.min() - if smallest < 0: - raise ValueError( - f"{name} holds negative labels (smallest {smallest}); label 0 is the " - "background and objects must have positive labels." - ) - largest = pixels.max() - if largest > LARGEST_LABEL: - raise ValueError( - f"{name} holds labels above {LARGEST_LABEL} (largest {largest}); MorphoBricks " - "stores labels as 32-bit signed numbers (int32), so relabel the objects from 1." - ) - - -def _folder_issues(folder: Path, experiment_root: Path) -> list[ValidationIssue]: - if not folder.is_dir(): - return [ - error( - "masks_folder_not_found", - f"The masks folder {folder} does not exist or is not a folder.", - action="Select the folder that holds one subfolder per mask type.", - ) - ] - if folder == experiment_root or experiment_root in folder.parents: - return [ - error( - "masks_inside_experiment", - "The masks folder is inside the experiment folder, where its folders would be " - "read as conditions and repeats.", - [folder.relative_to(experiment_root).as_posix() or "."], - "Move the masks folder next to the experiment folder and select it there.", - ) - ] - return [] - - -def _folder_masks( - folder: Path, discovery: DiscoveryResult, images: Mapping[str, TiffMetadata] -) -> tuple[list[MaskFile], list[ValidationIssue]]: - """Pair the TIFF files of the masks folder with the images, in either layout. - - A file is a mask of an image when its path is ``//`` (one - folder per mask type) or ``/_`` (the type after the - image name, as segmentation tools often write it). Each file is read the way that matches - an analysed image; a file matching two images or both layouts is refused, never guessed. - """ - by_key, by_leaf = {}, {} - for leaf in discovery.leaves: - prefix = "" if leaf.relative_path == "." else leaf.relative_path + "/" - for image in leaf.images: - if image in images: - stem = position_id(image.rsplit("/", 1)[-1]) - by_key[(prefix + stem).casefold()] = image - by_leaf.setdefault(prefix.casefold(), []).append((stem, image)) - files, unpaired, loose, ambiguous = [], [], [], [] - for path in sorted(folder.rglob("*")): - relative = path.relative_to(folder) - if any(is_system_name(part) for part in relative.parts) or not path.is_file(): - continue - mask_path = relative.as_posix() - if not is_tiff_name(path.name): - (loose if len(relative.parts) == 1 else unpaired).append(mask_path) - continue - found = _suffix_matches(relative, by_leaf) - if len(relative.parts) > 1: - key = Path(*relative.parts[1:]).with_name(position_id(path.name)).as_posix() - if key.casefold() in by_key: - found.append((by_key[key.casefold()], relative.parts[0])) - if len(found) == 1: - files.append(MaskFile(found[0][0], found[0][1], mask_path, FOLDER)) - else: - (ambiguous if found else unpaired).append(mask_path) - issues = _layout_issues(loose, unpaired) - if ambiguous: - issues.append( - error( - "ambiguous_mask", - f"{plural(len(ambiguous), 'mask')} could belong to more than one image or mask " - "type (an image name that starts another image's name, or a path that fits both " - "layouts), so they cannot be paired.", - ambiguous, - "Rename the masks (or the images) so that each mask names one image and one type.", - ) - ) - return files, issues - - -def _suffix_matches(relative: Path, by_leaf: Mapping[str, list]) -> list[tuple[str, str]]: - """Return ``(image, mask type)`` for every image whose name, then ``_``, starts the file's.""" - prefix = "" if len(relative.parts) == 1 else relative.parent.as_posix() + "/" - stem = position_id(relative.name) - found = [] - for image_stem, image in by_leaf.get(prefix.casefold(), []): - head = image_stem + "_" - if len(stem) > len(head) and stem.casefold().startswith(head.casefold()): - found.append((image, stem[len(head) :])) - return found - - -def _layout_issues(loose: list[str], unpaired: list[str]) -> list[ValidationIssue]: - """Warn about files that are not masks and masks that match no analysed image.""" - issues = [] - if loose: - issues.append( - warning( - "mask_outside_type_folder", - f"{plural(len(loose), 'file')} in the masks folder " - f"{agree(loose, 'is', 'are')} not masks " - "and will not be used.", - loose, - "Keep only masks there: Nuclei//.../.tif, or " - "/.../_Nuclei.tif.", - ) - ) - if unpaired: - issues.append( - warning( - "mask_without_image", - f"{plural(len(unpaired), 'file')} in the masks folder " - f"{agree(unpaired, 'matches', 'match')} no " - "analysed image (same folders and image name) and will not be used.", - unpaired, - "Check the folder names and file names against the experiment, or remove " - "masks of excluded images.", - ) - ) - return issues - - -def _label_masks( - root: Path, images: Mapping[str, TiffMetadata] -) -> tuple[list[MaskFile], list[str]]: - """Return the label images of every OME-Zarr image, as masks. - - Also return the ``/labels`` groups that could not be read: their masks would - otherwise vanish without a word (and with them the ``mask_missing`` warning), so they - are reported as ``unreadable_mask``. - """ - files, unreadable = [], [] - for image, meta in images.items(): - if meta.kind != "ome-zarr" or not meta.readable: - continue - try: - names = ngff_label_names(root / image) - except ZarrUnavailableError as exc: # documented: label images need the zarr extra - logger.info("No label images read from %s: %s", image, exc) - continue - except NgffError as exc: - logger.info("The label images of %s could not be read: %s", image, exc) - unreadable.append(f"{image}/labels") - continue - files.extend(MaskFile(image, name, f"{image}/labels/{name}", OME_ZARR) for name in names) - return files, unreadable - - -def _name_issues(files: list[MaskFile]) -> list[ValidationIssue]: - by_folded: dict[str, set[str]] = {} - for f in files: - by_folded.setdefault(f.mask_type.casefold(), set()).add(f.mask_type) - clashes = sorted(name for names in by_folded.values() if len(names) > 1 for name in names) - if not clashes: - return [] - return [ - error( - "mask_names_differ_in_case", - "Some mask types have names that differ only in upper and lower case, so they " - "cannot be told apart: " + ", ".join(clashes) + ".", - clashes, - "Give every mask type its own name.", - ) - ] - - -def _duplicate_issues(files: list[MaskFile]) -> list[ValidationIssue]: - seen: dict[tuple[str, str], MaskFile] = {} - duplicates = [] - for f in files: - key = (f.image, f.mask_type.casefold()) - if key in seen: - duplicates += [seen[key].path, f.path] - seen[key] = f - if not duplicates: - return [] - return [ - error( - "duplicate_mask", - "Some images have two masks of the same type (for example a file in the masks " - "folder and a label image of the same name, or two files with the same name and " - "different extensions).", - sorted(set(duplicates)), - "Keep one mask per image and type.", - ) - ] - - -def _header_issues( - files: list[MaskFile], - images: Mapping[str, TiffMetadata], - folder: Path | None, - root: Path, - unreadable: Iterable[str] = (), -) -> list[ValidationIssue]: - """Read each mask's header (axes, shape, pixel type) and check it against its image. - - Also check that the masks of one image have the same depth slices and time points. - ``unreadable`` are paths already known not to open (OME-Zarr ``labels`` groups). - """ - problems: dict[str, list[str]] = {} - if unreadable: - problems["unreadable_mask"] = list(unreadable) - for index, mask in enumerate(files): - try: - axes, shape, dtype = _mask_header(mask, folder, root) - except ( - OSError, - ValueError, - tifffile.TiffFileError, - NgffError, - ZarrUnavailableError, - ) as exc: - logger.info("Mask %s could not be opened: %s", mask.path, exc) - problems.setdefault("unreadable_mask", []).append(mask.path) - continue - if not _known_axes(axes): - problems.setdefault("mask_not_2d", []).append(mask.path) - continue - files[index] = MaskFile( - mask.image, mask.mask_type, mask.path, mask.source, shape, dtype, axes - ) - expected = _image_yx(images[mask.image]) - if expected is not None and _yx(axes, shape) != expected: - problems.setdefault("mask_size_mismatch", []).append(mask.path) - if dtype.kind not in "biu": - problems.setdefault("mask_pixel_type", []).append(mask.path) - by_image: dict[str, list[MaskFile]] = {} - for mask in files: - if mask.axes: - by_image.setdefault(mask.image, []).append(mask) - for masks in by_image.values(): - if len({tuple(m.planes.items()) for m in masks}) > 1: - problems.setdefault("mask_planes_differ", []).extend(m.path for m in masks) - return [ - error(code, _TEXTS[code][0], paths, _TEXTS[code][1]) - for code, paths in sorted(problems.items()) - ] - - -_TEXTS = { - "unreadable_mask": ( - "Some masks could not be opened.", - "Check that they are TIFF files (or OME-Zarr label images, listed in the image's " - "labels group) with one image each and not damaged; the technical details are in the " - "log.", - ), - "mask_not_2d": ( - "Some masks have dimensions other than Y and X that their metadata does not name as " - "depth (Z) or time (T), so their planes cannot be matched to the image's.", - "Save each mask as a 2D label image, or with its depth and time axes named in its " - "metadata (for example an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX).", - ), - "mask_planes_differ": ( - "The masks of some images have different numbers of depth slices or time points, so " - "they cannot all belong to the same planes of their image.", - "Make every mask of an image with the image's depth slices and time points.", - ), - "mask_size_mismatch": ( - "Some masks do not have the size (Y and X) of their image.", - "Check that each mask was made from its image, at full resolution.", - ), - "mask_pixel_type": ( - "Some masks do not hold whole numbers: mask pixels must be integers (labels) or " - "true/false.", - "Save the masks as integer label images.", - ), -} - - -def _mask_header(mask: MaskFile, folder: Path | None, root: Path) -> tuple: - """Return a mask's axes and shape without length-1 axes other than Y and X, and its dtype. - - The axes are the mask's own metadata (``tifffile``'s series axes, or the OME-NGFF axes of - the label image); no pixels are read. Y and X are kept whatever their length: a mask one - pixel high is ``YX`` with shape ``(1, N)``, never the 1D ``X`` (``tifffile`` and OME-NGFF - give both axes, so nothing is guessed). Only the other length-1 axes are dropped. - """ - if mask.source == FOLDER: - with tifffile.TiffFile(Path(folder) / mask.path) as tif: - if len(tif.series) != 1: - raise ValueError(f"{len(tif.series)} image series") - axes, shape = _tiff_axes(tif.series[0]) - dtype = tif.series[0].dtype - else: - image, name = mask.path.rsplit("/labels/", 1) - labels = open_ngff_labels(root / image, name) - axes, shape, dtype = labels.axes, labels.array.shape, np.dtype(labels.array.dtype) - kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1 or a in _YX] - return ( - "".join(a for a, _ in kept), - tuple(n for _, n in kept), - dtype.newbyteorder("="), - ) - - -def _axes_problem(axes: str, shape: tuple[int, ...]) -> str: - """Say why axes that :func:`_known_axes` refuses are not a mask's. - - The axes are :func:`_mask_header`'s: length-1 axes other than Y and X already left out. - """ - missing = [a for a in "YX" if a not in axes] - if missing: - return ( - f"its header gives no {' or '.join(missing)} axis, so its rows and columns are not " - "known" - ) - repeated = sorted({a for a in axes if axes.count(a) > 1}) - if repeated: - return f"its header gives {', '.join(repeated)} more than once" - other = [ - f"{a} (length {n})" - for a, n in zip(axes, shape, strict=True) - if a not in _YX and a not in PLANE_AXES - ] - several = len(other) > 1 - return ( - f"the dimension{'s' if several else ''} {', '.join(other)} {'are' if several else 'is'} " - f"neither Y nor X, and its metadata does not name {'them' if several else 'it'} depth " - "(Z) or time (T)" - ) - - -def _known_axes(axes: str) -> bool: - """Return True if the axes are Y and X, each once, plus at most one Z and one T.""" - return ( - axes.count("Y") == 1 - and axes.count("X") == 1 - and all(axes.count(a) <= 1 for a in PLANE_AXES) - and set(axes) <= {"Y", "X", *PLANE_AXES} - ) - - -def mask_plane_issues( - inventory: MaskInventory, - images: Iterable[TiffMetadata], - file_mappings: Mapping[str, AxisMapping], - *, - skip: Iterable[str] = (), -) -> list[ValidationIssue]: - """Check that every mask has the depth slices and time points of its image (issue #36). - - An image's depth slices and time points come from its confirmed axis roles; a mask's - from its own metadata (:attr:`MaskFile.planes`). They must be equal: a 2D mask belongs - to an image with one plane, never to every plane of a Z-stack or time-lapse. - - Parameters - ---------- - inventory - The masks found by :func:`find_masks`. - images - Header information of the analysed images. - file_mappings - Each image's own axis mapping (:attr:`.axes.AxisReview.file_mappings`); images - without one are left to the axis review's errors. - skip - Images not to check: those with an axis error, as in :func:`mask_size_issues`. - - Returns - ------- - list of ValidationIssue - The error ``mask_planes_mismatch`` naming the masks that do not fit, or nothing. - """ - headers = {meta.relative_path: meta for meta in images} - skip = set(skip) - mismatched = [] - for mask in inventory.files: - mapping = file_mappings.get(mask.image) - if not mask.axes or mapping is None or mask.image not in headers or mask.image in skip: - continue - shape = headers[mask.image].shape - planes = {} - for axis in PLANE_AXES: - index = mapping.index_of(axis) - planes[axis] = 1 if index is None else int(shape[index]) - if planes != mask.planes: - mismatched.append(mask.path) - if not mismatched: - return [] - return [ - error( - "mask_planes_mismatch", - f"{plural(len(mismatched), 'mask')} " - f"{agree(mismatched, 'does', 'do')} not have the depth slices " - f"and time points of " - f"{agree(mismatched, 'its', 'their')} image: a 2D mask fits only an image with " - "one plane, and a mask with depth (Z) or time (T) needs the same number of slices " - "or time points as its image.", - mismatched, - "Save one mask plane for every depth slice and time point of the image, with the " - "axes named in the mask's metadata, or check the image's axis roles.", - ) - ] - - -def _image_yx(meta: TiffMetadata) -> tuple[int, int] | None: - """Return the Y and X sizes the image's header names, or None when it does not name them. - - Used by :func:`find_masks`, before the axis roles are confirmed. A plain stack's ``Y`` - and ``X`` are only where its pages' rows and columns are stored - (:func:`.metadata.named_axes`), so its masks are compared by the confirmed roles only - (:func:`mask_size_issues`). - """ - named = named_axes(meta) - if "Y" not in named or "X" not in named: - return None - return (meta.shape[named.index("Y")], meta.shape[named.index("X")]) - - -def _missing_issues( - types: tuple[str, ...], files: list[MaskFile], images: Mapping[str, TiffMetadata] -) -> list[ValidationIssue]: - """Warn about images without a mask of a type that other images have.""" - have = {(f.image, f.mask_type) for f in files} - issues = [] - for mask_type in types: - missing = sorted(image for image in images if (image, mask_type) not in have) - if missing: - issues.append( - warning( - "mask_missing", - f"{plural(len(missing), 'image')} " - f"{agree(missing, 'has', 'have')} no {mask_type} mask; " - f"their {mask_type} objects cannot be measured.", - missing, - f"Add the {mask_type} masks of these images, or exclude the images.", - ) - ) - return issues diff --git a/src/morphobricks_preprocessing/masks/__init__.py b/src/morphobricks_preprocessing/masks/__init__.py new file mode 100644 index 00000000..75014aeb --- /dev/null +++ b/src/morphobricks_preprocessing/masks/__init__.py @@ -0,0 +1,54 @@ +"""Find the masks of every image and check them from their headers (``docs/contract.md`` §15). + +Masks (segmentations: label images of cells, nuclei, ...) are kept in a masks folder outside +the experiment, one subfolder per mask type, each mirroring the experiment's folders:: + + /////.tif + +A mask belongs to the image with the same position (same leaf folder, same ``position_id``, +contract §1). The type may also follow the image name instead of being a folder:: + + ////_.tif + +and the two layouts may be mixed. Its file may be a TIFF (``.tif``, ``.tiff``, any case, with +an optional ``.ome``). The label images of an OME-Zarr image (its OME-NGFF ``labels`` group) +are masks too, of the type named by the label image, and are always used when present +(design v0.4 §2). + +Any number of mask types is accepted; each is named by its folder, by the suffix after the +image name, or by its label image, and two +names may not differ only in case. Only headers are read here, never pixels: a mask has the +image's Y and X sizes and integer or boolean pixels. Its other dimensions (length-1 axes +aside; Y and X are kept whatever their length, so a mask one pixel high is a 2D ``YX`` mask) +must be labelled depth (Z) or time (T) in its own metadata, or stated once per mask type +(``--mask-axes Cells=TYX``; ``Cells=YX`` for one 2D mask shared by every plane), never +guessed. Masks are never converted or reordered (design v0.4 §3), and every +mask of one image has the same depth slices and time points. Whether those are the image's +planes needs the confirmed axis roles: :func:`mask_plane_issues`, run by +:func:`.run.run_experiment` (issue #36). Reading the pixels and relating the masks to each +other is :mod:`.mask_scan`. +""" + +from __future__ import annotations + +from .checks import mask_plane_issues +from .files import find_masks +from .labels import LARGEST_LABEL, check_labels +from .model import FOLDER, OME_ZARR, PLANE_AXES, MaskFile, MaskInventory +from .read import read_mask, read_mask_file +from .stated import mask_axes_option + +__all__ = [ + "FOLDER", + "LARGEST_LABEL", + "OME_ZARR", + "PLANE_AXES", + "MaskFile", + "MaskInventory", + "check_labels", + "find_masks", + "mask_axes_option", + "mask_plane_issues", + "read_mask", + "read_mask_file", +] diff --git a/src/morphobricks_preprocessing/masks/checks.py b/src/morphobricks_preprocessing/masks/checks.py new file mode 100644 index 00000000..4400e75e --- /dev/null +++ b/src/morphobricks_preprocessing/masks/checks.py @@ -0,0 +1,234 @@ +"""Check masks from their headers against their images (contract §15). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterable, Mapping +from dataclasses import replace +from pathlib import Path + +import tifffile + +from ..axes import AxisMapping +from ..metadata import TiffMetadata, named_axes +from ..validation import ValidationIssue, agree, error, plural +from ..zarr_input import ( + NgffError, + ZarrUnavailableError, +) +from .header import _mask_header +from .layout import _yx +from .model import FOLDER, PLANE_AXES, MaskFile, MaskInventory +from .stated import mask_named_axes, mask_roles, size_code + +logger = logging.getLogger(__name__) + + +def _header_issues( + files: list[MaskFile], + images: Mapping[str, TiffMetadata], + folder: Path | None, + root: Path, + unreadable: Iterable[str] = (), + stated: Mapping[str, str] | None = None, +) -> list[ValidationIssue]: + """Read each mask's header (axes, shape, pixel type) and check it against its image. + + The mask's roles are the axes its metadata names, or those stated for its mask type + (``stated``) where the metadata names none (:func:`.stated.mask_roles`); ``files`` is + updated in place with them. Also check that the masks of one image have the same depth + slices and time points (a stated shared 2-D mask aside). ``unreadable`` are paths + already known not to open (OME-Zarr ``labels`` groups). + """ + found: list[tuple[str, str, str]] = [("unreadable_mask", path, "") for path in unreadable] + for index, mask in enumerate(files): + checked, problems = _check_header(mask, images[mask.image], folder, root, stated or {}) + found.extend(problems) + if checked is not None: + files[index] = checked + by_image: dict[str, list[MaskFile]] = {} + for mask in files: + if mask.axes and not mask.shared: + by_image.setdefault(mask.image, []).append(mask) + for masks in by_image.values(): + if len({tuple(m.planes.items()) for m in masks}) > 1: + found.extend(("mask_planes_differ", m.path, "") for m in masks) + return _grouped(found) + + +def _check_header( + mask: MaskFile, + image: TiffMetadata, + folder: Path | None, + root: Path, + stated: Mapping[str, str], +) -> tuple[MaskFile | None, list[tuple[str, str, str]]]: + """Check one mask's header: its roles, its size against the image's named Y/X, its type. + + Returns the mask with its shape, pixel type and roles (None when its header cannot be + used), and its problems as ``(code, path, detail)``. + """ + try: + axes, shape, dtype = _mask_header(mask, folder, root) + named = mask_named_axes(axes, Path(folder) / mask.path if mask.source == FOLDER else None) + except (OSError, ValueError, tifffile.TiffFileError, NgffError, ZarrUnavailableError) as exc: + logger.info("Mask %s could not be opened: %s", mask.path, exc) + return None, [("unreadable_mask", mask.path, "")] + roles, code, detail = mask_roles(axes, named, stated.get(mask.mask_type)) + if code is not None: + return None, [(code, mask.path, detail)] + how = "stated" if mask.mask_type in stated else "header" + checked = replace(mask, shape=shape, dtype=dtype, axes=roles, header_axes=axes, axes_from=how) + problems = [] + expected = _image_yx(image) + if expected is not None and _yx(roles, shape) != expected: + problems.append((size_code(_yx(roles, shape), expected), mask.path, "")) + if dtype.kind not in "biu": + problems.append(("mask_pixel_type", mask.path, "")) + return checked, problems + + +def _grouped(found: Iterable[tuple[str, str, str]]) -> list[ValidationIssue]: + """Return one error per code, naming its masks and, where given, what is wrong.""" + paths: dict[str, list[str]] = {} + details: dict[str, list[str]] = {} + for code, path, detail in found: + paths.setdefault(code, []).append(path) + if detail: + details.setdefault(code, []).append(f"{path}: {detail}") + issues = [] + for code in sorted(paths): + message = _TEXTS[code][0] + listed = details.get(code, []) + if listed: + more = f"; and {len(listed) - 5} more" if len(listed) > 5 else "" + message += " " + "; ".join(listed[:5]) + more + "." + issues.append(error(code, message, paths[code], _TEXTS[code][1])) + return issues + + +_TEXTS = { + "unreadable_mask": ( + "Some masks could not be opened.", + "Check that they are TIFF files (or OME-Zarr label images, listed in the image's " + "labels group) with one image each and not damaged; the technical details are in the " + "log.", + ), + "mask_not_2d": ( + "Some masks have dimensions other than Y and X that their metadata does not name as " + "depth (Z) or time (T), so their planes cannot be matched to the image's.", + "State the axes of the mask type once (for example --mask-axes Cells=TYX): one role, " + "Y, X, Z or T, per axis of the mask. Or save each mask as a 2D label image, or with " + "its axes named in its metadata (an ImageJ hyperstack or OME-TIFF with axes TYX or " + "ZYX).", + ), + "invalid_mask_axes": ( + "The axes stated for some masks do not fit them.", + "State one role per axis of the mask's header (length-1 axes left out): Y and X once " + "each, and at most one Z and one T. Masks have no channels.", + ), + "mask_axes_contradict_header": ( + "The axes stated for some masks contradict the axes their metadata names.", + "State the axes only for masks whose metadata does not name them, or state the named ones.", + ), + "mask_layout": ( + "Some masks are stored with their height and width (Y and X) the other way round from " + "their image as it is read. Masks are never transposed.", + "Save each such mask again with its last two axes swapped, for example " + "numpy.swapaxes(mask, -1, -2) before saving it, or segment the image as it is read.", + ), + "mask_planes_differ": ( + "The masks of some images have different numbers of depth slices or time points, so " + "they cannot all belong to the same planes of their image.", + "Make every mask of an image with the image's depth slices and time points.", + ), + "mask_size_mismatch": ( + "Some masks do not have the size (Y and X) of their image.", + "Check that each mask was made from its image, at full resolution.", + ), + "mask_pixel_type": ( + "Some masks do not hold whole numbers: mask pixels must be integers (labels) or " + "true/false.", + "Save the masks as integer label images.", + ), +} + + +def mask_plane_issues( + inventory: MaskInventory, + images: Iterable[TiffMetadata], + file_mappings: Mapping[str, AxisMapping], + *, + skip: Iterable[str] = (), +) -> list[ValidationIssue]: + """Check that every mask has the depth slices and time points of its image (issue #36). + + An image's depth slices and time points come from its confirmed axis roles; a mask's + from its own metadata (:attr:`MaskFile.planes`). They must be equal: a 2D mask belongs + to an image with one plane, never to every plane of a Z-stack or time-lapse. + + Parameters + ---------- + inventory + The masks found by :func:`find_masks`. + images + Header information of the analysed images. + file_mappings + Each image's own axis mapping (:attr:`.axes.AxisReview.file_mappings`); images + without one are left to the axis review's errors. + skip + Images not to check: those with an axis error, as in :func:`mask_size_issues`. + + Returns + ------- + list of ValidationIssue + The error ``mask_planes_mismatch`` naming the masks that do not fit, or nothing. + """ + headers = {meta.relative_path: meta for meta in images} + skip = set(skip) + mismatched = [] + for mask in inventory.files: + mapping = file_mappings.get(mask.image) + unchecked = not mask.axes or mask.shared or mask.image in skip + if unchecked or mapping is None or mask.image not in headers: + continue + shape = headers[mask.image].shape + planes = {} + for axis in PLANE_AXES: + index = mapping.index_of(axis) + planes[axis] = 1 if index is None else int(shape[index]) + if planes != mask.planes: + mismatched.append(mask.path) + if not mismatched: + return [] + return [ + error( + "mask_planes_mismatch", + f"{plural(len(mismatched), 'mask')} " + f"{agree(mismatched, 'does', 'do')} not have the depth slices " + f"and time points of " + f"{agree(mismatched, 'its', 'their')} image: a 2D mask fits only an image with " + "one plane, and a mask with depth (Z) or time (T) needs the same number of slices " + "or time points as its image.", + mismatched, + "Save one mask plane for every depth slice and time point of the image, with the " + "axes named in the mask's metadata, or check the image's axis roles.", + ) + ] + + +def _image_yx(meta: TiffMetadata) -> tuple[int, int] | None: + """Return the Y and X sizes the image's header names, or None when it does not name them. + + Used by :func:`find_masks`, before the axis roles are confirmed. A plain stack's ``Y`` + and ``X`` are only where its pages' rows and columns are stored + (:func:`.metadata.named_axes`), so its masks are compared by the confirmed roles only + (:func:`mask_size_issues`). + """ + named = named_axes(meta) + if "Y" not in named or "X" not in named: + return None + return (meta.shape[named.index("Y")], meta.shape[named.index("X")]) diff --git a/src/morphobricks_preprocessing/masks/files.py b/src/morphobricks_preprocessing/masks/files.py new file mode 100644 index 00000000..3ed16ac2 --- /dev/null +++ b/src/morphobricks_preprocessing/masks/files.py @@ -0,0 +1,342 @@ +"""Find the masks of every image and pair them by name (contract §15). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +import logging +import os +from collections.abc import Iterable, Mapping +from pathlib import Path + +from ..discovery import DiscoveryResult, is_system_name, is_tiff_name +from ..image_names import position_id +from ..metadata import TiffMetadata +from ..validation import ValidationIssue, agree, error, has_errors, plural, warning +from ..zarr_input import ( + NgffError, + ZarrUnavailableError, + ngff_label_names, +) +from .checks import _header_issues +from .model import FOLDER, OME_ZARR, MaskFile, MaskInventory + +logger = logging.getLogger(__name__) + + +def find_masks( + masks_folder: str | os.PathLike | None, + discovery: DiscoveryResult, + images: Iterable[TiffMetadata], + *, + mask_axes: Mapping[str, str] | None = None, + output_folder: str | os.PathLike | None = None, +) -> MaskInventory: + """Pair every image with its masks and check them from their headers, without pixels. + + Parameters + ---------- + masks_folder + The masks folder (contract §15), or None to use only the OME-Zarr images' own label + images. The label images of OME-Zarr images are always used (design v0.4 §2). + discovery + The experiment's folders, which say where each image is. + images + Header information of the analysed images (excluded images left out); their Y and X + sizes are what each mask must match. + mask_axes + The axes the user states for the masks of a mask type, ``{type: roles}``: one role + (``Y``, ``X``, ``Z`` or ``T``) per axis of the mask's header, length-1 axes other + than Y and X left out (``"TYX"`` for a plain ``QYX`` stack of time points; ``"YX"`` + for a 2-D mask shared by every plane of its image). A statement may not contradict + an axis the metadata names (``mask_axes_contradict_header``). Never a reordering: a + mask is read as stored. + output_folder + The run's output folder: inside the masks folder it is the error + ``output_inside_masks`` (the run's files would be read as masks). + + Returns + ------- + MaskInventory + The masks found and every issue about them. + """ + images = {meta.relative_path: meta for meta in images} + issues: list[ValidationIssue] = [] + files: list[MaskFile] = [] + folder = None + if masks_folder is not None: + folder = Path(masks_folder).resolve() + issues.extend(_folder_issues(folder, discovery.root, output_folder)) + if not has_errors(issues): + found, extra = _folder_masks(folder, discovery, images) + files.extend(found) + issues.extend(extra) + found, unreadable = _label_masks(discovery.root, images) + files.extend(found) + issues.extend(_name_issues(files)) + files.sort(key=lambda f: (f.image, f.mask_type)) + issues.extend(_duplicate_issues(files)) + stated = dict(mask_axes or {}) + issues.extend(_header_issues(files, images, folder, discovery.root, unreadable, stated)) + types = tuple(sorted({f.mask_type for f in files})) + issues.extend(_unknown_type_issues(stated, types)) + issues.extend(_missing_issues(types, files, images)) + return MaskInventory( + folder=folder, + types=types, + files=tuple(files), + issues=tuple(sorted(issues, key=lambda issue: (not issue.is_error, issue.code))), + ) + + +def _folder_issues( + folder: Path, experiment_root: Path, output_folder: str | os.PathLike | None +) -> list[ValidationIssue]: + """Refuse a masks folder that is missing, or that holds or is held by the run's folders.""" + if not folder.is_dir(): + return [ + error( + "masks_folder_not_found", + f"The masks folder {folder} does not exist or is not a folder.", + action="Select the folder that holds one subfolder per mask type.", + ) + ] + if folder == experiment_root or experiment_root in folder.parents: + return [ + error( + "masks_inside_experiment", + "The masks folder is inside the experiment folder, where its folders would be " + "read as conditions and repeats.", + [folder.relative_to(experiment_root).as_posix() or "."], + "Move the masks folder next to the experiment folder and select it there.", + ) + ] + output = None if output_folder is None else Path(output_folder).resolve() + if output is not None and (output == folder or folder in output.parents): + return [ + error( + "output_inside_masks", + "The output folder is inside the masks folder, where the run's own files would " + "be read as masks.", + [output.relative_to(folder).as_posix() or "."], + "Choose an output folder outside the masks folder.", + ) + ] + return [] + + +def _folder_masks( + folder: Path, discovery: DiscoveryResult, images: Mapping[str, TiffMetadata] +) -> tuple[list[MaskFile], list[ValidationIssue]]: + """Pair the TIFF files of the masks folder with the images, in either layout. + + A file is a mask of an image when its path is ``//`` (one + folder per mask type) or ``/_`` (the type after the + image name, as segmentation tools often write it). Each file is read the way that matches + an analysed image; a file matching two images or both layouts is refused, never guessed. + A mask matching no image is the warning ``mask_without_image``, and a mask of an excluded + image the warning ``mask_of_excluded_image``: neither is used (design v0.4 §3 keeps the + severity of every existing check). + """ + by_key, by_leaf = {}, {} + for leaf in discovery.leaves: + prefix = "" if leaf.relative_path == "." else leaf.relative_path + "/" + for image in leaf.images: + # Excluded images pair too, so their masks are told apart from unpaired files. + stem = position_id(image.rsplit("/", 1)[-1]) + by_key[(prefix + stem).casefold()] = image + by_leaf.setdefault(prefix.casefold(), []).append((stem, image)) + files, unpaired, loose, ambiguous, excluded = [], [], [], [], [] + for path in sorted(folder.rglob("*")): + relative = path.relative_to(folder) + if any(is_system_name(part) for part in relative.parts) or not path.is_file(): + continue + mask_path = relative.as_posix() + if not is_tiff_name(path.name): + (loose if len(relative.parts) == 1 else unpaired).append(mask_path) + continue + found = _suffix_matches(relative, by_leaf) + if len(relative.parts) > 1: + key = Path(*relative.parts[1:]).with_name(position_id(path.name)).as_posix() + if key.casefold() in by_key: + found.append((by_key[key.casefold()], relative.parts[0])) + if len(found) == 1 and found[0][0] not in images: + excluded.append(mask_path) + elif len(found) == 1: + files.append(MaskFile(found[0][0], found[0][1], mask_path, FOLDER)) + else: + (ambiguous if found else unpaired).append(mask_path) + issues = [*_layout_issues(loose, unpaired), *_excluded_issues(excluded)] + if ambiguous: + issues.append( + error( + "ambiguous_mask", + f"{plural(len(ambiguous), 'mask')} could belong to more than one image or mask " + "type (an image name that starts another image's name, or a path that fits both " + "layouts), so they cannot be paired.", + ambiguous, + "Rename the masks (or the images) so that each mask names one image and one type.", + ) + ) + return files, issues + + +def _suffix_matches(relative: Path, by_leaf: Mapping[str, list]) -> list[tuple[str, str]]: + """Return ``(image, mask type)`` for every image whose name, then ``_``, starts the file's.""" + prefix = "" if len(relative.parts) == 1 else relative.parent.as_posix() + "/" + stem = position_id(relative.name) + found = [] + for image_stem, image in by_leaf.get(prefix.casefold(), []): + head = image_stem + "_" + if len(stem) > len(head) and stem.casefold().startswith(head.casefold()): + found.append((image, stem[len(head) :])) + return found + + +def _layout_issues(loose: list[str], unpaired: list[str]) -> list[ValidationIssue]: + """Warn about files that are not masks and masks that match no image.""" + issues = [] + if loose: + issues.append( + warning( + "mask_outside_type_folder", + f"{plural(len(loose), 'file')} in the masks folder " + f"{agree(loose, 'is', 'are')} not masks " + "and will not be used.", + loose, + "Keep only masks there: Nuclei//.../.tif, or " + "/.../_Nuclei.tif.", + ) + ) + if unpaired: + issues.append( + warning( + "mask_without_image", + f"{plural(len(unpaired), 'file')} in the masks folder " + f"{agree(unpaired, 'matches', 'match')} no " + "image of the experiment (same folders and image name) and will not be used.", + unpaired, + "Check the folder names and file names against the experiment.", + ) + ) + return issues + + +def _excluded_issues(excluded: list[str]) -> list[ValidationIssue]: + """Report the masks of excluded images apart from unpaired masks: they are not used.""" + if not excluded: + return [] + return [ + warning( + "mask_of_excluded_image", + f"{plural(len(excluded), 'mask')} belong to excluded images and are not used.", + excluded, + "Nothing to do; analyse the images again to use their masks.", + ) + ] + + +def _unknown_type_issues(stated: Mapping[str, str], types: tuple[str, ...]) -> list: + """Refuse axes stated for a mask type that is not found.""" + unknown = sorted(set(stated) - set(types)) + if not unknown: + return [] + return [ + error( + "invalid_mask_axes", + f"Mask axes were stated for {', '.join(unknown)}, which " + f"{agree(unknown, 'is not a mask type', 'are not mask types')} found " + f"(found: {', '.join(types) or 'none'}).", + unknown, + "State the axes by the mask type's name, as its folder or suffix gives it.", + ) + ] + + +def _label_masks( + root: Path, images: Mapping[str, TiffMetadata] +) -> tuple[list[MaskFile], list[str]]: + """Return the label images of every OME-Zarr image, as masks. + + Also return the ``/labels`` groups that could not be read: their masks would + otherwise vanish without a word (and with them the ``mask_missing`` warning), so they + are reported as ``unreadable_mask``. + """ + files, unreadable = [], [] + for image, meta in images.items(): + if meta.kind != "ome-zarr" or not meta.readable: + continue + try: + names = ngff_label_names(root / image) + except ZarrUnavailableError as exc: # documented: label images need the zarr extra + logger.info("No label images read from %s: %s", image, exc) + continue + except NgffError as exc: + logger.info("The label images of %s could not be read: %s", image, exc) + unreadable.append(f"{image}/labels") + continue + files.extend(MaskFile(image, name, f"{image}/labels/{name}", OME_ZARR) for name in names) + return files, unreadable + + +def _name_issues(files: list[MaskFile]) -> list[ValidationIssue]: + by_folded: dict[str, set[str]] = {} + for f in files: + by_folded.setdefault(f.mask_type.casefold(), set()).add(f.mask_type) + clashes = sorted(name for names in by_folded.values() if len(names) > 1 for name in names) + if not clashes: + return [] + return [ + error( + "mask_names_differ_in_case", + "Some mask types have names that differ only in upper and lower case, so they " + "cannot be told apart: " + ", ".join(clashes) + ".", + clashes, + "Give every mask type its own name.", + ) + ] + + +def _duplicate_issues(files: list[MaskFile]) -> list[ValidationIssue]: + seen: dict[tuple[str, str], MaskFile] = {} + duplicates = [] + for f in files: + key = (f.image, f.mask_type.casefold()) + if key in seen: + duplicates += [seen[key].path, f.path] + seen[key] = f + if not duplicates: + return [] + return [ + error( + "duplicate_mask", + "Some images have two masks of the same type (for example a file in the masks " + "folder and a label image of the same name, or two files with the same name and " + "different extensions).", + sorted(set(duplicates)), + "Keep one mask per image and type.", + ) + ] + + +def _missing_issues( + types: tuple[str, ...], files: list[MaskFile], images: Mapping[str, TiffMetadata] +) -> list[ValidationIssue]: + """Warn about images without a mask of a type that other images have.""" + have = {(f.image, f.mask_type) for f in files} + issues = [] + for mask_type in types: + missing = sorted(image for image in images if (image, mask_type) not in have) + if missing: + issues.append( + warning( + "mask_missing", + f"{plural(len(missing), 'image')} " + f"{agree(missing, 'has', 'have')} no {mask_type} mask; " + f"their {mask_type} objects cannot be measured.", + missing, + f"Add the {mask_type} masks of these images, or exclude the images.", + ) + ) + return issues diff --git a/src/morphobricks_preprocessing/masks/header.py b/src/morphobricks_preprocessing/masks/header.py new file mode 100644 index 00000000..3a2ee6f3 --- /dev/null +++ b/src/morphobricks_preprocessing/masks/header.py @@ -0,0 +1,80 @@ +"""A mask's header: its axes, shape and pixel type, read without pixels (contract §15). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import tifffile + +from ..zarr_input import ( + open_ngff_labels, +) +from .layout import _tiff_axes +from .model import _YX, FOLDER, PLANE_AXES, MaskFile + + +def _mask_header(mask: MaskFile, folder: Path | None, root: Path) -> tuple: + """Return a mask's axes and shape without length-1 axes other than Y and X, and its dtype. + + The axes are the mask's own metadata (``tifffile``'s series axes, or the OME-NGFF axes of + the label image); no pixels are read. Y and X are kept whatever their length: a mask one + pixel high is ``YX`` with shape ``(1, N)``, never the 1D ``X`` (``tifffile`` and OME-NGFF + give both axes, so nothing is guessed). Only the other length-1 axes are dropped. + """ + if mask.source == FOLDER: + with tifffile.TiffFile(Path(folder) / mask.path) as tif: + if len(tif.series) != 1: + raise ValueError(f"{len(tif.series)} image series") + axes, shape = _tiff_axes(tif.series[0]) + dtype = tif.series[0].dtype + else: + image, name = mask.path.rsplit("/labels/", 1) + labels = open_ngff_labels(root / image, name) + axes, shape, dtype = labels.axes, labels.array.shape, np.dtype(labels.array.dtype) + kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1 or a in _YX] + return ( + "".join(a for a, _ in kept), + tuple(n for _, n in kept), + dtype.newbyteorder("="), + ) + + +def _axes_problem(axes: str, shape: tuple[int, ...]) -> str: + """Say why axes that :func:`_known_axes` refuses are not a mask's. + + The axes are :func:`_mask_header`'s: length-1 axes other than Y and X already left out. + """ + missing = [a for a in "YX" if a not in axes] + if missing: + return ( + f"its header gives no {' or '.join(missing)} axis, so its rows and columns are not " + "known" + ) + repeated = sorted({a for a in axes if axes.count(a) > 1}) + if repeated: + return f"its header gives {', '.join(repeated)} more than once" + other = [ + f"{a} (length {n})" + for a, n in zip(axes, shape, strict=True) + if a not in _YX and a not in PLANE_AXES + ] + several = len(other) > 1 + return ( + f"the dimension{'s' if several else ''} {', '.join(other)} {'are' if several else 'is'} " + f"neither Y nor X, and its metadata does not name {'them' if several else 'it'} depth " + "(Z) or time (T)" + ) + + +def _known_axes(axes: str) -> bool: + """Return True if the axes are Y and X, each once, plus at most one Z and one T.""" + return ( + axes.count("Y") == 1 + and axes.count("X") == 1 + and all(axes.count(a) <= 1 for a in PLANE_AXES) + and set(axes) <= {"Y", "X", *PLANE_AXES} + ) diff --git a/src/morphobricks_preprocessing/masks/labels.py b/src/morphobricks_preprocessing/masks/labels.py new file mode 100644 index 00000000..bcaa634e --- /dev/null +++ b/src/morphobricks_preprocessing/masks/labels.py @@ -0,0 +1,58 @@ +"""Mask labels: whole numbers from 0 to the int32 limit (decision D-b, contract §16). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +import numpy as np + +LARGEST_LABEL = int(np.iinfo(np.int32).max) +"""The largest label a mask may hold, 2,147,483,647: MorphoBricks labels are int32 +(decision D-b, 30 October 2026), so every brick stores a label the same way.""" + + +def check_labels(pixels: np.ndarray, name: str = "the mask") -> None: + """Refuse a mask whose pixels are not labels from 0 to :data:`LARGEST_LABEL`. + + Label 0 is the background and every object has a positive label (contract §15). A + negative value is neither, so it is refused rather than read as an object (which would + also make the background one). A label above :data:`LARGEST_LABEL` cannot be stored as + an int32 label, the one label type of every MorphoBricks brick (decision D-b), so it is + refused rather than wrapped or widened. Pixels that are not whole numbers (floats) are + not labels. Boolean masks are labels 0 and 1. This is the one label check every brick + calls (MorphoBricks D86). + + Parameters + ---------- + pixels + The mask, or one plane of it. + name + The mask as the message names it, for example ``"the mask Cells/ctrl/a.tif"``. + + Raises + ------ + ValueError + Naming the mask and the offending value: a pixel below 0, a pixel above + :data:`LARGEST_LABEL`, or a pixel type other than integer or boolean. + """ + pixels = np.asarray(pixels) + if pixels.dtype.kind not in "biu": + raise ValueError( + f"{name} does not hold whole-number labels (pixel type {pixels.dtype}); save the " + "mask as integers (labels) or true/false." + ) + if pixels.dtype.kind == "b" or not pixels.size: + return + smallest = pixels.min() + if smallest < 0: + raise ValueError( + f"{name} holds negative labels (smallest {smallest}); label 0 is the " + "background and objects must have positive labels." + ) + largest = pixels.max() + if largest > LARGEST_LABEL: + raise ValueError( + f"{name} holds labels above {LARGEST_LABEL} (largest {largest}); MorphoBricks " + "stores labels as 32-bit signed numbers (int32), so relabel the objects from 1." + ) diff --git a/src/morphobricks_preprocessing/masks/layout.py b/src/morphobricks_preprocessing/masks/layout.py new file mode 100644 index 00000000..db422789 --- /dev/null +++ b/src/morphobricks_preprocessing/masks/layout.py @@ -0,0 +1,55 @@ +"""Where a mask's planes are: its axes as stored, and the index of one plane (contract §15). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +import math +from collections.abc import Mapping + +import tifffile + +from .model import _YX + + +def _tiff_axes(series: tifffile.TiffPageSeries) -> tuple[str, tuple[int, ...]]: + """Return the axes and shape of a TIFF mask's series, as ``tifffile`` gives them. + + One exception: a series without Y or X stored in one 2D page, for example a 1D array + written with ``tifffile``'s default metadata (``{"shape": [N]}``, axes ``X``), is that + page: ``YX`` with the page's shape, one row of N pixels. The page's rows and columns are + the TIFF's own (as for any plain 2D page, decision D-d), so nothing is guessed; any other + series without Y or X keeps its axes and is refused. + """ + axes, shape = str(series.axes), tuple(int(n) for n in series.shape) + if "Y" in axes and "X" in axes: + return axes, shape + page = series.keyframe + if len(series.pages) == 1 and page.axes == "YX" and math.prod(page.shape) == math.prod(shape): + return "YX", tuple(int(n) for n in page.shape) + return axes, shape + + +def _plane_index( + axes: str, shape: tuple[int, ...], wanted: Mapping[str, int | None] +) -> tuple[int | slice, ...]: + """Return the index of the plane at ``wanted`` Z and T: whole Y and X, 0 on other axes. + + Y and X are kept whatever their length, so a mask one pixel high (or wide) stays 2D; + every other length-1 axis is dropped (index 0). The header check (:func:`_mask_header`, + :func:`_known_axes`) has already refused any other axis longer than 1. + """ + index: list[int | slice] = [] + for a, n in zip(axes, shape, strict=True): + if a in _YX: + index.append(slice(None)) + elif a in wanted and n > 1: + index.append(wanted.get(a) or 0) + else: + index.append(0 if n == 1 else slice(None)) + return tuple(index) + + +def _yx(axes: str, shape: tuple[int, ...]) -> tuple[int, int]: + return shape[axes.index("Y")], shape[axes.index("X")] diff --git a/src/morphobricks_preprocessing/masks/model.py b/src/morphobricks_preprocessing/masks/model.py new file mode 100644 index 00000000..66d79743 --- /dev/null +++ b/src/morphobricks_preprocessing/masks/model.py @@ -0,0 +1,143 @@ +"""Masks as found: one :class:`MaskFile` per mask, the :class:`MaskInventory` of a run. + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from types import MappingProxyType + +import numpy as np + +from ..validation import ValidationIssue, has_errors + +FOLDER = "folder" +"""Source of a mask kept as a TIFF file in the masks folder.""" + + +OME_ZARR = "ome-zarr" +"""Source of a mask kept as a label image inside an OME-Zarr image.""" + + +PLANE_AXES = ("Z", "T") +"""The dimensions a mask may have besides Y and X: depth slices and time points.""" + + +_YX = frozenset("YX") +"""The axes of a mask plane, kept by the header and the readers even when of length 1.""" + + +@dataclass(frozen=True) +class MaskFile: + """One mask of one image. + + Attributes + ---------- + image + The image's path relative to the experiment root. + mask_type + The mask type: its folder name in the masks folder, or the label image's name. + path + The mask's path: relative to the masks folder for a TIFF mask; for a label image of + an OME-Zarr image, ``/labels/`` relative to the experiment root. + source + :data:`FOLDER` or :data:`OME_ZARR`. + shape + Shape of the mask without its length-1 axes other than Y and X, ``(Y, X)`` for a 2D + mask (``(1, 512)`` for a mask one pixel high). + dtype + Pixel type, or None when the header could not be read. + axes + The role of each entry of ``shape``: ``"YX"`` for a 2D mask, with ``Z`` and/or ``T`` + for a mask with several depth slices or time points (e.g. ``"TYX"``). They are the + axes the mask's own metadata names, or, for a mask whose metadata does not name them + (a plain stack), the axes the user stated for its mask type. Empty when the header + could not be read or the axes are not known. + header_axes + The mask's axes as its header gives them (``tifffile``'s letters, length-1 axes + other than Y and X left out), e.g. ``"QYX"`` for a plain stack; empty when unknown. + axes_from + ``"header"`` when the axes are those the mask's metadata names, ``"stated"`` when + the user stated the axes of its mask type (a plain stack, or a 2-D mask shared by + every plane of its image); empty when unknown. + """ + + image: str + mask_type: str + path: str + source: str + shape: tuple[int, ...] = () + dtype: np.dtype | None = None + axes: str = "" + header_axes: str = "" + axes_from: str = "" + + @property + def shared(self) -> bool: + """True for a 2-D mask stated for its type: one plane shared by every image plane.""" + return self.axes == "YX" and self.axes_from == "stated" + + @property + def planes(self) -> dict[str, int]: + """Number of depth slices (``"Z"``) and time points (``"T"``); 1 for an axis it lacks.""" + return { + axis: self.shape[self.axes.index(axis)] if axis in self.axes else 1 + for axis in PLANE_AXES + } + + +@dataclass(frozen=True) +class MaskInventory: + """Every mask found for the images of an experiment. + + Attributes + ---------- + folder + The masks folder, as an absolute path; None when only OME-Zarr label images were used. + types + The mask types found, sorted. + files + One entry per mask, sorted by image then type. + issues + Errors and warnings about the masks. + """ + + folder: Path | None + types: tuple[str, ...] = () + files: tuple[MaskFile, ...] = () + issues: tuple[ValidationIssue, ...] = () + + @property + def ok(self) -> bool: + """True if no issue blocks using the masks.""" + return not has_errors(self.issues) + + @property + def images(self) -> tuple[str, ...]: + """Images with at least one mask, sorted.""" + return tuple(sorted({f.image for f in self.files})) + + def for_image(self, image: str) -> Mapping[str, MaskFile]: + """Return the masks of one image, by type (empty when it has none).""" + return MappingProxyType({f.mask_type: f for f in self.files if f.image == image}) + + def to_manifest(self) -> dict: + """Return the inventory as recorded in the run manifest (contract §9).""" + return { + "folder": None if self.folder is None else self.folder.as_posix(), + "types": list(self.types), + "files": [ + { + "image": f.image, + "type": f.mask_type, + "path": f.path, + "source": f.source, + "axes": f.axes, + "shape": list(f.shape), + } + for f in self.files + ], + } diff --git a/src/morphobricks_preprocessing/masks/read.py b/src/morphobricks_preprocessing/masks/read.py new file mode 100644 index 00000000..9f785430 --- /dev/null +++ b/src/morphobricks_preprocessing/masks/read.py @@ -0,0 +1,178 @@ +"""Read one plane of a mask, as stored (contract §15). + +Moved from ``masks.py`` (PP-5). +""" + +from __future__ import annotations + +import math +import os +from collections.abc import Mapping +from pathlib import Path + +import numpy as np +import tifffile + +from ..zarr_input import ( + NgffError, + ZarrUnavailableError, + is_zarr_name, + open_ngff_labels, +) +from .header import _axes_problem, _known_axes, _mask_header +from .labels import check_labels +from .layout import _plane_index, _tiff_axes, _yx +from .model import _YX, FOLDER, OME_ZARR, MaskFile + + +def read_mask( + mask: MaskFile, + masks_folder: Path | None, + experiment_root: Path, + *, + z: int | None = None, + t: int | None = None, +) -> np.ndarray: + """Read one plane of a mask as a 2D array. + + Parameters + ---------- + mask + The mask, as found by :func:`find_masks`. + masks_folder, experiment_root + Where its file is (:attr:`MaskInventory.folder` for a TIFF mask, the experiment for + a label image). + z, t + The depth slice and time point to read, zero-based, the image's ``z`` and ``t``. Each + is required when the mask has that axis (``mask.axes``), and may only be None or 0 + otherwise: a mask's planes are never guessed (issue #36). + + Raises + ------ + ValueError + If the mask's dimensions are not the ones its header gave (``mask.axes``), or ``z`` + or ``t`` is missing or outside the mask, or the plane holds a label outside 0 to + :data:`LARGEST_LABEL` (:func:`check_labels`). + """ + # A shared 2-D mask (stated for its type) is one plane for every plane of its image. + wanted = {"Z": None, "T": None} if mask.shared else {"Z": z, "T": t} + for axis, index in wanted.items(): + length = mask.planes[axis] if mask.axes else 1 + if index is None and length > 1: + raise ValueError( + f"the mask {mask.path} has {length} planes along {axis}; say which one to read" + ) + if index is not None and not 0 <= index < length: + raise ValueError(f"the mask {mask.path} has no plane {axis}={index}") + if mask.source == FOLDER: + # A mask whose axes were stated (a plain stack) is read by those roles, as stored. + roles = mask.axes if mask.header_axes and mask.header_axes != mask.axes else None + pixels, axes = _read_tiff_plane(Path(masks_folder) / mask.path, wanted, roles) + else: + name = mask.path.rsplit("/labels/", 1)[1] + image = experiment_root / mask.path.rsplit("/labels/", 1)[0] + labels = open_ngff_labels(image, name) + selection = _plane_index(labels.axes, labels.array.shape, wanted) + pixels = np.asarray(labels.array[selection]) + axes = "".join(a for a, s in zip(labels.axes, selection, strict=True) if s == slice(None)) + if axes != "YX" or (mask.axes and pixels.shape != tuple(_yx(mask.axes, mask.shape))): + raise ValueError(f"the mask {mask.path} is not the 2D plane its header gave ({axes})") + check_labels(pixels, f"the mask {mask.path}") + return pixels + + +def read_mask_file( + path: str | os.PathLike, *, z: int | None = None, t: int | None = None +) -> np.ndarray: + """Read one plane of a mask file by its own header, without :func:`find_masks`. + + For bricks given a mask path directly (MorphoBricks D86): the mask is read as + :func:`read_mask` reads it, with the same rules. Its axes are those its metadata names + (length-1 axes aside, except Y and X: a mask one pixel high or wide is read as a 2D + array): Y and X, plus at most a depth axis Z and a time axis T; a + dimension the metadata does not name (a plain stack, ``IYX``) is refused, never taken + for depth, time or a channel. ``z`` and ``t`` select the plane and are required for a + mask with that axis. The labels are checked with :func:`check_labels`. + + Parameters + ---------- + path + A TIFF mask, or a label image of an OME-Zarr image, ``.zarr/labels/`` + (needs the ``zarr`` extra). + z, t + The depth slice and time point to read, zero-based. + + Returns + ------- + numpy.ndarray + The 2D plane, in the mask's own pixel type. + + Raises + ------ + ValueError + If the mask cannot be opened or holds several series, has a dimension its metadata + does not name as Z or T, is not of integers or booleans, ``z`` or ``t`` is missing + or outside the mask, or a label is outside 0 to :data:`LARGEST_LABEL`. + """ + path = Path(path).absolute() # the reader joins the mask's path to its folder + label_image = path.parent.name == "labels" and is_zarr_name(path.parent.parent.name) + mask = MaskFile("", "", path.as_posix(), OME_ZARR if label_image else FOLDER) + try: + axes, shape, dtype = _mask_header(mask, path.parent, path.parent) + except (OSError, ValueError, tifffile.TiffFileError, NgffError, ZarrUnavailableError) as exc: + raise ValueError(f"the mask {path} could not be opened: {exc}") from exc + if not _known_axes(axes): + raise ValueError( + f"the mask {path} has axes {axes!r} (shape {shape}): {_axes_problem(axes, shape)}" + ) + if dtype.kind not in "biu": + raise ValueError( + f"the mask {path} is of {dtype}, not integer labels or true/false; save the mask " + "as integers" + ) + return read_mask( + MaskFile("", "", mask.path, mask.source, shape, dtype, axes), + path.parent, + path.parent, + z=z, + t=t, + ) + + +def _read_tiff_plane( + path: Path, wanted: Mapping[str, int | None], roles: str | None = None +) -> tuple[np.ndarray, str]: + """Read the plane at ``wanted`` Z and T of a TIFF mask; return it and its axes. + + ``roles`` gives the roles of the header's axes without their length-1 axes other than + Y and X (the axes stated for a plain stack); by default the header's own letters. One + page is read when the planes are stored one per page (the usual layout); otherwise the + whole mask is read and the plane taken from it. + """ + with tifffile.TiffFile(path) as tif: + series = tif.series[0] + axes, shape = _tiff_axes(series) + if roles is not None: + given = iter(roles) + axes = "".join( + next(given) if n != 1 or a in _YX else "-" for a, n in zip(axes, shape, strict=True) + ) + index = _plane_index(axes, shape, wanted) + kept = "".join(a for a, i in zip(axes, index, strict=True) if i == slice(None)) + pages = len(series.pages) + split = next( + (k for k in range(len(shape) + 1) if math.prod(shape[:k]) == pages), + None, + ) + if ( + split is not None + and all(isinstance(i, int) for i in index[:split]) + and all(n == 1 or a in "YX" for a, n in zip(axes[split:], shape[split:], strict=True)) + and pages > 1 + ): + key = int(np.ravel_multi_index(index[:split], shape[:split])) + pixels = tif.asarray(key=key, series=0).reshape(shape[split:]) + pixels = pixels[index[split:]] + else: + pixels = series.asarray().reshape(shape)[index] + return pixels, kept diff --git a/src/morphobricks_preprocessing/masks/stated.py b/src/morphobricks_preprocessing/masks/stated.py new file mode 100644 index 00000000..a7a5732b --- /dev/null +++ b/src/morphobricks_preprocessing/masks/stated.py @@ -0,0 +1,156 @@ +"""A mask's roles: the axes its header names, or those stated for its mask type (§15). + +Masks are never converted or reordered; they are checked as stored against their image as +read (design v0.4 §3). The roles are the axes the mask's metadata names (the rule of +:func:`.axis_names.named_letters`, the one images use); where it names none (a plain stack), +the user states them once per mask type. A statement may not contradict a named axis. A +2-D mask stated ``YX`` for an image with several planes is one plane shared by all of them. +A mask stored with Y and X the other way round from its image is refused, with a +``numpy.swapaxes`` hint, never transposed. +""" + +from __future__ import annotations + +from collections.abc import Iterable +from pathlib import Path + +import tifffile + +from ..axis_names import PLACEHOLDER_AXES, named_letters +from ..metadata import series_axes_from_metadata +from .header import _known_axes + +MASK_ROLES = frozenset("YXZT") +"""The roles of a mask's axes: its plane (Y, X) and at most a depth (Z) and time (T) axis.""" + + +def mask_named_axes(axes: str, tiff: Path | None = None) -> str: + """Return a mask's header axes with ``?`` where its metadata does not name them. + + Parameters + ---------- + axes + The mask's header axes, length-1 axes other than Y and X left out. + tiff + The mask's TIFF file, read for its metadata kind only when ``axes`` has a + placeholder; None for an OME-Zarr label image, whose axes are its OME-NGFF axes. + + Returns + ------- + str + One character per axis, e.g. ``"TYX"``, ``"?YX"`` or ``"???"``. + """ + from_metadata = True + if tiff is not None and set(axes) & PLACEHOLDER_AXES: + with tifffile.TiffFile(tiff) as tif: + from_metadata = series_axes_from_metadata(tif.series[0]) + return named_letters(axes, from_metadata) + + +def mask_roles(header: str, named: str, stated: str | None) -> tuple[str, str | None, str]: + """Decide a mask's roles from its header and the axes stated for its mask type. + + Parameters + ---------- + header + The mask's header axes, length-1 axes other than Y and X left out. + named + The same axes with ``?`` for each one the metadata does not name. + stated + The roles stated for the mask's type (one per axis of ``header``), or None. + + Returns + ------- + tuple + ``(roles, None, "")`` when the roles are known, or ``("", code, detail)`` with the + error code (``mask_not_2d``, ``invalid_mask_axes``, ``mask_axes_contradict_header``, + ``mask_layout``) and what is wrong. + """ + if stated is None: + if "?" in named or not _known_axes(header): + return "", "mask_not_2d", f"header axes {header}" + roles = header + else: + roles = stated.upper() + problem = _stated_problem(roles, len(header)) + if problem: + return "", "invalid_mask_axes", f"header axes {header}, stated {stated}: {problem}" + contradicted = [ + f"{letter} stated {role}" + for letter, role in zip(named, roles, strict=True) + if letter not in ("?", role) + ] + if contradicted: + detail = f"header axes {header}, stated {stated}: {', '.join(contradicted)}" + return "", "mask_axes_contradict_header", detail + if roles.index("X") < roles.index("Y"): + return "", "mask_layout", f"axes {roles}: X is stored before Y" + return roles, None, "" + + +def _stated_problem(roles: str, count: int) -> str: + """Say what is wrong with roles stated for a mask's ``count`` axes, or ``""``.""" + if len(roles) != count: + return f"{len(roles)} roles for {count} axes" + others = sorted(set(roles) - MASK_ROLES) + if others: + return f"{', '.join(others)} is not a mask role (Y, X, Z or T; masks have no channels)" + if roles.count("Y") != 1 or roles.count("X") != 1: + return "Y and X are needed once each" + if roles.count("Z") > 1 or roles.count("T") > 1: + return "Z and T may appear once each" + return "" + + +def size_code(mask_yx: tuple[int, int], image_yx: tuple[int, int]) -> str: + """Name the error of a mask whose height and width are not its image's. + + Parameters + ---------- + mask_yx, image_yx + The (Y, X) sizes of the mask and of its image as read. + + Returns + ------- + str + ``"mask_layout"`` when they are the image's swapped (never transposed), otherwise + ``"mask_size_mismatch"``. + """ + return "mask_layout" if mask_yx == image_yx[::-1] else "mask_size_mismatch" + + +def mask_axes_option(statements: Iterable[str]) -> dict[str, str]: + """Read ``--mask-axes TYPE=ROLES`` statements into ``{type: roles}``. + + Parameters + ---------- + statements + The statements as given (``"Cells=TYX"``); the roles are checked against each mask + later (:func:`mask_roles`). + + Returns + ------- + dict + The roles by mask type. + + Raises + ------ + ValueError + For a statement without a type or roles, or two different statements for one type. + """ + stated: dict[str, str] = {} + for statement in statements: + mask_type, _, roles = statement.partition("=") + mask_type, roles = mask_type.strip(), roles.strip() + if not mask_type or not roles: + raise ValueError( + f"--mask-axes {statement}: give the mask type and its roles, for example " + "--mask-axes Cells=TYX." + ) + if stated.get(mask_type, roles) != roles: + raise ValueError( + f"--mask-axes gives two different roles for {mask_type}: " + f"{stated[mask_type]} and {roles}." + ) + stated[mask_type] = roles + return stated diff --git a/src/morphobricks_preprocessing/metadata.py b/src/morphobricks_preprocessing/metadata.py index b3402bb6..87192f21 100644 --- a/src/morphobricks_preprocessing/metadata.py +++ b/src/morphobricks_preprocessing/metadata.py @@ -27,6 +27,7 @@ import tifffile from morphobricks_normalization.dtypes import UnsupportedDtypeError, check_supported +from .axis_names import OPEN_AXIS, named_letters from .discovery import DiscoveryResult from .physical_sizes import ( PIXEL_SIZE_TOLERANCE, @@ -183,15 +184,6 @@ def dtype(self) -> np.dtype | None: return dtypes.pop() if len(dtypes) == 1 else None -PLACEHOLDER_AXES = frozenset("IQ") -"""``tifffile``'s letters for a dimension whose meaning the file does not give: ``I`` (a -plain stack of pages) and ``Q`` (a "shaped" file without axes, or an OME-NGFF axis of -another type). They are never metadata: the user gives such a dimension its role.""" - -OPEN_AXIS = "?" -"""Marks a dimension the metadata does not name in :func:`named_axes`.""" - - def named_axes(meta: TiffMetadata) -> str: """Return the image's axes with every dimension its metadata does not name as ``?``. @@ -220,13 +212,7 @@ def named_axes(meta: TiffMetadata) -> str: One character per dimension, e.g. ``"CYX"``, ``"?YX"`` (shaped axes ``QYX``) or ``"???"`` (a plain ``QYX`` stack); ``""`` for an unreadable image. """ - if not meta.readable: - return "" - plain_stack = not meta.axes_from_metadata and bool(PLACEHOLDER_AXES & set(meta.axes)) - return "".join( - OPEN_AXIS if axis in PLACEHOLDER_AXES or (plain_stack and axis in "YX") else axis - for axis in meta.axes - ) + return named_letters(meta.axes, meta.axes_from_metadata) if meta.readable else "" def header_axes(meta: TiffMetadata) -> str | None: diff --git a/src/morphobricks_preprocessing/run/pipeline.py b/src/morphobricks_preprocessing/run/pipeline.py index eb90fd29..8e9d8f5f 100644 --- a/src/morphobricks_preprocessing/run/pipeline.py +++ b/src/morphobricks_preprocessing/run/pipeline.py @@ -7,7 +7,7 @@ import logging import os -from collections.abc import Callable, Iterable +from collections.abc import Callable, Iterable, Mapping from dataclasses import asdict, replace from datetime import UTC, datetime from pathlib import Path @@ -77,7 +77,7 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 progress: Callable[[int, int, str], None] | None = None, design: ExperimentDesign | None = None, masks_folder: str | os.PathLike | None = None, - ome_zarr_labels: bool = False, + mask_axes: Mapping[str, str] | None = None, image_type: str | None = None, intensity_settings: IntensityCheckSettings | None = None, histogram_settings: HistogramSettings | None = None, @@ -118,11 +118,12 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 with the table, and its SHA-256 recorded in the manifest. Without a design nothing is asked, the manifest records none, and an existing ``experiment_design.csv`` is left as it is. - masks_folder, ome_zarr_labels - As :func:`preflight`. The masks are recorded in the manifest (``masks``, - contract §9); an error about them stops the run, including a mask whose - depth slices and time points are not its image's under the confirmed - roles (``mask_planes_mismatch``). Their pixels are not read: the mask + masks_folder, mask_axes + As :func:`preflight`; the label images of OME-Zarr images are always used. + The masks are recorded in the manifest (``masks``, contract §9); an error + about them stops the run, including a mask whose depth slices and time + points are not its image's under the confirmed roles + (``mask_planes_mismatch``). Their pixels are not read: the mask step (:func:`.mask_scan.scan_masks`) does that. image_type The image type the user confirmed, ``"fluorescence"`` or ``"rgb"`` @@ -163,9 +164,7 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 raise ValueError(f"{name} must be True or False, found {value!r}") root = Path(experiment_root).resolve() output = default_output_folder(root) if output_folder is None else Path(output_folder).resolve() - pre = preflight( - root, output, exclude=exclude, masks_folder=masks_folder, ome_zarr_labels=ome_zarr_labels - ) + pre = preflight(root, output, exclude=exclude, masks_folder=masks_folder, mask_axes=mask_axes) # A plain stack changed since its roles were confirmed is not covered (audit 2 B-1). mapping = unchanged_confirmation(mapping, root) review = review_axes(pre.metadata, mapping, missing_as_singleton=missing_as_singleton) diff --git a/src/morphobricks_preprocessing/run/preflight.py b/src/morphobricks_preprocessing/run/preflight.py index 9bab31ce..6fa4e6e6 100644 --- a/src/morphobricks_preprocessing/run/preflight.py +++ b/src/morphobricks_preprocessing/run/preflight.py @@ -8,8 +8,8 @@ import logging import os -from collections.abc import Iterable -from dataclasses import dataclass, field +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field, replace from pathlib import Path from ..axes import ( @@ -19,6 +19,7 @@ ) from ..dataset import DatasetReport, check_dataset from ..discovery import DiscoveryResult, default_output_folder, discover_experiment +from ..mask_names import MASK_NAME_CODES, mask_name_issues from ..masks import MaskInventory, find_masks from ..metadata import MetadataReport, inspect_images from ..planes import ( @@ -146,7 +147,7 @@ def preflight( *, exclude: Iterable[str] = (), masks_folder: str | os.PathLike | None = None, - ome_zarr_labels: bool = False, + mask_axes: Mapping[str, str] | None = None, ) -> Preflight: """Pass 1: check the folder and the image headers without reading pixels. @@ -162,11 +163,10 @@ def preflight( masks_folder The masks folder (contract §15): one subfolder per mask type, each mirroring the experiment's folders. The masks are paired with the images - and checked from their headers; the label images of OME-Zarr images are - included. - ome_zarr_labels - Use the label images of OME-Zarr images as masks, also without a masks - folder. + and checked from their headers. The label images of OME-Zarr images are + always used as masks, also without a masks folder (design v0.4 §2). + mask_axes + The axes stated per mask type, ``{type: roles}`` (:func:`.masks.find_masks`). Returns ------- @@ -176,11 +176,14 @@ def preflight( output = default_output_folder(root) if output_folder is None else Path(output_folder) discovery = discover_experiment(root, output) excluded = tuple(sorted({_normalise(path) for path in exclude})) + discovery = _without_excluded_masks(discovery, excluded) images = [path for path in discovery.images if path not in excluded] metadata = inspect_images(root, images) - masks = None - if masks_folder is not None or ome_zarr_labels: - masks = find_masks(masks_folder, discovery, metadata.files) + masks = find_masks( + masks_folder, discovery, metadata.files, mask_axes=mask_axes, output_folder=output + ) + if masks_folder is None and not masks.files and not masks.issues: + masks = None # no masks folder and no label images: the run has no masks return Preflight( discovery=discovery, metadata=metadata, @@ -192,6 +195,17 @@ def preflight( ) +def _without_excluded_masks( + discovery: DiscoveryResult, excluded: tuple[str, ...] +) -> DiscoveryResult: + """Screen the mask names again, leaving out excluded images: excluding one clears it.""" + if not excluded: + return discovery + kept = [issue for issue in discovery.issues if issue.code not in MASK_NAME_CODES] + names = mask_name_issues(discovery.leaves, excluded=excluded) + return replace(discovery, issues=(*kept, *names)) + + def _exclusion_issues(discovery: DiscoveryResult, excluded: tuple[str, ...]) -> list: issues = [] found = [path for path in excluded if path in discovery.images] diff --git a/src/morphobricks_preprocessing/sources.py b/src/morphobricks_preprocessing/sources.py index b69c526a..95bcaa52 100644 --- a/src/morphobricks_preprocessing/sources.py +++ b/src/morphobricks_preprocessing/sources.py @@ -11,7 +11,7 @@ from pathlib import Path from .axes import file_mapping_for -from .discovery import position_id +from .image_names import position_id from .manifest import RunManifest, source_fingerprint from .metadata import inspect_image from .validation import ValidationIssue, error, plural diff --git a/tests/test_axis_check.py b/tests/test_axis_check.py index 1dd705a5..6bf3a689 100644 --- a/tests/test_axis_check.py +++ b/tests/test_axis_check.py @@ -323,10 +323,10 @@ def test_mask_sizes_are_compared_by_the_confirmed_roles(tmp_path): files = pre.metadata.files review = review_axes(files, prefill_mapping("CYX")) assert mask_size_issues(pre.masks, files, review.file_mappings) == [] - # By roles that swap Y and X the image is 8 x 6, so the 6 x 8 mask does not fit... + # By roles that swap Y and X the image is 8 x 6, so the 6 x 8 mask is refused... swapped = review_axes(files, AxisMapping("CYX", "CXY")) found = mask_size_issues(pre.masks, files, swapped.file_mappings) - assert codes(found) == ["mask_size_mismatch"] + assert codes(found) == ["mask_layout"] # stored the other way round: never transposed # ...but the image is never read, so the run leaves its masks to the conversion. assert mask_size_issues(pre.masks, files, swapped.file_mappings, skip=["ctrl/a.tif"]) == [] diff --git a/tests/test_discovery.py b/tests/test_discovery.py index 8e29019f..2e126985 100644 --- a/tests/test_discovery.py +++ b/tests/test_discovery.py @@ -708,7 +708,7 @@ def test_issue_text_lists_paths_and_action(tmp_path): def test_naming_other_formats_does_not_make_them_images(tmp_path): """NAME_SUFFIXES only names positions: discovery still accepts TIFF and Zarr images only.""" - from morphobricks_preprocessing.discovery import IMAGE_SUFFIXES, NAME_SUFFIXES + from morphobricks_preprocessing.image_names import IMAGE_SUFFIXES, NAME_SUFFIXES assert set(IMAGE_SUFFIXES) < set(NAME_SUFFIXES) and ".npy" not in IMAGE_SUFFIXES make_tree(tmp_path, ["ctrl/bio1/img_01.tif", "ctrl/bio1/img_02.png"]) @@ -763,35 +763,13 @@ def test_a_masks_folder_without_mask_names_is_not_warned(tmp_path): assert warned(["labels/rep1/g1.tif", "ctrl/rep1/f1.tif"], tmp_path) == {} -def test_images_named_with_the_default_mask_suffixes_are_warned(tmp_path): - found = warned( - [ - "ctrl/rep1/f1.tif", - "ctrl/rep1/f1_mask.tif", - "ctrl/rep1/f2.tif", - "ctrl/rep1/f2_Labels.tif", - "drug/rep1/f2_SEG.ome.tif", - ], - tmp_path, - ) - issue_ = found["image_named_as_mask"] - assert issue_.paths == ( - "ctrl/rep1/f1_mask.tif", - "ctrl/rep1/f2_Labels.tif", - "drug/rep1/f2_SEG.ome.tif", - ) - assert "ctrl/rep1/f1_mask.tif (of ctrl/rep1/f1.tif)" in issue_.message - # across the whole experiment, an image of the same folder first otherwise - assert "drug/rep1/f2_SEG.ome.tif (of ctrl/rep1/f2.tif)" in issue_.message - - def test_numbered_and_channel_images_are_images(tmp_path): names = ["A1.tif", "A1_2.tif", "f1.tif", "f1_DNA.tif", "f1_segmented.tif", "g_mask.tif"] assert warned([f"ctrl/rep1/{name}" for name in names], tmp_path) == {} def test_mask_names_the_caller_expects(tmp_path): - from morphobricks_preprocessing.discovery import mask_name_issues + from morphobricks_preprocessing.mask_names import mask_name_issues make_tree(tmp_path, ["c/r/f1.tif", "c/r/f1_Cells.tif", "c/r/f1_nuclei.tif"]) result = discover_experiment(tmp_path) @@ -827,7 +805,7 @@ def test_images_named_as_masks_with_the_callers_suffixes_only(tmp_path): def test_the_default_mask_names_are_cellpatchers_former_defaults(): from morphobricks_preprocessing import MASK_FOLDER_WORDS, MASK_NAME_SUFFIXES - assert MASK_NAME_SUFFIXES == ("_mask", "_labels", "_label", "_seg", "_segmentation") + assert {"_mask", "_labels", "_label", "_seg", "_segmentation"} < set(MASK_NAME_SUFFIXES) assert {"mask", "masks", "label", "labels", "seg", "segmentation"} <= MASK_FOLDER_WORDS @@ -843,6 +821,6 @@ def test_the_default_mask_names_are_cellpatchers_former_defaults(): ], ) def test_mask_folder_part(path, folder): - from morphobricks_preprocessing.discovery import mask_folder_part + from morphobricks_preprocessing.mask_names import mask_folder_part assert mask_folder_part(path) == folder diff --git a/tests/test_masks.py b/tests/test_masks.py index b848f802..448aeacc 100644 --- a/tests/test_masks.py +++ b/tests/test_masks.py @@ -110,7 +110,7 @@ def test_missing_and_unpaired_masks_are_warnings(experiment, masks): def test_masks_of_excluded_images_are_not_used(experiment, masks): found = inventory(experiment, masks, exclude=["drug/rep1/a.tif"]) assert "drug/rep1/a.tif" not in found.images - assert codes(found.issues) == ["mask_without_image"] + assert codes(found.issues) == ["mask_of_excluded_image"] # The type after the image name ------------------------------------------------------- @@ -268,14 +268,14 @@ def test_label_images_of_ome_zarr_images_are_masks(tmp_path): path = write_ngff(tmp_path / "exp" / "ctrl" / "a.ome.zarr", np.zeros((16, 20), np.uint8), "yx") write_labels(path, {"cells": label_image(0), "nuclei": label_image(1)}, "yx") - found = preflight(tmp_path / "exp", ome_zarr_labels=True).masks + found = preflight(tmp_path / "exp").masks assert found.ok assert found.types == ("cells", "nuclei") assert [f.path for f in found.files] == [ "ctrl/a.ome.zarr/labels/cells", "ctrl/a.ome.zarr/labels/nuclei", ] - assert preflight(tmp_path / "exp").masks is None # only when masks are asked for + assert found.folder is None # always used, without a masks folder (design v0.4 §2) # Run, manifest and command line ------------------------------------------------------- @@ -460,7 +460,7 @@ def test_label_images_with_time_points(tmp_path): path = write_ngff(tmp_path / "exp" / "a.ome.zarr", np.zeros((3, 16, 20), np.uint8), "tyx") data = np.stack([label_image(t) for t in range(3)]) write_labels(path, {"cells": data}, "tyx") - found = preflight(tmp_path / "exp", ome_zarr_labels=True).masks + found = preflight(tmp_path / "exp").masks assert found.ok cells = found.for_image("a.ome.zarr")["cells"] assert (cells.axes, cells.planes) == ("TYX", {"Z": 1, "T": 3}) @@ -469,7 +469,7 @@ def test_label_images_with_time_points(tmp_path): tmp_path / "exp", prefill_mapping("TYX"), output_folder=tmp_path / "out", - ome_zarr_labels=True, + mask_axes=None, # label images are always used (design v0.4 §2) ) assert result.written, result.issues @@ -538,7 +538,7 @@ def test_a_malformed_labels_group_is_an_error_not_a_vanished_mask_type(tmp_path) for path in paths: group = zarr.open_group(str(path / "labels"), mode="a") group.attrs.update({"labels": "cells"}) - found = preflight(tmp_path / "exp", ome_zarr_labels=True).masks + found = preflight(tmp_path / "exp").masks assert not found.ok assert codes(found.issues) == ["unreadable_mask"] assert found.issues[0].paths == ("ctrl/a.ome.zarr/labels", "drug/a.ome.zarr/labels") @@ -546,7 +546,7 @@ def test_a_malformed_labels_group_is_an_error_not_a_vanished_mask_type(tmp_path) tmp_path / "exp", prefill_mapping("YX"), output_folder=tmp_path / "out", - ome_zarr_labels=True, + mask_axes=None, # label images are always used (design v0.4 §2) ) assert not result.written and "unreadable_mask" in codes(result.issues) diff --git a/tests/test_masks_v040.py b/tests/test_masks_v040.py new file mode 100644 index 00000000..96cc33bb --- /dev/null +++ b/tests/test_masks_v040.py @@ -0,0 +1,319 @@ +"""Masks in v0.4.0: stated axes, layout, pairing, names and label images (design v0.4 §2, §3). + +Masks are never converted or reordered: they are checked as stored against the images as +read. Plain-stack masks get their axes stated once per mask type; a 2-D mask is shared by +every plane of its image only when stated; a mask stored with Y and X the other way round +is refused with a ``swapaxes`` hint; files named like masks are blocked as images; the +label images of OME-Zarr images are always used and fingerprinted apart from their image. +""" + +import json + +import numpy as np +import pytest + +from morphobricks_preprocessing import ( + MASK_NAME_SUFFIXES, + AxisMapping, + find_masks, + mask_axes_option, + prefill_mapping, + preflight, + read_manifest, + read_mask, + run_experiment, +) +from morphobricks_preprocessing.cli import main +from morphobricks_preprocessing.manifest import source_fingerprint +from morphobricks_preprocessing.masks.stated import mask_roles +from tiff_tree import write_data, write_file, write_plain + + +def codes(issues): + return [i.code for i in issues] + + +def errors(issues): + return [i.code for i in issues if i.is_error] + + +def one(issues, code): + (found,) = [i for i in issues if i.code == code] + return found + + +def labels(shape, start=1): + return (np.arange(np.prod(shape)).reshape(shape) % 7 + start).astype(np.uint16) + + +@pytest.fixture +def timelapse(tmp_path): + """One 3-frame time-lapse (TYX, 6 x 8) and its masks folder.""" + root = tmp_path / "exp" + write_data(root / "ctrl/r1/a.tif", np.zeros((3, 6, 8), np.uint8), "TYX") + return root, tmp_path / "masks" + + +def inventory(root, masks, **kwargs): + pre = preflight(root) + return find_masks(masks, pre.discovery, pre.metadata.files, **kwargs) + + +# Stated mask axes ---------------------------------------------------------------------- + + +def test_a_plain_stack_mask_needs_its_axes_stated(timelapse): + root, masks = timelapse + write_plain(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8))) + found = inventory(root, masks) + issue = one(found.issues, "mask_not_2d") + assert "header axes IYX" in issue.message and "--mask-axes Cells=TYX" in issue.action + stated = inventory(root, masks, mask_axes={"Cells": "TYX"}) + assert stated.ok + (mask,) = stated.files + assert (mask.axes, mask.header_axes, mask.axes_from) == ("TYX", "IYX", "stated") + # Read by the stated roles, as stored: frame 2 is the file's third page. + plane = read_mask(mask, stated.folder, root, t=2) + np.testing.assert_array_equal(plane, labels((3, 6, 8))[2]) + + +def test_the_run_checks_stated_mask_planes_against_the_image(timelapse): + root, masks = timelapse + write_plain(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8))) + ok = run_experiment( + root, prefill_mapping("TYX"), masks_folder=masks, mask_axes={"Cells": "TYX"} + ) + assert ok.written + (recorded,) = read_manifest(ok.manifest_path).masks["files"] + assert (recorded["roles"], recorded["axes_from"], recorded["shared"]) == ( + "TYX", + "stated", + False, + ) + # The axes as stored, beside the roles they are read as: nothing is reordered. + assert (recorded["axes"], recorded["shape"]) == ("IYX", [3, 6, 8]) + # Stated as depth, the mask's planes are not the time-lapse's. + wrong = run_experiment( + root, prefill_mapping("TYX"), masks_folder=masks, mask_axes={"Cells": "ZYX"} + ) + assert errors(wrong.issues) == ["mask_planes_mismatch"] + + +def test_stated_axes_may_not_contradict_a_named_mask(timelapse): + root, masks = timelapse + write_data(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8)), "TYX") + found = inventory(root, masks, mask_axes={"Cells": "ZYX"}) + issue = one(found.issues, "mask_axes_contradict_header") + assert "T stated Z" in issue.message + # Stating what the header names is fine; a named TYX mask keeps its own axes. + assert inventory(root, masks, mask_axes={"Cells": "TYX"}).ok + assert inventory(root, masks).ok + + +@pytest.mark.parametrize( + ("stated", "problem"), + [("TY", "2 roles for 3 axes"), ("CYX", "C is not a mask role"), ("TYY", "Y and X")], +) +def test_invalid_stated_axes(timelapse, stated, problem): + root, masks = timelapse + write_plain(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8))) + issue = one(inventory(root, masks, mask_axes={"Cells": stated}).issues, "invalid_mask_axes") + assert problem in issue.message + + +def test_axes_stated_for_an_unknown_mask_type(timelapse): + root, masks = timelapse + write_data(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8)), "TYX") + issue = one(inventory(root, masks, mask_axes={"Nuclei": "TYX"}).issues, "invalid_mask_axes") + assert "Nuclei" in issue.message and "found: Cells" in issue.message + + +def test_a_2d_mask_on_a_stack_needs_a_shared_statement(timelapse): + root, masks = timelapse + write_data(masks / "Cells/ctrl/r1/a.tif", labels((6, 8)), "YX") + refused = run_experiment(root, prefill_mapping("TYX"), masks_folder=masks) + assert "mask_planes_mismatch" in errors(refused.issues) + # Stated YX for the type: one plane shared by every plane of the image. + shared = run_experiment( + root, prefill_mapping("TYX"), masks_folder=masks, mask_axes={"Cells": "YX"} + ) + assert shared.written + (recorded,) = read_manifest(shared.manifest_path).masks["files"] + assert (recorded["axes_from"], recorded["shared"]) == ("stated", True) + pre = preflight(root, masks_folder=masks, mask_axes={"Cells": "YX"}) + (mask,) = pre.masks.files + assert mask.shared and mask.axes_from == "stated" + np.testing.assert_array_equal(read_mask(mask, pre.masks.folder, root, t=2), labels((6, 8))) + + +def test_a_2d_mask_on_a_single_plane_image_needs_no_statement(tmp_path): + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + write_plain(tmp_path / "masks/Cells/ctrl/a.tif", labels((6, 8))) + result = run_experiment(root, prefill_mapping("YX"), masks_folder=tmp_path / "masks") + assert result.written + (recorded,) = read_manifest(result.manifest_path).masks["files"] + assert (recorded["axes_from"], recorded["shared"]) == ("header", False) + + +def test_a_named_tyx_mask_keeps_its_own_axes(timelapse): + root, masks = timelapse + write_data(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8)), "TYX") + assert run_experiment(root, AxisMapping("TYX", "TYX"), masks_folder=masks).written + + +def test_mask_roles(): + assert mask_roles("TYX", "TYX", None) == ("TYX", None, "") + assert mask_roles("QYX", "???", "TYX") == ("TYX", None, "") + assert mask_roles("QYX", "???", "TXY")[1] == "mask_layout" + assert mask_roles("QYX", "???", None)[1] == "mask_not_2d" + + +def test_the_mask_axes_option(): + assert mask_axes_option(["Cells=TYX", " Nuclei = YX "]) == {"Cells": "TYX", "Nuclei": "YX"} + assert mask_axes_option(["Cells=TYX", "Cells=TYX"]) == {"Cells": "TYX"} + with pytest.raises(ValueError, match="give the mask type and its roles"): + mask_axes_option(["TYX"]) + with pytest.raises(ValueError, match="two different roles for Cells"): + mask_axes_option(["Cells=TYX", "Cells=ZYX"]) + + +def test_the_command_line_states_mask_axes(timelapse, capsys): + root, masks = timelapse + write_plain(masks / "Cells/ctrl/r1/a.tif", labels((3, 6, 8))) + args = ["check", str(root), "--masks", str(masks)] + assert main(args) == 1 + assert "--mask-axes Cells=TYX" in capsys.readouterr().out + assert main([*args, "--mask-axes", "Cells=TYX"]) == 0 + assert main([*args, "--mask-axes", "TYX"]) == 1 + assert "give the mask type and its roles" in capsys.readouterr().out + + +# Layout: never transposed ------------------------------------------------------------ + + +def test_a_mask_stored_x_before_y_is_refused_with_a_swapaxes_hint(tmp_path): + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + write_data(tmp_path / "masks/Cells/ctrl/a.tif", labels((8, 6)), "XY") + pre = preflight(root, masks_folder=tmp_path / "masks") + issue = one(pre.masks.issues, "mask_layout") + assert "numpy.swapaxes(mask, -1, -2)" in issue.action + assert "X is stored before Y" in issue.message + + +def test_a_mask_with_the_images_height_and_width_swapped_is_a_layout_error(tmp_path): + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + write_data(tmp_path / "masks/Cells/ctrl/a.tif", labels((8, 6)), "YX") + result = run_experiment(root, prefill_mapping("YX"), masks_folder=tmp_path / "masks") + issue = one(result.issues, "mask_layout") + assert issue.paths == ("Cells/ctrl/a.tif",) and "swapaxes" in issue.action + assert "mask_size_mismatch" not in codes(result.issues) and not result.written + + +# Pairing ------------------------------------------------------------------------------- + + +def test_unpaired_masks_stay_a_warning(tmp_path): + # Design v0.4 §3: every existing check keeps its severity. + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + write_data(tmp_path / "masks/Cells/ctrl/a.tif", labels((6, 8)), "YX") + write_data(tmp_path / "masks/Cells/ctrl/zz.tif", labels((6, 8)), "YX") + write_file(tmp_path / "masks/Cells/ctrl/a.png") + pre = preflight(root, masks_folder=tmp_path / "masks") + issue = one(pre.masks.issues, "mask_without_image") + assert pre.masks.ok and issue.paths == ("Cells/ctrl/a.png", "Cells/ctrl/zz.tif") + + +def test_masks_of_excluded_images_are_reported_apart(tmp_path): + root = tmp_path / "exp" + for name in ("a", "b"): + write_data(root / f"ctrl/{name}.tif", np.zeros((6, 8), np.uint8), "YX") + write_data(tmp_path / f"masks/Cells/ctrl/{name}.tif", labels((6, 8)), "YX") + pre = preflight(root, masks_folder=tmp_path / "masks", exclude=["ctrl/b.tif"]) + assert pre.masks.ok and codes(pre.masks.issues) == ["mask_of_excluded_image"] + assert one(pre.masks.issues, "mask_of_excluded_image").paths == ("Cells/ctrl/b.tif",) + assert [f.image for f in pre.masks.files] == ["ctrl/a.tif"] + + +def test_the_output_folder_may_not_be_inside_the_masks_folder(tmp_path): + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + masks = tmp_path / "masks" + write_data(masks / "Cells/ctrl/a.tif", labels((6, 8)), "YX") + result = run_experiment( + root, prefill_mapping("YX"), masks_folder=masks, output_folder=masks / "out" + ) + assert errors(result.issues) == ["output_inside_masks"] and not result.written + assert one(result.issues, "output_inside_masks").paths == ("out",) + + +# Files named like masks are blocked as images (design v0.4 §3) --------------------------- + + +@pytest.mark.parametrize("name", ["a_cp_masks.tif", "a_masks.tif", "a_mask.tif", "a_SEG.tif"]) +def test_files_named_like_masks_are_blocked_as_images(tmp_path, name): + root = tmp_path / "exp" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + write_data(root / f"ctrl/{name}", labels((6, 8)), "YX") + result = run_experiment(root, prefill_mapping("YX")) + issue = one(result.issues, "image_named_as_mask") + assert issue.is_error and issue.paths == (f"ctrl/{name}",) and not result.written + assert "(of ctrl/a.tif)" in issue.message and "exclude them" in issue.action + # Excluded, it is no longer an image, so nothing is blocked. + assert run_experiment(root, prefill_mapping("YX"), exclude=[f"ctrl/{name}"]).written + + +def test_the_mask_name_suffixes_include_cellposes(): + assert {"_cp_masks", "_masks"} < set(MASK_NAME_SUFFIXES) + + +def test_a_mask_named_after_an_image_in_another_folder_is_blocked(tmp_path): + root = tmp_path / "exp" + write_data(root / "ctrl/rep1/f2.tif", np.zeros((6, 8), np.uint16), "YX") + write_data(root / "drug/rep1/f2_SEG.ome.tif", labels((6, 8)), "YX") + pre = preflight(root) + issue = one(pre.issues, "image_named_as_mask") + assert "drug/rep1/f2_SEG.ome.tif (of ctrl/rep1/f2.tif)" in issue.message + assert [i.code for i in pre.issues if i.is_error] == ["image_named_as_mask"] + + +# OME-Zarr label images: always used, fingerprinted on their own (design v0.4 §2) ----------- + + +def zarr_experiment(tmp_path): + pytest.importorskip("zarr") + from zarr_tree import write_labels, write_ngff + + path = write_ngff(tmp_path / "exp/ctrl/a.ome.zarr", np.zeros((6, 8), np.uint8), "yx") + write_labels(path, {"cells": labels((6, 8))}, "yx") + return tmp_path / "exp", path + + +def test_label_images_are_always_used(tmp_path): + root, _ = zarr_experiment(tmp_path) + with pytest.raises(TypeError, match="ome_zarr_labels"): + preflight(root, ome_zarr_labels=True) # removed: there is no flag any more + result = run_experiment(root, prefill_mapping("YX")) + assert result.written + recorded = read_manifest(result.manifest_path).masks + assert recorded["folder"] is None and recorded["types"] == ["cells"] + (entry,) = recorded["files"] + size, mtime_ns = source_fingerprint(root / "ctrl/a.ome.zarr/labels/cells") + assert (entry["size"], entry["mtime_ns"]) == (size, mtime_ns) + + +def test_label_images_are_left_out_of_the_image_fingerprint(tmp_path): + from zarr_tree import write_labels + + root, path = zarr_experiment(tmp_path) + before = source_fingerprint(path) + first = run_experiment(root, prefill_mapping("YX")) + write_labels(path, {"cells": labels((6, 8), 2), "nuclei": labels((6, 8))}, "yx") + assert source_fingerprint(path) == before + again = run_experiment(root, prefill_mapping("YX"), output_folder=first.output_folder) + assert again.read == () # the image's rows are reused + data = json.loads(again.manifest_path.read_text(encoding="utf-8")) + assert data["masks"]["types"] == ["cells", "nuclei"] diff --git a/tools/code_rules_baseline.json b/tools/code_rules_baseline.json index da575e7e..b9984c8f 100644 --- a/tools/code_rules_baseline.json +++ b/tools/code_rules_baseline.json @@ -12,10 +12,9 @@ "src/morphobricks_preprocessing/axis_review.py check_mapping: # executable lines (max #)": 72, "src/morphobricks_preprocessing/axis_review.py review_axes: # executable lines (max #)": 106, "src/morphobricks_preprocessing/cli.py _check: private function of # lines without a docstring": 52, - "src/morphobricks_preprocessing/cli.py _common: private function of # lines without a docstring": 19, - "src/morphobricks_preprocessing/cli.py _masks: private function of # lines without a docstring": 39, - "src/morphobricks_preprocessing/cli.py _run: # executable lines (max #)": 78, - "src/morphobricks_preprocessing/cli.py _run: private function of # lines without a docstring": 78, + "src/morphobricks_preprocessing/cli.py _masks: private function of # lines without a docstring": 34, + "src/morphobricks_preprocessing/cli.py _run: # executable lines (max #)": 75, + "src/morphobricks_preprocessing/cli.py _run: private function of # lines without a docstring": 75, "src/morphobricks_preprocessing/cli.py build_parser: # executable lines (max #)": 108, "src/morphobricks_preprocessing/dataset.py _missing_issues: private function of # lines without a docstring": 32, "src/morphobricks_preprocessing/dataset.py _unbalanced_issues: private function of # lines without a docstring": 19, @@ -25,20 +24,18 @@ "src/morphobricks_preprocessing/design.py: # lines (max #)": 964, "src/morphobricks_preprocessing/discovery.py _check_names: private function of # lines without a docstring": 14, "src/morphobricks_preprocessing/discovery.py finish: # executable lines (max #)": 145, - "src/morphobricks_preprocessing/discovery.py: # lines (max #)": 847, + "src/morphobricks_preprocessing/discovery.py: # lines (max #)": 610, "src/morphobricks_preprocessing/image_type.py _rgb_problem: private function of # lines without a docstring": 12, "src/morphobricks_preprocessing/manifest.py file_entry_for: # parameters (max #)": 9, - "src/morphobricks_preprocessing/masks.py _duplicate_issues: private function of # lines without a docstring": 19, - "src/morphobricks_preprocessing/masks.py _folder_issues: private function of # lines without a docstring": 19, - "src/morphobricks_preprocessing/masks.py _name_issues: private function of # lines without a docstring": 15, - "src/morphobricks_preprocessing/masks.py: # lines (max #)": 878, + "src/morphobricks_preprocessing/masks/files.py _duplicate_issues: private function of # lines without a docstring": 19, + "src/morphobricks_preprocessing/masks/files.py _name_issues: private function of # lines without a docstring": 15, "src/morphobricks_preprocessing/metadata.py _check_channel_names: private function of # lines without a docstring": 32, "src/morphobricks_preprocessing/metadata.py _check_dimensions: private function of # lines without a docstring": 30, "src/morphobricks_preprocessing/metadata.py _check_image_sizes: private function of # lines without a docstring": 15, "src/morphobricks_preprocessing/metadata.py _check_pixel_sizes: # executable lines (max #)": 71, "src/morphobricks_preprocessing/metadata.py _check_plane_counts: private function of # lines without a docstring": 27, "src/morphobricks_preprocessing/metadata.py check_metadata: # executable lines (max #)": 98, - "src/morphobricks_preprocessing/metadata.py: # lines (max #)": 934, + "src/morphobricks_preprocessing/metadata.py: # lines (max #)": 920, "src/morphobricks_preprocessing/notebook_ui.py __init__: # executable lines (max #)": 110, "src/morphobricks_preprocessing/notebook_ui.py _bins_help: private function of # lines without a docstring": 12, "src/morphobricks_preprocessing/notebook_ui.py _browse_csv: private function of # lines without a docstring": 11, @@ -87,12 +84,12 @@ "src/morphobricks_preprocessing/planes.py _zarr_layout: private function of # lines without a docstring": 40, "src/morphobricks_preprocessing/planes.py check_planes: # executable lines (max #)": 65, "src/morphobricks_preprocessing/planes.py: # lines (max #)": 1422, - "src/morphobricks_preprocessing/run/pipeline.py run_experiment: # executable lines (max #)": 161, + "src/morphobricks_preprocessing/run/pipeline.py run_experiment: # executable lines (max #)": 159, "src/morphobricks_preprocessing/run/pipeline.py run_experiment: # parameters (max #)": 15, "src/morphobricks_preprocessing/run/preflight.py _exclusion_issues: private function of # lines without a docstring": 34, "src/morphobricks_preprocessing/run/rows.py _bin_count: private function of # lines without a docstring": 12, "src/morphobricks_preprocessing/run/rows.py _pixel_issues: private function of # lines without a docstring": 38, - "tests/test_discovery.py: # lines (max #)": 848, + "tests/test_discovery.py: # lines (max #)": 826, "tests/test_masks.py: # lines (max #)": 612, "tests/test_metadata.py: # lines (max #)": 629, "tests/test_notebook_ui.py: # lines (max #)": 1604,