diff --git a/docs/contract.md b/docs/contract.md index 9ba34704..ff3fc068 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -163,7 +163,7 @@ file's schema or statistical interpretation. | `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. | | `csv_sha256` | SHA-256 of the table written with this manifest. | | `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. | -| `masks` | The masks paired with the images (§15), when the run was given a masks folder: `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes, `YX` for a 2D mask, with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the manifest stays version 1, and the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes. | +| `masks` | The masks paired with the images (§15), when the run was given a masks folder: `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes other than Y and X, `YX` for a 2D mask (`[1, 512]` for a mask one pixel high), with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the manifest stays version 1, and the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes. | | `image_type` | `fluorescence` or `rgb`, the image type the user confirmed (§13); null when the run was given none, or absent in manifests written before it. Optional key: the manifest stays version 1. Changing it recomputes nothing. | | `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. | | `intensity_checks` | The limits of the intensity checks run on the table (§14): `IntensityCheckSettings.to_dict()` of the normalization brick (`library_version`, `outlier_limit`, `saturation_limit`, `min_images`); absent in manifests written before it. Optional key. Changing the limits recomputes nothing. | @@ -315,7 +315,7 @@ masks/ masks folder, outside the experiment folder | Layout | Either one subfolder per mask type, each mirroring the experiment's folders (`//.../.tif`), or the experiment's folders with the type after the image name (`/.../_.tif`, as segmentation tools often write it, e.g. MorphoRelate's JUMP-Target-2 conversion); the two may be mixed. Any number of mask types. The label images of an OME-Zarr image (its OME-NGFF `labels` group, with the `zarr` extra) are masks too, of the type named by the label image; they are used whenever masks are asked for, with or without a masks folder. | | Pairing | A mask belongs to the image in the same leaf folder with the same `position_id` (§1), ignoring upper/lower case; its file is a TIFF (`.tif`, `.tiff`, optionally `.ome`, any case). Excluded images have no masks. | | Names | A mask type is named by its folder, by the suffix after the image name, or by its label image. A file that could belong to two images (an image name that starts another's, `a_b_cells.tif` beside images `a` and `a_b`) or fit both layouts is the error `ambiguous_mask`. Two names differing only in case are the error `mask_names_differ_in_case`; one image with two masks of one type (two extensions, or a file and a label image) is the error `duplicate_mask`. | -| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes are removed, a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed. A mask has the Y and X sizes of its image (`mask_size_mismatch`) and integer or boolean pixels (`mask_pixel_type`; a mask holding a negative label or a label above 2,147,483,647, the int32 limit every brick shares (§16, decision D-b), raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`, as is an OME-Zarr image's `labels` group that cannot be read or does not list its label images (path `/labels`), so its mask types never disappear unreported. All five are errors. | +| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes other than Y and X are removed (Y and X are kept whatever their length, so a mask one pixel high or wide, or a single pixel, is a 2D `YX` mask; a TIFF whose series has no Y or X but is one 2D page, such as a 1D array `tifffile` saved as `{"shape": [N]}` and reads as `X`, is that page, `YX` of 1 row; plan §12 question 44), a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed. A mask has the Y and X sizes of its image (`mask_size_mismatch`) and integer or boolean pixels (`mask_pixel_type`; a mask holding a negative label or a label above 2,147,483,647, the int32 limit every brick shares (§16, decision D-b), raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`, as is an OME-Zarr image's `labels` group that cannot be read or does not list its label images (path `/labels`), so its mask types never disappear unreported. All five are errors. | | Planes | A mask belongs to the image's planes (issue #36): its Z and T lengths (1 for an axis it lacks) must equal the image's under the confirmed axis roles, checked by `run` (`mask_planes_mismatch`, error; `mask_plane_issues`). So a 2D mask belongs only to an image with one depth slice and one time point, and is refused for a Z-stack or a time-lapse, even as a projection (a decision for the owner if wanted later); a mask's plane at index `i` along Z or T belongs to the image's rows with `z = i` or `t = i`. `read_mask(..., z=, t=)` reads one plane and requires `z` and `t` for a mask with those axes. `check` and the `masks` command have no confirmed roles and do not compare with the image. | | Folder | The masks folder must exist (`masks_folder_not_found`) and be outside the experiment folder (`masks_inside_experiment`), where its folders would be read as labels. Both errors. | | Warnings | `mask_missing` (images without a mask of a type that other images have: their objects of that type cannot be measured), `mask_without_image` (files matching no analysed image), `mask_outside_type_folder` (files directly in the masks folder that are not TIFF masks). | @@ -406,9 +406,9 @@ instead of keeping copies. | Length units (D-a2) | `space_unit(text)`, `to_micrometers(value, unit)`, `SPACE_UNITS`, `UnsupportedUnitError` (`physical_sizes`) | Microscopy units only: `pm`; `Å`/`angstrom`; `nm`; `µm`/`um`/`micron(s)`/`micrometer(s)`/`micrometre(s)`; `mm`; `cm`; `m`; with each name in US and British spelling and in the plural (`nanometre`, `millimeters`, ...). Symbols of two letters or fewer are matched exactly (`Mm` is not `mm`), longer names in any case; ImageJ's `\u00B5` is decoded. `space_unit` returns the OME-NGFF name (`"micrometer"`), None when no unit is given, and refuses any other unit by name (`UnsupportedUnitError`); `to_micrometers` converts a positive finite length, None when the length or its unit is not known. This brick's readers (§2) use the same table: since D86, ImageJ and OME-NGFF sizes in British spellings (`micrometre`) are calibrated, where they were unknown before. | | Plain-TIFF resolution (D-c) | `read_physical_sizes` | A TIFF's resolution tags give a pixel size only with OME or ImageJ metadata and a unit; a plain TIFF (scanner or image-editor dpi) is uncalibrated, never converted from inches or centimetres. | | Labels (D-b) | `check_labels(pixels, name)`, `LARGEST_LABEL` (`masks`) | A mask holds whole-number labels from 0 (background) to `LARGEST_LABEL` = 2,147,483,647: MorphoBricks labels are int32 in every brick. A negative label, a label above the limit, or a pixel type that is not integer or boolean is refused with a `ValueError` naming the mask and the value. `read_mask`, `piece_counts`, `pair_counts` and `scan_masks` call it. Since D86 the limit is int32; before, this brick refused only labels above 2^63 − 1 (int64), so a `uint32` or wider mask with a label of 3,000,000,000 is now refused. | -| Header axes (D-d) | `header_axes(meta)`, `named_axes(meta)`, `series_axes_from_metadata(series)`, `PLACEHOLDER_AXES`, `OPEN_AXIS` (`metadata`) | The axes an image's own metadata names, as `tifffile` reads them with length-1 axes left out: OME, ImageJ and `tifffile` "shaped" metadata (`{"axes": "CYX"}`, decision D-d), a plain 2D `YX` (or `YXS` RGB) TIFF, and an OME-Zarr image's OME-NGFF axes. None when an axis is a placeholder (`I`: a plain stack of pages; `Q`: shaped without axes, or an OME-NGFF axis of another type), or the image does not open: the file does not say which dimension is depth, time or channel. `named_axes` gives the rule position by position, with `?` (`OPEN_AXIS`) for a dimension the metadata does not name: a placeholder, and, in a plain stack of pages without axis metadata (`IYX`, `QYX`, `QQYX`; `TiffMetadata.axes_from_metadata` False, from `series_axes_from_metadata`; True, the strict reading, for a header built by hand), the `Y` and `X` too, which only say where each page's rows and columns are stored (a (Y, X, C) array written without metadata reads as `QYX`). `header_axes` gives axes exactly when nothing is open, and `check_declared_axes` then accepts those axes, them with `Y` and `X` swapped, and (without a C axis) `S` declared C; nothing else. | +| Header axes (D-d) | `header_axes(meta)`, `named_axes(meta)`, `series_axes_from_metadata(series)`, `PLACEHOLDER_AXES`, `OPEN_AXIS` (`metadata`) | The axes an image's own metadata names, as `tifffile` reads them with length-1 axes left out (`tifffile` keeps Y and X even of length 1, so an image one pixel high is `YX`): OME, ImageJ and `tifffile` "shaped" metadata (`{"axes": "CYX"}`, decision D-d), a plain 2D `YX` (or `YXS` RGB) TIFF, and an OME-Zarr image's OME-NGFF axes. None when an axis is a placeholder (`I`: a plain stack of pages; `Q`: shaped without axes, or an OME-NGFF axis of another type), or the image does not open: the file does not say which dimension is depth, time or channel. `named_axes` gives the rule position by position, with `?` (`OPEN_AXIS`) for a dimension the metadata does not name: a placeholder, and, in a plain stack of pages without axis metadata (`IYX`, `QYX`, `QQYX`; `TiffMetadata.axes_from_metadata` False, from `series_axes_from_metadata`; True, the strict reading, for a header built by hand), the `Y` and `X` too, which only say where each page's rows and columns are stored (a (Y, X, C) array written without metadata reads as `QYX`). `header_axes` gives axes exactly when nothing is open, and `check_declared_axes` then accepts those axes, them with `Y` and `X` swapped, and (without a C axis) `S` declared C; nothing else. | | Declared axes | `check_declared_axes(meta, declared)`, `declared_axes_transposed(meta, declared)` (`axes`) | Axes a brick or user declares for an image or mask (`image_axes="CYX"`) may name what the header leaves open, never contradict it: one letter per header axis (any case), roles X, Y, Z, C, T each at most once, with `Y` and `X` declared when the header has them (a plain stack too); an axis the metadata names (`named_axes`) keeps its letter, except that the named `Y` and `X` may be declared swapped (header `CYX` declared `CXY`, `ZYX` declared `ZXY`): the image is read transposed, and `declared_axes_transposed` returns True so the caller can record it (owner decision of 31 October 2026; it raises `ValueError` for a declaration that does not fit); any other disagreement is refused; a dimension it leaves open may be declared any role, so a plain `QYX` stack may be declared `YXC` as well as `CYX` (since v0.3.4; before, its positional `Y` and `X` were binding and a placeholder could not be declared X or Y), while an ImageJ `ZYX` declared `CYX`, an OME `CYX` declared `YXC` or a shaped `CYX` declared `YXC` are refused; the RGB samples `S` may be declared C when the header has no C; any other header axis (tile `R`, ...) cannot be declared. Otherwise the error `declared_axes_mismatch` naming the image and both axes (`unreadable_tiff` or `unreadable_ome_zarr` when the header does not open). | -| Mask files | `read_mask_file(path, z=, t=)` (`masks`) | Reads one plane of a TIFF mask or of an OME-Zarr label image (`.zarr/labels/`) given by its path, with `read_mask`'s rules (§15): its axes are its own metadata's, Y and X plus at most a Z and a T axis named so; any other or unnamed dimension (a plain stack, `IYX`; a `C` axis) is refused, never taken for depth or time; `z` and `t` are required for a mask with that axis; integer or boolean pixels; labels checked with `check_labels`. A `ValueError` names the file and the problem. | +| Mask files | `read_mask_file(path, z=, t=)` (`masks`) | Reads one plane of a TIFF mask or of an OME-Zarr label image (`.zarr/labels/`) given by its path, with `read_mask`'s rules (§15): its axes are its own metadata's, Y and X plus at most a Z and a T axis named so, length-1 axes other than Y and X left out (a mask one pixel high or wide is read as a 2D array, plan §12 question 44); any other or unnamed dimension (a plain stack, `IYX`; a `C` axis) is refused, the message giving the header's axes and shape and the dimension that is not named, never taken for depth or time; `z` and `t` are required for a mask with that axis; integer or boolean pixels; labels checked with `check_labels`. A `ValueError` names the file and the problem. | | Identity (D-e) | `FOV_KEY`, `check_identity_text(value, field)`, `check_plane_index(value, axis)`, `check_identity_case(keys)`, `MISSING_TEXT`, `IdentityError` (`discovery`) | `FOV_KEY` = (`condition`, `biological_repeat`, `technical_repeat`, `position_id`), the position key of §1. `check_identity_text` accepts a label or `position_id` that is text without leading or trailing whitespace and not `""`, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case (`MISSING_TEXT`); `check_plane_index` accepts None or an integer of at least 0, a Python `int` or a NumPy integer (returned as a plain `int`), and refuses everything else (bool, float, `"1"`) without converting it. Both raise `IdentityError` (a `ValueError`) naming the field and the value. Discovery applies the text rule to every folder and image name (`invalid_name`) and refuses names of one level that differ only in case anywhere in the experiment (`case_conflict`, §1). `position_id` keeps its own scope: unique within its technical repeat, ignoring case (`duplicate_position_id`). `check_identity_case` applies the same case rule to identities a brick reads from a table, one key per row in `FOV_KEY` order (or the three labels): a label of one level spelled two ways anywhere in the run, or two `position_id` values of one technical repeat equal ignoring case, raise `IdentityError` naming both spellings; the same key may repeat (several planes of one position). | | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | diff --git a/docs/plan.md b/docs/plan.md index df795a45..2b3448d8 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1013,3 +1013,25 @@ Raised during review; to be decided before the milestones that need them: default design: no file is the first run, not an unreadable one. The notebook keeps the behaviour of question 42 (file untouched, refusal shown, an answer replaces it), and `check` prints the refusal and writes nothing. Breaking change for the next release.* +44. How is a mask one pixel high or wide read (reported by the fiberfeatures brick)? A valid + 2D label mask of 1 x N, N x 1 or 1 x 1 pixels was refused: `read_mask_file` said it had + the axes `X` (or `Y`, or none) and `find_masks` gave `mask_not_2d`. `tifffile` (and + OME-NGFF) give such a mask the axes `YX`, with or without axis metadata; this brick's + own mask header dropped every length-1 axis, Y and X included, and the plane readers did + the same. Images were not affected: their axes are `tifffile`'s, which keep Y and X. + *Decided (31 October 2026, a fix within the existing rules): a mask's length-1 axes are + left out except Y and X, in the header (`MaskFile.axes`/`shape`, the manifest's `masks` + entries) and in `read_mask`/`read_mask_file`, which return the 2D (1, N), (N, 1) or + (1, 1) plane; a named Z or T stack of such planes keeps its planes. `tifffile` gives the + axes `X` alone only for a 1D array saved with its default metadata (`{"shape": [N]}`); + such a file is one TIFF page of 1 row and N columns, and a mask series without Y or X + stored in one 2D page is read as that page (`YX`, (1, N)), as any plain 2D page is + (decision D-d). Nothing is guessed: Y and X are the ones the file's header gives, and a + dimension it does not name (a plain stack `IYX`) is still refused, the message now + giving the header's axes and shape and the dimension that is not named instead of the + axes left after dropping Y or X. Masks + accepted before are recorded as before. Ships in the next release (v0.3.5), with the + fixes of questions 42 and 43. Open for the owner: an image (not a mask) saved as such a + 1D array keeps the header axes `X` (`header_axes`, `check_declared_axes`), so the folder + check suggests a mapping without Y and the run refuses it with `missing_xy_axis`; + reading it as one row too would change the shared header-axes rule (§16).* diff --git a/docs/user_guide.md b/docs/user_guide.md index a268f4d2..7f532ecd 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -596,7 +596,7 @@ continues. | `mask_names_differ_in_case` | Error | Two kinds of mask have names that differ only in upper/lower case. Rename one. | | `duplicate_mask` | Error | An image has two masks of one kind (two extensions, or a file and an OME-Zarr label image of the same name). Keep one. | | `unreadable_mask` | Error | A mask cannot be opened, or holds several images, or the `labels` group of an OME-Zarr image cannot be read (it must list its label images). Save the mask again as a TIFF label image, or repair the OME-Zarr labels. | -| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). | +| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). | | `mask_planes_differ` | Error | The masks of one image have different numbers of depth slices or time points. Make every mask of an image with the image's planes. | | `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. | | `mask_size_mismatch` | Error | A mask does not have the width and height of its image. Check that it was made from that image at full resolution. | diff --git a/src/morphobricks_preprocessing/masks.py b/src/morphobricks_preprocessing/masks.py index f03c779f..1e7be877 100644 --- a/src/morphobricks_preprocessing/masks.py +++ b/src/morphobricks_preprocessing/masks.py @@ -18,7 +18,8 @@ image name, or by its label image, and two names may not differ only in case. Only headers are read here, never pixels: a mask has the image's Y and X sizes and integer or boolean pixels. Its other dimensions (length-1 axes -aside) must be labelled depth (Z) or time (T) in its own metadata, never guessed, and every +aside; Y and X are kept whatever their length, so a mask one pixel high is a 2D ``YX`` mask) +must be labelled depth (Z) or time (T) in its own metadata, never guessed, and every mask of one image has the same depth slices and time points. Whether those are the image's planes needs the confirmed axis roles: :func:`mask_plane_issues`, run by :func:`.run.run_experiment` (issue #36). Reading the pixels and relating the masks to each @@ -61,6 +62,9 @@ PLANE_AXES = ("Z", "T") """The dimensions a mask may have besides Y and X: depth slices and time points.""" +_YX = frozenset("YX") +"""The axes of a mask plane, kept by the header and the readers even when of length 1.""" + @dataclass(frozen=True) class MaskFile: @@ -78,7 +82,8 @@ class MaskFile: source :data:`FOLDER` or :data:`OME_ZARR`. shape - Shape of the mask without its length-1 axes, ``(Y, X)`` for a 2D mask. + Shape of the mask without its length-1 axes other than Y and X, ``(Y, X)`` for a 2D + mask (``(1, 512)`` for a mask one pixel high). dtype Pixel type, or None when the header could not be read. axes @@ -259,10 +264,7 @@ def read_mask( name = mask.path.rsplit("/labels/", 1)[1] image = experiment_root / mask.path.rsplit("/labels/", 1)[0] labels = open_ngff_labels(image, name) - selection = tuple( - (wanted[a] or 0) if a in wanted and n > 1 else (0 if n == 1 else slice(None)) - for a, n in zip(labels.axes, labels.array.shape, strict=True) - ) + selection = _plane_index(labels.axes, labels.array.shape, wanted) pixels = np.asarray(labels.array[selection]) axes = "".join(a for a, s in zip(labels.axes, selection, strict=True) if s == slice(None)) if axes != "YX" or (mask.axes and pixels.shape != tuple(_yx(mask.axes, mask.shape))): @@ -278,7 +280,8 @@ def read_mask_file( For bricks given a mask path directly (MorphoBricks D86): the mask is read as :func:`read_mask` reads it, with the same rules. Its axes are those its metadata names - (length-1 axes aside): Y and X, plus at most a depth axis Z and a time axis T; a + (length-1 axes aside, except Y and X: a mask one pixel high or wide is read as a 2D + array): Y and X, plus at most a depth axis Z and a time axis T; a dimension the metadata does not name (a plain stack, ``IYX``) is refused, never taken for depth, time or a channel. ``z`` and ``t`` select the plane and are required for a mask with that axis. The labels are checked with :func:`check_labels`. @@ -312,8 +315,7 @@ def read_mask_file( raise ValueError(f"the mask {path} could not be opened: {exc}") from exc if not _known_axes(axes): raise ValueError( - f"the mask {path} has axes {axes!r}: dimensions other than Y and X must be named " - "depth (Z) or time (T) in its metadata" + f"the mask {path} has axes {axes!r} (shape {shape}): {_axes_problem(axes, shape)}" ) if dtype.kind not in "biu": raise ValueError( @@ -337,11 +339,8 @@ def _read_tiff_plane(path: Path, wanted: Mapping[str, int | None]) -> tuple[np.n """ with tifffile.TiffFile(path) as tif: series = tif.series[0] - axes, shape = series.axes, tuple(series.shape) - index = tuple( - (wanted.get(a) or 0) if n > 1 and a in wanted else (0 if n == 1 else slice(None)) - for a, n in zip(axes, shape, strict=True) - ) + axes, shape = _tiff_axes(series) + index = _plane_index(axes, shape, wanted) kept = "".join(a for a, i in zip(axes, index, strict=True) if i == slice(None)) pages = len(series.pages) split = next( @@ -358,10 +357,48 @@ def _read_tiff_plane(path: Path, wanted: Mapping[str, int | None]) -> tuple[np.n pixels = tif.asarray(key=key, series=0).reshape(shape[split:]) pixels = pixels[index[split:]] else: - pixels = series.asarray()[index] + pixels = series.asarray().reshape(shape)[index] return pixels, kept +def _tiff_axes(series: tifffile.TiffPageSeries) -> tuple[str, tuple[int, ...]]: + """Return the axes and shape of a TIFF mask's series, as ``tifffile`` gives them. + + One exception: a series without Y or X stored in one 2D page, for example a 1D array + written with ``tifffile``'s default metadata (``{"shape": [N]}``, axes ``X``), is that + page: ``YX`` with the page's shape, one row of N pixels. The page's rows and columns are + the TIFF's own (as for any plain 2D page, decision D-d), so nothing is guessed; any other + series without Y or X keeps its axes and is refused. + """ + axes, shape = str(series.axes), tuple(int(n) for n in series.shape) + if "Y" in axes and "X" in axes: + return axes, shape + page = series.keyframe + if len(series.pages) == 1 and page.axes == "YX" and math.prod(page.shape) == math.prod(shape): + return "YX", tuple(int(n) for n in page.shape) + return axes, shape + + +def _plane_index( + axes: str, shape: tuple[int, ...], wanted: Mapping[str, int | None] +) -> tuple[int | slice, ...]: + """Return the index of the plane at ``wanted`` Z and T: whole Y and X, 0 on other axes. + + Y and X are kept whatever their length, so a mask one pixel high (or wide) stays 2D; + every other length-1 axis is dropped (index 0). The header check (:func:`_mask_header`, + :func:`_known_axes`) has already refused any other axis longer than 1. + """ + index: list[int | slice] = [] + for a, n in zip(axes, shape, strict=True): + if a in _YX: + index.append(slice(None)) + elif a in wanted and n > 1: + index.append(wanted.get(a) or 0) + else: + index.append(0 if n == 1 else slice(None)) + return tuple(index) + + def _yx(axes: str, shape: tuple[int, ...]) -> tuple[int, int]: return shape[axes.index("Y")], shape[axes.index("X")] @@ -677,21 +714,24 @@ def _header_issues( def _mask_header(mask: MaskFile, folder: Path | None, root: Path) -> tuple: - """Return a mask's axes and shape without its length-1 axes, and its pixel type. + """Return a mask's axes and shape without length-1 axes other than Y and X, and its dtype. The axes are the mask's own metadata (``tifffile``'s series axes, or the OME-NGFF axes of - the label image); no pixels are read. + the label image); no pixels are read. Y and X are kept whatever their length: a mask one + pixel high is ``YX`` with shape ``(1, N)``, never the 1D ``X`` (``tifffile`` and OME-NGFF + give both axes, so nothing is guessed). Only the other length-1 axes are dropped. """ if mask.source == FOLDER: with tifffile.TiffFile(Path(folder) / mask.path) as tif: if len(tif.series) != 1: raise ValueError(f"{len(tif.series)} image series") - axes, shape, dtype = tif.series[0].axes, tif.series[0].shape, tif.series[0].dtype + axes, shape = _tiff_axes(tif.series[0]) + dtype = tif.series[0].dtype else: image, name = mask.path.rsplit("/labels/", 1) labels = open_ngff_labels(root / image, name) axes, shape, dtype = labels.axes, labels.array.shape, np.dtype(labels.array.dtype) - kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1] + kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1 or a in _YX] return ( "".join(a for a, _ in kept), tuple(n for _, n in kept), @@ -699,6 +739,33 @@ def _mask_header(mask: MaskFile, folder: Path | None, root: Path) -> tuple: ) +def _axes_problem(axes: str, shape: tuple[int, ...]) -> str: + """Say why axes that :func:`_known_axes` refuses are not a mask's. + + The axes are :func:`_mask_header`'s: length-1 axes other than Y and X already left out. + """ + missing = [a for a in "YX" if a not in axes] + if missing: + return ( + f"its header gives no {' or '.join(missing)} axis, so its rows and columns are not " + "known" + ) + repeated = sorted({a for a in axes if axes.count(a) > 1}) + if repeated: + return f"its header gives {', '.join(repeated)} more than once" + other = [ + f"{a} (length {n})" + for a, n in zip(axes, shape, strict=True) + if a not in _YX and a not in PLANE_AXES + ] + several = len(other) > 1 + return ( + f"the dimension{'s' if several else ''} {', '.join(other)} {'are' if several else 'is'} " + f"neither Y nor X, and its metadata does not name {'them' if several else 'it'} depth " + "(Z) or time (T)" + ) + + def _known_axes(axes: str) -> bool: """Return True if the axes are Y and X, each once, plus at most one Z and one T.""" return ( diff --git a/tests/test_header_axes.py b/tests/test_header_axes.py index 1b44f0e6..aadf94f9 100644 --- a/tests/test_header_axes.py +++ b/tests/test_header_axes.py @@ -424,3 +424,87 @@ def test_read_mask_file_reads_an_ome_zarr_label_image(tmp_path): data = np.stack([labels((8, 8), start=t + 1) for t in range(2)]) write_labels(image, {"cells": data}, "tyx") np.testing.assert_array_equal(read_mask_file(image / "labels" / "cells", t=1), data[1]) + + +# Masks one pixel high or wide (the fiberfeatures report, plan §12 question 44) ----------------- + + +ONE_PIXEL_SHAPES = [(1, 9), (9, 1), (1, 1)] +WRITERS = { + "plain": lambda path, data: plain(path, data), + "shaped YX": lambda path, data: write_data(path, data, "YX"), + "imagej": lambda path, data: imagej(path, data, "YX"), + "ome": lambda path, data: ome(path, data, "YX"), +} + + +@pytest.mark.parametrize("shape", ONE_PIXEL_SHAPES) +@pytest.mark.parametrize("writer", WRITERS) +def test_read_mask_file_reads_a_mask_one_pixel_high_or_wide(tmp_path, shape, writer): + # tifffile gives these masks the axes YX; the length-1 Y (or X) used to be dropped with + # the other length-1 axes, so the mask was refused as having the axes 'X' (or 'Y', ''). + data = labels(shape) + path = WRITERS[writer](tmp_path / "m.tif", data) + assert inspect_image(path, "m.tif").axes == "YX" + pixels = read_mask_file(path) + assert pixels.shape == shape + np.testing.assert_array_equal(pixels, data) + + +@pytest.mark.parametrize( + ("shape", "axis", "writer"), + [ + ((3, 1, 9), "Z", imagej), + ((3, 1, 9), "Z", ome), + # tifffile's ImageJ writer cannot store a trailing length-1 X (it writes one page), + # so these are OME-TIFF only. + ((3, 9, 1), "T", ome), + ((3, 1, 1), "Z", ome), + ], +) +def test_read_mask_file_reads_one_pixel_planes_of_a_named_stack(tmp_path, shape, axis, writer): + data = np.stack([labels(shape[1:], start=10 * i + 1) for i in range(shape[0])]) + path = writer(tmp_path / "m.tif", data, f"{axis}YX") + np.testing.assert_array_equal(read_mask_file(path, **{axis.lower(): 2}), data[2]) + + +def test_read_mask_file_reads_one_pixel_planes_of_an_ome_zarr_label_image(tmp_path): + pytest.importorskip("zarr") + from zarr_tree import write_labels, write_ngff + + image = write_ngff(tmp_path / "a.ome.zarr", np.zeros((2, 1, 9), np.uint8), "zyx") + data = np.stack([labels((1, 9), start=z + 1) for z in range(2)]) + write_labels(image, {"cells": data}, "zyx") + np.testing.assert_array_equal(read_mask_file(image / "labels" / "cells", z=1), data[1]) + + +def test_an_unnamed_stack_of_one_pixel_planes_is_refused_by_its_real_axes(tmp_path): + # Still never guessed, and the message names the header's axes, not 'IX'. + path = plain(tmp_path / "m.tif", np.stack([labels((1, 9))] * 3)) + with pytest.raises( + ValueError, match=r"'IYX' \(shape \(3, 1, 9\)\): the dimension I \(length 3\)" + ): + read_mask_file(path, z=0) + + +@pytest.mark.parametrize("shape", ONE_PIXEL_SHAPES) +def test_header_axes_of_an_image_one_pixel_high_or_wide(tmp_path, shape): + # Guard: the image side keeps Y and X of length 1 (tifffile never drops them). + for writer in WRITERS.values(): + meta = inspect_image(writer(tmp_path / "a.tif", labels(shape)), "a.tif") + assert (meta.axes, meta.shape) == ("YX", shape) + assert named_axes(meta) == header_axes(meta) == "YX" + assert check_declared_axes(meta, "YX") is None + assert check_declared_axes(meta, "ZX").code == "declared_axes_mismatch" + + +def test_read_mask_file_reads_a_1d_array_saved_as_one_row(tmp_path): + # tifffile's default metadata saves a 1D array as {"shape": [9]}, which it reads back with + # the axes 'X' and no Y; the file is one TIFF page of 1 row and 9 columns, so the mask + # is that page: YX, (1, 9). + data = labels((9,)) + path = tmp_path / "m.tif" + tifffile.imwrite(path, data) + with tifffile.TiffFile(path) as tif: + assert tif.series[0].axes == "X" + np.testing.assert_array_equal(read_mask_file(path), data.reshape(1, 9)) diff --git a/tests/test_masks.py b/tests/test_masks.py index ec428b21..d7069839 100644 --- a/tests/test_masks.py +++ b/tests/test_masks.py @@ -10,7 +10,9 @@ prefill_mapping, preflight, read_manifest, + read_mask, run_experiment, + scan_masks, ) from morphobricks_preprocessing.cli import main from tiff_tree import write_data, write_file @@ -194,6 +196,47 @@ def test_a_length_one_axis_and_boolean_masks_are_accepted(experiment, masks): assert inventory(experiment, masks).ok +@pytest.mark.parametrize("shape", [(1, 9), (9, 1), (1, 1)]) +def test_masks_one_pixel_high_or_wide_are_2d_masks(tmp_path, shape): + # A 1 x N mask is a 2D YX mask (tifffile gives YX); its length-1 Y was dropped and the + # mask refused as mask_not_2d. + root, folder = tmp_path / "exp", tmp_path / "masks" + write_data( + root / "ctrl" / "rep1" / "a.tif", np.arange(np.prod(shape), dtype=np.uint8).reshape(shape) + ) + mask = (np.arange(np.prod(shape)).reshape(shape) % 3).astype(np.uint16) + write_data(folder / "Cells" / "ctrl" / "rep1" / "a.tif", mask) + found = inventory(root, folder) + assert found.ok, codes(found.issues) + assert [(f.axes, f.shape) for f in found.files] == [("YX", shape)] + np.testing.assert_array_equal(read_mask(found.files[0], folder, root), mask) + assert scan_masks(found, root).types["Cells"] + + +def test_a_1d_array_saved_as_one_row_is_a_mask_one_pixel_high(tmp_path): + root, folder = tmp_path / "exp", tmp_path / "masks" + write_data(root / "ctrl" / "rep1" / "a.tif", np.zeros((1, 9), np.uint8)) + mask = np.arange(9, dtype=np.uint16) % 3 + path = folder / "Cells" / "ctrl" / "rep1" / "a.tif" + path.parent.mkdir(parents=True) + tifffile.imwrite(path, mask) # {"shape": [9]}: tifffile's axes 'X' + found = inventory(root, folder) + assert found.ok, codes(found.issues) + assert [(f.axes, f.shape) for f in found.files] == [("YX", (1, 9))] + np.testing.assert_array_equal(read_mask(found.files[0], folder, root), mask.reshape(1, 9)) + + +def test_a_named_stack_of_one_pixel_high_masks_keeps_its_planes(tmp_path): + root, folder = tmp_path / "exp", tmp_path / "masks" + write_data(root / "ctrl" / "rep1" / "a.tif", np.zeros((3, 1, 9), np.uint8), "TYX") + data = np.stack([np.full((1, 9), t + 1, np.uint16) for t in range(3)]) + write_data(folder / "Cells" / "ctrl" / "rep1" / "a.tif", data, "TYX") + found = inventory(root, folder) + assert found.ok, codes(found.issues) + assert [(f.axes, f.shape) for f in found.files] == [("TYX", (3, 1, 9))] + np.testing.assert_array_equal(read_mask(found.files[0], folder, root, t=2), data[2]) + + def test_unreadable_masks_are_errors(experiment, masks): write_file(masks / "Cells" / "ctrl" / "rep1" / "a.tif", b"not a tiff") assert codes(inventory(experiment, masks).issues) == ["unreadable_mask"]