diff --git a/docs/contract.md b/docs/contract.md index 69a54149..273fb17c 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -312,7 +312,7 @@ masks/ masks folder, outside the experiment folder | Layout | Either one subfolder per mask type, each mirroring the experiment's folders (`//.../.tif`), or the experiment's folders with the type after the image name (`/.../_.tif`, as segmentation tools often write it, e.g. MorphoRelate's JUMP-Target-2 conversion); the two may be mixed. Any number of mask types. The label images of an OME-Zarr image (its OME-NGFF `labels` group, with the `zarr` extra) are masks too, of the type named by the label image; they are used whenever masks are asked for, with or without a masks folder. | | Pairing | A mask belongs to the image in the same leaf folder with the same `position_id` (§1), ignoring upper/lower case; its file is a TIFF (`.tif`, `.tiff`, optionally `.ome`, any case). Excluded images have no masks. | | Names | A mask type is named by its folder, by the suffix after the image name, or by its label image. A file that could belong to two images (an image name that starts another's, `a_b_cells.tif` beside images `a` and `a_b`) or fit both layouts is the error `ambiguous_mask`. Two names differing only in case are the error `mask_names_differ_in_case`; one image with two masks of one type (two extensions, or a file and a label image) is the error `duplicate_mask`. | -| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes are removed, a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed. A mask has the Y and X sizes of its image (`mask_size_mismatch`) and integer or boolean pixels (`mask_pixel_type`; a `uint64` mask holding a label above 2^63 − 1, which later bricks cannot store as an int64 `label_id`, raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`. All five are errors. | +| Header checks | From the headers only, no pixels read (`check`, `run`, `preflight`): once its length-1 axes are removed, a mask has Y and X and at most a depth axis Z and a time axis T, each named so in its own metadata (TIFF axes as `tifffile` reads them, e.g. an ImageJ hyperstack or OME-TIFF; OME-NGFF axes for a label image); any other or unnamed axis, such as a plain stack that could be depth or time, is `mask_not_2d` and is never guessed. A mask has the Y and X sizes of its image (`mask_size_mismatch`) and integer or boolean pixels (`mask_pixel_type`; a `uint64` mask holding a label above 2^63 − 1, which later bricks cannot store as an int64 `label_id`, raises an error when its pixels are read); the masks of one image have the same depth slices and time points (`mask_planes_differ`); a mask that cannot be opened, or holds several series, is `unreadable_mask`, as is an OME-Zarr image's `labels` group that cannot be read or does not list its label images (path `/labels`), so its mask types never disappear unreported. All five are errors. | | Planes | A mask belongs to the image's planes (issue #36): its Z and T lengths (1 for an axis it lacks) must equal the image's under the confirmed axis roles, checked by `run` (`mask_planes_mismatch`, error; `mask_plane_issues`). So a 2D mask belongs only to an image with one depth slice and one time point, and is refused for a Z-stack or a time-lapse, even as a projection (a decision for the owner if wanted later); a mask's plane at index `i` along Z or T belongs to the image's rows with `z = i` or `t = i`. `read_mask(..., z=, t=)` reads one plane and requires `z` and `t` for a mask with those axes. `check` and the `masks` command have no confirmed roles and do not compare with the image. | | Folder | The masks folder must exist (`masks_folder_not_found`) and be outside the experiment folder (`masks_inside_experiment`), where its folders would be read as labels. Both errors. | | Warnings | `mask_missing` (images without a mask of a type that other images have: their objects of that type cannot be measured), `mask_without_image` (files matching no analysed image), `mask_outside_type_folder` (files directly in the masks folder that are not TIFF masks). | diff --git a/docs/user_guide.md b/docs/user_guide.md index 7957a4f1..854a5d7e 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -571,7 +571,7 @@ continues. | `masks_inside_experiment` | Error | The masks folder is inside the experiment folder, where its folders would be read as conditions and repeats. Move it next to the experiment folder. | | `mask_names_differ_in_case` | Error | Two kinds of mask have names that differ only in upper/lower case. Rename one. | | `duplicate_mask` | Error | An image has two masks of one kind (two extensions, or a file and an OME-Zarr label image of the same name). Keep one. | -| `unreadable_mask` | Error | A mask cannot be opened, or holds several images. Save it again as a TIFF label image. | +| `unreadable_mask` | Error | A mask cannot be opened, or holds several images, or the `labels` group of an OME-Zarr image cannot be read (it must list its label images). Save the mask again as a TIFF label image, or repair the OME-Zarr labels. | | `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). | | `mask_planes_differ` | Error | The masks of one image have different numbers of depth slices or time points. Make every mask of an image with the image's planes. | | `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. | diff --git a/src/morphobricks_preprocessing/masks.py b/src/morphobricks_preprocessing/masks.py index 87b26942..1780692b 100644 --- a/src/morphobricks_preprocessing/masks.py +++ b/src/morphobricks_preprocessing/masks.py @@ -191,12 +191,14 @@ def find_masks( found, extra = _folder_masks(folder, discovery, images) files.extend(found) issues.extend(extra) + unreadable: list[str] = [] if ome_zarr_labels: - files.extend(_label_masks(discovery.root, images)) + found, unreadable = _label_masks(discovery.root, images) + files.extend(found) issues.extend(_name_issues(files)) files.sort(key=lambda f: (f.image, f.mask_type)) issues.extend(_duplicate_issues(files)) - issues.extend(_header_issues(files, images, folder, discovery.root)) + issues.extend(_header_issues(files, images, folder, discovery.root, unreadable)) types = tuple(sorted({f.mask_type for f in files})) issues.extend(_missing_issues(types, files, images)) return MaskInventory( @@ -440,19 +442,30 @@ def _layout_issues(loose: list[str], unpaired: list[str]) -> list[ValidationIssu return issues -def _label_masks(root: Path, images: Mapping[str, TiffMetadata]) -> list[MaskFile]: - """Return the label images of every OME-Zarr image, as masks.""" - files = [] +def _label_masks( + root: Path, images: Mapping[str, TiffMetadata] +) -> tuple[list[MaskFile], list[str]]: + """Return the label images of every OME-Zarr image, as masks. + + Also return the ``/labels`` groups that could not be read: their masks would + otherwise vanish without a word (and with them the ``mask_missing`` warning), so they + are reported as ``unreadable_mask``. + """ + files, unreadable = [], [] for image, meta in images.items(): if meta.kind != "ome-zarr" or not meta.readable: continue try: names = ngff_label_names(root / image) - except (ZarrUnavailableError, NgffError) as exc: + except ZarrUnavailableError as exc: # documented: label images need the zarr extra logger.info("No label images read from %s: %s", image, exc) continue + except NgffError as exc: + logger.info("The label images of %s could not be read: %s", image, exc) + unreadable.append(f"{image}/labels") + continue files.extend(MaskFile(image, name, f"{image}/labels/{name}", OME_ZARR) for name in names) - return files + return files, unreadable def _name_issues(files: list[MaskFile]) -> list[ValidationIssue]: @@ -496,13 +509,20 @@ def _duplicate_issues(files: list[MaskFile]) -> list[ValidationIssue]: def _header_issues( - files: list[MaskFile], images: Mapping[str, TiffMetadata], folder: Path | None, root: Path + files: list[MaskFile], + images: Mapping[str, TiffMetadata], + folder: Path | None, + root: Path, + unreadable: Iterable[str] = (), ) -> list[ValidationIssue]: """Read each mask's header (axes, shape, pixel type) and check it against its image. Also check that the masks of one image have the same depth slices and time points. + ``unreadable`` are paths already known not to open (OME-Zarr ``labels`` groups). """ problems: dict[str, list[str]] = {} + if unreadable: + problems["unreadable_mask"] = list(unreadable) for index, mask in enumerate(files): try: axes, shape, dtype = _mask_header(mask, folder, root) @@ -542,8 +562,9 @@ def _header_issues( _TEXTS = { "unreadable_mask": ( "Some masks could not be opened.", - "Check that they are TIFF files (or OME-Zarr label images) with one image each and " - "not damaged; the technical details are in the log.", + "Check that they are TIFF files (or OME-Zarr label images, listed in the image's " + "labels group) with one image each and not damaged; the technical details are in the " + "log.", ), "mask_not_2d": ( "Some masks have dimensions other than Y and X that their metadata does not name as " diff --git a/tests/test_masks.py b/tests/test_masks.py index 2ebdce1f..e63b9413 100644 --- a/tests/test_masks.py +++ b/tests/test_masks.py @@ -467,3 +467,29 @@ def test_uint64_labels_above_the_int64_range_are_refused(experiment, masks, tmp_ out = tmp_path / "out" assert main(["masks", str(experiment), "--masks", str(masks), "--output", str(out)]) == 1 assert "cannot be stored as a 64-bit signed label number" in capsys.readouterr().out + + +def test_a_malformed_labels_group_is_an_error_not_a_vanished_mask_type(tmp_path): + zarr = pytest.importorskip("zarr") + from zarr_tree import write_labels, write_ngff + + paths = [] + for rel in ("ctrl/a.ome.zarr", "drug/a.ome.zarr"): + path = write_ngff(tmp_path / "exp" / rel, np.zeros((16, 20), np.uint8), "yx") + write_labels(path, {"cells": label_image(0)}, "yx") + paths.append(path) + # Both images' labels groups no longer list their label images: every cells mask is lost. + for path in paths: + group = zarr.open_group(str(path / "labels"), mode="a") + group.attrs.update({"labels": "cells"}) + found = preflight(tmp_path / "exp", ome_zarr_labels=True).masks + assert not found.ok + assert codes(found.issues) == ["unreadable_mask"] + assert found.issues[0].paths == ("ctrl/a.ome.zarr/labels", "drug/a.ome.zarr/labels") + result = run_experiment( + tmp_path / "exp", + prefill_mapping("YX"), + output_folder=tmp_path / "out", + ome_zarr_labels=True, + ) + assert not result.written and "unreadable_mask" in codes(result.issues)