Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/contract.md

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -1013,3 +1013,25 @@ Raised during review; to be decided before the milestones that need them:
default design: no file is the first run, not an unreadable one. The notebook keeps the
behaviour of question 42 (file untouched, refusal shown, an answer replaces it), and
`check` prints the refusal and writes nothing. Breaking change for the next release.*
44. How is a mask one pixel high or wide read (reported by the fiberfeatures brick)? A valid
2D label mask of 1 x N, N x 1 or 1 x 1 pixels was refused: `read_mask_file` said it had
the axes `X` (or `Y`, or none) and `find_masks` gave `mask_not_2d`. `tifffile` (and
OME-NGFF) give such a mask the axes `YX`, with or without axis metadata; this brick's
own mask header dropped every length-1 axis, Y and X included, and the plane readers did
the same. Images were not affected: their axes are `tifffile`'s, which keep Y and X.
*Decided (31 October 2026, a fix within the existing rules): a mask's length-1 axes are
left out except Y and X, in the header (`MaskFile.axes`/`shape`, the manifest's `masks`
entries) and in `read_mask`/`read_mask_file`, which return the 2D (1, N), (N, 1) or
(1, 1) plane; a named Z or T stack of such planes keeps its planes. `tifffile` gives the
axes `X` alone only for a 1D array saved with its default metadata (`{"shape": [N]}`);
such a file is one TIFF page of 1 row and N columns, and a mask series without Y or X
stored in one 2D page is read as that page (`YX`, (1, N)), as any plain 2D page is
(decision D-d). Nothing is guessed: Y and X are the ones the file's header gives, and a
dimension it does not name (a plain stack `IYX`) is still refused, the message now
giving the header's axes and shape and the dimension that is not named instead of the
axes left after dropping Y or X. Masks
accepted before are recorded as before. Ships in the next release (v0.3.5), with the
fixes of questions 42 and 43. Open for the owner: an image (not a mask) saved as such a
1D array keeps the header axes `X` (`header_axes`, `check_declared_axes`), so the folder
check suggests a mapping without Y and the run refuses it with `missing_xy_axis`;
reading it as one row too would change the shared header-axes rule (§16).*
2 changes: 1 addition & 1 deletion docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -596,7 +596,7 @@ continues.
| `mask_names_differ_in_case` | Error | Two kinds of mask have names that differ only in upper/lower case. Rename one. |
| `duplicate_mask` | Error | An image has two masks of one kind (two extensions, or a file and an OME-Zarr label image of the same name). Keep one. |
| `unreadable_mask` | Error | A mask cannot be opened, or holds several images, or the `labels` group of an OME-Zarr image cannot be read (it must list its label images). Save the mask again as a TIFF label image, or repair the OME-Zarr labels. |
| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). |
| `mask_not_2d` | Error | A mask has dimensions other than Y and X that its metadata does not name as depth (Z) or time (T), for example a plain stack. (A mask one pixel high or wide is a 2D mask, not this error.) Save it as a 2D label image, or with its axes named (an ImageJ hyperstack or OME-TIFF with axes TYX or ZYX). |
| `mask_planes_differ` | Error | The masks of one image have different numbers of depth slices or time points. Make every mask of an image with the image's planes. |
| `mask_planes_mismatch` | Error (`run`) | A mask does not have the depth slices and time points of its image under the confirmed axis roles: a 2D mask fits only an image with one plane (not a Z-stack or time-lapse), and a mask with Z or T needs as many slices or time points as its image. Save one mask plane per image plane, with the axes named, or check the axis roles. |
| `mask_size_mismatch` | Error | A mask does not have the width and height of its image. Check that it was made from that image at full resolution. |
Expand Down
105 changes: 86 additions & 19 deletions src/morphobricks_preprocessing/masks.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
image name, or by its label image, and two
names may not differ only in case. Only headers are read here, never pixels: a mask has the
image's Y and X sizes and integer or boolean pixels. Its other dimensions (length-1 axes
aside) must be labelled depth (Z) or time (T) in its own metadata, never guessed, and every
aside; Y and X are kept whatever their length, so a mask one pixel high is a 2D ``YX`` mask)
must be labelled depth (Z) or time (T) in its own metadata, never guessed, and every
mask of one image has the same depth slices and time points. Whether those are the image's
planes needs the confirmed axis roles: :func:`mask_plane_issues`, run by
:func:`.run.run_experiment` (issue #36). Reading the pixels and relating the masks to each
Expand Down Expand Up @@ -61,6 +62,9 @@
PLANE_AXES = ("Z", "T")
"""The dimensions a mask may have besides Y and X: depth slices and time points."""

_YX = frozenset("YX")
"""The axes of a mask plane, kept by the header and the readers even when of length 1."""


@dataclass(frozen=True)
class MaskFile:
Expand All @@ -78,7 +82,8 @@ class MaskFile:
source
:data:`FOLDER` or :data:`OME_ZARR`.
shape
Shape of the mask without its length-1 axes, ``(Y, X)`` for a 2D mask.
Shape of the mask without its length-1 axes other than Y and X, ``(Y, X)`` for a 2D
mask (``(1, 512)`` for a mask one pixel high).
dtype
Pixel type, or None when the header could not be read.
axes
Expand Down Expand Up @@ -259,10 +264,7 @@ def read_mask(
name = mask.path.rsplit("/labels/", 1)[1]
image = experiment_root / mask.path.rsplit("/labels/", 1)[0]
labels = open_ngff_labels(image, name)
selection = tuple(
(wanted[a] or 0) if a in wanted and n > 1 else (0 if n == 1 else slice(None))
for a, n in zip(labels.axes, labels.array.shape, strict=True)
)
selection = _plane_index(labels.axes, labels.array.shape, wanted)
pixels = np.asarray(labels.array[selection])
axes = "".join(a for a, s in zip(labels.axes, selection, strict=True) if s == slice(None))
if axes != "YX" or (mask.axes and pixels.shape != tuple(_yx(mask.axes, mask.shape))):
Expand All @@ -278,7 +280,8 @@ def read_mask_file(

For bricks given a mask path directly (MorphoBricks D86): the mask is read as
:func:`read_mask` reads it, with the same rules. Its axes are those its metadata names
(length-1 axes aside): Y and X, plus at most a depth axis Z and a time axis T; a
(length-1 axes aside, except Y and X: a mask one pixel high or wide is read as a 2D
array): Y and X, plus at most a depth axis Z and a time axis T; a
dimension the metadata does not name (a plain stack, ``IYX``) is refused, never taken
for depth, time or a channel. ``z`` and ``t`` select the plane and are required for a
mask with that axis. The labels are checked with :func:`check_labels`.
Expand Down Expand Up @@ -312,8 +315,7 @@ def read_mask_file(
raise ValueError(f"the mask {path} could not be opened: {exc}") from exc
if not _known_axes(axes):
raise ValueError(
f"the mask {path} has axes {axes!r}: dimensions other than Y and X must be named "
"depth (Z) or time (T) in its metadata"
f"the mask {path} has axes {axes!r} (shape {shape}): {_axes_problem(axes, shape)}"
)
if dtype.kind not in "biu":
raise ValueError(
Expand All @@ -337,11 +339,8 @@ def _read_tiff_plane(path: Path, wanted: Mapping[str, int | None]) -> tuple[np.n
"""
with tifffile.TiffFile(path) as tif:
series = tif.series[0]
axes, shape = series.axes, tuple(series.shape)
index = tuple(
(wanted.get(a) or 0) if n > 1 and a in wanted else (0 if n == 1 else slice(None))
for a, n in zip(axes, shape, strict=True)
)
axes, shape = _tiff_axes(series)
index = _plane_index(axes, shape, wanted)
kept = "".join(a for a, i in zip(axes, index, strict=True) if i == slice(None))
pages = len(series.pages)
split = next(
Expand All @@ -358,10 +357,48 @@ def _read_tiff_plane(path: Path, wanted: Mapping[str, int | None]) -> tuple[np.n
pixels = tif.asarray(key=key, series=0).reshape(shape[split:])
pixels = pixels[index[split:]]
else:
pixels = series.asarray()[index]
pixels = series.asarray().reshape(shape)[index]
return pixels, kept


def _tiff_axes(series: tifffile.TiffPageSeries) -> tuple[str, tuple[int, ...]]:
"""Return the axes and shape of a TIFF mask's series, as ``tifffile`` gives them.

One exception: a series without Y or X stored in one 2D page, for example a 1D array
written with ``tifffile``'s default metadata (``{"shape": [N]}``, axes ``X``), is that
page: ``YX`` with the page's shape, one row of N pixels. The page's rows and columns are
the TIFF's own (as for any plain 2D page, decision D-d), so nothing is guessed; any other
series without Y or X keeps its axes and is refused.
"""
axes, shape = str(series.axes), tuple(int(n) for n in series.shape)
if "Y" in axes and "X" in axes:
return axes, shape
page = series.keyframe
if len(series.pages) == 1 and page.axes == "YX" and math.prod(page.shape) == math.prod(shape):
return "YX", tuple(int(n) for n in page.shape)
return axes, shape


def _plane_index(
axes: str, shape: tuple[int, ...], wanted: Mapping[str, int | None]
) -> tuple[int | slice, ...]:
"""Return the index of the plane at ``wanted`` Z and T: whole Y and X, 0 on other axes.

Y and X are kept whatever their length, so a mask one pixel high (or wide) stays 2D;
every other length-1 axis is dropped (index 0). The header check (:func:`_mask_header`,
:func:`_known_axes`) has already refused any other axis longer than 1.
"""
index: list[int | slice] = []
for a, n in zip(axes, shape, strict=True):
if a in _YX:
index.append(slice(None))
elif a in wanted and n > 1:
index.append(wanted.get(a) or 0)
else:
index.append(0 if n == 1 else slice(None))
return tuple(index)


def _yx(axes: str, shape: tuple[int, ...]) -> tuple[int, int]:
return shape[axes.index("Y")], shape[axes.index("X")]

Expand Down Expand Up @@ -677,28 +714,58 @@ def _header_issues(


def _mask_header(mask: MaskFile, folder: Path | None, root: Path) -> tuple:
"""Return a mask's axes and shape without its length-1 axes, and its pixel type.
"""Return a mask's axes and shape without length-1 axes other than Y and X, and its dtype.

The axes are the mask's own metadata (``tifffile``'s series axes, or the OME-NGFF axes of
the label image); no pixels are read.
the label image); no pixels are read. Y and X are kept whatever their length: a mask one
pixel high is ``YX`` with shape ``(1, N)``, never the 1D ``X`` (``tifffile`` and OME-NGFF
give both axes, so nothing is guessed). Only the other length-1 axes are dropped.
"""
if mask.source == FOLDER:
with tifffile.TiffFile(Path(folder) / mask.path) as tif:
if len(tif.series) != 1:
raise ValueError(f"{len(tif.series)} image series")
axes, shape, dtype = tif.series[0].axes, tif.series[0].shape, tif.series[0].dtype
axes, shape = _tiff_axes(tif.series[0])
dtype = tif.series[0].dtype
else:
image, name = mask.path.rsplit("/labels/", 1)
labels = open_ngff_labels(root / image, name)
axes, shape, dtype = labels.axes, labels.array.shape, np.dtype(labels.array.dtype)
kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1]
kept = [(a, int(n)) for a, n in zip(axes, shape, strict=True) if n != 1 or a in _YX]
return (
"".join(a for a, _ in kept),
tuple(n for _, n in kept),
dtype.newbyteorder("="),
)


def _axes_problem(axes: str, shape: tuple[int, ...]) -> str:
"""Say why axes that :func:`_known_axes` refuses are not a mask's.

The axes are :func:`_mask_header`'s: length-1 axes other than Y and X already left out.
"""
missing = [a for a in "YX" if a not in axes]
if missing:
return (
f"its header gives no {' or '.join(missing)} axis, so its rows and columns are not "
"known"
)
repeated = sorted({a for a in axes if axes.count(a) > 1})
if repeated:
return f"its header gives {', '.join(repeated)} more than once"
other = [
f"{a} (length {n})"
for a, n in zip(axes, shape, strict=True)
if a not in _YX and a not in PLANE_AXES
]
several = len(other) > 1
return (
f"the dimension{'s' if several else ''} {', '.join(other)} {'are' if several else 'is'} "
f"neither Y nor X, and its metadata does not name {'them' if several else 'it'} depth "
"(Z) or time (T)"
)


def _known_axes(axes: str) -> bool:
"""Return True if the axes are Y and X, each once, plus at most one Z and one T."""
return (
Expand Down
84 changes: 84 additions & 0 deletions tests/test_header_axes.py
Original file line number Diff line number Diff line change
Expand Up @@ -424,3 +424,87 @@ def test_read_mask_file_reads_an_ome_zarr_label_image(tmp_path):
data = np.stack([labels((8, 8), start=t + 1) for t in range(2)])
write_labels(image, {"cells": data}, "tyx")
np.testing.assert_array_equal(read_mask_file(image / "labels" / "cells", t=1), data[1])


# Masks one pixel high or wide (the fiberfeatures report, plan §12 question 44) -----------------


ONE_PIXEL_SHAPES = [(1, 9), (9, 1), (1, 1)]
WRITERS = {
"plain": lambda path, data: plain(path, data),
"shaped YX": lambda path, data: write_data(path, data, "YX"),
"imagej": lambda path, data: imagej(path, data, "YX"),
"ome": lambda path, data: ome(path, data, "YX"),
}


@pytest.mark.parametrize("shape", ONE_PIXEL_SHAPES)
@pytest.mark.parametrize("writer", WRITERS)
def test_read_mask_file_reads_a_mask_one_pixel_high_or_wide(tmp_path, shape, writer):
# tifffile gives these masks the axes YX; the length-1 Y (or X) used to be dropped with
# the other length-1 axes, so the mask was refused as having the axes 'X' (or 'Y', '').
data = labels(shape)
path = WRITERS[writer](tmp_path / "m.tif", data)
assert inspect_image(path, "m.tif").axes == "YX"
pixels = read_mask_file(path)
assert pixels.shape == shape
np.testing.assert_array_equal(pixels, data)


@pytest.mark.parametrize(
("shape", "axis", "writer"),
[
((3, 1, 9), "Z", imagej),
((3, 1, 9), "Z", ome),
# tifffile's ImageJ writer cannot store a trailing length-1 X (it writes one page),
# so these are OME-TIFF only.
((3, 9, 1), "T", ome),
((3, 1, 1), "Z", ome),
],
)
def test_read_mask_file_reads_one_pixel_planes_of_a_named_stack(tmp_path, shape, axis, writer):
data = np.stack([labels(shape[1:], start=10 * i + 1) for i in range(shape[0])])
path = writer(tmp_path / "m.tif", data, f"{axis}YX")
np.testing.assert_array_equal(read_mask_file(path, **{axis.lower(): 2}), data[2])


def test_read_mask_file_reads_one_pixel_planes_of_an_ome_zarr_label_image(tmp_path):
pytest.importorskip("zarr")
from zarr_tree import write_labels, write_ngff

image = write_ngff(tmp_path / "a.ome.zarr", np.zeros((2, 1, 9), np.uint8), "zyx")
data = np.stack([labels((1, 9), start=z + 1) for z in range(2)])
write_labels(image, {"cells": data}, "zyx")
np.testing.assert_array_equal(read_mask_file(image / "labels" / "cells", z=1), data[1])


def test_an_unnamed_stack_of_one_pixel_planes_is_refused_by_its_real_axes(tmp_path):
# Still never guessed, and the message names the header's axes, not 'IX'.
path = plain(tmp_path / "m.tif", np.stack([labels((1, 9))] * 3))
with pytest.raises(
ValueError, match=r"'IYX' \(shape \(3, 1, 9\)\): the dimension I \(length 3\)"
):
read_mask_file(path, z=0)


@pytest.mark.parametrize("shape", ONE_PIXEL_SHAPES)
def test_header_axes_of_an_image_one_pixel_high_or_wide(tmp_path, shape):
# Guard: the image side keeps Y and X of length 1 (tifffile never drops them).
for writer in WRITERS.values():
meta = inspect_image(writer(tmp_path / "a.tif", labels(shape)), "a.tif")
assert (meta.axes, meta.shape) == ("YX", shape)
assert named_axes(meta) == header_axes(meta) == "YX"
assert check_declared_axes(meta, "YX") is None
assert check_declared_axes(meta, "ZX").code == "declared_axes_mismatch"


def test_read_mask_file_reads_a_1d_array_saved_as_one_row(tmp_path):
# tifffile's default metadata saves a 1D array as {"shape": [9]}, which it reads back with
# the axes 'X' and no Y; the file is one TIFF page of 1 row and 9 columns, so the mask
# is that page: YX, (1, 9).
data = labels((9,))
path = tmp_path / "m.tif"
tifffile.imwrite(path, data)
with tifffile.TiffFile(path) as tif:
assert tif.series[0].axes == "X"
np.testing.assert_array_equal(read_mask_file(path), data.reshape(1, 9))
43 changes: 43 additions & 0 deletions tests/test_masks.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
prefill_mapping,
preflight,
read_manifest,
read_mask,
run_experiment,
scan_masks,
)
from morphobricks_preprocessing.cli import main
from tiff_tree import write_data, write_file
Expand Down Expand Up @@ -194,6 +196,47 @@ def test_a_length_one_axis_and_boolean_masks_are_accepted(experiment, masks):
assert inventory(experiment, masks).ok


@pytest.mark.parametrize("shape", [(1, 9), (9, 1), (1, 1)])
def test_masks_one_pixel_high_or_wide_are_2d_masks(tmp_path, shape):
# A 1 x N mask is a 2D YX mask (tifffile gives YX); its length-1 Y was dropped and the
# mask refused as mask_not_2d.
root, folder = tmp_path / "exp", tmp_path / "masks"
write_data(
root / "ctrl" / "rep1" / "a.tif", np.arange(np.prod(shape), dtype=np.uint8).reshape(shape)
)
mask = (np.arange(np.prod(shape)).reshape(shape) % 3).astype(np.uint16)
write_data(folder / "Cells" / "ctrl" / "rep1" / "a.tif", mask)
found = inventory(root, folder)
assert found.ok, codes(found.issues)
assert [(f.axes, f.shape) for f in found.files] == [("YX", shape)]
np.testing.assert_array_equal(read_mask(found.files[0], folder, root), mask)
assert scan_masks(found, root).types["Cells"]


def test_a_1d_array_saved_as_one_row_is_a_mask_one_pixel_high(tmp_path):
root, folder = tmp_path / "exp", tmp_path / "masks"
write_data(root / "ctrl" / "rep1" / "a.tif", np.zeros((1, 9), np.uint8))
mask = np.arange(9, dtype=np.uint16) % 3
path = folder / "Cells" / "ctrl" / "rep1" / "a.tif"
path.parent.mkdir(parents=True)
tifffile.imwrite(path, mask) # {"shape": [9]}: tifffile's axes 'X'
found = inventory(root, folder)
assert found.ok, codes(found.issues)
assert [(f.axes, f.shape) for f in found.files] == [("YX", (1, 9))]
np.testing.assert_array_equal(read_mask(found.files[0], folder, root), mask.reshape(1, 9))


def test_a_named_stack_of_one_pixel_high_masks_keeps_its_planes(tmp_path):
root, folder = tmp_path / "exp", tmp_path / "masks"
write_data(root / "ctrl" / "rep1" / "a.tif", np.zeros((3, 1, 9), np.uint8), "TYX")
data = np.stack([np.full((1, 9), t + 1, np.uint16) for t in range(3)])
write_data(folder / "Cells" / "ctrl" / "rep1" / "a.tif", data, "TYX")
found = inventory(root, folder)
assert found.ok, codes(found.issues)
assert [(f.axes, f.shape) for f in found.files] == [("TYX", (3, 1, 9))]
np.testing.assert_array_equal(read_mask(found.files[0], folder, root, t=2), data[2])


def test_unreadable_masks_are_errors(experiment, masks):
write_file(masks / "Cells" / "ctrl" / "rep1" / "a.tif", b"not a tiff")
assert codes(inventory(experiment, masks).issues) == ["unreadable_mask"]
Expand Down
Loading