diff --git a/docs/contract.md b/docs/contract.md index ffa9f3db..0afd3ef3 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -178,6 +178,33 @@ information only; channel names must agree between images at each channel number later bricks that name channels should check their names against `channel_names` when it is not empty. +### 9.1 Reading a clean run (`RunInputs`, v0.4.0) + +Bricks read a run's files only through `RunInputs.open(output_folder, experiment_root=None, +masks_root=None)` (design v0.4 §6). It refuses (`RunInputsError`) a version 1 manifest, a +`plane_histograms.csv` or `experiment_design.csv` whose SHA-256 is not the manifest's, and an +image or mask changed or gone since the run; every read checks the file's fingerprint again. +It gives: + +- `fovs()`: `Fov` records with the FOV key, `plane_id`s, axes, roles, `axes_from`, channel + names, dtype, `significant_bits`, and the Z, T and C counts, from the manifest and the + table, never parsed; +- `pixel_sizes(fov)`: value, unit and source per role, or unknown; +- `read_plane(fov, z, t)`, a `(C, Y, X)` array of the stored values, and `read_image(fov)`, a + lazy `(T, Z, C, Y, X)` `LazyImage`. Both are in canonical order: an image whose own header + names X before Y (for example `ZXY`) is returned Y before X, as the run read it; nothing + else is reordered; +- `mask_types()` and `read_mask(fov, type, z=None, t=None)`: the recorded pairing only, read + as stored through the recorded `axes` and `roles` by preprocessing's reader with + `check_labels`; a mask changed since the run is refused; a stated shared 2-D mask serves + every plane; +- `histograms()` (the table as read when opened), `design()` and `image_type()`. + +A `RunInputs` is picklable, so worker processes can use it. +`morphobricks_preprocessing.testing.make_clean_run(folder, ...)` writes a tiny experiment of +named-axis OME-TIFFs (optionally with masks and a confirmed design) and its clean run, for +the bricks' tests. + ## 10. Converting TIFF images to OME-Zarr (removed in v0.4.0) Up to v0.3, a run could convert every TIFF image to OME-Zarr in `/ome_zarr` and @@ -391,7 +418,7 @@ instead of keeping copies. | Declared axes (removed) | `check_declared_axes`, `declared_axes_transposed` | **Removed in v0.4.0** (spec revision 4): they accepted a declaration with Y and X swapped and read it transposed, which v0.4 forbids. Bricks take no axis declarations; preprocessing's axis check (§2) compares the confirmed roles with the header. Former text, for reference: | **Deprecated in v0.4.0, removed in v0.5.0** (they warn with a `DeprecationWarning`): bricks take no axis declarations and read the files of a preprocessing run whose axes were checked (§2); nothing is read transposed. Until then: axes a brick or user declares for an image or mask (`image_axes="CYX"`) may name what the header leaves open, never contradict it: one letter per header axis (any case), roles X, Y, Z, C, T each at most once, with `Y` and `X` declared when the header has them (a plain stack too); an axis the metadata names (`named_axes`) keeps its letter, except that the named `Y` and `X` may be declared swapped (header `CYX` declared `CXY`, `ZYX` declared `ZXY`): the image is read transposed, and `declared_axes_transposed` returns True so the caller can record it (owner decision of 31 October 2026; it raises `ValueError` for a declaration that does not fit); any other disagreement is refused; a dimension it leaves open may be declared any role, so a plain `QYX` stack may be declared `YXC` as well as `CYX` (since v0.3.4; before, its positional `Y` and `X` were binding and a placeholder could not be declared X or Y), while an ImageJ `ZYX` declared `CYX`, an OME `CYX` declared `YXC` or a shaped `CYX` declared `YXC` are refused; the RGB samples `S` may be declared C when the header has no C; any other header axis (tile `R`, ...) cannot be declared. Otherwise the error `declared_axes_mismatch` naming the image and both axes (`unreadable_tiff` or `unreadable_ome_zarr` when the header does not open). | | Mask files | `read_mask_file(path, z=, t=)` (`masks`) | Reads one plane of a TIFF mask or of an OME-Zarr label image (`.zarr/labels/`) given by its path, with `read_mask`'s rules (§15): its axes are its own metadata's, Y and X plus at most a Z and a T axis named so, length-1 axes other than Y and X left out (a mask one pixel high or wide is read as a 2D array, plan §12 question 44); any other or unnamed dimension (a plain stack, `IYX`; a `C` axis) is refused, the message giving the header's axes and shape and the dimension that is not named, never taken for depth or time; `z` and `t` are required for a mask with that axis; integer or boolean pixels; labels checked with `check_labels`. A `ValueError` names the file and the problem. | | Identity (D-e) | `FOV_KEY`, `check_identity_text(value, field)`, `check_plane_index(value, axis)`, `check_identity_case(keys)`, `MISSING_TEXT`, `IdentityError` (`discovery`) | `FOV_KEY` = (`condition`, `biological_repeat`, `technical_repeat`, `position_id`), the position key of §1. `check_identity_text` accepts a label or `position_id` that is text without leading or trailing whitespace and not `""`, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case (`MISSING_TEXT`); `check_plane_index` accepts None or an integer of at least 0, a Python `int` or a NumPy integer (returned as a plain `int`), and refuses everything else (bool, float, `"1"`) without converting it. Both raise `IdentityError` (a `ValueError`) naming the field and the value. Discovery applies the text rule to every folder and image name (`invalid_name`) and refuses names of one level that differ only in case anywhere in the experiment (`case_conflict`, §1). `position_id` keeps its own scope: unique within its technical repeat, ignoring case (`duplicate_position_id`). `check_identity_case` applies the same case rule to identities a brick reads from a table, one key per row in `FOV_KEY` order (or the three labels): a label of one level spelled two ways anywhere in the run, or two `position_id` values of one technical repeat equal ignoring case, raise `IdentityError` naming both spellings; the same key may repeat (several planes of one position). | -| Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | +| Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`); from v0.4.0 file runs read through `RunInputs` (§9.1), which also checks masks. A version 1 manifest fails every image with `manifest_before_v040` alone (design v0.4 §6: every v1 run is redone). Otherwise each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. The folder check's `different_pixel_sizes` uses both (§11). | | Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order; one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking change for the next release:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | diff --git a/docs/user_guide.md b/docs/user_guide.md index 8c5529fd..97a1dbce 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -598,7 +598,7 @@ continues. | `mask_missing` | Warning | Some images have no mask of a kind that other images have; their objects of that kind cannot be measured. Add the masks, or exclude the images. | | `ambiguous_mask` | Error | A mask's name could belong to two images (one image name starts another's) or fit both layouts. Rename the masks or the images so each names one image and one kind. | | `mask_without_image`, `mask_outside_type_folder`, `mask_of_excluded_image` | Warning | Files in the masks folder that match no image of the experiment (check the folder and file names), other files directly in the masks folder, or masks of excluded images. They are not used. | -| `images_not_in_manifest`, `images_missing`, `images_changed`, `position_id_not_recorded`, `identity_differs`, `axis_mapping_mismatch` | Error (other bricks) | A later brick checked its images against this brick's run (`check_sources`): an image the run did not analyse, no longer there, changed since, recorded with other labels or position (or by a version that did not record positions), or read with axes the run gave other roles. Run this brick again on the experiment as it is, or use the run's labels and axis roles. | +| `manifest_before_v040`, `images_not_in_manifest`, `images_missing`, `images_changed`, `position_id_not_recorded`, `identity_differs`, `axis_mapping_mismatch` | Error (other bricks) | A later brick checked its images against this brick's run (`check_sources`): a run written before v0.4.0 (redo it), an image the run did not analyse, no longer there, changed since, recorded with other labels or position (or by a version that did not record positions), or read with axes the run gave other roles. Run this brick again on the experiment as it is, or use the run's labels and axis roles. | | `zarr_not_installed` | Error | OME-Zarr images need the `zarr` extra. Install it, or exclude the OME-Zarr images. | ## 9. Python interface diff --git a/pyproject.toml b/pyproject.toml index ff64add9..800c214c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -142,6 +142,8 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.fingerprint" = "io" "morphobricks_preprocessing.narrow_ranges" = "pure" "morphobricks_preprocessing.recorded_sizes" = "pure" +"morphobricks_preprocessing.run_inputs" = "io" +"morphobricks_preprocessing.testing" = "orchestration" "morphobricks_preprocessing.mask_scan" = "io" "morphobricks_preprocessing.masks" = "io" "morphobricks_preprocessing.masks.checks" = "io" diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 948c896a..d340d963 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -174,6 +174,7 @@ ) from .report import GroupSummary, group_summaries from .run import Preflight, RunResult, SamplePlane, preflight, run_experiment, sample_planes +from .run_inputs import Fov, LazyImage, RunInputs, RunInputsError from .sources import check_sources from .validation import Severity, ValidationIssue, has_errors from .zarr_input import ( @@ -256,6 +257,10 @@ "RepeatBox", "RunManifest", "RunResult", + "RunInputs", + "RunInputsError", + "Fov", + "LazyImage", "RunSettings", "SamplePlane", "Severity", diff --git a/src/morphobricks_preprocessing/run_inputs.py b/src/morphobricks_preprocessing/run_inputs.py new file mode 100644 index 00000000..694a7b3f --- /dev/null +++ b/src/morphobricks_preprocessing/run_inputs.py @@ -0,0 +1,421 @@ +"""What a clean preprocessing run gives the other bricks: :class:`RunInputs` (design v0.4 §6). + +A brick that works on files reads them only through a clean run: the images, their axis +roles, pixel sizes, masks, histograms, design and image type, all as the run recorded them. +Nothing is parsed from names, guessed or transposed, and no file is read once it changed: + +- :meth:`RunInputs.open` refuses a version 1 manifest (a run before the axis check), a table + or design whose SHA-256 is not the manifest's, and an image or mask changed or gone since + the run; +- every read checks the file's fingerprint again (:class:`.fingerprint.Fingerprint`); +- masks are read by the recorded pairing only, with preprocessing's reader and its label + check (:func:`.masks.check_labels`); a stated shared 2-D mask serves every plane. + +A :class:`RunInputs` is picklable: it holds paths and the records read when it opened (its +snapshot of the table), and opens files only to read them. +""" + +from __future__ import annotations + +import os +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path + +import numpy as np +from morphobricks_normalization.csv_io import OUTPUT_FILENAME, read_plane_histograms +from morphobricks_normalization.records import PlaneRecord + +from .axes import AxisMapping +from .design import DESIGN_FILENAME, ExperimentDesign, read_experiment_design +from .fingerprint import Fingerprint +from .manifest import ( + LEGACY_MANIFEST_VERSION, + MANIFEST_FILENAME, + FileEntry, + RunManifest, + file_sha256, + read_manifest, +) +from .masks import FOLDER, MaskFile, read_mask +from .masks.header import _mask_header +from .planes import DEFAULT_MEMORY_LIMIT, read_plane +from .recorded_sizes import RecordedSize + + +class RunInputsError(ValueError): + """The run cannot be used as it is: run preprocessing again (the message says why).""" + + +@dataclass(frozen=True) +class Fov: + """One position (field of view) of a clean run, as the manifest and the table record it. + + Attributes + ---------- + key + ``(condition, biological_repeat, technical_repeat, position_id)`` + (:data:`.discovery.FOV_KEY`). + relative_path + The image, relative to the experiment root. + plane_ids + The ``plane_id`` of each of its rows in the table, in table order. + axes, roles + The image's axes and the role of each (``None`` for a length-1 axis without one). + axes_from + ``"header"`` or ``"confirmed"`` (a plain stack whose roles the user confirmed). + channel_names + The channel names from the metadata; empty when not given. + dtype + The experiment's pixel type. + significant_bits + Significant bits from the metadata, or None. + z_count, t_count, c_count + Depth slices, time points and channels. + """ + + key: tuple[str, str, str, str] + relative_path: str + plane_ids: tuple[str, ...] + axes: str + roles: tuple[str | None, ...] + axes_from: str + channel_names: tuple[str | None, ...] + dtype: str + significant_bits: int | None + z_count: int + t_count: int + c_count: int + + +@dataclass(frozen=True) +class RunInputs: + """The inputs of a clean preprocessing run, read as the run recorded them. + + Use :meth:`open`; the attributes are the snapshot taken then. + + Attributes + ---------- + output_folder + The run's output folder. + experiment_root + Where the images are read from. + masks_root + Where the masks folder's masks are read from, or None. + manifest + The run manifest. + records + The table's rows, as read when opened. + """ + + output_folder: Path + experiment_root: Path + masks_root: Path | None + manifest: RunManifest + records: tuple[PlaneRecord, ...] + + @classmethod + def open( + cls, + output_folder: str | os.PathLike, + experiment_root: str | os.PathLike | None = None, + masks_root: str | os.PathLike | None = None, + ) -> RunInputs: + """Open a clean run, checking its table, design and images against its manifest. + + Parameters + ---------- + output_folder + The run's output folder (``run_manifest.json``, ``plane_histograms.csv``). + experiment_root + Where the experiment is now; by default where the run read it. + masks_root + Where the masks folder is now; by default where the run read it. + + Raises + ------ + RunInputsError + For a run before v0.4.0, a table or design changed since the run, or an image or + mask changed or gone. + """ + output = Path(output_folder).resolve() + manifest = _manifest(output) + _check_sha(output / OUTPUT_FILENAME, manifest.csv_sha256, "histogram table") + if manifest.experiment_design_sha256 is not None: + _check_sha(output / DESIGN_FILENAME, manifest.experiment_design_sha256, "design") + root = Path(experiment_root or manifest.experiment_root).resolve() + recorded_masks = (manifest.masks or {}).get("folder") + masks = masks_root or recorded_masks + inputs = cls( + output_folder=output, + experiment_root=root, + masks_root=None if masks is None else Path(masks).resolve(), + manifest=manifest, + records=tuple(read_plane_histograms(output / OUTPUT_FILENAME)), + ) + changed = [e.relative_path for e in manifest.files if not inputs._unchanged(e)] + if changed: + raise RunInputsError(_changed_text(changed)) + recorded = (manifest.masks or {}).get("files", ()) + changed = [m["path"] for m in recorded if not inputs._mask_unchanged(m)] + if changed: + raise RunInputsError(_changed_text(changed, "mask")) + return inputs + + def fovs(self) -> tuple[Fov, ...]: + """Return every position of the run, sorted by image path.""" + rows: dict[str, list[PlaneRecord]] = {} + for record in self.records: + rows.setdefault(record.source_relative_path, []).append(record) + return tuple(_fov(e, rows.get(e.relative_path, []), self.manifest) for e in self.files) + + @property + def files(self) -> tuple[FileEntry, ...]: + """The manifest entry of every image, sorted by path.""" + return tuple(sorted(self.manifest.files, key=lambda e: e.relative_path)) + + def pixel_sizes(self, fov: Fov | str) -> dict[str, RecordedSize]: + """Return the recorded size of one step along X, Y, Z and T, by role. + + A size the run does not know has no value and no source (uncalibrated). + """ + return {size.role: size for size in self._entry(fov).pixel_sizes} + + def read_plane( + self, fov: Fov | str, z: int = 0, t: int = 0, *, memory_limit: int = DEFAULT_MEMORY_LIMIT + ) -> np.ndarray: + """Read one plane, every channel, as a ``(C, Y, X)`` array of the stored values. + + Raises + ------ + RunInputsError + If the image changed since the run. + IndexError + If ``z`` or ``t`` is outside the image. + """ + entry = self._checked(fov) + mapping = _mapping(entry) + path = self.experiment_root / entry.relative_path + coordinates = {} + for role, index in (("Z", z), ("T", t)): + if mapping.index_of(role) is None: + if index != 0: + raise IndexError(f"{entry.relative_path} has no {role} {index}") + continue + coordinates[role.lower()] = index + has_c = mapping.index_of("C") is not None + channels = self._fov(entry).c_count if has_c else 1 + planes = [ + read_plane( + path, mapping, c=c if has_c else None, memory_limit=memory_limit, **coordinates + ).pixels + for c in range(channels) + ] + return np.stack(planes) + + def read_image(self, fov: Fov | str) -> LazyImage: + """Return the image as a lazy ``(T, Z, C, Y, X)`` array: planes are read when indexed.""" + entry = self._checked(fov) + first = self.read_plane(fov) + found = self._fov(entry) + shape = (found.t_count, found.z_count, *first.shape) + return LazyImage(self, entry.relative_path, shape, first.dtype) + + def mask_types(self) -> tuple[str, ...]: + """Return the mask types the run recorded, sorted.""" + return tuple((self.manifest.masks or {}).get("types", ())) + + def read_mask( + self, fov: Fov | str, mask_type: str, z: int | None = None, t: int | None = None + ) -> np.ndarray: + """Read one mask plane by the recorded pairing, checking its labels. + + A stated shared 2-D mask is the same plane for every ``z`` and ``t``. + + Raises + ------ + RunInputsError + If the image has no recorded mask of that type, or the mask changed since the run. + ValueError + For a label outside 0 to the int32 limit (:func:`.masks.check_labels`). + """ + entry = self._checked(fov) + recorded = self._mask_entry(entry.relative_path, mask_type) + if not self._mask_unchanged(recorded): + raise RunInputsError(_changed_text([recorded["path"]], "mask")) + mask = MaskFile(recorded["image"], recorded["type"], recorded["path"], recorded["source"]) + folder = self.masks_root if mask.source == FOLDER else None + stored, shape, dtype = _mask_header(mask, folder, self.experiment_root) + # The mask is read through the axes it is stored with, as the run recorded them. + if stored != recorded["axes"] or list(shape) != list(recorded["shape"]): + raise RunInputsError(_changed_text([mask.path], "mask")) + mask = MaskFile( + mask.image, + mask.mask_type, + mask.path, + mask.source, + tuple(shape), + dtype, + str(recorded["roles"]), + str(recorded["axes"]), + str(recorded["axes_from"]), + ) + return read_mask(mask, folder, self.experiment_root, z=z, t=t) + + def histograms(self) -> tuple[PlaneRecord, ...]: + """Return the table's rows, as read when the run was opened.""" + return self.records + + def design(self) -> ExperimentDesign | None: + """Return the run's experiment design, or None when it had none.""" + if self.manifest.experiment_design_sha256 is None: + return None + path = self.output_folder / DESIGN_FILENAME + _check_sha(path, self.manifest.experiment_design_sha256, "design") + return read_experiment_design(path) + + def image_type(self) -> str | None: + """Return the image type the user confirmed (``"fluorescence"`` or ``"rgb"``), or None.""" + return self.manifest.image_type + + def _fov(self, entry: FileEntry) -> Fov: + """Describe one position from its entry and its rows of the snapshot.""" + rows = [r for r in self.records if r.source_relative_path == entry.relative_path] + return _fov(entry, rows, self.manifest) + + def _entry(self, fov: Fov | str) -> FileEntry: + """Return the manifest entry of a position, given as a :class:`Fov` or its image path.""" + relative = fov.relative_path if isinstance(fov, Fov) else str(fov) + entry = self.manifest.file(relative) + if entry is None: + raise RunInputsError(f"{relative} is not an image of this run.") + return entry + + def _checked(self, fov: Fov | str) -> FileEntry: + """Return a position's entry after checking its image is unchanged since the run.""" + entry = self._entry(fov) + if not self._unchanged(entry): + raise RunInputsError(_changed_text([entry.relative_path])) + return entry + + def _unchanged(self, entry: FileEntry) -> bool: + """Return True when an image's fingerprint is the one the run recorded.""" + try: + now = Fingerprint.of(self.experiment_root / entry.relative_path) + except FileNotFoundError: + return False + return now == Fingerprint.recorded(entry) + + def _mask_unchanged(self, recorded: Mapping) -> bool: + """Return True when a mask's fingerprint is the one the run recorded.""" + if recorded["source"] == FOLDER: + if self.masks_root is None: + return False + where = self.masks_root / recorded["path"] + else: + where = self.experiment_root / recorded["path"] + try: + return Fingerprint.of(where) == Fingerprint.recorded(recorded) + except FileNotFoundError: + return False + + def _mask_entry(self, image: str, mask_type: str) -> Mapping: + """Return the recorded mask of an image and type.""" + for recorded in (self.manifest.masks or {}).get("files", ()): + if recorded["image"] == image and recorded["type"] == mask_type: + return recorded + raise RunInputsError(f"The run recorded no {mask_type} mask for {image}.") + + +@dataclass(frozen=True) +class LazyImage: + """An image of a clean run as a ``(T, Z, C, Y, X)`` array whose planes are read on demand. + + Attributes + ---------- + inputs + The run. + relative_path + The image. + shape + ``(T, Z, C, Y, X)``. + dtype + The stored pixel type. + """ + + inputs: RunInputs + relative_path: str + shape: tuple[int, ...] + dtype: np.dtype + + def __getitem__(self, index: tuple[int, int]) -> np.ndarray: + """Read the ``(C, Y, X)`` plane at ``[t, z]``.""" + t, z = index + return self.inputs.read_plane(self.relative_path, z=z, t=t) + + def __array__(self, dtype: np.dtype | None = None, copy: bool | None = None) -> np.ndarray: + """Read every plane into one array.""" + data = np.stack( + [np.stack([self[t, z] for z in range(self.shape[1])]) for t in range(self.shape[0])] + ) + return data if dtype is None else data.astype(dtype) + + +def _manifest(output: Path) -> RunManifest: + """Read an output folder's manifest, refusing runs written before v0.4.0.""" + path = output / MANIFEST_FILENAME + if not path.exists(): + raise RunInputsError(f"{output} holds no {MANIFEST_FILENAME}: run preprocessing first.") + manifest = read_manifest(path) + if manifest.manifest_version == LEGACY_MANIFEST_VERSION: + raise RunInputsError( + f"The run in {output} was written before v0.4.0, without the axis check. Run " + "preprocessing again on the experiment." + ) + return manifest + + +def _check_sha(path: Path, expected: str, what: str) -> None: + """Refuse a file whose SHA-256 is not the one the manifest recorded.""" + if not path.exists() or file_sha256(path) != expected: + raise RunInputsError( + f"The {what} {path.name} is not the one the run wrote (missing, or changed since). " + "Run preprocessing again." + ) + + +def _changed_text(paths: list[str], what: str = "image") -> str: + """Say which files changed or went missing since the run.""" + shown = ", ".join(sorted(paths)[:5]) + (" and more" if len(paths) > 5 else "") + where = "experiment_root=" if what == "image" else "masks_root=" + return ( + f"{len(paths)} {what}{'' if len(paths) == 1 else 's'} changed or missing since the " + f"run: {shown}. Run preprocessing again on the experiment as it is (or, if the " + f"{what}s were only moved, give {where} where they are now)." + ) + + +def _mapping(entry: FileEntry) -> AxisMapping: + """Return an image's own mapping as the run recorded it.""" + return AxisMapping(entry.axes, tuple(entry.roles or ()), entry.singleton_roles) + + +def _fov(entry: FileEntry, rows: list[PlaneRecord], manifest: RunManifest) -> Fov: + """Describe one position from its manifest entry and its table rows.""" + zs = {r.z for r in rows} or {0} + ts = {r.t for r in rows} or {0} + channels = rows[0].channel_count if rows else len(entry.channel_names) or 1 + return Fov( + key=(*entry.labels, entry.position_id or ""), + relative_path=entry.relative_path, + plane_ids=tuple(r.plane_id for r in rows), + axes=entry.axes, + roles=tuple(entry.roles or ()), + axes_from=entry.axes_from or "", + channel_names=entry.channel_names, + dtype=manifest.dtype, + significant_bits=entry.significant_bits, + z_count=max(zs) + 1, + t_count=max(ts) + 1, + c_count=channels, + ) diff --git a/src/morphobricks_preprocessing/sources.py b/src/morphobricks_preprocessing/sources.py index 95bcaa52..4bf44ba9 100644 --- a/src/morphobricks_preprocessing/sources.py +++ b/src/morphobricks_preprocessing/sources.py @@ -12,12 +12,12 @@ from .axes import file_mapping_for from .image_names import position_id -from .manifest import RunManifest, source_fingerprint +from .manifest import LEGACY_MANIFEST_VERSION, RunManifest, source_fingerprint from .metadata import inspect_image from .validation import ValidationIssue, error, plural -def check_sources( # noqa: C901 +def check_sources( manifest: RunManifest, experiment_root: str | os.PathLike, relative_paths: Iterable[str], @@ -33,7 +33,9 @@ def check_sources( # noqa: C901 modification time :func:`source_fingerprint` gives (an OME-Zarr folder: its files' total size and newest time); its recorded ``position_id`` is the one its name gives; with ``identities``, its labels and ``position_id`` are the caller's; with ``image_axes``, - the caller's axes give every axis the role the run confirmed for that image. + the caller's axes give every axis the role the run confirmed for that image. A version 1 + manifest (a run before v0.4.0, without the axis check) fails every image: such a run + must be redone (design v0.4 §6). Parameters ---------- @@ -56,7 +58,8 @@ def check_sources( # noqa: C901 Returns ------- list of ValidationIssue - Errors, each naming its images: ``images_not_in_manifest``, ``images_missing`` + Errors, each naming its images: ``manifest_before_v040`` (alone, for a version 1 + manifest), ``images_not_in_manifest``, ``images_missing`` (no longer on disk), ``images_changed``, ``position_id_not_recorded`` (a manifest from before ``position_id`` was recorded, when ``identities`` are checked), ``identity_differs`` and ``axis_mapping_mismatch``. Empty when all images pass. @@ -72,6 +75,8 @@ def check_sources( # noqa: C901 absent = [path for path in paths if given is not None and path not in given] if absent: raise ValueError(f"{name} has no entry for {', '.join(absent)}") + if manifest.manifest_version == LEGACY_MANIFEST_VERSION: + return [_issue("manifest_before_v040", [(path, "") for path in paths])] found: dict[str, list[tuple[str, str]]] = {} def report(code: str, path: str, detail: str = "") -> None: @@ -115,15 +120,17 @@ def report(code: str, path: str, detail: str = "") -> None: if not _roles_agree(axes, roles): described = "none fit" if roles is None else "".join(r or "-" for r in roles) report("axis_mapping_mismatch", path, f"given {axes}, preprocessing {described}") - issues = [] - for code, items in found.items(): - message, action = _SOURCE_TEXTS[code] - message = message.format(n=plural(len(items), "image")) - details = [f"{path}: {detail}" for path, detail in items if detail] - if details: - message += " " + "; ".join(details) + "." - issues.append(error(code, message, sorted(path for path, _ in items), action)) - return issues + return [_issue(code, items) for code, items in found.items()] + + +def _issue(code: str, items: list[tuple[str, str]]) -> ValidationIssue: + """Build the error of one code, naming its images and their details.""" + message, action = _SOURCE_TEXTS[code] + message = message.format(n=plural(len(items), "image")) + details = [f"{path}: {detail}" for path, detail in items if detail] + if details: + message += " " + "; ".join(details) + "." + return error(code, message, sorted(path for path, _ in items), action) def _roles_agree(axes: str, roles: Sequence[str | None] | None) -> bool: @@ -137,6 +144,11 @@ def _roles_agree(axes: str, roles: Sequence[str | None] | None) -> bool: _SOURCE_TEXTS = { + "manifest_before_v040": ( + "The manifest of {n} was written before v0.4.0, without the axis check, so its " + "results cannot be used.", + "Run the preprocessing brick again on this experiment.", + ), "images_not_in_manifest": ( "{n} not analysed by the preprocessing run of this manifest: added after it ran, " "excluded, or from another experiment.", diff --git a/src/morphobricks_preprocessing/testing.py b/src/morphobricks_preprocessing/testing.py new file mode 100644 index 00000000..975aa964 --- /dev/null +++ b/src/morphobricks_preprocessing/testing.py @@ -0,0 +1,128 @@ +"""A small clean preprocessing run for the other bricks' tests (design v0.4 §8, PP-6). + +:func:`make_clean_run` writes a tiny experiment of named-axis TIFFs (and, if asked, a masks +folder), runs preprocessing on it and returns where everything is, so a brick's tests read +their inputs through :class:`.run_inputs.RunInputs` exactly as a user's run would give them. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass +from pathlib import Path + +import numpy as np +import tifffile + +from .axes import prefill_mapping +from .dataset import biological_repeats +from .design import ExperimentDesign, default_design +from .discovery import discover_experiment +from .run import RunResult, run_experiment + + +@dataclass(frozen=True) +class CleanRun: + """A clean run made by :func:`make_clean_run`. + + Attributes + ---------- + experiment + The experiment folder. + output + The run's output folder. + masks + The masks folder, or None. + result + What :func:`.run.run_experiment` returned. + """ + + experiment: Path + output: Path + masks: Path | None + result: RunResult + + +def make_clean_run( + folder: str | os.PathLike, + *, + axes: str = "CYX", + shape: tuple[int, ...] = (2, 16, 20), + mask_types: tuple[str, ...] = (), + pixel_size: float | None = 0.5, + design: bool = False, + seed: int = 0, +) -> CleanRun: + """Write a two-condition experiment of named-axis TIFFs and run preprocessing on it. + + Each condition (``ctrl``, ``drug``) has one biological repeat (``rep1``) of two images + (``a.tif``, ``b.tif``) of random ``uint16`` pixels, with their axes named in their + metadata. Each mask type gets one 2-D label image per image (two square objects). + + Parameters + ---------- + folder + An empty folder; the experiment, masks and output are written inside it. + axes, shape + The images' axes (Y before X; ``YX`` planes with ``C``, ``Z`` and/or ``T``) and shape. + mask_types + Mask types to write (``("Cells",)``); 2-D masks need images with one plane. + pixel_size + The X and Y pixel size in micrometres written to the headers, or None for none. + design + Run with a confirmed experiment design (``ctrl`` the control, each ``rep1`` its own + repeat group), so the run writes ``experiment_design.csv``. + seed + The random seed of the pixels. + + Returns + ------- + CleanRun + + Raises + ------ + RuntimeError + If the run does not write its outputs (its issues are in the message). + """ + root = Path(folder) + experiment, masks = root / "experiment", (root / "masks" if mask_types else None) + rng = np.random.default_rng(seed) + metadata: dict = {"axes": axes} + if pixel_size is not None: + metadata.update(PhysicalSizeX=pixel_size, PhysicalSizeY=pixel_size) + for condition in ("ctrl", "drug"): + for name in ("a", "b"): + path = experiment / condition / "rep1" / f"{name}.ome.tif" + path.parent.mkdir(parents=True, exist_ok=True) + data = rng.integers(0, 4000, shape, dtype=np.uint16) + tifffile.imwrite(path, data, ome=True, photometric="minisblack", metadata=metadata) + for mask_type in mask_types: + _write_mask(masks / mask_type / condition / "rep1" / f"{name}.tif", shape[-2:]) + output = root / "output" + result = run_experiment( + experiment, + prefill_mapping(axes), + output_folder=output, + masks_folder=masks, + design=_confirmed_design(experiment) if design else None, + ) + if not result.written: + raise RuntimeError("The clean run was not written: " + "; ".join(map(str, result.issues))) + return CleanRun(experiment, output, masks, result) + + +def _confirmed_design(experiment: Path) -> ExperimentDesign: + """Confirm the default design of an experiment, with ``ctrl`` as the control.""" + found = default_design(biological_repeats(discover_experiment(experiment))) + for group in found.groups: + found = found.with_confirmed(group, True, "2026-01-01") + return found.with_control("ctrl", True) + + +def _write_mask(path: Path, yx: tuple[int, ...]) -> None: + """Write a 2-D label image with two square objects (labels 1 and 2).""" + labels = np.zeros(yx, np.uint16) + labels[1:4, 1:4] = 1 + labels[-4:-1, -4:-1] = 2 + path.parent.mkdir(parents=True, exist_ok=True) + tifffile.imwrite(path, labels, photometric="minisblack", metadata={"axes": "YX"}) diff --git a/tests/test_check_sources.py b/tests/test_check_sources.py index c573f76a..fb073968 100644 --- a/tests/test_check_sources.py +++ b/tests/test_check_sources.py @@ -160,3 +160,10 @@ def test_every_path_needs_its_axes_and_identity(run): check_sources(manifest, root, IMAGES, image_axes=dict.fromkeys(IMAGES[:2], "ZYX")) with pytest.raises(ValueError, match="identities has no entry"): check_sources(manifest, root, IMAGES, identities={}) + + +def test_a_version_1_manifest_fails_every_image(run): + root, manifest = run + issues = check_sources(replace(manifest, manifest_version="1"), root, IMAGES) + assert codes(issues) == {"manifest_before_v040": tuple(sorted(IMAGES))} + assert issues[0].is_error diff --git a/tests/test_run_inputs.py b/tests/test_run_inputs.py new file mode 100644 index 00000000..5f314e1b --- /dev/null +++ b/tests/test_run_inputs.py @@ -0,0 +1,308 @@ +"""The contract of RunInputs: what a clean run gives the bricks (design v0.4 §6, PP-6).""" + +import json +import multiprocessing +import os +import pickle +import shutil + +import numpy as np +import pytest +import tifffile + +from morphobricks_preprocessing import DESIGN_FILENAME, AxisMapping, prefill_mapping, run_experiment +from morphobricks_preprocessing.manifest import MANIFEST_FILENAME +from morphobricks_preprocessing.run_inputs import RunInputs, RunInputsError +from morphobricks_preprocessing.testing import make_clean_run +from tiff_tree import write_data, write_plain +from zarr_tree import write_labels, write_ngff + + +@pytest.fixture +def clean(tmp_path): + return make_clean_run(tmp_path, mask_types=("Cells",)) + + +def test_fovs_come_from_the_manifest_and_the_table(clean): + inputs = RunInputs.open(clean.output) + fovs = inputs.fovs() + assert [f.key for f in fovs] == [ + ("ctrl", "rep1", "single", "a"), + ("ctrl", "rep1", "single", "b"), + ("drug", "rep1", "single", "a"), + ("drug", "rep1", "single", "b"), + ] + first = fovs[0] + assert (first.axes, first.roles, first.axes_from) == ("CYX", ("C", "Y", "X"), "header") + assert (first.c_count, first.z_count, first.t_count, first.dtype) == (2, 1, 1, "uint16") + assert first.plane_ids == ("ctrl/rep1/a.ome.tif::z=0,t=0",) + + +def test_planes_and_images_are_the_stored_values(clean): + inputs = RunInputs.open(clean.output) + for fov in inputs.fovs(): + stored = tifffile.imread(clean.experiment / fov.relative_path) + np.testing.assert_array_equal(inputs.read_plane(fov), stored) + image = inputs.read_image(fov) + assert image.shape == (1, 1, 2, 16, 20) + np.testing.assert_array_equal(np.asarray(image)[0, 0], stored) + with pytest.raises(IndexError): + inputs.read_plane(fov, z=1) + + +def test_pixel_sizes_carry_their_unit_and_source(clean): + sizes = RunInputs.open(clean.output).pixel_sizes("ctrl/rep1/a.ome.tif") + assert (sizes["X"].value, sizes["X"].unit, sizes["X"].source) == (0.5, "micrometer", "header") + assert sizes["Z"].value is None and sizes["Z"].source is None + + +def test_masks_by_the_recorded_pairing(clean): + inputs = RunInputs.open(clean.output) + assert inputs.mask_types() == ("Cells",) + mask = inputs.read_mask("drug/rep1/b.ome.tif", "Cells") + assert mask.shape == (16, 20) and set(np.unique(mask)) == {0, 1, 2} + with pytest.raises(RunInputsError, match="no Nuclei mask"): + inputs.read_mask("drug/rep1/b.ome.tif", "Nuclei") + + +def test_histograms_design_and_image_type(clean): + inputs = RunInputs.open(clean.output) + assert len(inputs.histograms()) == 4 + assert {r.source_relative_path for r in inputs.histograms()} == { + f.relative_path for f in inputs.fovs() + } + assert inputs.design() is None and inputs.image_type() is None + + +def test_run_inputs_pickle(clean): + inputs = pickle.loads(pickle.dumps(RunInputs.open(clean.output))) + stored = tifffile.imread(clean.experiment / "ctrl/rep1/a.ome.tif") + np.testing.assert_array_equal(inputs.read_plane("ctrl/rep1/a.ome.tif"), stored) + assert inputs.read_mask("ctrl/rep1/a.ome.tif", "Cells").max() == 2 + + +# Refusals -------------------------------------------------------------------------------- + + +def test_a_version_1_run_is_refused(clean): + path = clean.output / MANIFEST_FILENAME + data = json.loads(path.read_text(encoding="utf-8")) + data["manifest_version"] = "1" + path.write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(RunInputsError, match="before v0.4.0"): + RunInputs.open(clean.output) + + +def test_a_changed_table_is_refused(clean): + table = clean.output / "plane_histograms.csv" + table.write_text(table.read_text(encoding="utf-8") + "\n", encoding="utf-8") + with pytest.raises(RunInputsError, match="histogram table"): + RunInputs.open(clean.output) + + +def test_an_image_changed_before_opening_is_refused(clean): + image = clean.experiment / "ctrl/rep1/a.ome.tif" + image.write_bytes(image.read_bytes() + b"\0") + with pytest.raises(RunInputsError, match="ctrl/rep1/a.ome.tif"): + RunInputs.open(clean.output) + + +def test_an_image_changed_after_opening_is_not_read(clean): + inputs = RunInputs.open(clean.output) + image = clean.experiment / "ctrl/rep1/b.ome.tif" + stat = image.stat() + os.utime(image, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**9)) + with pytest.raises(RunInputsError, match="changed or missing"): + inputs.read_plane("ctrl/rep1/b.ome.tif") + inputs.read_plane("ctrl/rep1/a.ome.tif") # the others are still read + + +def test_a_changed_mask_is_not_read(clean): + inputs = RunInputs.open(clean.output) + mask = clean.masks / "Cells/ctrl/rep1/a.tif" + tifffile.imwrite(mask, np.ones((16, 20), np.uint16), metadata={"axes": "YX"}) + with pytest.raises(RunInputsError, match="mask"): + inputs.read_mask("ctrl/rep1/a.ome.tif", "Cells") + + +def test_a_moved_experiment_is_opened_where_it_is(clean, tmp_path): + moved = tmp_path / "moved" + shutil.copytree(clean.experiment, moved, copy_function=shutil.copy2) + shutil.rmtree(clean.experiment) + with pytest.raises(RunInputsError): + RunInputs.open(clean.output) + inputs = RunInputs.open(clean.output, experiment_root=moved) + assert inputs.read_plane("ctrl/rep1/a.ome.tif").shape == (2, 16, 20) + + +# Plain stacks, stated masks and labels ------------------------------------------------------- + + +def test_a_confirmed_plain_stack_and_a_shared_mask(tmp_path): + root, masks = tmp_path / "exp", tmp_path / "masks" + stack = np.arange(3 * 6 * 8, dtype=np.uint8).reshape(3, 6, 8) + write_plain(root / "ctrl/a.tif", stack) + labels = np.zeros((6, 8), np.uint16) + labels[2:4, 2:5] = 7 + write_data(masks / "Cells/ctrl/a.tif", labels, "YX") + result = run_experiment( + root, + AxisMapping("IYX", "TYX"), + masks_folder=masks, + mask_axes={"Cells": "YX"}, + ) + inputs = RunInputs.open(result.output_folder) + (fov,) = inputs.fovs() + assert (fov.axes_from, fov.t_count, fov.c_count) == ("confirmed", 3, 1) + np.testing.assert_array_equal(inputs.read_plane(fov, t=2), stack[2:3]) + for t in range(3): + np.testing.assert_array_equal(inputs.read_mask(fov, "Cells", t=t), labels) + + +def test_labels_are_checked_on_every_read(tmp_path): + root, masks = tmp_path / "exp", tmp_path / "masks" + write_data(root / "ctrl/a.tif", np.zeros((6, 8), np.uint8), "YX") + signed = np.zeros((6, 8), np.int32) + signed[0, 0] = -5 + write_data(masks / "Cells/ctrl/a.tif", signed, "YX") + result = run_experiment(root, prefill_mapping("YX"), masks_folder=masks) + inputs = RunInputs.open(result.output_folder) + with pytest.raises(ValueError, match="-5|negative"): + inputs.read_mask("ctrl/a.tif", "Cells") + + +# Review additions: worker processes, design, canonical order, stored mask axes ----------------- + + +def test_run_inputs_work_in_spawned_worker_processes(clean): + inputs = RunInputs.open(clean.output) + paths = [f.relative_path for f in inputs.fovs()] + with multiprocessing.get_context("spawn").Pool(2) as pool: + planes = pool.starmap(RunInputs.read_plane, [(inputs, p) for p in paths]) + masks = pool.starmap(RunInputs.read_mask, [(inputs, p, "Cells") for p in paths]) + for path, plane, mask in zip(paths, planes, masks, strict=True): + np.testing.assert_array_equal(plane, tifffile.imread(clean.experiment / path)) + np.testing.assert_array_equal(mask, tifffile.imread(clean.masks / "Cells" / mask_of(path))) + + +def mask_of(image): + return image.replace(".ome.tif", ".tif") + + +def test_the_design_of_a_run_with_one(tmp_path): + clean = make_clean_run(tmp_path, design=True) + design = RunInputs.open(clean.output).design() + assert design.repeat_groups() == {("ctrl", "rep1"): "rep1", ("drug", "rep1"): "rep1"} + assert {row.condition: row.control for row in design.rows} == {"ctrl": True, "drug": False} + assert all(row.confirmed for row in design.rows) + + +def test_a_changed_or_missing_design_is_refused(tmp_path): + clean = make_clean_run(tmp_path, design=True) + path = clean.output / DESIGN_FILENAME + inputs = RunInputs.open(clean.output) + path.write_text(path.read_text(encoding="utf-8").replace("true", "false"), encoding="utf-8") + with pytest.raises(RunInputsError, match="design"): + inputs.design() + with pytest.raises(RunInputsError, match="design"): + RunInputs.open(clean.output) + path.unlink() + with pytest.raises(RunInputsError, match="design"): + RunInputs.open(clean.output) + + +def test_a_header_naming_x_before_y_is_read_y_before_x(tmp_path): + # Owner rule (design v0.4 change log): the header names the axes, so reordering is exact. + stored = np.arange(2 * 2 * 5 * 7, dtype=np.uint16).reshape(2, 2, 5, 7) # Z, C, X, Y + write_data(tmp_path / "exp/ctrl/a.tif", stored, "ZCXY") + result = run_experiment(tmp_path / "exp", prefill_mapping("ZCXY")) + assert result.written, result.issues + inputs = RunInputs.open(result.output_folder) + (fov,) = inputs.fovs() + canonical = stored.transpose(0, 1, 3, 2) # Z, C, Y, X + np.testing.assert_array_equal(inputs.read_plane(fov, z=1), canonical[1]) + image = inputs.read_image(fov) + assert image.shape == (1, 2, 2, 7, 5) + np.testing.assert_array_equal(np.asarray(image)[0], canonical) + + +def test_read_image_is_t_z_c_y_x_whatever_the_stored_order(tmp_path): + stored = np.random.default_rng(1).integers(0, 999, (3, 2, 2, 6, 8), dtype=np.uint16) + path = tmp_path / "exp/ctrl/a.ome.tif" + path.parent.mkdir(parents=True) + tifffile.imwrite(path, stored, ome=True, photometric="minisblack", metadata={"axes": "ZTCYX"}) + with tifffile.TiffFile(path) as tif: + assert tif.series[0].axes == "ZTCYX" + result = run_experiment(tmp_path / "exp", prefill_mapping("ZTCYX")) + inputs = RunInputs.open(result.output_folder) + (fov,) = inputs.fovs() + assert (fov.z_count, fov.t_count, fov.c_count) == (3, 2, 2) + np.testing.assert_array_equal( + np.asarray(inputs.read_image(fov)), stored.transpose(1, 0, 2, 3, 4) + ) + np.testing.assert_array_equal(inputs.read_plane(fov, z=2, t=1), stored[2, 1]) + + +@pytest.fixture +def stated(tmp_path): + """A TYX image with a plain-stack mask whose axes are stated TYX (stored IYX).""" + root, masks = tmp_path / "exp", tmp_path / "masks" + write_data(root / "ctrl/a.tif", np.zeros((3, 6, 8), np.uint8), "TYX") + labels = np.zeros((3, 6, 8), np.uint16) + labels[1, 1:3, 1:3], labels[2, 2:4, 2:4] = 4, 9 + write_plain(masks / "Cells/ctrl/a.tif", labels) + result = run_experiment( + root, prefill_mapping("TYX"), masks_folder=masks, mask_axes={"Cells": "TYX"} + ) + assert result.written, result.issues + return result.output_folder, masks, labels + + +def test_a_stated_mask_is_read_through_its_stored_axes(stated): + output, _, labels = stated + inputs = RunInputs.open(output) + (recorded,) = inputs.manifest.masks["files"] + assert (recorded["axes"], recorded["roles"], recorded["axes_from"]) == ("IYX", "TYX", "stated") + for t in range(3): + np.testing.assert_array_equal(inputs.read_mask("ctrl/a.tif", "Cells", t=t), labels[t]) + with pytest.raises(ValueError, match="say which one"): + inputs.read_mask("ctrl/a.tif", "Cells") + + +def test_a_mask_stored_otherwise_than_recorded_is_not_read(stated): + output, _, _ = stated + path = output / MANIFEST_FILENAME + data = json.loads(path.read_text(encoding="utf-8")) + data["masks"]["files"][0]["axes"] = "TYX" + path.write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(RunInputsError, match="mask"): + RunInputs.open(output).read_mask("ctrl/a.tif", "Cells", t=0) + + +def test_a_mask_changed_or_moved_before_opening_is_refused(stated, tmp_path): + output, masks, labels = stated + moved = tmp_path / "moved_masks" + shutil.copytree(masks, moved, copy_function=shutil.copy2) + shutil.rmtree(masks) + with pytest.raises(RunInputsError, match="1 mask changed or missing.*masks_root="): + RunInputs.open(output) + inputs = RunInputs.open(output, masks_root=moved) + np.testing.assert_array_equal(inputs.read_mask("ctrl/a.tif", "Cells", t=2), labels[2]) + write_plain(moved / "Cells/ctrl/a.tif", labels + 1) + with pytest.raises(RunInputsError, match="Cells/ctrl/a.tif"): + RunInputs.open(output, masks_root=moved) + + +def test_ome_zarr_label_images_are_read_by_their_pairing(tmp_path): + pytest.importorskip("zarr") + image = np.random.default_rng(2).integers(0, 99, (2, 4, 5)).astype(np.uint8) + path = write_ngff(tmp_path / "exp/ctrl/a.ome.zarr", image, "zyx") + labels = np.zeros((2, 4, 5), np.uint16) + labels[1, 1:3, 1:3] = 3 + write_labels(path, {"cells": labels}, "zyx") + result = run_experiment(tmp_path / "exp", prefill_mapping("ZYX")) + assert result.written, result.issues + inputs = RunInputs.open(result.output_folder) + assert inputs.mask_types() == ("cells",) + np.testing.assert_array_equal(inputs.read_mask("ctrl/a.ome.zarr", "cells", z=1), labels[1]) + np.testing.assert_array_equal(np.asarray(inputs.read_image("ctrl/a.ome.zarr"))[0, :, 0], image) diff --git a/tools/code_rules_baseline.json b/tools/code_rules_baseline.json index b9984c8f..e186a786 100644 --- a/tools/code_rules_baseline.json +++ b/tools/code_rules_baseline.json @@ -1,7 +1,7 @@ { "noqa": { "ANN": 57, - "C901": 2, + "C901": 1, "PLR0913": 3, "PLR0915": 2 },