diff --git a/CHANGELOG.md b/CHANGELOG.md index 180fea41..2489dcb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -130,6 +130,52 @@ contract §9.1 and §16). **API additions only.** `VolumeNotSupportedError`. Suite: `tests/test_spec_resolve_request.py`, a Spec module (`spec_gate.EXPECTED_MODULES`). +### PP-1: `Fov.image_type` per position, the partial-mask test helper, the pinned surface + +Stage 2 plan row PP-1 (MorphoBricks-Spec ADR 0002 AR8, contract §9.1 and §16). **Breaking** for a +reader of `RunInputs.image_type()`. + +- **`Fov.image_type`: each position's declared type.** `"rgb"` or `"channels"` (a unit that + declares no `image_type`, experiment-v1 §4.4b), from the published description only, never + from a header, and never None on a `Fov` of `fovs()`. One source: `FovIdentity.image_type` + (`fov_identity.declared_image_type`), which `Fov` and the resolver's `FovSelection` both carry, + so the two always agree; `resolve_request` reads the position's `Fov.image_type` instead of + looking the unit up again. `FovIdentity.image_type` and `FovSelection.image_type` are now `str` + (`"channels"` where PP-0 gave None; resolver contract still version 1, unreleased before + v0.5.1); `run_inputs_request.CHANNELS` and `IMAGE_TYPES` name the two values. +- **Removed: `RunInputs.image_type()`.** It returned the manifest's run-level `image_type` (the + command line's review, null on every described run), a run-wide value that would hide a + unit's own type. Read `Fov.image_type` (or `FovSelection.image_type`). The manifest field + stays as a record. Downstream: CellPatcher reads `inputs.image_type()` today + (`run_open.py`, `normalization.py`, `notebook_session.py`); its v0.5.1 pin (CP-4) moves to + `Fov.image_type`. +- **`make_described_run(..., masks=IDS, mask_images=PATHS)`** writes the masks of `IDS` for the + named images only (relative paths, as the new `testing.described_images(description)` lists + them), so a run with partial masks, such as Spec `v09` with `Nuclei` on 2 of 6 images, is + built as a real run; `None` (the default) is every image, as before. Explicit only: nothing + is chosen at random; a path outside the tree, an empty subset, a single text or a subset + without masks is a `ValueError` before anything is written. `write_described_tree` takes the + same `mask_images`. +- **The mask step's status file is parsed once per content.** `read_checked` still reads + `mask_representation.json` on every call but reuses the parse of the same bytes + (`lru_cache` keyed by the bytes, as `description_consume` already does for the description), + so the result is the one an uncached read gives and a rewritten file is parsed again; + `RunInputs` holds no memo and pickles as before. Measured on a 10,000-position `v09` run with + `Nuclei` masks (one machine, best of runs): `mask_check_status` for 1,000 positions 6.6 s + before, 0.85 s after, the same 1,000 answers and the same `resolve_request` result. The + description itself was already parsed once per content (`description_consume`), so it needed + no memo. Left as found (a later item): `resolve_request` takes about 3 s on that run and + `mask_check_status` for all 10,000 positions about 12 s, both from linear scans per file + (`file_status` over the status file's list, `_mask_entry` over the manifest's), not from + reading the description. +- **The pinned surface.** `tests/test_brick_api_surface.py` pins the names, fields, types and + signatures the bricks use (`RunInputs`, `Fov`, the resolver records, `resolve_request`, + `make_described_run`, `make_clean_run`, `described_images`, `RESOLVER_VERSION`); contract + §9.1 lists them. New Spec module `tests/test_spec_fov_image_type.py` (`v28` rgb on every + position; `v13`, `v01` channels; `v29`'s two units through the identity; `v09` on 2 of 6 + through the helper equal to PP-0's hand-built tree), in `spec_gate.EXPECTED_MODULES`; PP-0's + `v09` test now uses the helper. + ## 0.5.0 - 2026-11-09 Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`): diff --git a/docs/contract.md b/docs/contract.md index b8580929..7b80d4b9 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -330,7 +330,7 @@ It gives: (v0.4.0, plan §12 question 54). A `z` or `t` outside the image (`Fov.z_count`, `Fov.t_count`) is an `IndexError` for both reads (audit I5). There is no read by `plane_id`: a row gives its `z` and `t`; -- `histograms()` (the table as read when opened), `design()` and `image_type()`. `design()` +- `histograms()` (the table as read when opened) and `design()`. `design()` reads `experiment_design.csv` with the strict `read_confirmed_design` (§16) and the manifest's `experiment_design_sha256` (issue #83, plan §12 question 62; it used the lenient reader before): a design that reader refuses raises `RunInputsError`, with the refusal @@ -340,10 +340,17 @@ It gives: `DescriptionRequiredError`, `A_DESCRIPTION_REQUIRED`, and an older manifest with `ManifestFormatError`, "run preprocessing again"). `Fov.repeat_group`, `Fov.condition_id`, `Fov.technical_repeat_id` (the canonical IDs, added beside the unchanged `Fov.key`, whose condition, repeat and technical repeat stay - the source folder labels: owner decision Q8, experiment-v1 §3.4) and `Fov.channel_ids` (the + the source folder labels: owner decision Q8, experiment-v1 §3.4), `Fov.channel_ids` (the canonical channel ID of each C index, as the unit declares them; a unit whose channel count is not the image's, or folders that name no declared unit, is a `RunInputsError`: nothing is - guessed). `channel_index(fov, channel_id)` gives the C index, `ChannelNotFoundError` for a + guessed) and `Fov.image_type` (v0.5.1, PP-1: the unit's declared type, `"rgb"` or + `"channels"` for a unit that declares none, experiment-v1 §4.4b; from the published + description only, never from a header, and never None on a `Fov` of `fovs()`). Each position + carries its own unit's type, through `FovIdentity` (`fov_identity`), the one source of + `Fov.image_type` and `FovSelection.image_type` alike. There is no run-wide image type: + `RunInputs.image_type()` is removed (v0.5.1, PP-1; it returned the manifest's run-level + `image_type`, null on every described run, which would hide per-unit types). The manifest's + `image_type` (§9, §13) stays a record of the command line's review, not an input of a brick. `channel_index(fov, channel_id)` gives the C index, `ChannelNotFoundError` for a channel the image lacks (never a default). `references()` is the condition reference graph (condition ID to the ID it references, or `None`); `mask_semantics()` maps mask ID to `MaskSemantics(source_type, representation, source_channels)`; `mask_check_status(mask_id, @@ -365,7 +372,8 @@ It gives: one t), the consumer's `Capabilities` (`rgb_support`, `volume_support`, both False by default) and the explicit `device` (`cpu` or `cuda`; recorded, never chosen here). It returns an immutable, picklable `ResolvedRequest`: one `FovSelection` per position (the unchanged key, - `repeat_group`, `condition_id`, `technical_repeat_id`, the unit's declared `image_type`, the C + `repeat_group`, `condition_id`, `technical_repeat_id`, the unit's declared `image_type` + (`rgb` or `channels`, the position's `Fov.image_type`), the C index of every requested channel **in that image** (`channel_indices`, `index_of`), the masks as `MaskSelection(mask_id, source_type, representation, role, check_status, conversion)` and the `PlaneSelection`s to read), the `refusals`, the `coverage` of every biological repeat per @@ -380,7 +388,9 @@ It gives: (`VolumeNotSupportedError`, the resolver contract's own code: a volume without `volume_support`). `ResolvedRequest.require()` returns the selections or raises the first refusal's error class with every refusal in its text. The resolver reads the manifest, the - published description and `mask_representation.json` only: no pixel, no kernel, no brick, + published description and `mask_representation.json` only (both read on every call; the + parse of the same bytes is reused, by content, so a file rewritten since is parsed again: + PP-1): no pixel, no kernel, no brick, CellPatcher or provider import (a test scans its modules), and it writes nothing. It converts nothing: a recorded policy comes back as `MaskSelection.conversion` and each route applies it with the one shared labeller (CellPatcher's), so both routes give the same objects. The @@ -402,10 +412,25 @@ runs preprocessing and, with masks, the mask step: a described run, for the bric (v0.5.1; before, it was an undescribed run). `make_clean_run(folder, description=)` (also `testing.make_described_run`) writes instead the tree a published experiment description declares (`image_count` tiny images per -member, `rgb` units as RGB colour, one mask per image for each declared mask ID in `mask_types`), +member, `rgb` units as RGB colour, one mask per image for each declared mask ID in `mask_types`; +`make_described_run(..., masks=IDS, mask_images=PATHS)` writes the masks for the named images only, +a declared subset as `testing.described_images(description)` lists them, for a run with partial +masks such as Spec `v09` on 2 of 6: v0.5.1, PP-1; nothing is chosen at random, and a path outside +the tree, an empty subset or a subset without masks is a `ValueError` before anything is written), publishes the bytes as `morphobricks.toml`, runs preprocessing and the mask step, and returns the same `CleanRun` with `description` set; `axes`, `shape`, `pixel_size`, `design` and `seed` then -raise `ValueError`. Tested for the Spec fixtures `v03 v08 v13 v28 v30 v33 v34`. +raise `ValueError`. Tested for the Spec fixtures `v03 v08 v09 v13 v28 v30 v33 v34`. + +**The pinned surface (v0.5.1, PP-1).** `tests/test_brick_api_surface.py` pins, by name and type, +what the consuming bricks use: `RunInputs`'s public methods and their signatures (`open`, `fovs`, +`files`, `projection`, `pixel_sizes`, `read_plane`, `read_image`, `mask_types`, `read_mask`, +`histograms`, `design`, `description`, `section_digests`, `images_without_header_channel_names`, +`channel_index`, `references`, `mask_semantics`, `mask_check_status`, `require_checked`, +`resolve_request`), the fields and types of `Fov`, `RunInputs`, the resolver's records (`Request`, +`MaskRequest`, `Capabilities`, `ResolvedRequest`, `FovSelection`, `MaskSelection`, +`PlaneSelection`, `Provenance`, `Refusal`, `RepeatCoverage`) and `CleanRun`, `RESOLVER_VERSION`, +and the signatures of `make_described_run`, `make_clean_run` and `described_images`. A change to +any of them changes that test, this section and the CHANGELOG together. ### 9.2 Modes, recorded inputs, lock and report (v0.4.0, design v0.4 §1, §4) @@ -710,7 +735,7 @@ instead of keeping copies. | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. A size that cannot be converted to a positive finite length in micrometres (`1e308 m`) is unknown, never infinite, and an infinite or NaN value is never the same as any other (issue #93; before, `inf` matched every size). The folder check's `different_pixel_sizes` uses both (§11). | | Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order (a column named twice or without a name is refused naming it, as by the lenient reader, §12); one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking in v0.4.0:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | -| Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are found by `resolve_request` (§9.1) against the consumer's declared capability, and a brick raises them through `ResolvedRequest.require()` (or reports its `refusals`). A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. `RunInputs.resolve_request()` (§9.1) is the one interpreter of a described run for every route: a brick builds a `Request` and reads through the `FovSelection`s it returns, so channel-by-ID, scope, mask-representation and RGB rules have one implementation; `VolumeNotSupportedError` (`A_VOLUME_NOT_SUPPORTED`) and `DescriptionRequiredError` (`A_DESCRIPTION_REQUIRED`, raised by `RunInputs.open`) are the resolver contract's own codes (MorphoBricks-Spec ADR 0002), not experiment-v1 §8's. | +| Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are found by `resolve_request` (§9.1) against the consumer's declared capability, and a brick raises them through `ResolvedRequest.require()` (or reports its `refusals`). A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. `Fov.image_type` (§9.1) is each position's declared `rgb` or `channels`; there is no run-wide image type. `RunInputs.resolve_request()` (§9.1) is the one interpreter of a described run for every route: a brick builds a `Request` and reads through the `FovSelection`s it returns, so channel-by-ID, scope, mask-representation and RGB rules have one implementation; `VolumeNotSupportedError` (`A_VOLUME_NOT_SUPPORTED`) and `DescriptionRequiredError` (`A_DESCRIPTION_REQUIRED`, raised by `RunInputs.open`) are the resolver contract's own codes (MorphoBricks-Spec ADR 0002), not experiment-v1 §8's. | | Experiment description (Stage 1) | `read_description(path)`, `description_consume.read_published_description(path)` (the published `morphobricks.toml` only), `run.description_file.publish_description` (the only writer), `draft.draft_from_discovery`, `draft.migrate_legacy_design`, `description_reconcile.reconcile` | A brick reads the **published** `morphobricks.toml` through `RunInputs` (§9.1) and never a draft: a draft is internal to preprocessing, is never named `morphobricks.toml` and is refused with `A_DRAFT_REFUSED` (experiment-v1 §12). The command line shows the draft (`describe`, §12) and confirms nothing: the user's answers are the only way an ID, a repeat or a removal becomes confirmed. The owner's answers Q1 to Q8 of the Stage 1 design are in plan §12, question 78. | ## 17. Named-axis copies (`convert`, v0.4.0, design v0.4 §5) diff --git a/src/morphobricks_preprocessing/fov_identity.py b/src/morphobricks_preprocessing/fov_identity.py index 19228434..73302ba3 100644 --- a/src/morphobricks_preprocessing/fov_identity.py +++ b/src/morphobricks_preprocessing/fov_identity.py @@ -4,15 +4,18 @@ function of the key: the repeat ID (``repeat_group``) of the member folder, the condition ID of the condition folder, the technical repeat ID of the unit folder, and the unit's channel IDs by C index. Nothing is looked up by the label alone: a label that names no declared member, or -two, is refused. The unit's declared image type rides along (``image_type``) for the -request resolver (:mod:`.run_inputs_resolve`). +two, is refused. The unit's declared image type rides along (``image_type``: ``rgb``, or +``channels`` for a unit that declares none, experiment-v1 section 4.4b): the one source of +``Fov.image_type`` and of the request resolver's ``FovSelection.image_type`` +(:mod:`.run_inputs_resolve`). It is never read from a header. """ from __future__ import annotations from dataclasses import dataclass -from .description import Description +from .description import Description, Unit +from .run_inputs_request import CHANNELS, RGB @dataclass(frozen=True) @@ -23,8 +26,17 @@ class FovIdentity: condition_id: str technical_repeat_id: str channel_ids: tuple[str, ...] - image_type: str | None = None - """The unit's declared image type (``rgb``), or None for an ordinary channel image.""" + image_type: str + """The unit's declared image type: ``rgb``, or ``channels`` for an ordinary channel image.""" + + +def declared_image_type(unit: Unit) -> str: + """Return a unit's declared image type: ``rgb``, or ``channels`` when it declares none. + + The description's only value is ``rgb``; an absent ``image_type`` is an ordinary channel + image (experiment-v1 section 4.4b). Nothing is inferred from the images. + """ + return RGB if unit.image_type == RGB else CHANNELS def fov_identity(description: Description, labels: tuple[str, str, str]) -> FovIdentity: @@ -44,7 +56,7 @@ def fov_identity(description: Description, labels: tuple[str, str, str]) -> FovI """ condition, repeat, technical = labels found = [ - FovIdentity(repeat_id, condition_id, unit_id, unit.channels, unit.image_type) + FovIdentity(repeat_id, condition_id, unit_id, unit.channels, declared_image_type(unit)) for repeat_id, entry in description.repeats.items() for condition_id, member in entry.members.items() if description.conditions[condition_id].source_folder == condition diff --git a/src/morphobricks_preprocessing/mask_representation.py b/src/morphobricks_preprocessing/mask_representation.py index 2520ba6e..cb28f8ab 100644 --- a/src/morphobricks_preprocessing/mask_representation.py +++ b/src/morphobricks_preprocessing/mask_representation.py @@ -23,6 +23,7 @@ import os from collections.abc import Mapping from dataclasses import dataclass +from functools import lru_cache from pathlib import Path import numpy as np @@ -133,17 +134,32 @@ def check_representation( def read_checked(output_folder: str | os.PathLike) -> Mapping[str, Mapping]: - """Return the status file's ``masks`` table, or empty when there is none to trust.""" + """Return the status file's ``masks`` table, or empty when there is none to trust. + + The file is read on every call; its parse is reused for the same bytes only + (:func:`_parsed_checked`, PP-1), so a status file the mask step rewrote is parsed again. + The table is shared between calls on the same bytes: read it, never change it. + """ path = Path(output_folder) / REPRESENTATION_FILENAME try: if path.is_symlink(): return {} - document = json.loads(path.read_text(encoding="utf-8")) + data = path.read_bytes() + except OSError: + return {} + return _parsed_checked(data) + + +@lru_cache(maxsize=8) +def _parsed_checked(data: bytes) -> Mapping[str, Mapping]: + """Return the ``masks`` table of a status file's bytes (cached by content), else empty.""" + try: + document = json.loads(data.decode("utf-8")) masks = document["masks"] if document["version"] != REPRESENTATION_VERSION or not isinstance(masks, dict): return {} return masks - except (OSError, ValueError, KeyError, TypeError): + except (ValueError, KeyError, TypeError): return {} diff --git a/src/morphobricks_preprocessing/run_inputs.py b/src/morphobricks_preprocessing/run_inputs.py index c85c065e..a290a62f 100644 --- a/src/morphobricks_preprocessing/run_inputs.py +++ b/src/morphobricks_preprocessing/run_inputs.py @@ -1,8 +1,9 @@ """What a clean preprocessing run gives the other bricks: :class:`RunInputs` (design v0.4 §6). A brick that works on files reads them only through a clean run: the images, their axis -roles, pixel sizes, masks, histograms, design and image type, all as the run recorded them. -Nothing is parsed from names, guessed or transposed, and no file is read once it changed: +roles, pixel sizes, masks, histograms, design and each unit's declared image type +(``Fov.image_type``), all as the run recorded them. Nothing is parsed from names, guessed or +transposed, and no file is read once it changed: - :meth:`RunInputs.open` refuses an output folder without a published description (``A_DESCRIPTION_REQUIRED``), a manifest of an older version (run preprocessing again), a table, @@ -133,6 +134,11 @@ class Fov: channel_ids The canonical channel ID of each C index, in order (None only in a ``Fov`` built by hand). + image_type + The unit's declared image type, ``"rgb"`` or ``"channels"`` (an ordinary channel + image), from the published description only (experiment-v1 §4.4b): never read from a + header. Each position carries its own unit's type; there is no run-wide value. None + only in a ``Fov`` built by hand. """ key: tuple[str, str, str, str] @@ -151,6 +157,7 @@ class Fov: condition_id: str | None = None technical_repeat_id: str | None = None channel_ids: tuple[str, ...] | None = None + image_type: str | None = None @dataclass(frozen=True) @@ -399,10 +406,6 @@ def design(self) -> ExperimentDesign | None: "Run preprocessing again." ) from exc - def image_type(self) -> str | None: - """Return the image type the user confirmed (``"fluorescence"`` or ``"rgb"``), or None.""" - return self.manifest.image_type - def _check_files(self) -> None: """Refuse changed or gone files, masks without axes or shared, unrecorded files.""" changed = [e.relative_path for e in self.manifest.files if not self._unchanged(e)] diff --git a/src/morphobricks_preprocessing/run_inputs_request.py b/src/morphobricks_preprocessing/run_inputs_request.py index c70e228b..16d5e7a4 100644 --- a/src/morphobricks_preprocessing/run_inputs_request.py +++ b/src/morphobricks_preprocessing/run_inputs_request.py @@ -65,6 +65,10 @@ BINARY = "binary" INSTANCES = "instances" RGB = "rgb" +CHANNELS = "channels" +IMAGE_TYPES = (CHANNELS, RGB) +"""A unit's declared image type: ``rgb``, or ``channels`` for an ordinary channel image (the +description's absent ``image_type``, experiment-v1 section 4.4b). Never read from a header.""" CHECKED = "checked" @@ -195,7 +199,8 @@ class FovSelection: """What to read of one position, with its canonical identity. ``channel_indices`` holds the C index of each ``channel_ids`` entry **in this image**; - :meth:`index_of` looks one up. ``image_type`` is the unit's declared type (``rgb`` or None). + :meth:`index_of` looks one up. ``image_type`` is the unit's declared type, ``rgb`` or + ``channels`` (:data:`IMAGE_TYPES`), the same as the position's ``Fov.image_type``. """ key: tuple[str, str, str, str] @@ -203,7 +208,7 @@ class FovSelection: repeat_group: str condition_id: str technical_repeat_id: str - image_type: str | None + image_type: str channel_ids: tuple[str, ...] channel_indices: tuple[int, ...] masks: tuple[MaskSelection, ...] diff --git a/src/morphobricks_preprocessing/run_inputs_resolve.py b/src/morphobricks_preprocessing/run_inputs_resolve.py index 8d93fc92..d8e88368 100644 --- a/src/morphobricks_preprocessing/run_inputs_resolve.py +++ b/src/morphobricks_preprocessing/run_inputs_resolve.py @@ -31,7 +31,6 @@ from typing import TYPE_CHECKING from .description import Description -from .fov_identity import fov_identity from .manifest import RunManifest from .mask_representation import CHECKED, file_status, read_checked from .run_inputs_request import ( @@ -111,11 +110,10 @@ def resolve_request(self, request: Request) -> ResolvedRequest: found, more = self._mask(mask, description, fovs, repeats, masks_table) refusals.extend(found) covered.extend(more) - types = {f.relative_path: _image_type(description, f) for f in fovs} - rgb = [path for path, kind in types.items() if kind == RGB] + rgb = [f.relative_path for f in fovs if f.image_type == RGB] refusals.extend(_some(rgb_refusal(request.capabilities, rgb))) refusals.extend(_some(volume_refusal(request))) - selections = () if refusals else self._selections(request, description, fovs, types) + selections = () if refusals else self._selections(request, description, fovs) return ResolvedRequest( selections, tuple(refusals), tuple(covered), self._provenance(request) ) @@ -155,7 +153,6 @@ def _selections( request: Request, description: Description, fovs: tuple[Fov, ...], - types: Mapping[str, str | None], ) -> tuple[FovSelection, ...]: """Build one selection per position; nothing here can refuse any more.""" masks = tuple( @@ -176,7 +173,7 @@ def _selections( repeat_group=fov.repeat_group, condition_id=fov.condition_id, technical_repeat_id=fov.technical_repeat_id, - image_type=types[fov.relative_path], + image_type=fov.image_type, channel_ids=request.channel_ids, channel_indices=tuple(fov.channel_ids.index(c) for c in request.channel_ids), masks=masks, @@ -209,8 +206,3 @@ def _mask_entries(manifest: RunManifest, source_type: str) -> dict[str, Mapping] """Return the recorded mask entries of one source type, by image path.""" files = (manifest.masks or {}).get("files", ()) return {entry["image"]: entry for entry in files if entry["type"] == source_type} - - -def _image_type(description: Description, fov: Fov) -> str | None: - """Return the declared image type of a position's unit (``rgb`` or None).""" - return fov_identity(description, fov.key[:3]).image_type diff --git a/src/morphobricks_preprocessing/run_inputs_semantic.py b/src/morphobricks_preprocessing/run_inputs_semantic.py index d29c55f7..8dc7be37 100644 --- a/src/morphobricks_preprocessing/run_inputs_semantic.py +++ b/src/morphobricks_preprocessing/run_inputs_semantic.py @@ -8,7 +8,8 @@ - canonical channel IDs per position and ``channel_index(fov, id)``: by ID, never by C index alone, and an absent channel is an error, never a default; - ``repeat_group``, ``condition_id`` and ``technical_repeat_id`` beside the unchanged FOV key - (:mod:`.fov_identity`), the condition reference graph and the masks' semantics; + and the unit's declared ``image_type`` (:mod:`.fov_identity`), the condition reference + graph and the masks' semantics; - refusals: ``A_DRAFT_REFUSED`` for a draft or an unconfirmed file, ``A_MASK_UNCHECKED`` for a mask file whose representation the mask step has not checked. @@ -99,6 +100,7 @@ def with_identity(fov: Fov, entry: FileEntry, description: Description) -> Fov: condition_id=identity.condition_id, technical_repeat_id=identity.technical_repeat_id, channel_ids=identity.channel_ids, + image_type=identity.image_type, ) diff --git a/src/morphobricks_preprocessing/testing.py b/src/morphobricks_preprocessing/testing.py index fbc55cb4..2f418e39 100644 --- a/src/morphobricks_preprocessing/testing.py +++ b/src/morphobricks_preprocessing/testing.py @@ -18,6 +18,7 @@ from .axes import prefill_mapping from .description import Channel, Condition, Description, Mask, Member, Repeat, Unit from .testing_described import CleanRun as CleanRun +from .testing_described import described_images as described_images from .testing_described import make_described_run as make_described_run from .testing_described import publish_and_run diff --git a/src/morphobricks_preprocessing/testing_described.py b/src/morphobricks_preprocessing/testing_described.py index 37565781..ddfc543b 100644 --- a/src/morphobricks_preprocessing/testing_described.py +++ b/src/morphobricks_preprocessing/testing_described.py @@ -4,10 +4,10 @@ (``morphobricks.toml``, experiment-v1) declares: one folder per condition, repeat and technical repeat, ``image_count`` tiny images per member (its total, spread over its technical repeats; each image has as many channels as the unit declares, or is RGB colour -for an ``rgb`` unit), and, when asked, one mask per image for each declared mask. It -publishes the description beside the output, runs preprocessing on it and, -unless told not to, runs the mask step, so a brick's tests read their inputs through -:class:`.run_inputs.RunInputs` exactly as a user's described run gives them. +for an ``rgb`` unit), and, when asked, one mask per image for each declared mask (or only for +the images ``mask_images`` names). It publishes the description beside the output, runs +preprocessing on it and, unless told not to, runs the mask step, so a brick's tests read their +inputs through :class:`.run_inputs.RunInputs` exactly as a user's described run gives them. :func:`.testing.make_clean_run` calls it for ``make_clean_run(description=...)``, and :func:`publish_and_run` for its own tree. @@ -16,7 +16,7 @@ from __future__ import annotations import os -from collections.abc import Iterator +from collections.abc import Iterable, Iterator from dataclasses import dataclass from pathlib import Path @@ -154,28 +154,102 @@ def _write_mask(path: Path, representation: str) -> None: tifffile.imwrite(path, labels, photometric="minisblack", metadata={"axes": "YX"}) -def write_described_tree( - root: Path, masks_root: Path | None, description: Description, mask_ids: tuple[str, ...] = () -) -> list[str]: - """Write the images (and the masks of ``mask_ids``) of a description; return image paths. +def described_images(description: Description) -> tuple[str, ...]: + """Return the relative path of every image the description's tree holds, sorted. + + These are the images :func:`make_described_run` writes (``img0.tif``, ``img1.tif``, ... + in each unit's folder), so a test can name the subset ``mask_images`` gives masks to + before anything is written. Raises ------ ValueError - If a mask ID is not declared by the description, or masks are asked without a folder. + For a member with no technical repeat or fewer images than technical repeats. + """ + return tuple( + sorted( + _image_name(folder, index) + for folder, _, count in _units(description) + for index in range(count) + ) + ) + + +def _image_name(folder: Path, index: int) -> str: + """Return the relative path of a unit folder's ``index``-th image.""" + return (folder / f"img{index}.tif").as_posix() + + +def _masked_images( + description: Description, mask_ids: tuple[str, ...], mask_images: Iterable[str] | None +) -> frozenset[str]: + """Return the images that get masks, refusing an undeclared ID or an unknown image. + + Raises + ------ + ValueError + If a mask ID is not declared, ``mask_images`` is given without masks or as one text, + names no image, or names an image the tree does not hold. """ unknown = [m for m in mask_ids if m not in description.masks] - if unknown or (mask_ids and masks_root is None): + if unknown: raise ValueError( f"mask IDs {unknown} are not declared by the description " f"(declared: {sorted(description.masks)})" ) + every = described_images(description) + if mask_images is None: + return frozenset(every) if mask_ids else frozenset() + if isinstance(mask_images, str) or not mask_ids: + raise ValueError( + "mask_images names the images (a collection of relative paths) that get the masks " + "of mask_ids; it needs at least one mask ID" + ) + chosen = frozenset(mask_images) + if not chosen: + raise ValueError( + "mask_images names no image: a declared mask with files for none of them is the " + "run without masks (leave masks and mask_images out)" + ) + outside = sorted(chosen - set(every)) + if outside: + raise ValueError( + f"mask_images {outside} are not images of the described tree " + "(testing_described.described_images gives them)" + ) + return chosen + + +def write_described_tree( + root: Path, + masks_root: Path | None, + description: Description, + mask_ids: tuple[str, ...] = (), + mask_images: Iterable[str] | None = None, +) -> list[str]: + """Write the images (and the masks of ``mask_ids``) of a description; return image paths. + + Each mask ID gets one file per image of ``mask_images`` (relative paths, as + :func:`described_images` gives them), or per image when it is None. Everything is checked + before the first file is written. + + Raises + ------ + ValueError + If a mask ID is not declared by the description, masks are asked without a folder, or + ``mask_images`` is not a subset of the tree's images (:func:`_masked_images`). + """ + masked = _masked_images(description, mask_ids, mask_images) + if mask_ids and masks_root is None: + raise ValueError("masks are asked without a masks folder") + if mask_ids: + masks_root.mkdir(parents=True, exist_ok=True) found: list[str] = [] for folder, unit, count in _units(description): for index in range(count): - rel = (folder / f"img{index}.tif").as_posix() + rel = _image_name(folder, index) _write_image(root / rel, unit, seed=len(found) + 1) - for mask_id in mask_ids: + for mask_id in mask_ids if rel in masked else (): mask = description.masks[mask_id] _write_mask(masks_root / mask.source_type / rel, mask.representation) found.append(rel) @@ -187,6 +261,7 @@ def make_described_run( description: Description | bytes | str, *, masks: tuple[str, ...] = (), + mask_images: Iterable[str] | None = None, mask_step: bool = True, ) -> CleanRun: """Write the experiment a description declares, publish it and run preprocessing. @@ -203,6 +278,12 @@ def make_described_run( masks IDs of the description's masks to write, one file per image and ID (instances: two labelled squares; binary: one region of 1s). None by default. + mask_images + The images that get the masks, as relative paths (:func:`described_images`), for a + run whose masks cover a declared subset (experiment-v1 §4.5: progressive + segmentation, Spec ``v09``); None (the default) writes them for every image. Only the + images named get masks: nothing is chosen at random. At least one image: a mask on + none of them is the run without ``masks``. mask_step Run the mask step (:func:`.run.mask_step.describe_masks`) after the run, so the masks are ``checked``; False leaves them ``not yet checked``. Ignored without ``masks``. @@ -215,7 +296,8 @@ def make_described_run( Raises ------ ValueError - For an unknown mask ID, a description that mixes rgb and fluorescence units, or a + For an unknown mask ID, ``mask_images`` naming an image the tree does not hold (or + given without ``masks``), a description that mixes rgb and fluorescence units, or a member with no technical repeat or whose ``image_count`` is below its number of technical repeats (nothing is written then). RuntimeError @@ -224,7 +306,7 @@ def make_described_run( root = Path(folder) parsed, data = description_of(description) masks_root = root / "masks" if masks else None - write_described_tree(root / "experiment", masks_root, parsed, tuple(masks)) + write_described_tree(root / "experiment", masks_root, parsed, tuple(masks), mask_images) return publish_and_run(root, root / "experiment", masks_root, _mapping(parsed), data, mask_step) diff --git a/tests/spec_gate.py b/tests/spec_gate.py index 6fcb1eeb..ee04277d 100644 --- a/tests/spec_gate.py +++ b/tests/spec_gate.py @@ -31,6 +31,7 @@ "test_spec_derived_design.py", "test_spec_discovery.py", "test_spec_encoding.py", + "test_spec_fov_image_type.py", "test_spec_gate.py", "test_spec_no_fallback.py", "test_spec_pin.py", diff --git a/tests/test_brick_api_surface.py b/tests/test_brick_api_surface.py new file mode 100644 index 00000000..a50a282c --- /dev/null +++ b/tests/test_brick_api_surface.py @@ -0,0 +1,224 @@ +"""The API surface the other bricks use (PP-1, contract §9.1 and §16): names and types pinned. + +The consuming bricks read a run only through ``RunInputs``, its ``Fov``s and the request +resolver, and build their test runs with ``make_described_run`` and ``make_clean_run``. An +accidental rename, a removed method, a changed field type or a changed signature fails here; a +deliberate change updates this test, the contract and the CHANGELOG together. +""" + +from __future__ import annotations + +import dataclasses +import inspect + +import pytest + +import morphobricks_preprocessing as package +from morphobricks_preprocessing import ( + Capabilities, + Fov, + FovSelection, + MaskRequest, + MaskSelection, + PlaneSelection, + Provenance, + Refusal, + RepeatCoverage, + Request, + ResolvedRequest, + RunInputs, +) +from morphobricks_preprocessing.run_inputs_request import COVERAGE, IMAGE_TYPES +from morphobricks_preprocessing.testing import ( + CleanRun, + described_images, + make_clean_run, + make_described_run, +) + +PATH = "str | os.PathLike" +RUN_INPUTS_METHODS = { + "open": ( + f"(output_folder: '{PATH}', experiment_root: '{PATH} | None' = None, " + f"masks_root: '{PATH} | None' = None) -> 'RunInputs'" + ), + "fovs": "(self) -> 'tuple[Fov, ...]'", + "pixel_sizes": "(self, fov: 'Fov | str') -> 'dict[str, RecordedSize]'", + "read_plane": ( + "(self, fov: 'Fov | str', z: 'int' = 0, t: 'int' = 0, *, " + "memory_limit: 'int' = 1073741824) -> 'np.ndarray'" + ), + "read_image": "(self, fov: 'Fov | str') -> 'LazyImage'", + "mask_types": "(self) -> 'tuple[str, ...]'", + "read_mask": ( + "(self, fov: 'Fov | str', mask_type: 'str', z: 'int | None' = None, " + "t: 'int | None' = None) -> 'np.ndarray'" + ), + "histograms": "(self) -> 'tuple[PlaneRecord, ...]'", + "design": "(self) -> 'ExperimentDesign | None'", + "description": "(self) -> 'Description'", + "section_digests": "(self) -> 'dict[str, str]'", + "images_without_header_channel_names": "(self) -> 'dict[str, tuple[int, ...]]'", + "channel_index": "(self, fov: 'object', channel_id: 'str') -> 'int'", + "references": "(self) -> 'dict[str, str | None]'", + "mask_semantics": "(self) -> 'dict[str, MaskSemantics]'", + "mask_check_status": "(self, mask_id: 'str', fov: 'object') -> 'str'", + "require_checked": "(self, recorded: 'Mapping', mask_type: 'str') -> 'None'", + "resolve_request": "(self, request: 'Request') -> 'ResolvedRequest'", +} +RUN_INPUTS_PROPERTIES = {"files", "projection"} + +FIELDS = { + RunInputs: [ + ("output_folder", "Path"), + ("experiment_root", "Path"), + ("masks_root", "Path | None"), + ("manifest", "RunManifest"), + ("records", "tuple[PlaneRecord, ...]"), + ], + Fov: [ + ("key", "tuple[str, str, str, str]"), + ("relative_path", "str"), + ("plane_ids", "tuple[str, ...]"), + ("axes", "str"), + ("roles", "tuple[str | None, ...]"), + ("axes_from", "str"), + ("channel_names", "tuple[str | None, ...]"), + ("dtype", "str"), + ("significant_bits", "int | None"), + ("z_count", "int"), + ("t_count", "int"), + ("c_count", "int"), + ("repeat_group", "str | None"), + ("condition_id", "str | None"), + ("technical_repeat_id", "str | None"), + ("channel_ids", "tuple[str, ...] | None"), + ("image_type", "str | None"), + ], + Request: [ + ("channel_ids", "tuple[str, ...]"), + ("masks", "tuple[MaskRequest, ...]"), + ("planes", "str"), + ("capabilities", "Capabilities"), + ("device", "str"), + ], + MaskRequest: [("mask_id", "str"), ("role", "str"), ("binary_mask_policy", "str | None")], + Capabilities: [("rgb_support", "bool"), ("volume_support", "bool")], + ResolvedRequest: [ + ("selections", "tuple[FovSelection, ...]"), + ("refusals", "tuple[Refusal, ...]"), + ("coverage", "tuple[RepeatCoverage, ...]"), + ("provenance", "Provenance"), + ], + FovSelection: [ + ("key", "tuple[str, str, str, str]"), + ("relative_path", "str"), + ("repeat_group", "str"), + ("condition_id", "str"), + ("technical_repeat_id", "str"), + ("image_type", "str"), + ("channel_ids", "tuple[str, ...]"), + ("channel_indices", "tuple[int, ...]"), + ("masks", "tuple[MaskSelection, ...]"), + ("planes", "tuple[PlaneSelection, ...]"), + ], + MaskSelection: [ + ("mask_id", "str"), + ("source_type", "str"), + ("representation", "str"), + ("role", "str"), + ("check_status", "str"), + ("conversion", "str | None"), + ], + PlaneSelection: [("t", "int"), ("z", "tuple[int, ...]")], + Provenance: [ + ("resolver_version", "int"), + ("description_sha256", "str"), + ("section_digests", "tuple[tuple[str, str], ...]"), + ("request", "Request"), + ], + Refusal: [ + ("code", "str"), + ("subject", "str"), + ("message", "str"), + ("files", "tuple[str, ...]"), + ], + RepeatCoverage: [ + ("repeat_group", "str"), + ("subject", "str"), + ("status", "str"), + ("present", "int"), + ("total", "int"), + ], + CleanRun: [ + ("experiment", "Path"), + ("output", "Path"), + ("masks", "Path | None"), + ("result", "RunResult"), + ("description", "Description"), + ], +} + +HELPERS = { + make_described_run: ( + f"(folder: '{PATH}', description: 'Description | bytes | str', *, " + "masks: 'tuple[str, ...]' = (), mask_images: 'Iterable[str] | None' = None, " + "mask_step: 'bool' = True) -> 'CleanRun'" + ), + make_clean_run: ( + f"(folder: '{PATH}', *, axes: 'str' = 'CYX', shape: 'tuple[int, ...]' = (2, 16, 20), " + "mask_types: 'tuple[str, ...]' = (), pixel_size: 'float | None' = 0.5, " + "design: 'bool' = False, seed: 'int' = 0, " + "description: 'Description | bytes | str | None' = None) -> 'CleanRun'" + ), + described_images: "(description: 'Description') -> 'tuple[str, ...]'", +} + + +def test_run_inputs_has_exactly_the_public_methods_bricks_use(): + public = {name for name in dir(RunInputs) if not name.startswith("_")} + fields = {f.name for f in dataclasses.fields(RunInputs)} + assert public - fields == set(RUN_INPUTS_METHODS) | RUN_INPUTS_PROPERTIES + for name in RUN_INPUTS_PROPERTIES: + assert isinstance(inspect.getattr_static(RunInputs, name), property), name + assert isinstance(inspect.getattr_static(RunInputs, "open"), classmethod) + + +@pytest.mark.parametrize("name", sorted(RUN_INPUTS_METHODS)) +def test_run_inputs_method_signatures_are_pinned(name): + assert str(inspect.signature(getattr(RunInputs, name))) == RUN_INPUTS_METHODS[name] + + +def test_the_run_wide_image_type_is_gone(): + # PP-1: the image type is each unit's declaration, on Fov and FovSelection only. + assert not hasattr(RunInputs, "image_type") + assert IMAGE_TYPES == ("channels", "rgb") + + +@pytest.mark.parametrize("cls", list(FIELDS), ids=lambda cls: cls.__name__) +def test_record_fields_and_types_are_pinned(cls): + assert [(f.name, f.type) for f in dataclasses.fields(cls)] == FIELDS[cls] + assert cls.__dataclass_params__.frozen + + +@pytest.mark.parametrize("helper", list(HELPERS), ids=lambda helper: helper.__name__) +def test_test_run_helper_signatures_are_pinned(helper): + assert str(inspect.signature(helper)) == HELPERS[helper] + + +def test_the_resolver_names_are_exported_and_versioned(): + assert package.RESOLVER_VERSION == 1 + assert COVERAGE == ("full", "partial", "unavailable") + for cls in [*FIELDS, Fov]: + if cls is not CleanRun: + assert getattr(package, cls.__name__) is cls, cls.__name__ + for name in ( + "ChannelNotFoundError", + "DescriptionRequiredError", + "MaskNotInstancesError", + "MaskUncheckedError", + "RgbNotSupportedError", + "VolumeNotSupportedError", + "RunInputsError", + ): + assert name in package.__all__, name diff --git a/tests/test_checked_status_memo.py b/tests/test_checked_status_memo.py new file mode 100644 index 00000000..ae52c76e --- /dev/null +++ b/tests/test_checked_status_memo.py @@ -0,0 +1,63 @@ +"""PP-1: the mask step's status file is parsed once per content, never reused when it changes. + +``read_checked`` reads ``mask_representation.json`` on every call and reuses the parse of the +same bytes only, so a brick that asks ``mask_check_status`` for every position does not decode +the whole file each time, and a status file the mask step rewrote is read again (the result is +the one an uncached read gives). ``RunInputs`` holds no memo: its pickle is unchanged. +""" + +from __future__ import annotations + +import dataclasses +import json +import pickle + +from morphobricks_preprocessing import RunInputs +from morphobricks_preprocessing.mask_representation import ( + REPRESENTATION_FILENAME, + read_checked, +) +from morphobricks_preprocessing.run.mask_step import describe_masks, prepare_mask_step +from morphobricks_preprocessing.testing import make_described_run +from spec_pin import fixtures + +V08 = next(item for item in fixtures("valid") if item.name.startswith("v08")).data + + +def test_the_same_bytes_are_parsed_once_and_new_bytes_again(tmp_path, monkeypatch): + run = make_described_run(tmp_path / "run", V08, masks=("Nuclei",)) + first = read_checked(run.output) + assert read_checked(run.output) is first # the same bytes: the parse is reused + calls = [] + real = json.loads + monkeypatch.setattr(json, "loads", lambda text: calls.append(1) or real(text)) + path = run.output / REPRESENTATION_FILENAME + document = real(path.read_text(encoding="utf-8")) + document["masks"]["Nuclei"]["files"] = document["masks"]["Nuclei"]["files"][:1] + path.write_text(json.dumps(document), encoding="utf-8") + changed = read_checked(run.output) + assert calls == [1] and changed == document["masks"] and changed != first + path.write_text("{not json", encoding="utf-8") + assert read_checked(run.output) == {} + path.unlink() + assert read_checked(run.output) == {} + + +def test_a_mask_step_after_open_is_seen_by_the_same_run_inputs(tmp_path): + run = make_described_run(tmp_path / "run", V08, masks=("Nuclei",), mask_step=False) + inputs = RunInputs.open(run.output) + fov = inputs.fovs()[0] + assert inputs.mask_check_status("Nuclei", fov) == "not yet checked" + assert describe_masks(prepare_mask_step(None, run.output)).issues == () + assert inputs.mask_check_status("Nuclei", fov) == "checked" + + +def test_run_inputs_pickles_as_before(tmp_path): + run = make_described_run(tmp_path / "run", V08, masks=("Nuclei",)) + inputs = RunInputs.open(run.output) + statuses = [inputs.mask_check_status("Nuclei", f) for f in inputs.fovs()] + copy = pickle.loads(pickle.dumps(inputs)) + # Only the dataclass fields are pickled: no memo rides along. + assert set(vars(copy)) == {f.name for f in dataclasses.fields(RunInputs)} + assert copy.manifest == inputs.manifest and copy.fovs() == inputs.fovs() + assert [copy.mask_check_status("Nuclei", f) for f in copy.fovs()] == statuses diff --git a/tests/test_run_inputs.py b/tests/test_run_inputs.py index 744a6066..a1476d1d 100644 --- a/tests/test_run_inputs.py +++ b/tests/test_run_inputs.py @@ -73,7 +73,10 @@ def test_histograms_design_and_image_type(clean): f.relative_path for f in inputs.fovs() } # The design derived from the description (an output of the run, v0.5.1). - assert inputs.design().confirmed and inputs.image_type() is None + assert inputs.design().confirmed + # The image type is each unit's declaration (PP-1): no run-wide value hides it. + assert not hasattr(inputs, "image_type") + assert {f.image_type for f in inputs.fovs()} == {"channels"} def test_run_inputs_pickle(clean): diff --git a/tests/test_spec_fov_image_type.py b/tests/test_spec_fov_image_type.py new file mode 100644 index 00000000..c0b77355 --- /dev/null +++ b/tests/test_spec_fov_image_type.py @@ -0,0 +1,181 @@ +"""PP-1: each position's declared image type and the partial-mask run helper (Stage 2). + +- ``Fov.image_type`` is the unit's declaration, ``rgb`` or ``channels`` (experiment-v1 §4.4b), + from the published description only, through :class:`.fov_identity.FovIdentity` (the one + source; the resolver's ``FovSelection.image_type`` is the same value). ``v28`` is ``rgb`` + on every position, ``v13`` and ``v01`` are ``channels``. A file holding an RGB and a channels + unit (``v29``) is valid but refused by the Stage 1-2 runtime + (``R_HETEROGENEOUS_LAYOUT_NOT_ENABLED``), so the two types are shown in one description + through the identity and in one run each through ``RunInputs``. +- There is no run-wide image type on ``RunInputs`` any more. +- ``make_described_run(mask_images=...)`` writes masks for a declared subset of the images: + ``v09`` with ``Nuclei`` masks on 2 of 6 resolves to "present in 2 of 6" and ``rep1`` + partial, exactly as the tree PP-0's test built by hand. +""" + +from __future__ import annotations + +import warnings + +import pytest + +from morphobricks_preprocessing import MaskRequest, Request, RunInputs, run_experiment +from morphobricks_preprocessing.fov_identity import fov_identity +from morphobricks_preprocessing.run.mask_step import describe_masks, prepare_mask_step +from morphobricks_preprocessing.testing import described_images, make_described_run +from morphobricks_preprocessing.testing_described import ( + PUBLISHED_NAME, + _mapping, + description_of, + write_described_tree, +) +from spec_pin import fixtures + +VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} + + +def _small(name): + """Return a fixture's bytes with 3 images per member instead of 500 (``v09``).""" + return VALID[name].data.replace(b"image_count = 500", b"image_count = 3") + + +def _open(tmp_path, data, **kwargs): + run = make_described_run(tmp_path / "run", data, **kwargs) + return RunInputs.open(run.output) + + +# --- Fov.image_type ------------------------------------------------------------------------- + + +def test_v28_every_fov_is_rgb(tmp_path): + fovs = _open(tmp_path, VALID["v28"].data).fovs() + assert len(fovs) == 32 + assert {f.image_type for f in fovs} == {"rgb"} + assert {f.channel_ids for f in fovs} == {("R", "G", "B")} + + +@pytest.mark.parametrize("name", ["v13", "v01"]) +def test_a_unit_without_a_declared_type_is_channels_on_every_fov(tmp_path, name): + fovs = _open(tmp_path, VALID[name].data).fovs() + assert fovs and {f.image_type for f in fovs} == {"channels"} + + +def test_an_rgb_and_a_channels_unit_keep_their_own_types(tmp_path): + # v29 (one repeat, an RGB unit beside a two-channel unit) is valid; its identities give + # each unit its own declaration. + description, _ = description_of(VALID["v29"].data) + types = { + unit_id: fov_identity( + description, + ( + description.conditions[condition].source_folder, + member.source_repeat, + unit.source_folder, + ), + ).image_type + for entry in description.repeats.values() + for condition, member in entry.members.items() + for unit_id, unit in member.technical_repeats.items() + } + assert sorted(types.values()) == ["channels", "rgb"] + # As runs: the runtime refuses the mixed layout, so one run of each type. + rgb = _open(tmp_path / "rgb", VALID["v28"].data).fovs() + channels = _open(tmp_path / "channels", VALID["v01"].data).fovs() + assert {f.image_type for f in rgb} == {"rgb"} + assert {f.image_type for f in channels} == {"channels"} + + +def test_run_inputs_has_no_run_wide_image_type(tmp_path): + inputs = _open(tmp_path, VALID["v28"].data) + assert not hasattr(inputs, "image_type") + # The run-level record of the command line's review stays a manifest field only. + assert inputs.manifest.image_type is None + + +# --- make_described_run(mask_images=...) ---------------------------------------------------- + + +def test_described_images_are_the_images_the_helper_writes(tmp_path): + data = _small("v09") + images = described_images(description_of(data)[0]) + run = make_described_run(tmp_path / "run", data) + assert images == tuple(f.relative_path for f in RunInputs.open(run.output).fovs()) + assert len(images) == 6 + + +def test_v09_masks_on_2_of_6_resolve_to_partial_coverage(tmp_path): + data = _small("v09") + images = described_images(description_of(data)[0]) + run = make_described_run(tmp_path / "run", data, masks=("Nuclei",), mask_images=images[:2]) + assert sorted( + p.relative_to(run.masks / "nuclei").as_posix() for p in run.masks.rglob("*.tif") + ) == list(images[:2]) + resolved = RunInputs.open(run.output).resolve_request( + Request(("DNA",), (MaskRequest("Nuclei"),)) + ) + assert [(r.code, r.subject) for r in resolved.refusals] == [("A_SCOPE_INCOMPLETE", "Nuclei")] + assert "present in 2 of 6 acquisitions" in resolved.refusals[0].message + assert resolved.refusals[0].files == images[2:] + found = [(c.subject, c.repeat_group, c.status, c.present, c.total) for c in resolved.coverage] + assert found == [("DNA", "rep1", "full", 6, 6), ("Nuclei", "rep1", "partial", 2, 6)] + + +def _hand_built(tmp_path, data, keep): + """PP-0's tree: every mask written, then all but the first ``keep`` deleted.""" + description, raw = description_of(data) + root = tmp_path / "hand" + experiment, output, masks = root / "experiment", root / "output", root / "masks" + images = write_described_tree(experiment, masks, description, ("Nuclei",)) + for rel in images[keep:]: + (masks / "nuclei" / rel).unlink() + output.mkdir(parents=True) + (output / PUBLISHED_NAME).write_bytes(raw) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + result = run_experiment( + experiment, _mapping(description), output_folder=output, masks_folder=masks + ) + assert result.written + assert describe_masks(prepare_mask_step(None, output)).issues == () + return RunInputs.open(output) + + +def test_the_helpers_v09_run_equals_the_hand_built_one(tmp_path): + data = _small("v09") + images = described_images(description_of(data)[0]) + hand = _hand_built(tmp_path, data, keep=2) + helper = _open(tmp_path, data, masks=("Nuclei",), mask_images=images[:2]) + request = Request(("DNA",), (MaskRequest("Nuclei"),)) + assert helper.resolve_request(request) == hand.resolve_request(request) + assert helper.fovs() == hand.fovs() + pairs = [ + sorted((m["image"], m["type"], m["path"]) for m in inputs.manifest.masks["files"]) + for inputs in (helper, hand) + ] + assert pairs[0] == pairs[1] and len(pairs[0]) == 2 + + +def test_a_declared_mask_without_files_is_present_in_0_of_6(tmp_path): + # Masks on none of the images: the run without masks (mask_images=() is refused below). + inputs = _open(tmp_path, _small("v09")) + resolved = inputs.resolve_request(Request(("DNA",), (MaskRequest("Nuclei"),))) + assert "present in 0 of 6 acquisitions" in resolved.refusals[0].message + assert {c.status for c in resolved.coverage if c.subject == "Nuclei"} == {"unavailable"} + + +@pytest.mark.parametrize( + ("masks", "mask_images", "match"), + [ + (("Nuclei",), ("CTRL/rep1/img9.tif",), "not images of the described tree"), + (("Nuclei",), "CTRL/rep1/img0.tif", "a collection of relative paths"), + ((), ("CTRL/rep1/img0.tif",), "needs at least one mask ID"), + (("Nuclei",), (), "names no image"), + (("Cells",), None, "not declared by the description"), + ], +) +def test_a_subset_outside_the_tree_is_refused_before_anything_is_written( + tmp_path, masks, mask_images, match +): + with pytest.raises(ValueError, match=match): + make_described_run(tmp_path / "run", _small("v09"), masks=masks, mask_images=mask_images) + assert not (tmp_path / "run").exists() or not any((tmp_path / "run").rglob("*.tif")) diff --git a/tests/test_spec_resolve_request.py b/tests/test_spec_resolve_request.py index 0afa22bf..dec76010 100644 --- a/tests/test_spec_resolve_request.py +++ b/tests/test_spec_resolve_request.py @@ -28,7 +28,6 @@ import pickle import subprocess import sys -import warnings from pathlib import Path import pytest @@ -48,22 +47,15 @@ RunInputs, RunInputsError, VolumeNotSupportedError, - run_experiment, ) from morphobricks_preprocessing.manifest import read_manifest from morphobricks_preprocessing.run.mask_step import describe_masks, prepare_mask_step -from morphobricks_preprocessing.run_inputs import Fov from morphobricks_preprocessing.run_inputs_request import ( PlaneSelection, plane_selections, ) -from morphobricks_preprocessing.testing import make_described_run -from morphobricks_preprocessing.testing_described import ( - PUBLISHED_NAME, - _mapping, - description_of, - write_described_tree, -) +from morphobricks_preprocessing.testing import described_images, make_described_run +from morphobricks_preprocessing.testing_described import PUBLISHED_NAME, description_of from spec_pin import fixtures VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} @@ -88,25 +80,11 @@ def _open(tmp_path, name, masks=(), mask_step=True): def _partial_mask_run(tmp_path, name, mask_id, keep): - """Write a described run whose ``mask_id`` masks exist for the first ``keep`` images only.""" + """A described run whose ``mask_id`` masks exist for the first ``keep`` images only (PP-1).""" data = VALID[name].data.replace(b"image_count = 500", b"image_count = 3") - description, raw = description_of(data) - root = tmp_path / "run" - experiment, output, masks = root / "experiment", root / "output", root / "masks" - images = write_described_tree(experiment, masks, description, (mask_id,)) - source_type = description.masks[mask_id].source_type - for rel in images[keep:]: - (masks / source_type / rel).unlink() - output.mkdir(parents=True) - (output / PUBLISHED_NAME).write_bytes(raw) - with warnings.catch_warnings(): - warnings.simplefilter("ignore") - result = run_experiment( - experiment, _mapping(description), output_folder=output, masks_folder=masks - ) - assert result.written - assert describe_masks(prepare_mask_step(None, output)).issues == () - return RunInputs.open(output), images + images = described_images(description_of(data)[0]) + run = make_described_run(tmp_path / "run", data, masks=(mask_id,), mask_images=images[:keep]) + return RunInputs.open(run.output), list(images) def _codes(resolved): @@ -126,7 +104,7 @@ def test_v08_instances_mask_as_objects_resolves_every_position(tmp_path): assert selection.channel_indices == (0, 1) assert selection.index_of("ACTIN") == 1 assert selection.repeat_group == "rep1" and selection.condition_id == "CTRL" - assert selection.technical_repeat_id == "single" and selection.image_type is None + assert selection.technical_repeat_id == "single" and selection.image_type == "channels" (mask,) = selection.masks assert mask.mask_id == "Nuclei" and mask.source_type == "nuclei" assert mask.representation == "instances" and mask.check_status == "checked" @@ -317,10 +295,10 @@ def test_v28_rgb_units_are_refused_unless_the_consumer_declares_rgb_support(tmp_ assert accepted.selections[0].channel_indices == (0, 1, 2) -def test_v01_fluorescence_units_have_no_image_type_and_need_no_capability(tmp_path): +def test_v01_channel_units_are_channels_and_need_no_capability(tmp_path): inputs, _ = _open(tmp_path, "v01") resolved = inputs.resolve_request(Request(("DNA",))) - assert not resolved.refused and {s.image_type for s in resolved.selections} == {None} + assert not resolved.refused and {s.image_type for s in resolved.selections} == {"channels"} # --- planes and volumes ----------------------------------------------------------------------- @@ -521,6 +499,10 @@ def test_a_fov_selection_carries_the_unchanged_key_beside_the_canonical_ids(tmp_ assert all(f.key == s.key for f, s in zip(inputs.fovs(), resolved.selections, strict=True)) -def test_fov_is_unchanged_by_the_resolver(): - names = {f.name for f in dataclasses.fields(Fov)} - assert "image_type" not in names # PP-1 adds it; the resolver reads the unit's declaration +def test_a_selection_carries_its_fovs_image_type(tmp_path): + # PP-1: one source, the unit's declaration through FovIdentity, for Fov and FovSelection. + inputs, _ = _open(tmp_path, "v28") + capable = Request(("R", "G", "B"), capabilities=Capabilities(rgb_support=True)) + resolved = inputs.resolve_request(capable) + by_path = {f.relative_path: f.image_type for f in inputs.fovs()} + assert {s.relative_path: s.image_type for s in resolved.selections} == by_path