diff --git a/CHANGELOG.md b/CHANGELOG.md index ebe27636..180fea41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,44 @@ published `morphobricks.toml` first), `run_experiment(design=...)` and the `desc of `PreprocessingNotebook` are gone, and a run written by v0.5.0 or earlier is refused until it is run again. +### PP-0: `RunInputs.resolve_request`, the shared request resolver (#256) + +Resolver contract version 1 (`RESOLVER_VERSION = 1`; MorphoBricks-Spec ADR 0002 AR1-AR11, +contract §9.1 and §16). **API additions only.** + +- **One interpreter of a described run for every analysis route** (a brick's direct run and a + run through MorphoFeature alike). A `Request` (canonical channel IDs; masks by ID with a role, + `objects` or `region`, and, for a `binary` mask used as objects, the explicit + `binary_mask_policy`; `plane` or `volume`; the consumer's `Capabilities`, `rgb_support` and + `volume_support`, both False by default; the explicit `device`, `cpu` or `cuda`) resolves to + an immutable, picklable `ResolvedRequest`: one `FovSelection` per position with the C index of + every requested channel **in that image**, the masks as `MaskSelection`s, the planes to read + and the canonical IDs beside the unchanged key; every applicable refusal at once + (`A_SCOPE_INCOMPLETE` "present in k of n acquisitions", `A_MASK_UNCHECKED` "k of n files", + `A_MASK_NOT_INSTANCES`, `A_RGB_NOT_SUPPORTED`, and the contract's own + `A_VOLUME_NOT_SUPPORTED`); the coverage of every biological repeat per channel and mask + (`full`, `partial`, `unavailable`), refused or not; and the `Provenance` (the resolver + version, the description's SHA-256 and section digests as the manifest recorded them, and the + request with its device). A refused request selects nothing; `require()` raises the first + refusal's error class with every refusal in its text. +- **It only classifies.** It reads the manifest, the published description (checked against + the run's SHA-256) and `mask_representation.json`; it reads no pixel, runs no kernel, writes + nothing, converts no mask (a recorded policy comes back as `MaskSelection.conversion` for the + route's shared labeller), checks only the device's spelling (the consumer refuses `cuda` + without a GPU) and imports no brick, CellPatcher or provider module, directly or through the + package (tested in a fresh interpreter). A malformed request is a `ValueError`: an ID twice; + an unknown role, policy, plane kind or device; a policy on a mask that is not `binary` or in a + `region` role; channels or masks not a tuple (of `str` IDs, of `MaskRequest`), or a capability + that is not a `bool` (the request is recorded as given, so it must be immutable). +- **Every run it sees is described** (AR10): a run without `morphobricks.toml` is refused by + `RunInputs.open` with `A_DESCRIPTION_REQUIRED` (PP-C), so the resolver has no undescribed + branch. +- `FovIdentity` gains the unit's `image_type`; `Fov` is unchanged. New exports: + `RESOLVER_VERSION`, `Request`, `MaskRequest`, `Capabilities`, `ResolvedRequest`, + `FovSelection`, `MaskSelection`, `PlaneSelection`, `Refusal`, `RepeatCoverage`, `Provenance`, + `VolumeNotSupportedError`. Suite: `tests/test_spec_resolve_request.py`, a Spec module + (`spec_gate.EXPECTED_MODULES`). + ## 0.5.0 - 2026-11-09 Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`): diff --git a/docs/contract.md b/docs/contract.md index 944e619e..b8580929 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -355,6 +355,44 @@ It gives: of a mask file the mask step has not checked (§15; `mask_representation.json`) is refused with `MaskUncheckedError` (`A_MASK_UNCHECKED`); the error classes carry the Spec code as `code`. `ChannelNotFoundError` is `A_SCOPE_INCOMPLETE` (Stage 2, owner answer 2): one rule and one code for a channel that no image has and for one that some images lack, and its message names the image. `section_digests()` returns a copy of the manifest's `experiment_description.section_digests` (SHA-256 of the `conditions`, `channels`, `repeats` and `masks` sections; the published file is not read again). +- **The request resolver (resolver contract version 1, `RESOLVER_VERSION`; MorphoBricks + architecture decision of 10 November 2026).** `resolve_request(request)` is the one + interpreter of a described run for an analysis, on every route (a brick's direct run and a run + through MorphoFeature alike; `run_inputs_request`, `run_inputs_resolve`). A `Request` names + canonical channel IDs, masks by ID with a role (`objects` or `region`) and, for a `binary` + mask used as objects, the explicit `binary_mask_policy` (`connected_components`, + experiment-v1 §4.5c), the planes (`plane`, one selection per (t, z), or `volume`, every Z of + one t), the consumer's `Capabilities` (`rgb_support`, `volume_support`, both False by + default) and the explicit `device` (`cpu` or `cuda`; recorded, never chosen here). It returns + an immutable, picklable `ResolvedRequest`: one `FovSelection` per position (the unchanged key, + `repeat_group`, `condition_id`, `technical_repeat_id`, the unit's declared `image_type`, the C + index of every requested channel **in that image** (`channel_indices`, `index_of`), the masks + as `MaskSelection(mask_id, source_type, representation, role, check_status, conversion)` and + the `PlaneSelection`s to read), the `refusals`, the `coverage` of every biological repeat per + channel and mask (`full`, `partial`, `unavailable`; experiment-v1 §8) and the `Provenance` + (the resolver version, the description's SHA-256 and section digests as the manifest recorded + them, and the request). A refused request has no selections: no subset is taken. Every + applicable refusal is reported, each a `Refusal(code, subject, message, files)`: + `A_SCOPE_INCOMPLETE` for a channel or mask some or all images lack ("present in k of n + acquisitions", k may be 0; an undeclared mask ID is 0 of n), `A_MASK_UNCHECKED` ("k of n + files"), `A_MASK_NOT_INSTANCES` (a non-`instances` mask as objects without the policy), + `A_RGB_NOT_SUPPORTED` (`rgb` units without `rgb_support`) and `A_VOLUME_NOT_SUPPORTED` + (`VolumeNotSupportedError`, the resolver contract's own code: a volume without + `volume_support`). `ResolvedRequest.require()` returns the selections or raises the first + refusal's error class with every refusal in its text. The resolver reads the manifest, the + published description and `mask_representation.json` only: no pixel, no kernel, no brick, + CellPatcher or provider import (a test scans its modules), and it writes nothing. It converts + nothing: a recorded policy comes back as `MaskSelection.conversion` and each route applies it + with the one shared labeller (CellPatcher's), so both routes give the same objects. The + `device` is checked for its spelling only and recorded in the provenance; the consumer refuses + `cuda` without a GPU and never switches. A malformed request (an ID twice, an unknown role, + policy, plane kind or device, a policy on a mask that is not `binary` or in a `region` role, + channels or masks not a tuple of `str` IDs or of `MaskRequest`, a capability that is not a + `bool`) is a `ValueError`, not a refusal. Every run the resolver sees is described: a run + without a published description is refused by `open` with `A_DESCRIPTION_REQUIRED` (§0; + `DescriptionRequiredError`, the resolver contract's other own code, MorphoBricks-Spec ADR 0002 + AR10, AR11 (1)), so there is no undescribed path. + A `RunInputs` is picklable, so worker processes can use it. `morphobricks_preprocessing.testing.make_clean_run(folder, ...)` writes a tiny experiment of named-axis OME-TIFFs (optionally with masks), publishes its description @@ -672,7 +710,7 @@ instead of keeping copies. | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. A size that cannot be converted to a positive finite length in micrometres (`1e308 m`) is unknown, never infinite, and an infinite or NaN value is never the same as any other (issue #93; before, `inf` matched every size). The folder check's `different_pixel_sizes` uses both (§11). | | Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order (a column named twice or without a name is refused naming it, as by the lenient reader, §12); one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking in v0.4.0:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | -| Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are for the bricks to raise, and no check for them lives here. A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. | +| Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are found by `resolve_request` (§9.1) against the consumer's declared capability, and a brick raises them through `ResolvedRequest.require()` (or reports its `refusals`). A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. `RunInputs.resolve_request()` (§9.1) is the one interpreter of a described run for every route: a brick builds a `Request` and reads through the `FovSelection`s it returns, so channel-by-ID, scope, mask-representation and RGB rules have one implementation; `VolumeNotSupportedError` (`A_VOLUME_NOT_SUPPORTED`) and `DescriptionRequiredError` (`A_DESCRIPTION_REQUIRED`, raised by `RunInputs.open`) are the resolver contract's own codes (MorphoBricks-Spec ADR 0002), not experiment-v1 §8's. | | Experiment description (Stage 1) | `read_description(path)`, `description_consume.read_published_description(path)` (the published `morphobricks.toml` only), `run.description_file.publish_description` (the only writer), `draft.draft_from_discovery`, `draft.migrate_legacy_design`, `description_reconcile.reconcile` | A brick reads the **published** `morphobricks.toml` through `RunInputs` (§9.1) and never a draft: a draft is internal to preprocessing, is never named `morphobricks.toml` and is refused with `A_DRAFT_REFUSED` (experiment-v1 §12). The command line shows the draft (`describe`, §12) and confirms nothing: the user's answers are the only way an ID, a repeat or a removal becomes confirmed. The owner's answers Q1 to Q8 of the Stage 1 design are in plan §12, question 78. | ## 17. Named-axis copies (`convert`, v0.4.0, design v0.4 §5) diff --git a/pyproject.toml b/pyproject.toml index a619fa83..1ab9077d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -236,6 +236,8 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.run_inputs_error" = "pure" "morphobricks_preprocessing.required_description" = "io" "morphobricks_preprocessing.run_inputs_semantic" = "io" +"morphobricks_preprocessing.run_inputs_request" = "pure" +"morphobricks_preprocessing.run_inputs_resolve" = "io" "morphobricks_preprocessing.run_inputs_image" = "pure" "morphobricks_preprocessing.description_layouts" = "pure" "morphobricks_preprocessing.description_record" = "pure" diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 7de33fc3..b0302294 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -216,6 +216,20 @@ MaskUncheckedError, RgbNotSupportedError, ) +from .run_inputs_request import ( + RESOLVER_VERSION, + Capabilities, + FovSelection, + MaskRequest, + MaskSelection, + PlaneSelection, + Provenance, + Refusal, + RepeatCoverage, + Request, + ResolvedRequest, + VolumeNotSupportedError, +) from .sources import check_sources from .unrecorded import UnrecordedFiles, unrecorded_files from .validation import Severity, ValidationIssue, has_errors @@ -260,6 +274,7 @@ "OPEN_AXIS", "PIXEL_SIZE_TOLERANCE", "PLACEHOLDER_AXES", + "RESOLVER_VERSION", "RGB", "SPACE_UNITS", "AxisMapping", @@ -324,6 +339,17 @@ "MaskNotInstancesError", "MaskUncheckedError", "RgbNotSupportedError", + "VolumeNotSupportedError", + "Capabilities", + "FovSelection", + "MaskRequest", + "MaskSelection", + "PlaneSelection", + "Provenance", + "Refusal", + "RepeatCoverage", + "Request", + "ResolvedRequest", "UnrecordedFiles", "UnrecordedFilesError", "unrecorded_files", diff --git a/src/morphobricks_preprocessing/fov_identity.py b/src/morphobricks_preprocessing/fov_identity.py index e1ba4f74..19228434 100644 --- a/src/morphobricks_preprocessing/fov_identity.py +++ b/src/morphobricks_preprocessing/fov_identity.py @@ -4,7 +4,8 @@ function of the key: the repeat ID (``repeat_group``) of the member folder, the condition ID of the condition folder, the technical repeat ID of the unit folder, and the unit's channel IDs by C index. Nothing is looked up by the label alone: a label that names no declared member, or -two, is refused. +two, is refused. The unit's declared image type rides along (``image_type``) for the +request resolver (:mod:`.run_inputs_resolve`). """ from __future__ import annotations @@ -22,6 +23,8 @@ class FovIdentity: condition_id: str technical_repeat_id: str channel_ids: tuple[str, ...] + image_type: str | None = None + """The unit's declared image type (``rgb``), or None for an ordinary channel image.""" def fov_identity(description: Description, labels: tuple[str, str, str]) -> FovIdentity: @@ -41,7 +44,7 @@ def fov_identity(description: Description, labels: tuple[str, str, str]) -> FovI """ condition, repeat, technical = labels found = [ - FovIdentity(repeat_id, condition_id, unit_id, unit.channels) + FovIdentity(repeat_id, condition_id, unit_id, unit.channels, unit.image_type) for repeat_id, entry in description.repeats.items() for condition_id, member in entry.members.items() if description.conditions[condition_id].source_folder == condition diff --git a/src/morphobricks_preprocessing/run_inputs.py b/src/morphobricks_preprocessing/run_inputs.py index 80f0f279..c85c065e 100644 --- a/src/morphobricks_preprocessing/run_inputs.py +++ b/src/morphobricks_preprocessing/run_inputs.py @@ -65,6 +65,7 @@ from .run_inputs_error import RunInputsError as RunInputsError # re-exported (moved) from .run_inputs_image import LazyImage as LazyImage # re-exported (moved) from .run_inputs_projection import ProjectionInputs +from .run_inputs_resolve import ResolvingInputs from .run_inputs_semantic import SemanticInputs, check_description, with_identity from .run_inputs_text import ( _ZARR_MISSING, @@ -153,7 +154,7 @@ class Fov: @dataclass(frozen=True) -class RunInputs(SemanticInputs, ProjectionInputs): +class RunInputs(SemanticInputs, ProjectionInputs, ResolvingInputs): """The inputs of a clean preprocessing run, read as the run recorded them. Use :meth:`open`; the attributes are the snapshot taken then. diff --git a/src/morphobricks_preprocessing/run_inputs_request.py b/src/morphobricks_preprocessing/run_inputs_request.py new file mode 100644 index 00000000..c70e228b --- /dev/null +++ b/src/morphobricks_preprocessing/run_inputs_request.py @@ -0,0 +1,465 @@ +"""The request resolver's types and pure rules (resolver contract version 1). + +``RunInputs.resolve_request`` (:mod:`.run_inputs_resolve`) is the one interpreter of an +experiment description for an analysis run, on every route (a direct brick run and a run +through MorphoFeature alike; MorphoBricks architecture decision of 10 November 2026). This +module holds what it takes and gives, and the rules that need no file: + +- a :class:`Request` names canonical channel IDs, masks with their role and, for a ``binary`` + mask used as objects, the explicit ``binary_mask_policy`` (experiment-v1 section 4.5c), the + plane selection (one plane, or a Z-volume) and the consumer's :class:`Capabilities`; +- a :class:`ResolvedRequest` is immutable and picklable: one :class:`FovSelection` per position + with the C index of every requested channel in **that** image, the masks as the description + declares them and the planes to read, plus the :class:`Refusal` list, the coverage of every + biological repeat (full, partial or unavailable) and the :class:`Provenance`; +- the resolver classifies and refuses; it never labels, converts, reads a pixel or chooses a + device. A recorded ``binary_mask_policy`` comes back as ``MaskSelection.conversion`` for the + consumer to apply with the one shared labeller (CellPatcher's), on every route. + +Codes (experiment-v1 section 8, as the classes of :mod:`.run_inputs_error`): ``A_SCOPE_INCOMPLETE`` +("present in k of n acquisitions", k may be 0, one rule for channels and masks, owner decision +SI1), ``A_MASK_NOT_INSTANCES``, ``A_MASK_UNCHECKED``, ``A_RGB_NOT_SUPPORTED`` and, from the +resolver contract, ``A_VOLUME_NOT_SUPPORTED`` (a volume asked of a consumer without +``volume_support``). A refused request selects nothing: no subset is taken for the user. The +contract's other code, ``A_DESCRIPTION_REQUIRED``, is raised before a ``RunInputs`` exists +(``RunInputs.open``): every run the resolver sees is described. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field + +from .run_inputs_error import RunInputsError + +RESOLVER_VERSION = 1 +"""The resolver contract version a :class:`Provenance` records.""" + +OBJECTS = "objects" +REGION = "region" +MASK_ROLES = (OBJECTS, REGION) +"""How a request uses a mask: as an object set (instances) or as a region (any pixel set).""" + +CONNECTED_COMPONENTS = "connected_components" +BINARY_MASK_POLICIES = (CONNECTED_COMPONENTS,) +"""The recorded conversions of a ``binary`` mask to objects (experiment-v1 section 4.5c).""" + +PLANE = "plane" +VOLUME = "volume" +PLANE_KINDS = (PLANE, VOLUME) +"""What one selection reads: one (t, z) plane, or every Z slice of one time point.""" + +CPU = "cpu" +CUDA = "cuda" +DEVICES = (CPU, CUDA) +"""The explicit device of a request (owner decision of 10 November 2026): chosen by the user or +the notebook default, recorded in the provenance on every route, never switched silently. The +resolver records it; the consumer refuses ``cuda`` without a GPU.""" + +FULL = "full" +PARTIAL = "partial" +UNAVAILABLE = "unavailable" +COVERAGE = (FULL, PARTIAL, UNAVAILABLE) +"""A biological repeat's coverage for one requested channel or mask (experiment-v1 section 8).""" + +BINARY = "binary" +INSTANCES = "instances" +RGB = "rgb" +CHECKED = "checked" + + +class VolumeNotSupportedError(RunInputsError): + """A Z-volume was requested from a consumer that does not declare ``volume_support``. + + The resolver contract's own code: 3-D is enabled later, by its own decision, for the + consumers that declare the capability; nothing enables it by default. + """ + + code = "A_VOLUME_NOT_SUPPORTED" + + +@dataclass(frozen=True) +class Capabilities: + """What the consumer declares it can take; the resolver refuses what it does not. + + Attributes + ---------- + rgb_support + True when the consumer declares RGB support (its provider manifest's + ``rgb_support = true``, experiment-v1 section 4.4b). False refuses ``rgb`` units. + volume_support + True when the consumer takes a Z-volume as one selection. False refuses a ``volume`` + request. No consumer declares it today. + """ + + rgb_support: bool = False + volume_support: bool = False + + +@dataclass(frozen=True) +class MaskRequest: + """One mask of the request, by its declared ID. + + Attributes + ---------- + mask_id + The mask ID of the description (``masks.``), never a source type. + role + ``objects`` (an object set: the mask must be ``instances``, or converted by the + recorded policy) or ``region``. + binary_mask_policy + The explicit per-request conversion of a ``binary`` mask used as objects + (``connected_components``), or None. Never a default. + """ + + mask_id: str + role: str = OBJECTS + binary_mask_policy: str | None = None + + +@dataclass(frozen=True) +class Request: + """What an analysis asks of a described run. + + Attributes + ---------- + channel_ids + Canonical channel IDs, each resolved to its C index per image. May be empty. + masks + The masks used, with their roles and conversions. + planes + ``plane`` (one selection per time point and Z slice, today's behaviour) or + ``volume`` (one selection per time point holding every Z slice). + capabilities + The consumer's declared capabilities. + device + ``cpu`` or ``cuda``, the explicit device the kernels run on. Recorded; the resolver + runs no kernel and does not check that a GPU exists (the consumer refuses, never + switches). + """ + + channel_ids: tuple[str, ...] = () + masks: tuple[MaskRequest, ...] = () + planes: str = PLANE + capabilities: Capabilities = field(default_factory=Capabilities) + device: str = CPU + + +@dataclass(frozen=True) +class Refusal: + """One reason the request is refused, with its Spec code.""" + + code: str + subject: str + message: str + files: tuple[str, ...] = () + + +@dataclass(frozen=True) +class RepeatCoverage: + """One biological repeat's coverage of one requested channel or mask.""" + + repeat_group: str + subject: str + status: str + present: int + total: int + + +@dataclass(frozen=True) +class PlaneSelection: + """The planes of one read: time point ``t`` and the Z slices, one for a plane.""" + + t: int + z: tuple[int, ...] + + +@dataclass(frozen=True) +class MaskSelection: + """One mask as the consumer reads it: ``read_mask(fov, source_type, ...)``. + + ``conversion`` is the recorded ``binary_mask_policy`` the consumer applies with the shared + labeller (None when the mask is used as declared). ``check_status`` is ``checked``. + """ + + mask_id: str + source_type: str + representation: str + role: str + check_status: str + conversion: str | None = None + + +@dataclass(frozen=True) +class FovSelection: + """What to read of one position, with its canonical identity. + + ``channel_indices`` holds the C index of each ``channel_ids`` entry **in this image**; + :meth:`index_of` looks one up. ``image_type`` is the unit's declared type (``rgb`` or None). + """ + + key: tuple[str, str, str, str] + relative_path: str + repeat_group: str + condition_id: str + technical_repeat_id: str + image_type: str | None + channel_ids: tuple[str, ...] + channel_indices: tuple[int, ...] + masks: tuple[MaskSelection, ...] + planes: tuple[PlaneSelection, ...] + + def index_of(self, channel_id: str) -> int: + """Return the C index of a requested channel in this image. + + Raises + ------ + KeyError + For a channel the request did not name. + """ + try: + return self.channel_indices[self.channel_ids.index(channel_id)] + except ValueError: + raise KeyError(f"{channel_id!r} is not a channel of this request") from None + + +@dataclass(frozen=True) +class Provenance: + """What the selection was resolved from, for the consumer's own records. + + ``section_digests`` are the manifest's (``(section, hex)`` pairs); ``description_sha256`` + is provenance, never a cache key (experiment-v1 section 9.3). + """ + + resolver_version: int + description_sha256: str + section_digests: tuple[tuple[str, str], ...] + request: Request + + +@dataclass(frozen=True) +class ResolvedRequest: + """The resolver's answer: selections when nothing refuses, else the refusals. + + ``selections`` is empty whenever ``refusals`` is not: a refused request takes no subset. + ``coverage`` is reported either way, so the orchestrator can record the scope. + """ + + selections: tuple[FovSelection, ...] + refusals: tuple[Refusal, ...] + coverage: tuple[RepeatCoverage, ...] + provenance: Provenance + + @property + def refused(self) -> bool: + """True when at least one refusal applies.""" + return bool(self.refusals) + + def require(self) -> tuple[FovSelection, ...]: + """Return the selections, or raise the first refusal as its error class. + + Raises + ------ + RunInputsError + The subclass of the first refusal's code (``error.code``), its message listing + every refusal. + """ + if not self.refusals: + return self.selections + first = self.refusals[0] + text = "\n".join(r.message for r in self.refusals) + raise _error_class(first.code)(text) + + +def _error_class(code: str) -> type[RunInputsError]: + """Return the ``RunInputsError`` subclass carrying ``code``.""" + pending = [RunInputsError, *RunInputsError.__subclasses__()] + while pending: + cls = pending.pop() + if cls.code == code: + return cls + pending.extend(cls.__subclasses__()) + return RunInputsError + + +def check_request(request: Request, declared_masks: Mapping[str, str]) -> None: + """Refuse a malformed request before anything is classified. + + Parameters + ---------- + request + The request. + declared_masks + Mask ID to declared representation, for the policy rules. + + Raises + ------ + ValueError + Channels or masks not given as a tuple (of ``str`` IDs, of :class:`MaskRequest`), a + capability that is not a ``bool``, a channel ID named twice, a mask ID named twice, a + role or policy outside the contract, a policy on a mask that is not ``binary`` (nothing + to convert) or on a ``region`` role, an unknown plane kind, or a device that is neither + ``cpu`` nor ``cuda``. + """ + _check_shape(request) + if len(set(request.channel_ids)) != len(request.channel_ids): + raise ValueError("a channel ID is requested twice") + ids = [m.mask_id for m in request.masks] + if len(set(ids)) != len(ids): + raise ValueError("a mask ID is requested twice") + if request.planes not in PLANE_KINDS: + raise ValueError(f"planes must be one of {PLANE_KINDS}, not {request.planes!r}") + if request.device not in DEVICES: + raise ValueError(f"device must be one of {DEVICES}, not {request.device!r}") + for mask in request.masks: + if mask.role not in MASK_ROLES: + raise ValueError(f"mask role must be one of {MASK_ROLES}, not {mask.role!r}") + if mask.binary_mask_policy is None: + continue + if mask.binary_mask_policy not in BINARY_MASK_POLICIES: + raise ValueError( + f"binary_mask_policy must be one of {BINARY_MASK_POLICIES}, " + f"not {mask.binary_mask_policy!r}" + ) + if mask.role != OBJECTS or declared_masks.get(mask.mask_id, BINARY) != BINARY: + raise ValueError( + f"binary_mask_policy applies to a binary mask used as objects; {mask.mask_id!r} " + f"is {declared_masks.get(mask.mask_id, 'undeclared')} in the role {mask.role!r}" + ) + + +def _check_shape(request: Request) -> None: + """Refuse a request whose fields are not the frozen types the result records. + + The request is recorded in the provenance as given, so a list would leave the result + mutable (and unhashable), and a string would be read as one ID per character. + """ + if not isinstance(request.channel_ids, tuple) or not all( + isinstance(c, str) for c in request.channel_ids + ): + raise ValueError(f"channel_ids must be a tuple of str, not {request.channel_ids!r}") + if not isinstance(request.masks, tuple) or not all( + isinstance(m, MaskRequest) and isinstance(m.mask_id, str) for m in request.masks + ): + raise ValueError(f"masks must be a tuple of MaskRequest (str IDs), not {request.masks!r}") + capabilities = request.capabilities + if not isinstance(capabilities, Capabilities) or not all( + isinstance(v, bool) for v in (capabilities.rgb_support, capabilities.volume_support) + ): + raise ValueError(f"capabilities must be Capabilities of bool, not {capabilities!r}") + + +def scope_refusal(subject: str, kind: str, missing: Iterable[str], total: int) -> Refusal | None: + """Return the ``A_SCOPE_INCOMPLETE`` refusal of a channel or mask, or None when complete. + + Parameters + ---------- + subject + The channel or mask ID. + kind + ``channel`` or ``mask``, for the message. + missing + The images that lack it. + total + The images of the run. + """ + lacking = tuple(sorted(missing)) + if not lacking: + return None + present = total - len(lacking) + shown = "\n".join(f" - {path}" for path in lacking[:10]) + more = f"\n ... and {len(lacking) - 10} more" if len(lacking) > 10 else "" + return Refusal( + "A_SCOPE_INCOMPLETE", + subject, + f"The {kind} {subject!r} is present in {present} of {total} acquisitions; the images " + f"without it:\n{shown}{more}\nNo subset is taken: select and record an eligible " + "subset, or complete the data (A_SCOPE_INCOMPLETE).", + lacking, + ) + + +def coverage( + subject: str, repeats: Mapping[str, Iterable[str]], present: Iterable[str] +) -> tuple[RepeatCoverage, ...]: + """Classify every biological repeat for one subject: full, partial or unavailable. + + Parameters + ---------- + subject + The channel or mask ID. + repeats + ``repeat_group`` to the images of that repeat. + present + The images that have the subject. + """ + have = set(present) + found = [] + for group, images in sorted(repeats.items()): + paths = list(images) + count = sum(1 for path in paths if path in have) + status = FULL if count == len(paths) else UNAVAILABLE if count == 0 else PARTIAL + found.append(RepeatCoverage(group, subject, status, count, len(paths))) + return tuple(found) + + +def plane_selections(t_count: int, z_count: int, kind: str) -> tuple[PlaneSelection, ...]: + """Return the planes to read of one image: per (t, z), or every Z per t for a volume.""" + if kind == VOLUME: + return tuple(PlaneSelection(t, tuple(range(z_count))) for t in range(t_count)) + return tuple(PlaneSelection(t, (z,)) for t in range(t_count) for z in range(z_count)) + + +def mask_representation_refusal(mask: MaskRequest, representation: str) -> Refusal | None: + """Return ``A_MASK_NOT_INSTANCES`` for a non-instances mask used as objects without a policy.""" + if mask.role != OBJECTS or representation == INSTANCES or mask.binary_mask_policy: + return None + return Refusal( + "A_MASK_NOT_INSTANCES", + mask.mask_id, + f"The mask {mask.mask_id!r} is declared {representation}, so it has no object identity " + "and cannot be an object set. Ask for the recorded conversion " + f"binary_mask_policy={CONNECTED_COMPONENTS!r} in this request, or declare the mask " + "instances (A_MASK_NOT_INSTANCES).", + ) + + +def unchecked_refusal(mask_id: str, unchecked: Iterable[str], total: int) -> Refusal | None: + """Return ``A_MASK_UNCHECKED`` naming the files the mask step has not read, or None.""" + files = tuple(sorted(unchecked)) + if not files: + return None + return Refusal( + "A_MASK_UNCHECKED", + mask_id, + f"The mask {mask_id!r} is not yet checked against its declared representation for " + f"{len(files)} of {total} files. Run the masks step on this output folder first " + "(A_MASK_UNCHECKED).", + files, + ) + + +def rgb_refusal(capabilities: Capabilities, rgb_images: Iterable[str]) -> Refusal | None: + """Return ``A_RGB_NOT_SUPPORTED`` when ``rgb`` units are requested without the capability.""" + images = tuple(sorted(rgb_images)) + if not images or capabilities.rgb_support: + return None + return Refusal( + "A_RGB_NOT_SUPPORTED", + RGB, + f"{len(images)} images belong to units declared rgb and this consumer does not declare " + "rgb_support; nothing is converted to grey or split (A_RGB_NOT_SUPPORTED).", + images, + ) + + +def volume_refusal(request: Request) -> Refusal | None: + """Return ``A_VOLUME_NOT_SUPPORTED`` for a volume asked without ``volume_support``.""" + if request.planes != VOLUME or request.capabilities.volume_support: + return None + return Refusal( + "A_VOLUME_NOT_SUPPORTED", + VOLUME, + "A Z-volume selection was requested and this consumer does not declare volume_support; " + "ask for planes (A_VOLUME_NOT_SUPPORTED).", + ) diff --git a/src/morphobricks_preprocessing/run_inputs_resolve.py b/src/morphobricks_preprocessing/run_inputs_resolve.py new file mode 100644 index 00000000..8d93fc92 --- /dev/null +++ b/src/morphobricks_preprocessing/run_inputs_resolve.py @@ -0,0 +1,216 @@ +"""``RunInputs.resolve_request``: the one experiment interpreter for analysis runs. + +:class:`ResolvingInputs` is mixed into :class:`.run_inputs.RunInputs`. From a published +description and the run's manifest it classifies what a :class:`.run_inputs_request.Request` +asks for and gives back an immutable :class:`.run_inputs_request.ResolvedRequest`: + +- every requested channel resolved to its C index **per image** (by canonical ID, never by a + shared index), refused as ``A_SCOPE_INCOMPLETE`` when any image lacks it (k of n, k may be 0); +- every requested mask by its declared ID, refused as ``A_SCOPE_INCOMPLETE`` when some images + have no file of it, ``A_MASK_UNCHECKED`` when the mask step has not read every file, and + ``A_MASK_NOT_INSTANCES`` for a non-instances mask used as objects without the recorded + ``binary_mask_policy`` (experiment-v1 sections 4.5a, 4.5c, 8); +- ``rgb`` units refused as ``A_RGB_NOT_SUPPORTED`` unless the consumer declares ``rgb_support`` + (section 4.4b), and a Z-volume refused as ``A_VOLUME_NOT_SUPPORTED`` unless it declares + ``volume_support``; +- the coverage of every biological repeat (``repeat_group``) per channel and mask: full, + partial or unavailable, reported whether or not the request is refused; +- the provenance: the resolver contract version, the description's SHA-256 and section + digests as the manifest recorded them, and the request itself (its device included). + +It reads only the manifest, the published description (checked against the run's SHA-256) and +the mask step's status file: no pixel, no brick, no CellPatcher module. Every run is described: +a run without a published description is refused before a ``RunInputs`` exists +(``RunInputs.open``, ``A_DESCRIPTION_REQUIRED``), so there is no undescribed path here. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import TYPE_CHECKING + +from .description import Description +from .fov_identity import fov_identity +from .manifest import RunManifest +from .mask_representation import CHECKED, file_status, read_checked +from .run_inputs_request import ( + RESOLVER_VERSION, + RGB, + FovSelection, + MaskRequest, + MaskSelection, + Provenance, + Refusal, + RepeatCoverage, + Request, + ResolvedRequest, + check_request, + coverage, + mask_representation_refusal, + plane_selections, + rgb_refusal, + scope_refusal, + unchecked_refusal, + volume_refusal, +) + +if TYPE_CHECKING: + from .run_inputs import Fov + + +class ResolvingInputs: + """The ``resolve_request`` method of ``RunInputs`` (resolver contract version 1).""" + + output_folder: Path + manifest: RunManifest + + def description(self) -> Description: # pragma: no cover - SemanticInputs provides it + """Return the published description (``SemanticInputs``).""" + raise NotImplementedError + + def fovs(self) -> tuple[Fov, ...]: # pragma: no cover - RunInputs provides it + """Return every position (``RunInputs``).""" + raise NotImplementedError + + def resolve_request(self, request: Request) -> ResolvedRequest: + """Resolve an analysis request against this described run. + + Parameters + ---------- + request + The channels, masks, planes, capabilities and device of the analysis. + + Returns + ------- + ResolvedRequest + One selection per position when nothing refuses; else the refusals and no + selection. The coverage per repeat and the provenance are given either way. + + Raises + ------ + RunInputsError + When the published description is gone or changed since the run + (:meth:`.run_inputs_semantic.SemanticInputs.description`). + ValueError + For a malformed request (:func:`.run_inputs_request.check_request`). + """ + description = self.description() + check_request(request, {k: m.representation for k, m in description.masks.items()}) + fovs = self.fovs() + repeats = _repeats(fovs) + refusals: list[Refusal] = [] + covered: list[RepeatCoverage] = [] + for channel_id in request.channel_ids: + present = [f.relative_path for f in fovs if channel_id in f.channel_ids] + missing = [f.relative_path for f in fovs if channel_id not in f.channel_ids] + refusals.extend(_some(scope_refusal(channel_id, "channel", missing, len(fovs)))) + covered.extend(coverage(channel_id, repeats, present)) + masks_table = read_checked(self.output_folder) if request.masks else None + for mask in request.masks: + found, more = self._mask(mask, description, fovs, repeats, masks_table) + refusals.extend(found) + covered.extend(more) + types = {f.relative_path: _image_type(description, f) for f in fovs} + rgb = [path for path, kind in types.items() if kind == RGB] + refusals.extend(_some(rgb_refusal(request.capabilities, rgb))) + refusals.extend(_some(volume_refusal(request))) + selections = () if refusals else self._selections(request, description, fovs, types) + return ResolvedRequest( + selections, tuple(refusals), tuple(covered), self._provenance(request) + ) + + def _mask( + self, + mask: MaskRequest, + description: Description, + fovs: tuple[Fov, ...], + repeats: Mapping[str, tuple[str, ...]], + table: object, + ) -> tuple[list[Refusal], tuple[RepeatCoverage, ...]]: + """Classify one requested mask: presence per image, check status, representation.""" + declared = description.masks.get(mask.mask_id) + if declared is None: + missing = [f.relative_path for f in fovs] + refusal = scope_refusal(mask.mask_id, "mask", missing, len(fovs)) + return _some(refusal), coverage(mask.mask_id, repeats, ()) + recorded = _mask_entries(self.manifest, declared.source_type) + present = [f.relative_path for f in fovs if f.relative_path in recorded] + missing = [f.relative_path for f in fovs if f.relative_path not in recorded] + refusals = _some(scope_refusal(mask.mask_id, "mask", missing, len(fovs))) + unchecked = [ + recorded[path]["path"] + for path in present + if file_status( + table, mask.mask_id, declared.source_type, declared.representation, recorded[path] + ) + != CHECKED + ] + refusals.extend(_some(unchecked_refusal(mask.mask_id, unchecked, len(present)))) + refusals.extend(_some(mask_representation_refusal(mask, declared.representation))) + return refusals, coverage(mask.mask_id, repeats, present) + + def _selections( + self, + request: Request, + description: Description, + fovs: tuple[Fov, ...], + types: Mapping[str, str | None], + ) -> tuple[FovSelection, ...]: + """Build one selection per position; nothing here can refuse any more.""" + masks = tuple( + MaskSelection( + m.mask_id, + description.masks[m.mask_id].source_type, + description.masks[m.mask_id].representation, + m.role, + CHECKED, + m.binary_mask_policy, + ) + for m in request.masks + ) + return tuple( + FovSelection( + key=fov.key, + relative_path=fov.relative_path, + repeat_group=fov.repeat_group, + condition_id=fov.condition_id, + technical_repeat_id=fov.technical_repeat_id, + image_type=types[fov.relative_path], + channel_ids=request.channel_ids, + channel_indices=tuple(fov.channel_ids.index(c) for c in request.channel_ids), + masks=masks, + planes=plane_selections(fov.t_count, fov.z_count, request.planes), + ) + for fov in fovs + ) + + def _provenance(self, request: Request) -> Provenance: + """Return the provenance from the manifest's description record (always present).""" + recorded = self.manifest.experiment_description + digests = tuple(sorted(recorded["section_digests"].items())) + return Provenance(RESOLVER_VERSION, recorded["sha256"], digests, request) + + +def _some(refusal: Refusal | None) -> list[Refusal]: + """Return a one-item list for a refusal, an empty one for None.""" + return [] if refusal is None else [refusal] + + +def _repeats(fovs: tuple[Fov, ...]) -> dict[str, tuple[str, ...]]: + """Return the images of every biological repeat (``repeat_group``), sorted.""" + groups: dict[str, list[str]] = {} + for fov in fovs: + groups.setdefault(fov.repeat_group, []).append(fov.relative_path) + return {group: tuple(sorted(paths)) for group, paths in groups.items()} + + +def _mask_entries(manifest: RunManifest, source_type: str) -> dict[str, Mapping]: + """Return the recorded mask entries of one source type, by image path.""" + files = (manifest.masks or {}).get("files", ()) + return {entry["image"]: entry for entry in files if entry["type"] == source_type} + + +def _image_type(description: Description, fov: Fov) -> str | None: + """Return the declared image type of a position's unit (``rgb`` or None).""" + return fov_identity(description, fov.key[:3]).image_type diff --git a/tests/spec_gate.py b/tests/spec_gate.py index 2583d8de..6fcb1eeb 100644 --- a/tests/spec_gate.py +++ b/tests/spec_gate.py @@ -42,6 +42,7 @@ "test_spec_records.py", "test_spec_records_goldens.py", "test_spec_refval_crosscheck.py", + "test_spec_resolve_request.py", "test_spec_rules_table.py", "test_spec_run_aliases.py", "test_spec_run_description.py", diff --git a/tests/test_spec_resolve_request.py b/tests/test_spec_resolve_request.py new file mode 100644 index 00000000..0afa22bf --- /dev/null +++ b/tests/test_spec_resolve_request.py @@ -0,0 +1,526 @@ +"""``RunInputs.resolve_request``: the shared request resolver on the Spec fixtures (contract v1). + +The architecture decision of 10 November 2026: one experiment interpreter in preprocessing for +every route. Shown here on the vendored fixtures through ``make_described_run``: + +- ``v08`` (P27): ``TumorRegion`` as objects is ``A_MASK_NOT_INSTANCES`` unless the request + records ``binary_mask_policy``; as a region it is accepted; the conversion is returned, never + applied here. ``A_MASK_UNCHECKED`` before the mask step (P28, "k of n files"). +- ``v09`` (P11, GS2): masks on 2 of 6 images are ``A_SCOPE_INCOMPLETE`` "present in 2 of 6" + with ``rep1`` partial; ``v01``: a channel no image has is "present in 0 of n" and every repeat + unavailable (SI1). +- ``v13``: 90 positions resolved by channel ID per image; a reordered channel list on one + position changes that position's indices only. +- ``v28`` (P24): ``rgb`` units are ``A_RGB_NOT_SUPPORTED`` unless the capability is declared. +- A volume is refused unless ``volume_support`` is declared; a one-slice volume reads what a + plane reads. +- The result is immutable and picklable; the provenance is the manifest's record; the resolver + writes nothing and imports no brick, CellPatcher or provider module, not even indirectly. +- Every run the resolver sees is described: a run without ``morphobricks.toml`` is refused + with ``A_DESCRIPTION_REQUIRED`` before a ``RunInputs`` exists (ADR 0002 AR10). +""" + +from __future__ import annotations + +import ast +import dataclasses +import json +import pickle +import subprocess +import sys +import warnings +from pathlib import Path + +import pytest + +import morphobricks_preprocessing.run_inputs_request as request_module +import morphobricks_preprocessing.run_inputs_resolve as resolve_module +from morphobricks_preprocessing import ( + RESOLVER_VERSION, + Capabilities, + ChannelNotFoundError, + DescriptionRequiredError, + MaskNotInstancesError, + MaskRequest, + MaskUncheckedError, + Request, + RgbNotSupportedError, + RunInputs, + RunInputsError, + VolumeNotSupportedError, + run_experiment, +) +from morphobricks_preprocessing.manifest import read_manifest +from morphobricks_preprocessing.run.mask_step import describe_masks, prepare_mask_step +from morphobricks_preprocessing.run_inputs import Fov +from morphobricks_preprocessing.run_inputs_request import ( + PlaneSelection, + plane_selections, +) +from morphobricks_preprocessing.testing import make_described_run +from morphobricks_preprocessing.testing_described import ( + PUBLISHED_NAME, + _mapping, + description_of, + write_described_tree, +) +from spec_pin import fixtures + +VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} +POLICY = "connected_components" +FORBIDDEN_IMPORTS = ( + "cellpatcher", + "morphofeature", + "morphoobjects", + "morphorelate", + "fiberfeatures", + "morphoembed", + "morphocoverage", + "torch", + "skimage", + "scipy", +) + + +def _open(tmp_path, name, masks=(), mask_step=True): + run = make_described_run(tmp_path / "run", VALID[name].data, masks=masks, mask_step=mask_step) + return RunInputs.open(run.output), run + + +def _partial_mask_run(tmp_path, name, mask_id, keep): + """Write a described run whose ``mask_id`` masks exist for the first ``keep`` images only.""" + data = VALID[name].data.replace(b"image_count = 500", b"image_count = 3") + description, raw = description_of(data) + root = tmp_path / "run" + experiment, output, masks = root / "experiment", root / "output", root / "masks" + images = write_described_tree(experiment, masks, description, (mask_id,)) + source_type = description.masks[mask_id].source_type + for rel in images[keep:]: + (masks / source_type / rel).unlink() + output.mkdir(parents=True) + (output / PUBLISHED_NAME).write_bytes(raw) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + result = run_experiment( + experiment, _mapping(description), output_folder=output, masks_folder=masks + ) + assert result.written + assert describe_masks(prepare_mask_step(None, output)).issues == () + return RunInputs.open(output), images + + +def _codes(resolved): + return [(r.code, r.subject) for r in resolved.refusals] + + +# --- v08: masks (P27, P28) --------------------------------------------------------------- + + +def test_v08_instances_mask_as_objects_resolves_every_position(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + resolved = inputs.resolve_request(Request(("DNA", "ACTIN"), (MaskRequest("Nuclei"),))) + assert not resolved.refused and len(resolved.selections) == 4 + assert resolved.require() == resolved.selections + for selection in resolved.selections: + assert selection.channel_ids == ("DNA", "ACTIN") + assert selection.channel_indices == (0, 1) + assert selection.index_of("ACTIN") == 1 + assert selection.repeat_group == "rep1" and selection.condition_id == "CTRL" + assert selection.technical_repeat_id == "single" and selection.image_type is None + (mask,) = selection.masks + assert mask.mask_id == "Nuclei" and mask.source_type == "nuclei" + assert mask.representation == "instances" and mask.check_status == "checked" + assert mask.conversion is None and mask.role == "objects" + assert selection.planes == (PlaneSelection(0, (0,)),) + assert [(c.subject, c.status, c.present, c.total) for c in resolved.coverage] == [ + ("DNA", "full", 4, 4), + ("ACTIN", "full", 4, 4), + ("Nuclei", "full", 4, 4), + ] + + +def test_v08_p27a_binary_mask_as_objects_without_a_policy_is_refused(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + resolved = inputs.resolve_request(Request(("DNA",), (MaskRequest("TumorRegion"),))) + assert _codes(resolved) == [("A_MASK_NOT_INSTANCES", "TumorRegion")] + assert resolved.selections == () # a refused request takes no subset + assert "binary_mask_policy" in resolved.refusals[0].message + with pytest.raises(MaskNotInstancesError, match="A_MASK_NOT_INSTANCES") as raised: + resolved.require() + assert raised.value.code == "A_MASK_NOT_INSTANCES" + + +def test_v08_p27b_the_recorded_policy_is_returned_for_the_consumer_to_apply(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + request = Request(("DNA",), (MaskRequest("TumorRegion", binary_mask_policy=POLICY),)) + resolved = inputs.resolve_request(request) + assert not resolved.refused and len(resolved.selections) == 4 + for selection in resolved.selections: + (mask,) = selection.masks + assert mask.conversion == POLICY and mask.representation == "binary" + assert mask.source_type == "tumor_region" + # The resolver converts nothing: the mask is still read as stored (0/1 region). + assert set(inputs.read_mask(inputs.fovs()[0], "tumor_region").ravel()) <= {0, 1} + # P27c: a later request without the policy is refused again. + again = inputs.resolve_request(Request(("DNA",), (MaskRequest("TumorRegion"),))) + assert _codes(again) == [("A_MASK_NOT_INSTANCES", "TumorRegion")] + + +def test_v08_binary_mask_as_a_region_is_accepted(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + resolved = inputs.resolve_request(Request((), (MaskRequest("TumorRegion", role="region"),))) + assert not resolved.refused and len(resolved.selections) == 4 + assert resolved.selections[0].masks[0].conversion is None + assert resolved.selections[0].channel_indices == () + + +def test_v08_p28_unchecked_masks_are_refused_with_k_of_n_files(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion"), mask_step=False) + request = Request(("DNA",), (MaskRequest("Nuclei"), MaskRequest("TumorRegion", role="region"))) + resolved = inputs.resolve_request(request) + assert _codes(resolved) == [("A_MASK_UNCHECKED", "Nuclei"), ("A_MASK_UNCHECKED", "TumorRegion")] + assert "4 of 4 files" in resolved.refusals[0].message + assert len(resolved.refusals[0].files) == 4 + with pytest.raises(MaskUncheckedError, match="A_MASK_UNCHECKED"): + resolved.require() + assert describe_masks(prepare_mask_step(None, inputs.output_folder)).issues == () + assert not RunInputs.open(inputs.output_folder).resolve_request(request).refused + + +def test_v08_p28_a_mask_checked_for_some_files_is_unchecked_for_k_of_n(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei",)) + status = inputs.output_folder / "mask_representation.json" + document = json.loads(status.read_text(encoding="utf-8")) + dropped = document["masks"]["Nuclei"]["files"].pop(0)["path"] + status.write_text(json.dumps(document), encoding="utf-8") + resolved = inputs.resolve_request(Request(("DNA",), (MaskRequest("Nuclei"),))) + assert _codes(resolved) == [("A_MASK_UNCHECKED", "Nuclei")] + assert "for 1 of 4 files" in resolved.refusals[0].message + assert resolved.refusals[0].files == (dropped,) + assert resolved.selections == () + assert [(c.status, c.present, c.total) for c in resolved.coverage if c.subject == "Nuclei"] == [ + ("full", 4, 4) + ] # coverage is presence; the check status is the refusal's + + +def test_v08_every_applicable_refusal_is_reported_together(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("TumorRegion",), mask_step=False) + request = Request(("PHASE",), (MaskRequest("TumorRegion"), MaskRequest("Nuclei"))) + resolved = inputs.resolve_request(request) + assert _codes(resolved) == [ + ("A_SCOPE_INCOMPLETE", "PHASE"), + ("A_MASK_UNCHECKED", "TumorRegion"), + ("A_MASK_NOT_INSTANCES", "TumorRegion"), + ("A_SCOPE_INCOMPLETE", "Nuclei"), # declared, but no file recorded: 0 of 4 + ] + with pytest.raises(ChannelNotFoundError) as raised: + resolved.require() + assert raised.value.code == "A_SCOPE_INCOMPLETE" + assert "A_MASK_UNCHECKED" in str(raised.value) # the text lists every refusal + + +# --- scope and coverage (SI1, GS2) ------------------------------------------------------- + + +def test_v01_a_channel_no_image_has_is_present_in_0_of_n_and_every_repeat_unavailable(tmp_path): + inputs, _ = _open(tmp_path, "v01") + fovs = inputs.fovs() + resolved = inputs.resolve_request(Request(("DNA", "PHASE"))) + assert _codes(resolved) == [("A_SCOPE_INCOMPLETE", "PHASE")] + assert f"present in 0 of {len(fovs)} acquisitions" in resolved.refusals[0].message + assert set(resolved.refusals[0].files) == {f.relative_path for f in fovs} + by_subject = {} + for item in resolved.coverage: + by_subject.setdefault(item.subject, set()).add(item.status) + assert by_subject == {"DNA": {"full"}, "PHASE": {"unavailable"}} + assert {c.repeat_group for c in resolved.coverage} == {f.repeat_group for f in fovs} + + +def test_an_undeclared_mask_id_is_present_in_0_of_n(tmp_path): + inputs, _ = _open(tmp_path, "v01") + resolved = inputs.resolve_request(Request((), (MaskRequest("Nuclei", role="region"),))) + assert _codes(resolved) == [("A_SCOPE_INCOMPLETE", "Nuclei")] + assert "present in 0 of" in resolved.refusals[0].message + assert {c.status for c in resolved.coverage} == {"unavailable"} + + +def test_v09_partial_masks_are_present_in_k_of_n_and_the_repeat_is_partial(tmp_path): + inputs, images = _partial_mask_run(tmp_path, "v09", "Nuclei", keep=2) + assert len(images) == 6 + resolved = inputs.resolve_request(Request(("DNA",), (MaskRequest("Nuclei"),))) + assert _codes(resolved) == [("A_SCOPE_INCOMPLETE", "Nuclei")] + refusal = resolved.refusals[0] + assert "present in 2 of 6 acquisitions" in refusal.message + assert refusal.files == tuple(images[2:]) + found = [(c.subject, c.repeat_group, c.status, c.present, c.total) for c in resolved.coverage] + assert found == [("DNA", "rep1", "full", 6, 6), ("Nuclei", "rep1", "partial", 2, 6)] + assert resolved.selections == () + # The same run without the mask resolves in full: the refusal is the mask's, not the run's. + assert len(inputs.resolve_request(Request(("DNA",))).selections) == 6 + + +# --- v13: channels by ID per image ---------------------------------------------------------- + + +def test_v13_resolves_every_position_by_channel_id(tmp_path): + inputs, _ = _open(tmp_path, "v13") + resolved = inputs.resolve_request(Request(("ACTIN", "DNA"))) + assert not resolved.refused and len(resolved.selections) == 90 + fovs = {f.relative_path: f for f in inputs.fovs()} + for selection in resolved.selections: + fov = fovs[selection.relative_path] + assert selection.key == fov.key + assert selection.channel_indices == tuple( + inputs.channel_index(fov, c) for c in selection.channel_ids + ) + assert selection.channel_indices == (1, 0) # the request's order, not the image's + assert [c.repeat_group for c in resolved.coverage if c.subject == "DNA"] == [ + "rep1", + "rep2", + "rep3", + ] + assert {c.status for c in resolved.coverage} == {"full"} + + +def test_v13_a_reordered_channel_list_changes_only_that_positions_indices(tmp_path, monkeypatch): + # Preprocessing processes one channel layout per experiment today (a unit declaring another + # order is refused before the run is written), so the resolver's per-image rule is shown on + # the run's own positions with one position's channel order reversed. + inputs, _ = _open(tmp_path, "v13") + original = inputs.fovs() + swapped = original[7] + reordered = dataclasses.replace(swapped, channel_ids=tuple(reversed(swapped.channel_ids))) + changed = tuple(reordered if f is swapped else f for f in original) + monkeypatch.setattr(RunInputs, "fovs", lambda self: changed) + resolved = inputs.resolve_request(Request(("DNA", "ACTIN"))) + by_path = {s.relative_path: s for s in resolved.selections} + assert by_path[swapped.relative_path].channel_indices == (1, 0) + others = {s.channel_indices for p, s in by_path.items() if p != swapped.relative_path} + assert others == {(0, 1)} + + +# --- v28: RGB (P24) ------------------------------------------------------------------------- + + +def test_v28_rgb_units_are_refused_unless_the_consumer_declares_rgb_support(tmp_path): + inputs, _ = _open(tmp_path, "v28") + resolved = inputs.resolve_request(Request(("R", "G", "B"))) + assert _codes(resolved) == [("A_RGB_NOT_SUPPORTED", "rgb")] + assert len(resolved.refusals[0].files) == len(inputs.fovs()) == 32 + with pytest.raises(RgbNotSupportedError, match="A_RGB_NOT_SUPPORTED"): + resolved.require() + accepted = inputs.resolve_request( + Request(("R", "G", "B"), capabilities=Capabilities(rgb_support=True)) + ) + assert not accepted.refused and len(accepted.selections) == 32 + assert {s.image_type for s in accepted.selections} == {"rgb"} + assert accepted.selections[0].channel_indices == (0, 1, 2) + + +def test_v01_fluorescence_units_have_no_image_type_and_need_no_capability(tmp_path): + inputs, _ = _open(tmp_path, "v01") + resolved = inputs.resolve_request(Request(("DNA",))) + assert not resolved.refused and {s.image_type for s in resolved.selections} == {None} + + +# --- planes and volumes ----------------------------------------------------------------------- + + +def test_a_volume_is_refused_by_default_and_accepted_with_volume_support(tmp_path): + inputs, _ = _open(tmp_path, "v01") + refused = inputs.resolve_request(Request(("DNA",), planes="volume")) + assert _codes(refused) == [("A_VOLUME_NOT_SUPPORTED", "volume")] + with pytest.raises(VolumeNotSupportedError, match="A_VOLUME_NOT_SUPPORTED") as raised: + refused.require() + assert raised.value.code == "A_VOLUME_NOT_SUPPORTED" + assert isinstance(raised.value, RunInputsError) + accepted = inputs.resolve_request( + Request(("DNA",), planes="volume", capabilities=Capabilities(volume_support=True)) + ) + plane = inputs.resolve_request(Request(("DNA",))) + # A one-slice volume reads what the plane selection reads (today's runs are 2-D). + assert [s.planes for s in accepted.selections] == [s.planes for s in plane.selections] + assert accepted.selections[0].planes == (PlaneSelection(0, (0,)),) + + +def test_plane_selections_enumerate_planes_or_one_volume_per_time_point(): + assert plane_selections(2, 3, "plane") == ( + PlaneSelection(0, (0,)), + PlaneSelection(0, (1,)), + PlaneSelection(0, (2,)), + PlaneSelection(1, (0,)), + PlaneSelection(1, (1,)), + PlaneSelection(1, (2,)), + ) + assert plane_selections(2, 3, "volume") == ( + PlaneSelection(0, (0, 1, 2)), + PlaneSelection(1, (0, 1, 2)), + ) + assert plane_selections(1, 1, "volume") == plane_selections(1, 1, "plane") + + +# --- malformed requests ----------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("request_", "match"), + [ + (Request(("DNA", "DNA")), "channel ID is requested twice"), + (Request((), (MaskRequest("Nuclei"), MaskRequest("Nuclei"))), "mask ID is requested twice"), + (Request(("DNA",), planes="stack"), "planes must be one of"), + (Request(("DNA",), device="gpu"), "device must be one of"), + (Request(("DNA",), device="CUDA"), "device must be one of"), + (Request(("DNA",), device="cuda:0"), "device must be one of"), + (Request(["DNA"]), "channel_ids must be a tuple of str"), + (Request("DNA"), "channel_ids must be a tuple of str"), + (Request((), [MaskRequest("Nuclei")]), "masks must be a tuple of MaskRequest"), + (Request((), ("Nuclei",)), "masks must be a tuple of MaskRequest"), + (Request(("DNA",), capabilities=Capabilities(rgb_support=1)), "capabilities must be"), + (Request(("DNA",), capabilities=Capabilities(volume_support="no")), "capabilities must be"), + (Request((), (MaskRequest("Nuclei", role="cells"),)), "mask role must be one of"), + ( + Request((), (MaskRequest("TumorRegion", binary_mask_policy="threshold"),)), + "binary_mask_policy must be one of", + ), + ( + Request((), (MaskRequest("Nuclei", binary_mask_policy=POLICY),)), + "applies to a binary mask used as objects", + ), + ( + Request((), (MaskRequest("TumorRegion", role="region", binary_mask_policy=POLICY),)), + "applies to a binary mask used as objects", + ), + ], +) +def test_a_malformed_request_is_a_value_error_not_a_refusal(tmp_path, request_, match): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + with pytest.raises(ValueError, match=match): + inputs.resolve_request(request_) + + +# --- provenance, immutability, no writes ----------------------------------------------------- + + +def test_provenance_is_the_manifests_record_and_the_request_with_its_device(tmp_path): + inputs, run = _open(tmp_path, "v01") + request = Request(("DNA",), device="cuda") + resolved = inputs.resolve_request(request) + recorded = read_manifest(run.output / "run_manifest.json").experiment_description + assert resolved.provenance.resolver_version == RESOLVER_VERSION == 1 + assert resolved.provenance.description_sha256 == recorded["sha256"] + assert dict(resolved.provenance.section_digests) == recorded["section_digests"] + assert resolved.provenance.request == request and resolved.provenance.request.device == "cuda" + assert inputs.resolve_request(Request(("DNA",))).provenance.request.device == "cpu" + + +def test_a_description_changed_since_the_run_is_refused_not_resolved(tmp_path): + inputs, run = _open(tmp_path, "v01") + published = run.output / PUBLISHED_NAME + published.write_bytes(published.read_bytes() + b"# edited after the run\n") + with pytest.raises(RunInputsError, match="not the one the run read"): + inputs.resolve_request(Request(("DNA",))) + + +def test_the_result_is_immutable_and_picklable(tmp_path): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei",)) + resolved = inputs.resolve_request(Request(("DNA",), (MaskRequest("Nuclei"),))) + with pytest.raises(dataclasses.FrozenInstanceError): + resolved.selections = () + with pytest.raises(dataclasses.FrozenInstanceError): + resolved.selections[0].channel_indices = (1,) + with pytest.raises(dataclasses.FrozenInstanceError): + resolved.provenance.request.channel_ids = () + assert pickle.loads(pickle.dumps(resolved)) == resolved + refused = inputs.resolve_request(Request(("PHASE",))) + assert pickle.loads(pickle.dumps(refused)) == refused + with pytest.raises(KeyError): + resolved.selections[0].index_of("ACTIN") + + +def test_resolving_writes_nothing(tmp_path): + inputs, run = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + before = {p: (p.stat().st_size, p.stat().st_mtime_ns) for p in run.output.rglob("*")} + converted = MaskRequest("TumorRegion", binary_mask_policy=POLICY) + inputs.resolve_request(Request(("DNA",), (converted,))) + inputs.resolve_request(Request(("PHASE",), (MaskRequest("TumorRegion"),))) + assert {p: (p.stat().st_size, p.stat().st_mtime_ns) for p in run.output.rglob("*")} == before + assert RunInputs.open(run.output).fovs() == inputs.fovs() + + +# --- every run the resolver sees is described (AR10) ---------------------------------------- + + +def test_a_run_without_a_published_description_never_reaches_the_resolver(tmp_path): + inputs, run = _open(tmp_path, "v01") + (run.output / PUBLISHED_NAME).unlink() + with pytest.raises(DescriptionRequiredError, match="A_DESCRIPTION_REQUIRED") as raised: + RunInputs.open(run.output) + assert raised.value.code == "A_DESCRIPTION_REQUIRED" + + +# --- the resolver imports no brick, CellPatcher or provider module ------------------------- + + +@pytest.mark.parametrize("module", [request_module, resolve_module]) +def test_the_resolver_modules_import_no_brick_cellpatcher_or_provider_module(module): + source = Path(module.__file__).read_text(encoding="utf-8") + imported = set() + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + elif ( + isinstance(node, ast.ImportFrom) and node.level == 0 + ): # relative ones are the package's + imported.add(node.module or "") + allowed = ("__future__", "collections", "dataclasses", "pathlib", "typing") + for name in imported: + assert not name.startswith(FORBIDDEN_IMPORTS), name + assert name.startswith(allowed), name + + +def test_importing_the_resolver_loads_no_brick_cellpatcher_or_provider_module(): + # A fresh interpreter: the package's own relative imports must not pull any in either. + code = ( + "import sys, morphobricks_preprocessing.run_inputs_resolve, " + "morphobricks_preprocessing.run_inputs_request; " + "print(sorted({n.split('.')[0] for n in sys.modules}))" + ) + found = subprocess.run( + [sys.executable, "-c", code], check=True, capture_output=True, text=True + ).stdout + loaded = set(ast.literal_eval(found)) + # scipy is preprocessing's own dependency (the package loads it); everything else is barred. + barred = set(FORBIDDEN_IMPORTS) - {"scipy"} + assert not loaded & barred, loaded & barred + + +def test_the_resolver_runs_no_kernel_and_reads_no_pixel(tmp_path, monkeypatch): + inputs, _ = _open(tmp_path, "v08", masks=("Nuclei", "TumorRegion")) + import tifffile + + def fail(*args, **kwargs): + raise AssertionError("the resolver read pixels") + + for cls in (tifffile.TiffFile, tifffile.TiffPage, tifffile.TiffFrame, tifffile.TiffPageSeries): + monkeypatch.setattr(cls, "asarray", fail) + monkeypatch.setattr(RunInputs, "read_plane", fail) + monkeypatch.setattr(RunInputs, "read_mask", fail) + resolved = inputs.resolve_request( + Request(("DNA",), (MaskRequest("TumorRegion", binary_mask_policy=POLICY),)) + ) + assert not resolved.refused and len(resolved.selections) == 4 + + +def test_a_fov_selection_carries_the_unchanged_key_beside_the_canonical_ids(tmp_path): + inputs, _ = _open(tmp_path, "v33") + resolved = inputs.resolve_request(Request(("DNA",))) + assert not resolved.refused + by_condition = {s.key[0]: s.condition_id for s in resolved.selections} + assert by_condition == {"control": "ctrl", "DrugA": "DrugA"} # the key keeps the folder + assert all(isinstance(s, type(resolved.selections[0])) for s in resolved.selections) + assert all(f.key == s.key for f, s in zip(inputs.fovs(), resolved.selections, strict=True)) + + +def test_fov_is_unchanged_by_the_resolver(): + names = {f.name for f in dataclasses.fields(Fov)} + assert "image_type" not in names # PP-1 adds it; the resolver reads the unit's declaration