Semantic RunInputs from a published description (Stage 1, PR 12)

CHANGELOG.md

@@ -8,6 +8,19 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **Semantic `RunInputs` (Stage 1, PR 12; stacked on PR 11).** For a run with a published
+  description, each `Fov` carries `repeat_group`, `condition_id`, `technical_repeat_id` and
+  `channel_ids` beside the unchanged key (owner decision Q8: the key keeps the source folder
+  labels); `RunInputs.channel_index(fov, id)` finds a channel by canonical ID and refuses an
+  absent one; `references()` gives the condition reference graph, `mask_semantics()` and
+  `mask_check_status()` the masks' meaning and check status. `RunInputs.open` and
+  `description()` read only the published `morphobricks.toml`, check it is the file the run
+  recorded, and refuse a draft or an unconfirmed file with `A_DRAFT_REFUSED`; `read_mask` of a
+  file the mask step has not checked is `A_MASK_UNCHECKED`. A run without a description is
+  unchanged. New modules `fov_identity`, `description_consume`, `run_inputs_error`,
+  `run_inputs_semantic`; `LazyImage` moved to `run_inputs_image` (still exported from
+  `run_inputs`). Conformance P14, P25 and the `A_MASK_UNCHECKED` part of P28 in
+  `tests/test_spec_run_inputs_semantic.py`.
 - **A run with a published experiment description (Stage 1, PR 11).** When the output folder
   holds a published `morphobricks.toml`, `run` reads it with the strict reader (a refusal is
   reported with its Spec code, table path and a plain message), runs the physical checks

docs/contract.md

@@ -301,7 +301,26 @@ It gives:
   reads `experiment_design.csv` with the strict `read_confirmed_design` (§16) and the
   manifest's `experiment_design_sha256` (issue #83, plan §12 question 62; it used the lenient
   reader before): a design that reader refuses raises `RunInputsError`, with the refusal
-  as its `__cause__`.
+  as its `__cause__`;
+- **semantics of a published description** (Stage 1 PR 12; only for a run whose manifest has
+  `experiment_description`, §9; a run without one is unchanged: the fields below are `None`,
+  `description()`, `references()` and `mask_semantics()` return `None`, and `read_mask` is as
+  before). `Fov.repeat_group`, `Fov.condition_id`, `Fov.technical_repeat_id` (the canonical
+  IDs, added beside the unchanged `Fov.key`, whose condition, repeat and technical repeat stay
+  the source folder labels: owner decision Q8, experiment-v1 §3.4) and `Fov.channel_ids` (the
+  canonical channel ID of each C index, as the unit declares them; a unit whose channel count
+  is not the image's, or folders that name no declared unit, is a `RunInputsError`: nothing is
+  guessed). `channel_index(fov, channel_id)` gives the C index, `ChannelNotFoundError` for a
+  channel the image lacks (never a default). `references()` is the condition reference graph
+  (condition ID to the ID it references, or `None`); `mask_semantics()` maps mask ID to
+  `MaskSemantics(source_type, representation, source_channels)`; `mask_check_status(mask_id,
+  fov)` is `checked` or `not yet checked`. `description()` reads the published
+  `morphobricks.toml` again and returns it only if it is the file whose SHA-256 the run
+  recorded (`RunInputsError` otherwise, also at `open`); a draft, a file not named
+  `morphobricks.toml` (`description_consume.read_published_description`) or a published file
+  with an unconfirmed repeat is refused with `DraftRefusedError` (`A_DRAFT_REFUSED`). `read_mask`
+  of a mask file the mask step has not checked (§15; `mask_representation.json`) is refused with
+  `MaskUncheckedError` (`A_MASK_UNCHECKED`); the error classes carry the Spec code as `code`.
 
 A `RunInputs` is picklable, so worker processes can use it.
 `morphobricks_preprocessing.testing.make_clean_run(folder, ...)` writes a tiny experiment of

pyproject.toml

@@ -184,6 +184,11 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.physical_check" = "io"
 "morphobricks_preprocessing.description" = "pure"
 "morphobricks_preprocessing.description_write" = "pure"
+"morphobricks_preprocessing.fov_identity" = "pure"
+"morphobricks_preprocessing.description_consume" = "io"
+"morphobricks_preprocessing.run_inputs_error" = "pure"
+"morphobricks_preprocessing.run_inputs_semantic" = "io"
+"morphobricks_preprocessing.run_inputs_image" = "pure"
 "morphobricks_preprocessing.description_layouts" = "pure"
 "morphobricks_preprocessing.description_record" = "pure"
 "morphobricks_preprocessing.description_run_text" = "pure"

src/morphobricks_preprocessing/description_consume.py

@@ -0,0 +1,59 @@
+"""Reading the published description as a consumer (experiment-v1 sections 10 and 12).
+
+A consumer reads only the published ``morphobricks.toml``. A draft (``morphobricks.draft.json``,
+a reconciliation proposal, anything not named ``morphobricks.toml``) and a file with an
+unconfirmed repeat are refused with ``A_DRAFT_REFUSED``; nothing is read from them.
+"""
+
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+from pathlib import Path
+
+from .description import Description, DescriptionError, description_from_bytes
+from .description_file import DESCRIPTION_FILENAME, read_file_bytes, sha256_hex
+from .description_rules import Code
+from .run_inputs_error import DraftRefusedError, RunInputsError
+
+
+@lru_cache(maxsize=8)
+def _parsed(data: bytes) -> Description:
+    """Return the model of valid bytes (cached by content: a read is then a read and a hash)."""
+    return description_from_bytes(data)
+
+
+def read_published_description(path: str | os.PathLike) -> tuple[Description, str]:
+    """Read the published description at ``path``; return it and the SHA-256 of its bytes.
+
+    Raises
+    ------
+    DraftRefusedError
+        ``A_DRAFT_REFUSED``: the file is not named ``morphobricks.toml`` (a draft or proposal)
+        or holds a repeat that is not confirmed.
+    RunInputsError
+        When the file is missing, a symbolic link, or not a valid description.
+    """
+    path = Path(path)
+    if path.name != DESCRIPTION_FILENAME:
+        raise DraftRefusedError(
+            f"{path.name} is not the published {DESCRIPTION_FILENAME}: a draft or proposal "
+            "is never read (A_DRAFT_REFUSED). Publish the description first."
+        )
+    try:
+        data = read_file_bytes(path)
+        if data is None:
+            raise RunInputsError(f"{path} is gone: the run recorded a description.")
+        return _parsed(data), sha256_hex(data)
+    except DescriptionError as exc:
+        codes = sorted({issue.code.value for issue in exc.issues})
+        if any(issue.code is Code.E_REPEAT_UNCONFIRMED for issue in exc.issues):
+            raise DraftRefusedError(
+                f"{path.name} has a repeat that is not confirmed, so it is a draft "
+                "(A_DRAFT_REFUSED). Confirm and publish it first."
+            ) from exc
+        raise RunInputsError(
+            f"{path.name} is not a valid experiment description ({', '.join(codes)})."
+        ) from exc
+    except OSError as exc:
+        raise RunInputsError(f"{path.name} cannot be read: {exc}") from exc

src/morphobricks_preprocessing/fov_identity.py

@@ -0,0 +1,57 @@
+"""A position's canonical identity from the description (experiment-v1 section 3.4).
+
+The FOV key keeps the source folder labels. The canonical IDs are added beside it, each a
+function of the key: the repeat ID (``repeat_group``) of the member folder, the condition ID of
+the condition folder, the technical repeat ID of the unit folder, and the unit's channel IDs
+by C index. Nothing is looked up by the label alone: a label that names no declared member, or
+two, is refused.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from .description import Description
+
+
+@dataclass(frozen=True)
+class FovIdentity:
+    """The canonical identity of one position."""
+
+    repeat_group: str
+    condition_id: str
+    technical_repeat_id: str
+    channel_ids: tuple[str, ...]
+
+
+def fov_identity(description: Description, labels: tuple[str, str, str]) -> FovIdentity:
+    """Return the identity of the position whose source folder labels are ``labels``.
+
+    Parameters
+    ----------
+    description
+        The published description.
+    labels
+        ``(condition folder, biological repeat folder, technical repeat folder)``.
+
+    Raises
+    ------
+    LookupError
+        When the labels name no declared unit, or more than one.
+    """
+    condition, repeat, technical = labels
+    found = [
+        FovIdentity(repeat_id, condition_id, unit_id, unit.channels)
+        for repeat_id, entry in description.repeats.items()
+        for condition_id, member in entry.members.items()
+        if description.conditions[condition_id].source_folder == condition
+        and member.source_repeat == repeat
+        for unit_id, unit in member.technical_repeats.items()
+        if unit.source_folder == technical
+    ]
+    if len(found) != 1:
+        raise LookupError(
+            f"the folders {condition!r} / {repeat!r} / {technical!r} name "
+            f"{len(found)} units of the description, not one"
+        )
+    return found[0]

src/morphobricks_preprocessing/run_inputs.py

@@ -41,6 +41,7 @@
 from .axes import AxisMapping
 from .calibration import CALIBRATION_FILENAME
 from .conversion_check import conversion_problem
+from .description import Description
 from .design import DESIGN_FILENAME, DesignCheckError, ExperimentDesign, read_confirmed_design
 from .fingerprint import Fingerprint
 from .folder_rules import masks_folder_issues
@@ -59,6 +60,9 @@
 from .planes import DEFAULT_MEMORY_LIMIT, read_plane
 from .recorded_sizes import RecordedSize
 from .run.generation import published_path, read_stable
+from .run_inputs_error import RunInputsError as RunInputsError  # re-exported (moved)
+from .run_inputs_image import LazyImage as LazyImage  # re-exported (moved)
+from .run_inputs_semantic import SemanticInputs, check_description, with_identity
 from .run_inputs_text import (
     _ZARR_MISSING,
     _changed_text,
@@ -72,10 +76,6 @@
 from .zarr_input import NgffError, ZarrUnavailableError
 
 
-class RunInputsError(ValueError):
-    """The run cannot be used as it is: run preprocessing again (the message says why)."""
-
-
 class UnrecordedFilesError(RunInputsError):
     """The experiment holds images or masks the run neither recorded nor excluded.
 
@@ -122,6 +122,12 @@ class Fov:
         Significant bits from the metadata, or None.
     z_count, t_count, c_count
         Depth slices, time points and channels.
+    repeat_group, condition_id, technical_repeat_id
+        The canonical repeat, condition and technical repeat IDs of the position, added
+        beside the unchanged ``key`` (experiment-v1 §3.4). None for a run without a
+        published description.
+    channel_ids
+        The canonical channel ID of each C index, in order; None without a description.
     """
 
     key: tuple[str, str, str, str]
@@ -136,10 +142,14 @@ class Fov:
     z_count: int
     t_count: int
     c_count: int
+    repeat_group: str | None = None
+    condition_id: str | None = None
+    technical_repeat_id: str | None = None
+    channel_ids: tuple[str, ...] | None = None
 
 
 @dataclass(frozen=True)
-class RunInputs:
+class RunInputs(SemanticInputs):
     """The inputs of a clean preprocessing run, read as the run recorded them.
 
     Use :meth:`open`; the attributes are the snapshot taken then.
@@ -224,6 +234,7 @@ def _read(
             design = published_path(output, DESIGN_FILENAME, manifest.experiment_design_sha256)
             _check_sha(design, manifest.experiment_design_sha256, "design")
         _check_calibration(output / CALIBRATION_FILENAME, manifest.calibration_sha256)
+        check_description(output, manifest)  # A_DRAFT_REFUSED, or changed since the run
         root = Path(experiment_root or manifest.experiment_root).resolve()
         _check_conversion(root, manifest)
         masks = masks_root or (manifest.masks or {}).get("folder")
@@ -244,7 +255,10 @@ def fovs(self) -> tuple[Fov, ...]:
         rows: dict[str, list[PlaneRecord]] = {}
         for record in self.records:
             rows.setdefault(record.source_relative_path, []).append(record)
-        return tuple(_fov(e, rows.get(e.relative_path, []), self.manifest) for e in self.files)
+        described = self.description()
+        return tuple(
+            _fov(e, rows.get(e.relative_path, []), self.manifest, described) for e in self.files
+        )
 
     @property
     def files(self) -> tuple[FileEntry, ...]:
@@ -330,6 +344,7 @@ def read_mask(
         entry = self._checked(fov)
         _check_planes(entry.relative_path, self._fov(entry), z, t)
         recorded = self._mask_entry(entry.relative_path, mask_type)
+        self.require_checked(recorded, mask_type)  # A_MASK_UNCHECKED (a described run)
         if not self._mask_unchanged(recorded):
             raise RunInputsError(_changed_text([recorded["path"]], "mask"))
         mask = MaskFile(recorded["image"], recorded["type"], recorded["path"], recorded["source"])
@@ -454,47 +469,6 @@ def _mask_unchanged(self, recorded: Mapping) -> bool:
         except FileNotFoundError:
             return False
 
-    def _mask_entry(self, image: str, mask_type: str) -> Mapping:
-        """Return the recorded mask of an image and type."""
-        for recorded in (self.manifest.masks or {}).get("files", ()):
-            if recorded["image"] == image and recorded["type"] == mask_type:
-                return recorded
-        raise RunInputsError(f"The run recorded no {mask_type} mask for {image}.")
-
-
-@dataclass(frozen=True)
-class LazyImage:
-    """An image of a clean run as a ``(T, Z, C, Y, X)`` array whose planes are read on demand.
-
-    Attributes
-    ----------
-    inputs
-        The run.
-    relative_path
-        The image.
-    shape
-        ``(T, Z, C, Y, X)``.
-    dtype
-        The stored pixel type.
-    """
-
-    inputs: RunInputs
-    relative_path: str
-    shape: tuple[int, ...]
-    dtype: np.dtype
-
-    def __getitem__(self, index: tuple[int, int]) -> np.ndarray:
-        """Read the ``(C, Y, X)`` plane at ``[t, z]``."""
-        t, z = index
-        return self.inputs.read_plane(self.relative_path, z=z, t=t)
-
-    def __array__(self, dtype: np.dtype | None = None, copy: bool | None = None) -> np.ndarray:
-        """Read every plane into one array."""
-        data = np.stack(
-            [np.stack([self[t, z] for z in range(self.shape[1])]) for t in range(self.shape[0])]
-        )
-        return data if dtype is None else data.astype(dtype)
-
 
 def _check_planes(relative_path: str, fov: Fov, z: int | None, t: int | None) -> None:
     """Refuse a ``z`` or ``t`` outside the image (audit I5)."""
@@ -566,14 +540,23 @@ def _mapping(entry: FileEntry) -> AxisMapping:
     return AxisMapping(entry.axes, tuple(entry.roles or ()), entry.singleton_roles)
 
 
-def _fov(entry: FileEntry, rows: list[PlaneRecord], manifest: RunManifest) -> Fov:
-    """Describe one position from its manifest entry and its table rows."""
+def _fov(
+    entry: FileEntry,
+    rows: list[PlaneRecord],
+    manifest: RunManifest,
+    description: Description | None = None,
+) -> Fov:
+    """Describe one position from its manifest entry and its table rows.
+
+    With the published ``description`` the canonical identity is added beside the key; the
+    unit's channels must be the image's channels (nothing is guessed when they are not).
+    """
     zs = {r.z for r in rows} or {0}
     ts = {r.t for r in rows} or {0}
     channels = rows[0].channel_count if rows else len(entry.channel_names) or 1
     if entry.position_id is None:  # never an empty name in the key (round-2 audit A-m5)
         raise RunInputsError(f"The run records no position_id for {entry.relative_path}.")
-    return Fov(
+    fov = Fov(
         key=(*entry.labels, entry.position_id),
         relative_path=entry.relative_path,
         plane_ids=tuple(r.plane_id for r in rows),
@@ -587,3 +570,4 @@ def _fov(entry: FileEntry, rows: list[PlaneRecord], manifest: RunManifest) -> Fo
         t_count=max(ts) + 1,
         c_count=channels,
     )
+    return fov if description is None else with_identity(fov, entry, description)

src/morphobricks_preprocessing/run_inputs_error.py

@@ -0,0 +1,32 @@
+"""The errors of :class:`.run_inputs.RunInputs` (moved from ``run_inputs``, which re-exports them).
+
+``RunInputsError`` is the general refusal: run preprocessing again. The Spec's analysis-layer
+refusals (experiment-v1 section 8) are subclasses with their code, so a caller can match them:
+:class:`DraftRefusedError` (``A_DRAFT_REFUSED``) and :class:`MaskUncheckedError`
+(``A_MASK_UNCHECKED``).
+"""
+
+from __future__ import annotations
+
+
+class RunInputsError(ValueError):
+    """The run cannot be used as it is: run preprocessing again (the message says why)."""
+
+    code: str | None = None
+    """The Spec's code when the refusal has one (``A_*``), else None."""
+
+
+class DraftRefusedError(RunInputsError):
+    """A draft or an unconfirmed description was given: only the published file is read."""
+
+    code = "A_DRAFT_REFUSED"
+
+
+class MaskUncheckedError(RunInputsError):
+    """A mask file whose representation the mask step has not checked yet was used."""
+
+    code = "A_MASK_UNCHECKED"
+
+
+class ChannelNotFoundError(RunInputsError):
+    """A canonical channel ID is not one of the image's channels (never a default index)."""