diff --git a/CHANGELOG.md b/CHANGELOG.md index f0d8fb07..e35fd32a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the ## Unreleased +- **Images without header channel names are listed in a described run (follow-up of #218, owner + decision 2026-11-08).** An image whose header names no channel, or not every one, still passes + the per-unit header checks (§4.3, fixture v12: no evidence needed, the confirmed layout is + used), with no warning, but is now recorded: the unit's record in the manifest's + `experiment_description` entry gets `images_without_header_channel_names` (image, `none` or + `partial`, and the zero-based channel positions without a name; omitted for a unit whose images + are all named), `read_manifest` reads it strictly, `RunInputs.images_without_header_channel_names()` + returns it read-only (None without a description), and the report text and JSON list the images. + A run without a description is unchanged: no new manifest or report key. - **A draft written under another Spec commit is refused at resume (#223, owner decision of 2026-11-08).** `load_draft` and `publish_draft` raise `DraftFileError` with the new code `spec_commit_mismatch` when the draft's `spec_commit` is not the pinned `SPEC_COMMIT`; diff --git a/docs/contract.md b/docs/contract.md index 82fb18e3..bdc98b18 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -216,7 +216,7 @@ interpretation. | `narrow_histogram_ranges` | The channel planes whose range check found the central range too narrow (§4), sorted by plane and channel: `relative_path`, `plane_id`, `channel` and the fields of the normalization brick's `RangeCheck` (`histogram_range`, `central_percentiles`, `central_bins`, `robust_quantiles`, `robust_limits`, `outside_robust_count`, `robust_central_bins`, `min_central_bins`). Empty when there are none; absent in manifests written before follow-up P6. Optional key: the manifest stays version 1. | | `conversion_record_sha256` | SHA-256 of the conversion record (`.morphobricks-conversion.json`) in the experiment root when the experiment is a tree of named-axis copies (design v0.4 §5, §17); null otherwise. `RunInputs` checks it (owner decision of 2 November 2026, plan question 59): a record changed or gone since the run is refused, as for `calibration_sha256`. A changed record changes no row (the copies are checked by their fingerprints like any image). | | `calibration_sha256` | SHA-256 of the output folder's `calibration.csv` as the run read it (§9.3); null when there was none (or absent in manifests written before PP-8). `RunInputs` checks it. | -| `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). | +| `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation; and, only for a unit that has such images, `images_without_header_channel_names`: a list sorted by `image` (relative path) of objects `image`, `header_names` (`none` when the header names no channel, `partial` when only some) and `channel_positions` (the zero-based C positions without a header name). Such an image passes the header checks, needs no evidence and the confirmed layout is used (§4.3); it is listed, not warned about, and the key is omitted for a unit whose images are all fully named; RGB units list nothing), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). | | Version 1 | A manifest written before v0.4.0 (`"1"`) is still read, for reporting, but no row of it is reused: its axes were never checked. The next run recomputes every row ("the earlier run was written before v0.4.0, without the axis check"). Bricks refuse it. | | Row reuse | A file's rows are reused only when its row key is the earlier run's (`fingerprint.RowKey`, one rule for "the same file, read the same way", audit D5): its fingerprint (`fingerprint.Fingerprint`: size and modification time, and for an OME-Zarr image the inventory digest), labels, axes, roles, `singleton_roles`, `axes_from` and the path it is read from (so moving or converting the experiment recomputes it), and the settings that change rows (bins, histogram range, robust quantiles, range-check limit). **The identity rule is the fingerprint, which is metadata** (design v0.4 §1): its known blind spot is a file rewritten with the same size whose modification time (for an OME-Zarr image, every changed file's) is set back to its earlier value; its earlier rows are then reused, its staged rows resumed, and `RunInputs` accepts it. | | Content verification (on by default, owner 7 November 2026, #159; opt-in before) | Every run does it unless it opts out with `RunMode(name, verify_content=False)` (`run_experiment(mode=...)`, CLI `--no-verify-content`, the notebook's "Verify each image's content" box unticked); the opt-out is recorded as `settings.verify_content: false`. It closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (true without one: a first run, or a version 1 manifest); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the cost is in the pull request that made it the default (about 3 seconds per 1.5 GB read, measured at 1 GB/s of SHA-256). A run that verifies writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). | diff --git a/pyproject.toml b/pyproject.toml index 57779466..f252cba6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -201,6 +201,7 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.run_inputs_image" = "pure" "morphobricks_preprocessing.description_layouts" = "pure" "morphobricks_preprocessing.description_record" = "pure" +"morphobricks_preprocessing.description_unnamed" = "pure" "morphobricks_preprocessing.description_run_text" = "pure" "morphobricks_preprocessing.mask_representation" = "io" "morphobricks_preprocessing.description_transition" = "pure" diff --git a/src/morphobricks_preprocessing/description_record.py b/src/morphobricks_preprocessing/description_record.py index 75cb9fea..f999a777 100644 --- a/src/morphobricks_preprocessing/description_record.py +++ b/src/morphobricks_preprocessing/description_record.py @@ -13,6 +13,7 @@ from collections.abc import Mapping from .description import Description +from .description_unnamed import FIELD, checked_items, unnamed_items from .description_write import section_digests KEY = "experiment_description" @@ -22,7 +23,9 @@ def unit_records( - description: Description, verified: Mapping[tuple[str, str, str], bool] + description: Description, + verified: Mapping[tuple[str, str, str], bool], + unnamed: Mapping[tuple[str, str, str], Mapping[str, tuple[int, ...]]] | None = None, ) -> list[dict[str, object]]: """Return every declared unit with whether its image type was checked against a header. @@ -30,25 +33,30 @@ def unit_records( check's outcome; a unit it does not name is not verified. It is False for a unit whose format states no colour interpretation (OME-Zarr: the declaration stands on the user's confirmation) and for a unit with no readable image. + + ``unnamed`` maps the same keys to the images whose header names not every channel and the + positions without a name (:mod:`.description_unnamed`); a unit with such images gets the + list under ``images_without_header_channel_names``, any other unit has no such key. """ - return [ - { - "repeat": repeat_id, - "condition": condition_id, - "technical_repeat": unit_id, - "image_type_header_verified": verified.get( - ( + out = [] + for repeat_id, repeat in sorted(description.repeats.items()): + for condition_id, member in sorted(repeat.members.items()): + for unit_id, unit in sorted(member.technical_repeats.items()): + key = ( description.conditions[condition_id].source_folder, member.source_repeat, unit.source_folder, - ), - False, - ), - } - for repeat_id, repeat in sorted(description.repeats.items()) - for condition_id, member in sorted(repeat.members.items()) - for unit_id, unit in sorted(member.technical_repeats.items()) - ] + ) + record: dict[str, object] = { + "repeat": repeat_id, + "condition": condition_id, + "technical_repeat": unit_id, + "image_type_header_verified": verified.get(key, False), + } + if (unnamed or {}).get(key): + record[FIELD] = unnamed_items(unnamed[key], len(unit.channels)) + out.append(record) + return out def description_record( @@ -94,4 +102,7 @@ def recorded_description(value: object) -> dict | None: ): if not isinstance(value[name], kind): raise ValueError(f"{KEY}.{name} is not a {kind.__name__}") + for index, unit in enumerate(value["units"]): + if isinstance(unit, dict) and FIELD in unit: + checked_items(unit[FIELD], f"{KEY}.units[{index}]") return value diff --git a/src/morphobricks_preprocessing/description_unnamed.py b/src/morphobricks_preprocessing/description_unnamed.py new file mode 100644 index 00000000..1ebbac18 --- /dev/null +++ b/src/morphobricks_preprocessing/description_unnamed.py @@ -0,0 +1,95 @@ +"""Images of a described run whose header names no channel, or not every one (experiment-v1 §4.3). + +Such an image passes the per-unit header checks: it gives no evidence against the user's +confirmed layout, which is used. It is not warned about; it is *listed*, so a reader can see +which images stand on the confirmation alone. The list is part of the unit's record in the +manifest's ``experiment_description`` entry (:mod:`.description_record`), under +:data:`FIELD`, and only for a unit that has such images: a unit whose images are all fully +named has no such key. Each item names the image (relative to the experiment root), whether +its header names no channel (``"none"``) or only some (``"partial"``), and the zero-based C +positions without a name. A run without a description records nothing of this. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping + +from .validation import plural + +FIELD = "images_without_header_channel_names" +"""Key of a unit record that has images with unnamed channels.""" +NONE = "none" +PARTIAL = "partial" +_KEYS = {"image", "header_names", "channel_positions"} + + +def unnamed_items( + unnamed: Mapping[str, tuple[int, ...]], declared_channels: int +) -> list[dict[str, object]]: + """Return the manifest items of one unit's unnamed images, sorted by path.""" + return [ + { + "image": path, + "header_names": NONE if len(positions) == declared_channels else PARTIAL, + "channel_positions": list(positions), + } + for path, positions in sorted(unnamed.items()) + ] + + +def checked_items(value: object, where: str) -> None: + """Refuse a malformed list of unnamed images of one unit record. + + Raises + ------ + ValueError + When it is not a non-empty list of items with exactly the recorded keys, a string + image, ``none`` or ``partial``, and ascending distinct non-negative whole positions + (all of the unit's positions are not known here, so only ``none`` and ``partial`` + are told apart by the writer). + """ + if not isinstance(value, list) or not value: + raise ValueError(f"{where}.{FIELD} must be a non-empty list") + for item in value: + if not isinstance(item, dict) or set(item) != _KEYS: + raise ValueError(f"{where}.{FIELD} items must hold exactly {', '.join(sorted(_KEYS))}") + if not isinstance(item["image"], str) or not item["image"]: + raise ValueError(f"{where}.{FIELD} image is not a path") + if item["header_names"] not in (NONE, PARTIAL): + raise ValueError(f"{where}.{FIELD} header_names is not {NONE} or {PARTIAL}") + positions = item["channel_positions"] + whole = isinstance(positions, list) and all( + isinstance(p, int) and not isinstance(p, bool) and p >= 0 for p in positions + ) + if not whole or not positions or positions != sorted(set(positions)): + raise ValueError(f"{where}.{FIELD} channel_positions must be ascending positions") + paths = [item["image"] for item in value] + if paths != sorted(set(paths)): + raise ValueError(f"{where}.{FIELD} images must be sorted and distinct") + + +def listed_units(units: Iterable[Mapping[str, object]]) -> list[Mapping[str, object]]: + """Return the unit records that list unnamed images.""" + return [unit for unit in units if unit.get(FIELD)] + + +def text_lines(units: Iterable[Mapping[str, object]]) -> list[str]: + """Describe the listed images for the report text; empty when there are none.""" + listed = listed_units(units) + if not listed: + return [] + count = sum(len(unit[FIELD]) for unit in listed) # type: ignore[arg-type] + lines = [ + f"Images without header channel names: {plural(count, 'image')} in " + f"{plural(len(listed), 'unit')} (checked as passing; your confirmed layout is used)" + ] + for unit in listed: + lines.append( + f" repeat {unit['repeat']} / condition {unit['condition']} / " + f"technical repeat {unit['technical_repeat']}" + ) + for item in unit[FIELD]: # type: ignore[union-attr] + spots = ", ".join(str(p) for p in item["channel_positions"]) + what = "no channel named" if item["header_names"] == NONE else "unnamed positions" + lines.append(f" {item['image']}: {what} (channel positions {spots}, from 0)") + return lines diff --git a/src/morphobricks_preprocessing/discovery_report.py b/src/morphobricks_preprocessing/discovery_report.py index 1ef09f6a..9beab215 100644 --- a/src/morphobricks_preprocessing/discovery_report.py +++ b/src/morphobricks_preprocessing/discovery_report.py @@ -24,6 +24,8 @@ from pathlib import Path from typing import TYPE_CHECKING +from .description_unnamed import FIELD as UNNAMED_FIELD +from .description_unnamed import listed_units, text_lines from .image_names import position_id from .recorded_sizes import HEADER, PIXELS, STATED, RecordedSize from .validation import ValidationIssue, plural @@ -148,6 +150,7 @@ def text(self) -> str: lines.extend(_folder_lines(data)) lines.extend(_image_lines(data)) lines.extend(_mask_lines(data)) + lines.extend(text_lines(data.get(UNNAMED_FIELD, ()))) lines.extend(_outcome_lines(self.outcome)) lines.extend(_issue_lines(self.issues)) return "\n".join(lines) + "\n" @@ -162,6 +165,7 @@ def build_report( design: ExperimentDesign | None = None, image_type: str | None = None, outcome: RunOutcome | None = None, + unit_records: Sequence[Mapping[str, object]] | None = None, ) -> DiscoveryReport: """Build the report of one run from what it found and did. @@ -181,6 +185,10 @@ def build_report( The image type, or None. outcome What the run did with the earlier results. + unit_records + The manifest's unit records of a run with a published description, or None. Their + images without header channel names are listed under ``images_without_header_channel_names`` + (:mod:`.description_unnamed`); the key is left out for a run without a description. Returns ------- @@ -204,6 +212,11 @@ def build_report( "image_type": image_type, "masks": _masks(pre), } + if unit_records is not None: + data[UNNAMED_FIELD] = [ + {k: v for k, v in unit.items() if k != "image_type_header_verified"} + for unit in listed_units(unit_records) + ] ordered = tuple(sorted(issues, key=lambda issue: not issue.is_error)) return DiscoveryReport(data, ordered, outcome) diff --git a/src/morphobricks_preprocessing/physical_check.py b/src/morphobricks_preprocessing/physical_check.py index fb49325b..3847874c 100644 --- a/src/morphobricks_preprocessing/physical_check.py +++ b/src/morphobricks_preprocessing/physical_check.py @@ -131,6 +131,7 @@ class PhysicalReport: colour interpretation (TIFF), False for a format that states none (OME-Zarr: it stands on the user's confirmation). ``unchecked`` lists images whose header could not be read or whose channel count is not stated by the header: they were not checked, which is not a pass. + ``unnamed`` lists the images that pass without header channel names (:attr:`unnamed`). """ issues: tuple[PhysicalIssue, ...] = () @@ -140,6 +141,12 @@ class PhysicalReport: default_factory=lambda: MappingProxyType({}) ) unchecked: tuple[str, ...] = () + unnamed: Mapping[UnitKey, Mapping[str, tuple[int, ...]]] = field( + default_factory=lambda: MappingProxyType({}) + ) + """Per unit that passed, the images whose header names not every channel, each with the C + positions without a name (all of them when the header names none). Such an image passes + (§4.3: it needs no evidence, the user's confirmed layout is used) and is listed, not warned.""" @property def codes(self) -> frozenset[str]: @@ -289,6 +296,19 @@ def _evidence_issues( return out +def _unnamed(unit: Unit, layouts: Mapping[str, Layout]) -> dict[str, tuple[int, ...]]: + """Return per image the declared channel positions its header gives no name (not RGB).""" + if unit.image_type == RGB: # R, G and B take no header names (§4.4b) + return {} + found = { + path: tuple( + i for i in range(len(unit.channels)) if i >= len(layout.names) or not layout.names[i] + ) + for path, layout in layouts.items() + } + return {path: positions for path, positions in found.items() if positions} + + def _check_unit( key: UnitKey, repeat_unit: tuple[str, Unit], @@ -296,25 +316,30 @@ def _check_unit( paths: Sequence[str], metas: Mapping[str, TiffMetadata], roles: Mapping[str, Sequence[str | None]], -) -> tuple[list[PhysicalIssue], Layout | None, bool, list[str]]: - """Check one declared unit that has images; return issues, layout, verified, unchecked.""" +) -> tuple[list[PhysicalIssue], Layout | None, bool, list[str], dict[str, tuple[int, ...]]]: + """Check one declared unit that has images. + + Returns the issues, the common layout, verified, the unchecked images and, for a unit + without issues, the images whose header names not every channel (:func:`_unnamed`). + """ _, unit = repeat_unit readable = [p for p in paths if p in metas and metas[p].readable] layouts = {p: layout_of(metas[p], roles.get(p)) for p in readable} unchecked = [p for p in paths if p not in layouts] verified = bool(readable) and all(metas[p].kind != OME_ZARR for p in readable) if not layouts: - return [], None, verified, unchecked + return [], None, verified, unchecked, {} reference, issues = _homogeneity(key, layouts) if issues: - return issues, None, verified, unchecked + return issues, None, verified, unchecked, {} issues = _type_issues(key, unit, layouts, metas) if issues: - return issues, None, verified, unchecked + return issues, None, verified, unchecked, {} issues = _evidence_issues(key, unit, description, layouts) if reference.channels is None: # the header does not say; never a guessed count unchecked += [p for p in layouts if p not in unchecked] - return issues, None if issues else reference, verified, unchecked + unnamed = {} if issues else _unnamed(unit, layouts) + return issues, None if issues else reference, verified, unchecked, unnamed def check_physical( @@ -361,15 +386,18 @@ def check_physical( layouts: dict[UnitKey, Layout] = {} verified: dict[UnitKey, bool] = {} unchecked: list[str] = [] + unnamed: dict[UnitKey, Mapping[str, tuple[int, ...]]] = {} for key, paths in found.items(): if key not in declared: continue - found_issues, layout, ok, missing = _check_unit( + found_issues, layout, ok, missing, nameless = _check_unit( key, declared[key], description, sorted(paths), metas, roles or {} ) issues += found_issues unchecked += missing verified[key] = ok + if nameless: + unnamed[key] = MappingProxyType(nameless) if layout is not None: layouts[key] = layout if mask_types is not None: @@ -383,6 +411,7 @@ def check_physical( MappingProxyType(layouts), MappingProxyType(verified), tuple(dict.fromkeys(unchecked)), + MappingProxyType(unnamed), ) diff --git a/src/morphobricks_preprocessing/run/description_run.py b/src/morphobricks_preprocessing/run/description_run.py index 26089fde..e1b67f24 100644 --- a/src/morphobricks_preprocessing/run/description_run.py +++ b/src/morphobricks_preprocessing/run/description_run.py @@ -182,18 +182,24 @@ def mask_records( return out +def unit_entries(found: RunDescription) -> list[dict[str, object]]: + """Return the manifest's unit records: image type verification and unnamed header images.""" + report = found.physical + if report is None: + return unit_records(found.description, {}) + verified = {astuple(k): v for k, v in report.image_type_verified.items()} + unnamed = {astuple(k): v for k, v in report.unnamed.items()} + return unit_records(found.description, verified, unnamed) + + def manifest_entry( found: RunDescription, masks: Mapping | None, output: Path, images: int ) -> dict[str, object]: """Return the manifest's ``experiment_description`` entry of a run.""" - report = found.physical - verified = ( - {} if report is None else {astuple(k): v for k, v in report.image_type_verified.items()} - ) return description_record( found.description, found.sha256, - unit_records(found.description, verified), + unit_entries(found), mask_records(found.description, masks, output, images), found.records, ) diff --git a/src/morphobricks_preprocessing/run/pipeline.py b/src/morphobricks_preprocessing/run/pipeline.py index ada6cf82..9c84e57c 100644 --- a/src/morphobricks_preprocessing/run/pipeline.py +++ b/src/morphobricks_preprocessing/run/pipeline.py @@ -64,6 +64,7 @@ RunDescription, checked_description, manifest_entry, + unit_entries, ) from .design_file import design_for_run from .generation import staged_generation @@ -333,6 +334,7 @@ def _report( design=run.checked.design, image_type=run.inputs.image_type, outcome=outcome, + unit_records=None if run.description is None else unit_entries(run.description), ) diff --git a/src/morphobricks_preprocessing/run_inputs_semantic.py b/src/morphobricks_preprocessing/run_inputs_semantic.py index f4a2991c..986f259d 100644 --- a/src/morphobricks_preprocessing/run_inputs_semantic.py +++ b/src/morphobricks_preprocessing/run_inputs_semantic.py @@ -26,6 +26,7 @@ from .description import Description from .description_consume import read_published_description from .description_file import DESCRIPTION_FILENAME +from .description_unnamed import FIELD as UNNAMED_FIELD from .fov_identity import FovIdentity, fov_identity from .manifest import FileEntry, RunManifest from .mask_representation import CHECKED, file_status, read_checked @@ -122,6 +123,22 @@ def description(self) -> Description | None: """ return check_description(self.output_folder, self.manifest) + def images_without_header_channel_names(self) -> dict[str, tuple[int, ...]] | None: + """Return the images the run passed without header channel names, as it recorded them. + + Each image's relative path maps to its zero-based C positions whose header gives no + name (all of them when the header names none). Empty when every header names all its + channels; None when the run had no description. Read from the manifest only. + """ + recorded = self.manifest.experiment_description + if recorded is None: + return None + return { + item["image"]: tuple(item["channel_positions"]) + for unit in recorded["units"] + for item in unit.get(UNNAMED_FIELD, ()) + } + def channel_index(self, fov: object, channel_id: str) -> int: """Return the C index of a canonical channel in one position's image. diff --git a/tests/test_unnamed_header_images.py b/tests/test_unnamed_header_images.py new file mode 100644 index 00000000..24b754ca --- /dev/null +++ b/tests/test_unnamed_header_images.py @@ -0,0 +1,163 @@ +"""Images of a described run whose header names no channel (or not every one) are listed. + +Owner decision 2026-11-08, on the P26 per-unit header checks: such an image passes (§4.3, fixture +v12: it needs no evidence, the user's confirmed layout is used) and is *listed* in the manifest's +``experiment_description`` entry and the report text, not warned about. A run without a +description records nothing of this. +""" + +from __future__ import annotations + +import json + +import numpy as np +import pytest +import tifffile + +from description_tree import MAPPING, build_experiment +from morphobricks_preprocessing import RunInputs, run_experiment +from morphobricks_preprocessing.description import description_from_bytes +from morphobricks_preprocessing.discovery import discover_experiment +from morphobricks_preprocessing.manifest import ManifestFormatError, read_manifest +from morphobricks_preprocessing.metadata import inspect_images +from morphobricks_preprocessing.physical_check import UnitKey, check_physical +from spec_pin import fixtures + +V01 = next(f for f in fixtures("valid") if f.name.startswith("v01_")) +DESCRIPTION = description_from_bytes(V01.data) +FIELD = "images_without_header_channel_names" +NAMES = ["DAPI", "Phalloidin-488"] # the confirmed header names of v01's DNA and ACTIN + + +def _named(path, names): + tifffile.imwrite( + path, + np.random.default_rng(3).integers(0, 200, (len(names), 8, 8), dtype=np.uint8), + ome=True, + photometric="minisblack", + metadata={"axes": "CYX", "Channel": {"Name": names}}, + ) + + +def _run(tmp_path, *, described=True): + """One image per unit: CTRL/rep1 named, DrugA/rep1 partly, CTRL/rep2 none, DrugA/rep2 named.""" + root, out = tmp_path / "exp", tmp_path / "out" + out.mkdir() + paths = build_experiment(root, DESCRIPTION, images=1) + by_unit = {p.rsplit("/", 1)[0]: p for p in paths} + _named(root / by_unit["CTRL/rep1"], NAMES) + _named(root / by_unit["DrugA/rep1"], [NAMES[0], ""]) + _named(root / by_unit["DrugA/rep2"], NAMES) + if described: + (out / "morphobricks.toml").write_bytes(V01.data) + return root, out, by_unit + + +def _entry(out): + return json.loads((out / "run_manifest.json").read_text(encoding="utf-8"))[ + "experiment_description" + ] + + +def _listed(entry): + return {(u["condition"], u["repeat"]): u[FIELD] for u in entry["units"] if FIELD in u} + + +def test_the_run_lists_the_unnamed_and_the_partly_named_image_exactly(tmp_path): + root, out, by_unit = _run(tmp_path) + assert run_experiment(root, MAPPING, output_folder=out).written + entry = _entry(out) + assert len(entry["units"]) == 4 + listed = _listed(entry) + assert listed == { + ("DrugA", "rep1"): [ + {"image": by_unit["DrugA/rep1"], "header_names": "partial", "channel_positions": [1]} + ], + ("CTRL", "rep2"): [ + {"image": by_unit["CTRL/rep2"], "header_names": "none", "channel_positions": [0, 1]} + ], + } + assert read_manifest(out / "run_manifest.json").experiment_description == entry + inputs = RunInputs.open(out) + assert inputs.images_without_header_channel_names() == { + by_unit["DrugA/rep1"]: (1,), + by_unit["CTRL/rep2"]: (0, 1), + } + + +def test_the_report_text_and_json_show_them(tmp_path): + root, out, by_unit = _run(tmp_path) + run_experiment(root, MAPPING, output_folder=out) + text = (out / "discovery_report.txt").read_text(encoding="utf-8") + assert "Images without header channel names: 2 images in 2 units" in text + assert f"{by_unit['CTRL/rep2']}: no channel named (channel positions 0, 1" in text + assert f"{by_unit['DrugA/rep1']}: unnamed positions (channel positions 1" in text + assert by_unit["CTRL/rep1"] not in text and by_unit["DrugA/rep2"] not in text + data = json.loads((out / "discovery_report.json").read_text(encoding="utf-8")) + assert {u["condition"] for u in data[FIELD]} == {"DrugA", "CTRL"} + assert data["issues"] == [] # listed, not warned + + +def test_a_fully_named_run_lists_nothing_and_has_no_such_key(tmp_path): + root, out = tmp_path / "exp", tmp_path / "out" + out.mkdir() + for path in build_experiment(root, DESCRIPTION, images=1): + _named(root / path, NAMES) + (out / "morphobricks.toml").write_bytes(V01.data) + assert run_experiment(root, MAPPING, output_folder=out).written + entry = _entry(out) + assert not any(FIELD in unit for unit in entry["units"]) # omitted, never an empty list + data = json.loads((out / "discovery_report.json").read_text(encoding="utf-8")) + assert data[FIELD] == [] + assert "without header channel names" not in (out / "discovery_report.txt").read_text() + assert RunInputs.open(out).images_without_header_channel_names() == {} + + +def test_an_undescribed_run_has_no_new_key_anywhere(tmp_path): + root, out, _ = _run(tmp_path, described=False) + with pytest.warns(UserWarning, match="No experiment description"): + assert run_experiment(root, MAPPING, output_folder=out).written + assert "experiment_description" not in json.loads((out / "run_manifest.json").read_text()) + assert FIELD not in json.loads((out / "discovery_report.json").read_text()) + assert "without header channel names" not in (out / "discovery_report.txt").read_text() + assert RunInputs.open(out).images_without_header_channel_names() is None + + +@pytest.mark.parametrize( + "bad", + [ + [], + "x", + [{"image": "a.tif"}], + [{"image": "a.tif", "header_names": "some", "channel_positions": [0]}], + [{"image": "a.tif", "header_names": "none", "channel_positions": [1, 0]}], + [{"image": "a.tif", "header_names": "none", "channel_positions": [True]}], + [{"image": "a.tif", "header_names": "none", "channel_positions": []}], + [ + {"image": "b.tif", "header_names": "none", "channel_positions": [0]}, + {"image": "a.tif", "header_names": "none", "channel_positions": [0]}, + ], + ], +) +def test_a_malformed_list_is_refused_on_read(tmp_path, bad): + root, out, _ = _run(tmp_path) + run_experiment(root, MAPPING, output_folder=out) + data = json.loads((out / "run_manifest.json").read_text(encoding="utf-8")) + data["experiment_description"]["units"][0][FIELD] = bad + (out / "run_manifest.json").write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(ManifestFormatError, match=FIELD): + read_manifest(out / "run_manifest.json") + + +def test_the_physical_report_gives_the_positions_per_unit(tmp_path): + root, _, by_unit = _run(tmp_path) + discovery = discover_experiment(root) + files = inspect_images(root, discovery.images).files + report = check_physical( + DESCRIPTION, discovery, files, roles={f.relative_path: "CYX" for f in files} + ) + assert report.ok + assert dict(report.unnamed) == { + UnitKey("DrugA", "rep1", "single"): {by_unit["DrugA/rep1"]: (1,)}, + UnitKey("CTRL", "rep2", "single"): {by_unit["CTRL/rep2"]: (0, 1)}, + }