diff --git a/CHANGELOG.md b/CHANGELOG.md index b390c0fc..2491af36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,22 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the ## Unreleased +- **The run side of the Z projection (part of #176).** A run on projected copies records a + `projection` summary in its manifest (method, slices requested, and for every image the + slices used, types, dtype rule, rounding and the source's content SHA-256, with the + conversion record it comes from), `RunInputs.projection` returns it (None for an unprojected + run, whose manifest is byte for byte as before), and the report says "max projections of + slices 1:4". 2-D masks next to projected images are paired as usual and the run warns + `masks_on_projected_images` (report and manifest); 3-D masks still stop the run + (`mask_planes_mismatch`). `RunInputs.open` refuses a manifest whose `projection` is not + what the conversion record gives, projected copies recorded without it, and a `projection` + without a record: nothing is read as a plain image. +- **`projection_record_missing` (owner decision, 8 November 2026).** `check` and `run` read the + OME `Image/Description` header (the first page's description tag, no pixels) of every + analysed TIFF the conversion record does not list as `projected`. A description that begins + "MorphoBricks Z projection:" in a tree with no record, or whose record does not list the + image as projected, is an error naming the images and saying to rerun `convert --project` + or restore the record. Plain images are unaffected. - **`A_*` error classes and `RunInputs.section_digests()` (Stage 2 design PR 2, owner answer 2).** `ChannelNotFoundError` now has `code = "A_SCOPE_INCOMPLETE"`: one rule and one code for a channel no image has and for one some images lack; the message names the image and ends with the code. diff --git a/docs/contract.md b/docs/contract.md index 8f4990c8..7b511199 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -217,6 +217,7 @@ interpretation. | `conversion_record_sha256` | SHA-256 of the conversion record (`.morphobricks-conversion.json`) in the experiment root when the experiment is a tree of named-axis copies (design v0.4 §5, §17); null otherwise. `RunInputs` checks it (owner decision of 2 November 2026, plan question 59): a record changed or gone since the run is refused, as for `calibration_sha256`. A changed record changes no row (the copies are checked by their fingerprints like any image). | | `calibration_sha256` | SHA-256 of the output folder's `calibration.csv` as the run read it (§9.3); null when there was none (or absent in manifests written before PP-8). `RunInputs` checks it. | | `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation; and, only for a unit that has such images, `images_without_header_channel_names`: a list sorted by `image` (relative path) of objects `image`, `header_names` (`none` when the header names no channel, `partial` when only some) and `channel_positions` (the zero-based C positions without a header name). Such an image passes the header checks, needs no evidence and the confirmed layout is used (§4.3); it is listed, not warned about, and the key is omitted for a unit whose images are all fully named; RGB units list nothing), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). | +| `projection` | Only for a run on a tree of Z-projected copies (the key is left out otherwise, so a run without projection is byte for byte as before; issue #176, run side). An object with exactly `method` (`max`, `mean` or `sum`), `z_range` (`[start, stop]` requested for the experiment, or null for every slice), `conversion_record` (`{file, sha256}`: the record these facts come from, the same SHA-256 as `conversion_record_sha256`), `source_root`, `images` (one per analysed image, by path: `relative_path`, `source_relative_path`, `source_content_sha256`, `source_z_size`, `z_range` (the slices used), `source_dtype`, `output_dtype`, `dtype_rule`, `rounding`, `significant_bits_dropped`), `masks_paired` (the masks paired with projected images, sorted) and `warnings` (`["masks_on_projected_images"]` exactly when `masks_paired` is not empty). A reader that knows nothing of the key ignores it, but it also refuses the conversion record version 3 the run names, so projected copies are never read by it as plain images; `manifest_version` is not changed. | | Version 1 | A manifest written before v0.4.0 (`"1"`) is still read, for reporting, but no row of it is reused: its axes were never checked. The next run recomputes every row ("the earlier run was written before v0.4.0, without the axis check"). Bricks refuse it. | | Row reuse | A file's rows are reused only when its row key is the earlier run's (`fingerprint.RowKey`, one rule for "the same file, read the same way", audit D5): its fingerprint (`fingerprint.Fingerprint`: size and modification time, and for an OME-Zarr image the inventory digest), labels, axes, roles, `singleton_roles`, `axes_from` and the path it is read from (so moving or converting the experiment recomputes it), and the settings that change rows (bins, histogram range, robust quantiles, range-check limit). **The identity rule is the fingerprint, which is metadata** (design v0.4 §1): its known blind spot is a file rewritten with the same size whose modification time (for an OME-Zarr image, every changed file's) is set back to its earlier value; its earlier rows are then reused, its staged rows resumed, and `RunInputs` accepts it. | | Content verification (on by default, owner 7 November 2026, #159; opt-in before) | Every run does it unless it opts out with `RunMode(name, verify_content=False)` (`run_experiment(mode=...)`, CLI `--no-verify-content`, the notebook's "Verify each image's content" box unticked); the opt-out is recorded as `settings.verify_content: false`. It closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (true without one: a first run, or a version 1 manifest); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the cost is in the pull request that made it the default (about 3 seconds per 1.5 GB read, measured at 1 GB/s of SHA-256). A run that verifies writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). | @@ -238,7 +239,7 @@ manifest's (also a `calibration.csv` added since the run), the conversion record on named-axis copies whose SHA-256 is not the manifest's `conversion_record_sha256` (changed or gone since the run; §17), a conversion record of version 1 (no copy SHA-256: convert again), a copy of a recorded image whose fingerprint or SHA-256 is not the record's (or that -the record does not list; PP-11c, issue #87), a run whose experiment (as recorded or +the record does not list; PP-11c, issue #87), a manifest whose `projection` is not exactly what the record gives for the recorded images and masks (or that has none for projected copies, or a `projection` without a conversion record: projected copies are never read as plain images; issue #176), a run whose experiment (as recorded or given), named-axis copies or output folder is inside its masks folder, or whose masks folder is inside its experiment (§15's folder rules, audit B1: a run recorded before them is refused as a new one would be), an image or mask changed or gone since the run, and a mask recorded without axes (a v0.4.0rc1 run wrote `axes: ""` for a @@ -250,7 +251,7 @@ The copies' SHA-256 is checked once, when the run is opened, for every image the run recorded (`conversion_check`): each copy is read whole then, so opening a run on named-axis copies costs one read of those copies (the excluded images' copies are not read). Checking on every read instead would read a copy whole for each of its planes. -Reads after `open` check the fingerprint, as for any image. +Reads after `open` check the fingerprint, as for any image. `RunInputs.projection` is the run's `RunProjection` (`method`, `z_range`, `conversion_record`, `source_root`, `images` and `image(relative_path)` with each image's slices used, types, rule and source SHA-256, `masks_paired`, `warnings`), or None for a run on images that are not projected copies, so a brick can carry the projection into its own records without reading the conversion record. It also refuses an experiment holding files the run neither recorded nor excluded (`UnrecordedFilesError`, a `RunInputsError`, PP-8b): images that preprocessing's own @@ -652,7 +653,7 @@ exclude=, missing_as_singleton=, memory_limit=, progress=)` (`conversion`), the | Repeat | A copy already in the tree is reused when its source (fingerprint and axes), roles, copy axes, pixel sizes, unknown sizes (roles and reasons), channel names, significant bits, the copy's fingerprint and the copy's SHA-256 (read whole) are as recorded; a new image is added; when every copy is reused the record is kept as it was (same SHA-256). Anything else (a source changed, other roles, a copy changed, an image no longer in the experiment, another experiment's copy tree, a file at a new image's place that the record does not list) is `copy_differs`: nothing is overwritten. A conversion record that cannot be read, or of version 1, is `conversion_record_unreadable`. | | Conversion record | `.morphobricks-conversion.json` in the copy tree's root (a dot-file, so discovery skips it), record version 3: `source_root`, `mapping`, `projection` (null, or `{method, z_range}`: the method and slice range of a tree of Z-projected copies, see "Projected copies" below), `missing_as_singleton`, `written_at`, `versions`, and per image `relative_path`, `action` (`converted` or `copied`), `reason` (the axis code, `excluded`, or null), `source` and `copy` fingerprints (`size`, `mtime_ns`), `copy_sha256` (the copy's SHA-256, computed in one pass after it is written; an OME-Zarr copy's is the SHA-256 of its files' relative paths and SHA-256s in path order, its `labels` group left out as for fingerprints: `content_hash`), `source_axes`, `roles`, `copy_axes`, `pixel_sizes` (by role, as §9), `unknown_sizes` (role: why), `channel_names` (as the copy holds them; `[]` for none), `significant_bits` (or null) and `projection` (null, or the projected copy's provenance, below). Sources keep the fingerprint rule (design v0.4 §1); the copies' SHA-256 is the owner's decision of 2 November 2026 (plan §12 question 59 (16)). Version 2 (PP-11c) added `copy_sha256`, `channel_names` and `significant_bits`; version 3 (issue #176) added the record-level `projection` and the entry's `projection`, and the action `projected`; a version 2 record is still read (a tree without a projection; it may not hold a projection field), and a version 3 record is refused by a reader that knows only version 2; a version 1 record (v0.4.0rc3 or earlier) is refused everywhere it is read (`convert`, the run's `review_conversion`, `RunInputs`) with a message to convert the originals again into a new folder: its copies cannot be checked. `read_conversion_record(folder)` reads it with its SHA-256. | | Projected copies (issue #176, record version 3) | A tree made with a Z projection holds, for every image, a 2-D copy of its Z-stack made with the record-level `projection` `{method, z_range}` (`max`, `mean` or `sum`; there is no default method; `z_range` is `[start, stop]`, zero-based, stop excluded, the same for every image, or null for every slice). An entry has the action `projected` and a `projection` object with exactly: `source_relative_path`, `source_content_sha256` (the source's content as read while writing), `source_z_axis` (`{letter, index}` of the header's Z axis, or null for an image without one, which is one slice), `source_z_size`, `z_range` (the slices actually used), `method`, `source_dtype`, `output_dtype`, `dtype_rule`, `rounding` (`none`, `float64, one division` or `float64 addition, z ascending`), `nan_policy` (`propagate`), `dropped_sizes` (`{"Z": {value, unit, source}}`, or `{"Z": null}` when the source gave no Z size: the Z size is never written to the copy) and `significant_bits_dropped` (a reason, or null: kept for `max`, dropped for `sum` and `mean`). Excluded images stay plain copies (`copied`, reason `excluded`). The output type depends only on the input type, the method and the slice count (never on the data): `max` keeps the type; `sum` of `uint8`/`uint16` is `uint32`, of `int8`/`int16` is `int32`, of `uint32`/`int32` is `float64` holding the exact integer, of floats is `float64`; `mean` is `float64`; 64-bit integers are refused (`cannot_convert_pixel_type`). A sum or mean whose worst case (the type's extreme times the slices) could not be held exactly is refused (`projection_cannot_be_exact`), never wrapped or clipped. A NaN makes its pixel NaN, infinities are kept. The reader refuses an entry whose output type or rounding is not the table's for its method, a projected entry in a record without a projection (and the reverse), and a method or range that is not the record's. The reuse key holds the projection (everything of it except the content SHA-256, which the write computes), so another method or range into the same tree is `copy_differs`. Masks are never projected: `convert_experiment(..., projection=Projection(...), masks_folder=)` scans the masks folder from the headers only and refuses, before writing anything, masks of the images to project that have depth slices (`projection_masks_3d`) or more than Y and X with axes that are not named (`projection_masks_axes_unknown`); 2-D masks are left alone. The other refusals, all before anything is written: `nothing_to_project` (no image has a Z role), `projection_z_range_outside`, `projection_cannot_be_exact`, `cannot_convert_pixel_type`, `projection_over_memory_limit` (one input plane, the running plane and, when it is another type, the output plane must fit `memory_limit`) and `projection_zarr_not_supported` (OME-Zarr images are not projected yet). The copy's OME-XML holds an `Image/Description` with the method, slices, source, source SHA-256 and types, and a UUID derived from it, so one projection of one source is the same bytes; the source's content SHA-256 is taken before and after the projection and the source fingerprint before and after, and a source that changed is refused. A sum that overflows to infinity (`projection_overflow`) or adds an infinity to its opposite (`projection_undefined`) stops the conversion and is never written. Excluded images are copied byte for byte. | -| The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and refuse a copy that changed (its fingerprint or its SHA-256: each copy is read whole, so `check` and `run` on a copy tree cost one more read of the copies), went missing or was added since (`copy_changed_after_conversion`: an error since PP-12, owner decision of 2 November 2026; the run stops after the header checks, before any pixel is read, writing only its report, since `RunInputs` would refuse the run; the message says to convert the originals again into a new folder); an unreadable record stops the run (`conversion_record_unreadable`). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`), is of version 1, or whose recorded images' copies no longer have the recorded fingerprint and SHA-256 (§9.1). | +| The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`, and for projected copies the `projection` summary (§9); the report says "max projections of slices 1:4 (2 images) of " (and `conversion.projection` in `discovery_report.json`). Masks are paired as usual (the image has no Z now, so 2-D masks fit; a mask with depth slices stays `mask_planes_mismatch` and stops the run), and the run warns `masks_on_projected_images`, naming the method, for masks paired with projections, in the report and the manifest's `projection.warnings`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and refuse a copy that changed (its fingerprint or its SHA-256: each copy is read whole, so `check` and `run` on a copy tree cost one more read of the copies), went missing or was added since (`copy_changed_after_conversion`: an error since PP-12, owner decision of 2 November 2026; the run stops after the header checks, before any pixel is read, writing only its report, since `RunInputs` would refuse the run; the message says to convert the originals again into a new folder); an unreadable record stops the run (`conversion_record_unreadable`). `check` and `run` also read the OME `Image/Description` header (first page's description tag only, no pixels) of every analysed TIFF the record does not list as `projected`: a description that begins "MorphoBricks Z projection:" in a tree with no record, or whose record does not list the image as projected, is refused as `projection_record_missing` (owner decision of 8 November 2026; plain images are unaffected). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`), is of version 1, or whose recorded images' copies no longer have the recorded fingerprint and SHA-256 (§9.1). | | Notebook (PP-11b) | Step 2's **Make named-axis copies (optional)** card, under the axis check, shown only when it reports images with `axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y` or `axes_not_named` for the roles being edited (design v0.4 §5: offered only for files with errors), and not with `folder_step(show_output=False)`. The roles are the user's: those confirmed in step 2, or the roles shown once the preview is checked and the card's own question is answered; any change of a role or of the missing-axes answer takes it back. A required folder for the copies (Browse…, or pasted) checked with the command's rule (`copy_folder_issues`); `preview_conversion` (`conversion_plan`, headers only, writes nothing) gives what will be converted and copied byte for byte (the command's `plan_conversion`), the space needed and the free space; then `convert_experiment` with a progress bar, under the copy tree's lock. Refusals use the existing cards (the lock card with **Remove the lock of a stopped run**, the rule of `--remove-stale-lock`). After the copies: the record's path, and **Check the copies in step 1**, which sets the experiment folder to the copy tree, keeps the output folder and checks again; the card says that the missing-axes answer must be given again and lists the copies whose relabelled sizes are unknown (`ConversionResult.unknown_sizes`), to state under **Pixel sizes**. | ## 18. The description draft (`morphobricks.draft.json`, Stage 1) diff --git a/docs/user_guide.md b/docs/user_guide.md index 3f17b2c7..d366ef97 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -929,7 +929,7 @@ they are. OME-Zarr images are not projected yet (`projection_zarr_not_supported` them). One folder holds one method and one slice range (`copy_differs` otherwise). The command prints the next commands with a new output folder (by default the copy folder's name with `_MorphoBricks_Preprocessing`): the projected copies are a different source, so the description starts again there (the -design and the stated pixel sizes are entered again). The notebook card follows. +design and the stated pixel sizes are entered again). A run on the copies records the projection in its manifest (`projection`: the method, the slices used, the source SHA-256 of every image) and the report says "max projections of slices 1:4". Masks next to projected images are used as they are, with the warning `masks_on_projected_images`: check that they were made on, or for, the projection. A run on projected copies reads the conversion record strictly: a record that is missing or does not match the run is refused, never read as a plain image. The notebook card follows. **In the notebook.** When step 2's axis check reports these images, a **Make named-axis copies (optional)** card appears under it, calling the same functions as the command: diff --git a/pyproject.toml b/pyproject.toml index 58b7d16b..5b766400 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -175,9 +175,12 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.copy_axes" = "pure" "morphobricks_preprocessing.projection" = "pure" "morphobricks_preprocessing.projection_record" = "pure" +"morphobricks_preprocessing.projection_run" = "pure" +"morphobricks_preprocessing.run_inputs_projection" = "pure" "morphobricks_preprocessing.projection_issues" = "pure" "morphobricks_preprocessing.projection_layout" = "pure" "morphobricks_preprocessing.projection_masks" = "io" +"morphobricks_preprocessing.projection_marker" = "io" "morphobricks_preprocessing.projection_plan" = "orchestration" "morphobricks_preprocessing.planned_copy" = "pure" "morphobricks_preprocessing.ome_tiff_copy" = "io" diff --git a/src/morphobricks_preprocessing/conversion_check.py b/src/morphobricks_preprocessing/conversion_check.py index f8917f27..963a3638 100644 --- a/src/morphobricks_preprocessing/conversion_check.py +++ b/src/morphobricks_preprocessing/conversion_check.py @@ -10,20 +10,28 @@ from __future__ import annotations -from collections.abc import Iterable +from collections.abc import Iterable, Mapping from pathlib import Path from .conversion_record import ( RECORD_FILENAME, + ConversionRecord, ConversionRecordError, changed_copies, read_record, ) from .manifest import file_sha256 +from .projection_run import run_projection from .validation import agree, plural -def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | None: +def conversion_problem( + root: Path, + expected: str | None, + paths: Iterable[str], + projection: Mapping | None = None, + masks: Iterable[tuple[str, str]] = (), +) -> str | None: """Return why a run on named-axis copies cannot be used, or None when it can. Parameters @@ -34,6 +42,11 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | The record's SHA-256 the run recorded. paths The images the run recorded, relative to ``root``. + projection + The manifest's ``projection`` entry, or None. It must be exactly what the record + gives for ``paths`` and ``masks``: projected copies are never read as plain images. + masks + ``(image, mask path)`` of the masks the run recorded. Returns ------- @@ -41,6 +54,8 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | The message for the user, or None. """ path = root / RECORD_FILENAME + if expected is None: + return None if projection is None else _no_record(path, root) if not path.exists() or file_sha256(path) != expected: return ( f"The conversion record {path.name} in {root} is not the one the run read " @@ -57,7 +72,7 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | ) changed = changed_copies(root, found[0], paths) if found is not None else [] if not changed: - return None + return _projection_problem(found[0], expected, list(paths), projection, masks) them = agree(changed, "it", "them") return ( f"{plural(len(changed), 'named-axis copy', 'named-axis copies')} changed since the " @@ -65,3 +80,35 @@ def conversion_problem(root: Path, expected: str, paths: Iterable[str]) -> str | f"{', '.join(changed[:5])}{' and more' if len(changed) > 5 else ''}. Convert the " "originals again into a new folder and run preprocessing on that one." ) + + +def _no_record(path: Path, root: Path) -> str: + """Say that a run recorded a projection but no conversion record to read it from.""" + return ( + f"The run recorded Z-projected images, but no conversion record {path.name} (a run " + f"records the record's SHA-256), so it is not known how the images in {root} were " + "made. Run preprocessing again on the projected copies." + ) + + +def _projection_problem( + record: ConversionRecord, + sha256: str, + paths: list[str], + recorded: Mapping | None, + masks: Iterable[tuple[str, str]], +) -> str | None: + """Return why the manifest's projection is not what the record says, or None. + + Projected copies recorded without their projection (or the reverse, or with other + slices, method, types or source SHA-256) are refused, never read as plain images. + """ + expected = run_projection(record, sha256, paths, masks) + if (None if expected is None else expected.to_dict()) == recorded: + return None + what = "no projection" if recorded is None else "a projection that differs from the record" + return ( + f"The run recorded {what}, but its conversion record says how these images were " + "made (Z-projected copies, or not): the manifest and the record disagree, so these " + "images are not read as plain images. Run preprocessing again on the projected copies." + ) diff --git a/src/morphobricks_preprocessing/conversion_record.py b/src/morphobricks_preprocessing/conversion_record.py index 15937524..e2272474 100644 --- a/src/morphobricks_preprocessing/conversion_record.py +++ b/src/morphobricks_preprocessing/conversion_record.py @@ -32,7 +32,9 @@ from .fingerprint import Fingerprint, current_fingerprint from .manifest import file_sha256 from .projection import Projection +from .projection_marker import marked_images, marker_issue from .projection_record import ProjectionEntry, projection_from_data, projection_to_data +from .projection_run import RunProjection, masks_issue, run_projection from .recorded_sizes import RecordedSize, sizes_from_data, sizes_to_data from .validation import ValidationIssue, error, plural, warning @@ -369,9 +371,15 @@ class ConversionReview: source_root: str | None issues: tuple[ValidationIssue, ...] = () record: str = RECORD_FILENAME + projection: RunProjection | None = None -def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> ConversionReview | None: +def review_conversion( + root: str | os.PathLike, + images: Iterable[str], + analysed: Iterable[str] | None = None, + masks: Iterable[tuple[str, str]] = (), +) -> ConversionReview | None: """Check a copy tree against its conversion record; None for an ordinary experiment. Parameters @@ -380,18 +388,26 @@ def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> Convers The experiment folder of the run. images The images discovery found in it, relative to it. + analysed + The images the run analyses (not excluded); by default ``images``. + masks + ``(image, mask path)`` of every mask found, for ``masks_on_projected_images``. Returns ------- ConversionReview or None - None when the folder holds no conversion record. + None when the folder holds no conversion record. For projected copies it carries + the :class:`.projection_run.RunProjection` and, with masks, the warning. """ + images = list(images) try: found = read_record(root) except ConversionRecordError as exc: return ConversionReview(None, None, (_unreadable(exc),)) - if found is None: - return None + analysed = images if analysed is None else list(analysed) + if found is None: # a projected copy needs its record: refuse it, nothing is guessed + marked = marked_images(root, analysed) + return ConversionReview(None, None, (marker_issue(marked),)) if marked else None record, sha256 = found sources = Path(record.source_root) changed = [ @@ -402,7 +418,14 @@ def review_conversion(root: str | os.PathLike, images: Iterable[str]) -> Convers listed = {e.relative_path for e in record.files} copies = sorted(set(changed_copies(root, record)) | (set(images) - listed)) issues = [*_source_issue(changed, record.source_root), *_copy_issue(copies)] - return ConversionReview(sha256, record.source_root, tuple(issues)) + done = {e.relative_path for e in record.files if e.action == PROJECTED} + marked = marked_images(root, [name for name in analysed if name not in done]) + if marked: + issues.append(marker_issue(marked)) + projection = run_projection(record, sha256, analysed, masks) + if projection is not None: + issues.extend(masks_issue(projection)) + return ConversionReview(sha256, record.source_root, tuple(issues), projection=projection) def changed_copies( diff --git a/src/morphobricks_preprocessing/discovery_report.py b/src/morphobricks_preprocessing/discovery_report.py index 9beab215..3477deb6 100644 --- a/src/morphobricks_preprocessing/discovery_report.py +++ b/src/morphobricks_preprocessing/discovery_report.py @@ -147,6 +147,8 @@ def text(self) -> str: lines.append(f" changed from {self.outcome.previous_root}") if data.get("conversion"): lines.append(f" named-axis copies of {data['conversion']['source_root']}") + if "projection" in data["conversion"]: + lines.extend(_projection_lines(data["conversion"]["projection"])) lines.extend(_folder_lines(data)) lines.extend(_image_lines(data)) lines.extend(_mask_lines(data)) @@ -225,11 +227,26 @@ def _conversion(pre: Preflight) -> dict | None: """Return the conversion record of a tree of named-axis copies, or None (design §5).""" if pre.conversion is None: return None - return { + found = pre.conversion.projection + data = { "record": pre.conversion.record, "sha256": pre.conversion.sha256, "source_root": pre.conversion.source_root, } + if found is not None: # the key is left out for copies that are not projected + data["projection"] = {**found.to_dict(), "summary": found.describe()} + return data + + +def _projection_lines(projection: dict) -> list[str]: + """Return the report lines of projected copies: the method and what masks are paired.""" + lines = [f" {projection['summary']} of {projection['source_root']}"] + if projection["masks_paired"]: + lines.append( + f" {len(projection['masks_paired'])} masks are paired with the {projection['method']} " + "projections (masks_on_projected_images; masks are never projected)" + ) + return lines def _issue_dict(issue: ValidationIssue) -> dict: diff --git a/src/morphobricks_preprocessing/manifest.py b/src/morphobricks_preprocessing/manifest.py index 81854be1..5046336c 100644 --- a/src/morphobricks_preprocessing/manifest.py +++ b/src/morphobricks_preprocessing/manifest.py @@ -1,23 +1,20 @@ """The run manifest: what a run read, with which settings, and what it wrote. -``run_manifest.json`` sits next to ``plane_histograms.csv``. It records the -confirmed folder labels and axis mapping, the settings, the package versions, -and each source file's fingerprint (size and modification time, and for an -OME-Zarr folder its inventory). The next run compares these with the experiment -folder to find new, changed and deleted files, and reads again only those -(``docs/plan.md`` §7). The fingerprint is metadata: a file rewritten with the same -size and its modification time set back is not seen as changed, unless the run -verifies content (``verify_content``: each source's SHA-256 is recorded and must -match). The format is described in ``docs/contract.md`` §9. +``run_manifest.json`` sits next to ``plane_histograms.csv``. It records the confirmed folder +labels and axis mapping, the settings, the package versions, and each source file's +fingerprint (size and modification time, and for an OME-Zarr folder its inventory). The next +run compares these with the experiment folder to find new, changed and deleted files, and +reads again only those (``docs/plan.md`` §7). The fingerprint is metadata: a file rewritten +with the same size and its modification time set back is not seen as changed, unless the run +verifies content (``verify_content``: each source's SHA-256 is recorded and must match). The +format is described in ``docs/contract.md`` §9. The manifest also records, for information only, each file's channel names and -significant bits when its metadata provides them, the SHA-256 of the -experiment design used, so later bricks can check that it was not changed, and -the image type the user confirmed, which later bricks normalize by, and the -limits of the intensity checks run on the table. It also records the histogram -range the table was binned with, and the planes whose central intensity range -was too narrow (follow-up P6), so a later run that keeps their rows still -reports them. +significant bits, the SHA-256 of the experiment design used (so later bricks can check it +was not changed), the image type the user confirmed (which later bricks normalize by), the +limits of the intensity checks, the histogram range the table was binned with, and the +planes whose central intensity range was too narrow (follow-up P6), so a later run that +keeps their rows still reports them. For projected copies it records the projection. """ from __future__ import annotations @@ -58,11 +55,14 @@ from .manifest_versions import MANIFEST_VERSION as MANIFEST_VERSION from .manifest_versions import STRICT_MANIFEST_VERSION as STRICT_MANIFEST_VERSION from .narrow_ranges import NarrowRange +from .projection_run import KEY as PROJECTION_KEY +from .projection_run import recorded_projection from .recorded_sizes import RecordedSize, sizes_from_data, sizes_to_data from .versions import package_versions as package_versions # re-exported (moved) MANIFEST_FILENAME = "run_manifest.json" AXES_FROM = ("header", "confirmed") +_LABEL_KEYS = ("folder", "condition", "biological_repeat", "technical_repeat") """Where a file's roles come from (:attr:`FileEntry.axes_from`).""" @@ -254,6 +254,8 @@ class RunManifest: experiment_description What a run with a published ``morphobricks.toml`` records about it (:mod:`.description_record`); None, and the key left out, for a run without one. + projection + The Z projection of the images (:mod:`.projection_run`); None (key left out) if none. schema_version Version of the CSV format. manifest_version @@ -280,6 +282,7 @@ class RunManifest: conversion_record_sha256: str | None = None calibration_sha256: str | None = None experiment_description: dict | None = None + projection: dict | None = None schema_version: str = SCHEMA_VERSION manifest_version: str = MANIFEST_VERSION @@ -339,12 +342,7 @@ def to_dict(self) -> dict: }, "axis_mapping": {"axes": self.mapping.axes, "roles": list(self.mapping.roles)}, "labels": [ - { - "folder": folder, - "condition": labels[0], - "biological_repeat": labels[1], - "technical_repeat": labels[2], - } + dict(zip(_LABEL_KEYS, (folder, *labels), strict=True)) for folder, labels in self.leaves.items() ], "files": [_file_dict(f, strict=self.manifest_version != "1") for f in self.files], @@ -365,6 +363,7 @@ def to_dict(self) -> dict: "versions": dict(self.versions), } optional = {"masks": self.masks, DESCRIPTION_KEY: self.experiment_description} + optional[PROJECTION_KEY] = self.projection data.update({key: value for key, value in optional.items() if value is not None}) return data @@ -425,6 +424,7 @@ def from_dict(cls, data: dict) -> RunManifest: conversion_record_sha256=optional_text(data.get("conversion_record_sha256")), calibration_sha256=optional_text(data.get("calibration_sha256")), experiment_description=recorded_description(data.get(DESCRIPTION_KEY)), + projection=recorded_projection(data.get(PROJECTION_KEY)), schema_version=str(data["schema_version"]), manifest_version=str(version), ) @@ -471,15 +471,9 @@ def file_entry_with( ) -> FileEntry: """Describe a source image with a fingerprint already taken, as :func:`file_entry_for`. - Parameters - ---------- - signature - ``(size, mtime_ns, inventory)`` (:func:`.inventory.source_signature`), for example - the one taken when its header was read. - relative_path, labels, mapping - As :func:`file_entry_for`. - **header - The keyword arguments of :func:`file_entry_for` (channel names, bits, ...). + ``signature`` is ``(size, mtime_ns, inventory)`` (:func:`.inventory.source_signature`), for + example the one taken when the header was read; ``relative_path``, ``labels``, ``mapping`` + and the ``header`` keywords (channel names, bits, ...) are those of :func:`file_entry_for`. Returns ------- diff --git a/src/morphobricks_preprocessing/projection_marker.py b/src/morphobricks_preprocessing/projection_marker.py new file mode 100644 index 00000000..aa34b065 --- /dev/null +++ b/src/morphobricks_preprocessing/projection_marker.py @@ -0,0 +1,68 @@ +"""Find projected copies by their OME header, for a tree whose record does not vouch for them. + +The writer (:func:`.ome_tiff_copy.write_projection`) gives every projected copy an OME +``Image/Description`` that begins with :data:`.projection_record.MARKER`. An image with that +description in a tree with no conversion record, or whose record does not list it as +projected, is refused (``projection_record_missing``): without the record it would pass as +a plain 2-D image and nothing would say it is a projection. + +Cost: one TIFF header read per image the record does not already list as projected (the +first page's description tag; no pixel is read). Nothing else is inferred from the header, +and an image that is not an OME-TIFF, or whose header cannot be read, is not marked. +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterable +from pathlib import Path +from xml.etree import ElementTree + +import tifffile + +from .projection_record import MARKER +from .validation import ValidationIssue, error, plural + +logger = logging.getLogger(__name__) + +_SUFFIXES = (".tif", ".tiff") + + +def is_projected_copy(path: Path) -> bool: + """Return True if the OME ``Image/Description`` of the TIFF at ``path`` is the marker's.""" + if path.suffix.lower() not in _SUFFIXES: + return False + try: + with tifffile.TiffFile(path) as tif: + text = tif.pages.first.description + root = ElementTree.fromstring(text) + except (OSError, ValueError, ElementTree.ParseError, tifffile.TiffFileError, AttributeError): + logger.debug("Cannot read the description of %s", path, exc_info=True) + return False + for image in root: + if image.tag.rsplit("}", 1)[-1] != "Image": + continue + for child in image: + if child.tag.rsplit("}", 1)[-1] == "Description": + return (child.text or "").startswith(MARKER) + return False + + +def marked_images(root: str | Path, images: Iterable[str]) -> list[str]: + """Return the images (relative to ``root``) whose header says they are projected copies.""" + return sorted(name for name in images if is_projected_copy(Path(root) / name)) + + +def marker_issue(images: list[str]) -> ValidationIssue: + """Refuse projected copies that no conversion record lists as projected.""" + first = images[0] + return error( + "projection_record_missing", + f"{plural(len(images), 'image')} (first: {first}) say in their OME description that " + "they are Z-projected copies, but the folder has no conversion record that lists " + f"{'it' if len(images) == 1 else 'them'} as projected, so it is not known how they were " + "made and they are not read as plain images.", + images, + "Run `convert --project` again from the original images into a new folder, or restore " + "the conversion record (.morphobricks-conversion.json) that was written with these copies.", + ) diff --git a/src/morphobricks_preprocessing/projection_record.py b/src/morphobricks_preprocessing/projection_record.py index 7e0783c4..2885ebcb 100644 --- a/src/morphobricks_preprocessing/projection_record.py +++ b/src/morphobricks_preprocessing/projection_record.py @@ -215,6 +215,10 @@ def from_dict(cls, data: Mapping) -> ProjectionEntry: ) +MARKER = "MorphoBricks Z projection:" +"""How the OME ``Image/Description`` of a projected copy begins (:mod:`.projection_marker`).""" + + def projection_description(entry: ProjectionEntry) -> str: """Return the text written as the copy's OME-XML ``Image/Description``. @@ -224,7 +228,7 @@ def projection_description(entry: ProjectionEntry) -> str: """ start, stop = entry.z_range return ( - f"MorphoBricks Z projection: {entry.method} of slices {start}:{stop} " + f"{MARKER} {entry.method} of slices {start}:{stop} " f"(of {entry.source_z_size}) of {entry.source_relative_path}; " f"source content SHA-256 {entry.source_content_sha256}; " f"{entry.source_dtype} to {entry.output_dtype} ({entry.dtype_rule}; " diff --git a/src/morphobricks_preprocessing/projection_run.py b/src/morphobricks_preprocessing/projection_run.py new file mode 100644 index 00000000..a6d2a377 --- /dev/null +++ b/src/morphobricks_preprocessing/projection_run.py @@ -0,0 +1,304 @@ +"""What a run records about Z-projected copies (issue #176, run side). + +A run on a tree of projected copies (:mod:`.projection_record`) copies a ``projection`` +summary into its manifest, so a later brick can carry it into its own records without +reading the conversion record: the method, the slices requested, and for each image the +slices used, the types and the rule, the source's content SHA-256, and which conversion +record it came from. A run without projected copies has no such key: its manifest is +byte for byte what it was. + +Masks are never projected (design §7): a 2-D mask next to a projected image is paired as +usual, and the run records and reports ``masks_on_projected_images``, which names the +method. Everything here is pure data, read strictly: a summary that does not match the +conversion record it names is refused by :class:`.run_inputs.RunInputs`. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from .projection import METHODS +from .validation import ValidationIssue, plural, warning + +if TYPE_CHECKING: # the record module imports this one + from .conversion_record import ConversionRecord, CopyEntry + +KEY = "projection" +"""The manifest's key; absent for a run without projected copies.""" +MASKS_WARNING = "masks_on_projected_images" +"""The warning code: masks drawn on (or for) the unprojected images are paired with copies.""" + +_SHA256 = re.compile(r"[0-9a-f]{64}") +_IMAGE_KEYS = ( + "relative_path", + "source_relative_path", + "source_content_sha256", + "source_z_size", + "z_range", + "source_dtype", + "output_dtype", + "dtype_rule", + "rounding", + "significant_bits_dropped", +) +_KEYS = frozenset( + {"method", "z_range", "conversion_record", "source_root", "images", "masks_paired", "warnings"} +) + + +@dataclass(frozen=True) +class ProjectedImage: + """How one analysed image was projected, as the run recorded it. + + Attributes + ---------- + relative_path + The projected copy, relative to the experiment root (the run's image). + source_relative_path + The source image it was made from, relative to the source root. + source_content_sha256 + The SHA-256 of the source's content as read when the copy was written. + source_z_size + The number of slices the source had. + z_range + ``(start, stop)``, the slices used (zero-based, stop excluded). + source_dtype, output_dtype + The pixel types read and written. + dtype_rule, rounding + How the output type was decided and how the values were rounded. + significant_bits_dropped + Why the source's significant bits are not kept, or None when they are. + """ + + relative_path: str + source_relative_path: str + source_content_sha256: str + source_z_size: int + z_range: tuple[int, int] + source_dtype: str + output_dtype: str + dtype_rule: str + rounding: str + significant_bits_dropped: str | None + + def to_dict(self) -> dict: + """Return the image as JSON-compatible data.""" + return {**{k: getattr(self, k) for k in _IMAGE_KEYS}, "z_range": list(self.z_range)} + + @classmethod + def from_dict(cls, data: object) -> ProjectedImage: + """Read an image written by :meth:`to_dict`, refusing anything else.""" + if not isinstance(data, Mapping) or set(data) != set(_IMAGE_KEYS): + raise ValueError(f"a projected image needs exactly the keys {list(_IMAGE_KEYS)}") + numbers = ("z_range", "source_z_size", "significant_bits_dropped") + bounds, size = data["z_range"], data["source_z_size"] + dropped = data["significant_bits_dropped"] + if any(not isinstance(data[k], str) for k in _IMAGE_KEYS if k not in numbers): + raise ValueError(f"{data.get('relative_path')!r}: a projected image field is not text") + if not _SHA256.fullmatch(str(data["source_content_sha256"])): + raise ValueError(f"{data['relative_path']!r}: the source SHA-256 is not a SHA-256") + ints = isinstance(bounds, list) and len(bounds) == 2 and all(type(b) is int for b in bounds) + if not ints or type(size) is not int or not 0 <= bounds[0] < bounds[1] <= size: + raise ValueError(f"{data['relative_path']!r}: the slice range does not fit the size") + if dropped is not None and not isinstance(dropped, str): + raise ValueError(f"{data['relative_path']!r}: significant_bits_dropped is not text") + fields = {k: data[k] for k in _IMAGE_KEYS} + return cls(**{**fields, "z_range": (bounds[0], bounds[1])}) + + +@dataclass(frozen=True) +class RunProjection: + """The projection of a run's images (``RunInputs.projection``; None for no projection). + + Attributes + ---------- + method + ``"max"``, ``"mean"`` or ``"sum"``: the one method of the copy tree. + z_range + The slice range asked for (every image), or None for every slice. The slices each + image used are in :attr:`images`. + conversion_record + The conversion record's ``(file name, SHA-256)``, the source of these facts. + source_root + The experiment folder the copies were made from. + images + One entry per analysed image, sorted by path. + masks_paired + The masks paired with projected images, sorted (never projected themselves). + warnings + The recorded warning codes: :data:`MASKS_WARNING` when masks are paired. + """ + + method: str + z_range: tuple[int, int] | None + conversion_record: tuple[str, str] + source_root: str + images: tuple[ProjectedImage, ...] + masks_paired: tuple[str, ...] = () + warnings: tuple[str, ...] = () + + def image(self, relative_path: str) -> ProjectedImage | None: + """Return how an analysed image was projected, or None if it is not listed.""" + return next((i for i in self.images if i.relative_path == relative_path), None) + + def describe(self) -> str: + """Return the report line, e.g. ``max projections of slices 1:4 (2 images)``.""" + used = {i.z_range for i in self.images} + shown = self.z_range or (next(iter(used)) if len(used) == 1 else None) + slices = "all slices" if shown is None else f"slices {shown[0]}:{shown[1]}" + return f"{self.method} projections of {slices} ({plural(len(self.images), 'image')})" + + def to_dict(self) -> dict: + """Return the summary as the manifest's ``projection`` object.""" + name, sha256 = self.conversion_record + return { + "method": self.method, + "z_range": None if self.z_range is None else list(self.z_range), + "conversion_record": {"file": name, "sha256": sha256}, + "source_root": self.source_root, + "images": [i.to_dict() for i in self.images], + "masks_paired": list(self.masks_paired), + "warnings": list(self.warnings), + } + + @classmethod + def from_dict(cls, data: object) -> RunProjection: + """Read the manifest's ``projection`` object strictly. + + Raises + ------ + ValueError + For missing or extra keys, an unknown method, a bad range or SHA-256, or a + warning list that does not match the masks listed. + """ + if not isinstance(data, Mapping) or set(data) != _KEYS: + raise ValueError(f"{KEY} must hold exactly {', '.join(sorted(_KEYS))}") + bounds, record = data["z_range"], data["conversion_record"] + if data["method"] not in METHODS: + raise ValueError(f"{KEY}.method {data['method']!r} is not max, mean or sum") + if bounds is not None and not (isinstance(bounds, list) and len(bounds) == 2): + raise ValueError(f"{KEY}.z_range is not null or two numbers") + if not isinstance(record, Mapping) or set(record) != {"file", "sha256"}: + raise ValueError(f"{KEY}.conversion_record must be {{file, sha256}}") + if not isinstance(record["file"], str) or not _SHA256.fullmatch(str(record["sha256"])): + raise ValueError(f"{KEY}.conversion_record is not a file name and a SHA-256") + masks, warns = data["masks_paired"], data["warnings"] + if not isinstance(masks, list) or not all(isinstance(m, str) for m in masks): + raise ValueError(f"{KEY}.masks_paired is not a list of paths") + if warns != ([MASKS_WARNING] if masks else []) or not isinstance(data["source_root"], str): + raise ValueError( + f"{KEY}.warnings is not {MASKS_WARNING!r} exactly when masks are paired" + ) + if not isinstance(data["images"], list): + raise ValueError(f"{KEY}.images is not a list") + return cls( + method=data["method"], + z_range=None if bounds is None else (bounds[0], bounds[1]), + conversion_record=(record["file"], record["sha256"]), + source_root=data["source_root"], + images=tuple(ProjectedImage.from_dict(i) for i in data["images"]), + masks_paired=tuple(masks), + warnings=tuple(warns), + ) + + +def recorded_projection(value: object) -> dict | None: + """Return the manifest's ``projection`` entry as read (None when absent), strictly. + + Raises + ------ + ValueError + When the entry is not a valid :class:`RunProjection`. + """ + if value is None: + return None + RunProjection.from_dict(value) + return dict(value) # type: ignore[arg-type] + + +def run_projection( + record: ConversionRecord, + sha256: str, + analysed: Iterable[str], + masks: Iterable[tuple[str, str]] = (), + file_name: str = ".morphobricks-conversion.json", +) -> RunProjection | None: + """Summarise the projection a run analyses, from the conversion record it reads. + + Parameters + ---------- + record + The tree's conversion record. + sha256 + The record file's SHA-256 (what the manifest names). + analysed + The images the run analyses, relative to the copy tree. + masks + ``(image, mask path)`` of every mask found for the images. + file_name + The record's file name. + + Returns + ------- + RunProjection or None + None when the record holds no projection. + """ + if record.projection is None: + return None + wanted = set(analysed) + entries = sorted( + (e for e in record.files if e.projection is not None and e.relative_path in wanted), + key=lambda e: e.relative_path, + ) + images = tuple(_image(e) for e in entries) + projected = {i.relative_path for i in images} + paired = tuple(sorted({path for image, path in masks if image in projected})) + return RunProjection( + method=record.projection.method, + z_range=record.projection.z_range, + conversion_record=(file_name, sha256), + source_root=record.source_root, + images=images, + masks_paired=paired, + warnings=(MASKS_WARNING,) if paired else (), + ) + + +def masks_issue(projection: RunProjection) -> list[ValidationIssue]: + """Return the ``masks_on_projected_images`` warning, or nothing when no mask is paired.""" + if not projection.masks_paired: + return [] + method, count = projection.method, len(projection.masks_paired) + return [ + warning( + MASKS_WARNING, + f"{plural(count, 'mask')} {'is' if count == 1 else 'are'} paired with images that " + f"are {method} projections of Z-stacks. Masks are never projected: they are used " + f"as they are, and nothing says they were drawn on the {method} projection. Each " + "is a 2-D mask; a label image has no meaningful maximum, mean or sum, so a mask " + "made on the stack would not describe these images.", + projection.masks_paired, + f"Check that the masks were made on, or for, the {method} projections. The " + "run records this warning and the method in the manifest.", + ) + ] + + +def _image(entry: CopyEntry) -> ProjectedImage: + """Return the run's record of one projected copy.""" + done = entry.projection + return ProjectedImage( + relative_path=entry.relative_path, + source_relative_path=done.source_relative_path, + source_content_sha256=done.source_content_sha256, + source_z_size=done.source_z_size, + z_range=done.z_range, + source_dtype=done.source_dtype, + output_dtype=done.output_dtype, + dtype_rule=done.dtype_rule, + rounding=done.rounding, + significant_bits_dropped=done.significant_bits_dropped, + ) diff --git a/src/morphobricks_preprocessing/run/pipeline.py b/src/morphobricks_preprocessing/run/pipeline.py index ba3fe5f3..011f2b45 100644 --- a/src/morphobricks_preprocessing/run/pipeline.py +++ b/src/morphobricks_preprocessing/run/pipeline.py @@ -502,6 +502,7 @@ def _write_generation( narrow_ranges=tuple(rows.narrow), calibration_sha256=pre.calibration.sha256, conversion_record_sha256=None if pre.conversion is None else pre.conversion.sha256, + projection=_projection(pre), experiment_description=( None if described is None @@ -513,6 +514,12 @@ def _write_generation( return file_sha256(write_manifest(manifest, staged / MANIFEST_FILENAME)) +def _projection(pre: Preflight) -> dict | None: + """Return the manifest's ``projection`` entry for projected copies, else None.""" + found = None if pre.conversion is None else pre.conversion.projection + return None if found is None else found.to_dict() + + def _kept_time(manifest: RunManifest, earlier: RunManifest | None) -> RunManifest: """Return the manifest with the earlier run's ``written_at`` when nothing else changed. diff --git a/src/morphobricks_preprocessing/run/preflight.py b/src/morphobricks_preprocessing/run/preflight.py index ccbe2c88..b9d8ccdf 100644 --- a/src/morphobricks_preprocessing/run/preflight.py +++ b/src/morphobricks_preprocessing/run/preflight.py @@ -26,7 +26,7 @@ from ..discovery import DiscoveryResult, default_output_folder, discover_experiment from ..fingerprint import Fingerprint, current_fingerprint from ..mask_names import MASK_NAME_CODES, mask_name_issues -from ..masks import MaskInventory, find_masks +from ..masks import MaskFile, MaskInventory, find_masks from ..metadata import MetadataReport, check_metadata, inspect_image, inspect_images from ..name_issues import non_nfc_folders_issue from ..pixel_size_checks import PIXEL_SIZE_CODES, pixel_size_issues @@ -246,13 +246,20 @@ def preflight( dataset=check_dataset(discovery, excluded=excluded), masks=masks, calibration=calibration, - conversion=review_conversion(root, discovery.images), + conversion=review_conversion( + root, discovery.images, images, [(m.image, m.path) for m in _mask_files(masks)] + ), header_fingerprints=bound, ) report("acquisitions", 1, 1, "") return result +def _mask_files(masks: MaskInventory | None) -> tuple[MaskFile, ...]: + """Return the masks found for the analysed images (none when masks were not asked for).""" + return () if masks is None else masks.files + + def _no_progress(phase: str, done: int, total: int, path: str) -> None: """Report nothing (the default of :func:`preflight`'s ``progress``).""" diff --git a/src/morphobricks_preprocessing/run_inputs.py b/src/morphobricks_preprocessing/run_inputs.py index 1b8833ba..e629fcc6 100644 --- a/src/morphobricks_preprocessing/run_inputs.py +++ b/src/morphobricks_preprocessing/run_inputs.py @@ -62,6 +62,7 @@ from .run.generation import published_path, read_stable from .run_inputs_error import RunInputsError as RunInputsError # re-exported (moved) from .run_inputs_image import LazyImage as LazyImage # re-exported (moved) +from .run_inputs_projection import ProjectionInputs from .run_inputs_semantic import SemanticInputs, check_description, with_identity from .run_inputs_text import ( _ZARR_MISSING, @@ -149,7 +150,7 @@ class Fov: @dataclass(frozen=True) -class RunInputs(SemanticInputs): +class RunInputs(SemanticInputs, ProjectionInputs): """The inputs of a clean preprocessing run, read as the run recorded them. Use :meth:`open`; the attributes are the snapshot taken then. @@ -527,10 +528,14 @@ def _check_calibration(path: Path, expected: str | None) -> None: def _check_conversion(root: Path, manifest: RunManifest) -> None: """Refuse a changed or gone conversion record, or a copy changed since it was written.""" - if manifest.conversion_record_sha256 is None: - return paths = [e.relative_path for e in manifest.files] - problem = conversion_problem(root, manifest.conversion_record_sha256, paths) + problem = conversion_problem( + root, + manifest.conversion_record_sha256, + paths, + manifest.projection, + [(m["image"], m["path"]) for m in (manifest.masks or {}).get("files", [])], + ) if problem is not None: raise RunInputsError(problem) diff --git a/src/morphobricks_preprocessing/run_inputs_projection.py b/src/morphobricks_preprocessing/run_inputs_projection.py new file mode 100644 index 00000000..45cc9e1d --- /dev/null +++ b/src/morphobricks_preprocessing/run_inputs_projection.py @@ -0,0 +1,30 @@ +"""The Z projection of a run for :class:`.run_inputs.RunInputs` (issue #176, run side). + +A run on projected copies records a ``projection`` summary in its manifest +(:mod:`.projection_run`). :meth:`.run_inputs.RunInputs.open` checks it against the conversion +record the run names (:func:`.conversion_check.conversion_problem`): the summary must be +exactly what the record says for the recorded images and masks, so projected copies are never +read as plain images and a record changed by hand is refused. +""" + +from __future__ import annotations + +from .manifest import RunManifest +from .projection_run import RunProjection + + +class ProjectionInputs: + """Methods of ``RunInputs`` that need the projection.""" + + manifest: RunManifest + + @property + def projection(self) -> RunProjection | None: + """Return how the run's images were projected, or None for an unprojected run. + + CellPatcher and MorphoFeature carry this into their own records (method, slices + used per image, source SHA-256, the conversion record's SHA-256, and the warning + ``masks_on_projected_images`` when masks are paired with the projections). + """ + recorded = self.manifest.projection + return None if recorded is None else RunProjection.from_dict(recorded) diff --git a/tests/test_projection_marker.py b/tests/test_projection_marker.py new file mode 100644 index 00000000..dddfd5bd --- /dev/null +++ b/tests/test_projection_marker.py @@ -0,0 +1,82 @@ +"""Projected copies whose conversion record is gone are refused (owner decision, 2026-11-08). + +The regression is ``test_projected_copies_without_their_record_are_refused``: before, a copy +tree whose record was deleted looked like plain 2-D images, and ``check`` and ``run`` accepted +it with no trace that it was a projection. +""" + +import json + +import numpy as np +import pytest +import tifffile + +from morphobricks_preprocessing import AxisMapping, Projection, convert_experiment, run_experiment +from morphobricks_preprocessing.conversion_record import RECORD_FILENAME +from morphobricks_preprocessing.run.preflight import preflight +from morphobricks_preprocessing.testing import _confirmed_design +from test_projection_convert import NAMES, ZCYX, codes, stack_experiment + +CYX = AxisMapping("CYX", tuple("CYX")) +CODE = "projection_record_missing" +pytestmark = pytest.mark.filterwarnings("ignore:No experiment description:UserWarning") + + +def projected_tree(tmp_path): + exp, copies = tmp_path / "exp", tmp_path / "copies" + stack_experiment(exp) + done = convert_experiment(exp, ZCYX, copies, projection=Projection("max", None)) + assert done.written, done.issues + return copies + + +def test_projected_copies_without_their_record_are_refused(tmp_path): + copies = projected_tree(tmp_path) + assert CODE not in codes(preflight(copies).issues) + (copies / RECORD_FILENAME).unlink() + pre = preflight(copies) + assert not pre.ok and CODE in codes(pre.issues) + issue = next(i for i in pre.issues if i.code == CODE) + assert set(issue.paths) == set(NAMES) and "convert --project" in issue.action + out = tmp_path / "out" + run = run_experiment(copies, CYX, output_folder=out, design=_confirmed_design(copies)) + assert ( + not run.written and CODE in codes(run.issues) and not (out / "run_manifest.json").exists() + ) + + +def test_a_record_that_lists_the_image_as_copied_does_not_vouch_for_it(tmp_path): + copies = projected_tree(tmp_path) + path = copies / RECORD_FILENAME + record = json.loads(path.read_text(encoding="utf-8")) + record["files"][0].update(action="copied", projection=None) + path.write_text(json.dumps(record), encoding="utf-8") + pre = preflight(copies) + assert not pre.ok + assert CODE in codes(pre.issues) or "conversion_record_unreadable" in codes(pre.issues) + + +def test_a_plain_ome_tiff_without_the_marker_is_unaffected(tmp_path): + folder = tmp_path / "plain" + for name in NAMES: + data = np.zeros((3, 8, 9), np.uint16) + (folder / name).parent.mkdir(parents=True, exist_ok=True) + tifffile.imwrite( + folder / name, data, ome=True, photometric="minisblack", + metadata={"axes": "CYX", "Description": "an ordinary description"}, + ) # fmt: skip + pre = preflight(folder) + assert CODE not in codes(pre.issues) and pre.conversion is None + + +def test_the_check_reads_no_pixels(tmp_path, monkeypatch): + copies = projected_tree(tmp_path) + (copies / RECORD_FILENAME).unlink() + + def no_pixels(self, *args, **kwargs): + raise AssertionError("a pixel read") + + monkeypatch.setattr(tifffile.TiffPage, "asarray", no_pixels) + from morphobricks_preprocessing.projection_marker import marked_images + + assert marked_images(copies, NAMES) == sorted(NAMES) diff --git a/tests/test_run_projection.py b/tests/test_run_projection.py new file mode 100644 index 00000000..384cb049 --- /dev/null +++ b/tests/test_run_projection.py @@ -0,0 +1,178 @@ +"""The run side of the Z projection (issue #176): manifest, ``RunInputs``, warning, report. + +The regression is ``test_a_run_on_projected_copies_records_and_exposes_the_projection``: +before, the manifest had no ``projection`` key and ``RunInputs`` had no ``projection``, so a +brick reading the run saw plain 2-D images with no trace of the method or the source. +""" + +import json + +import numpy as np +import pytest + +from morphobricks_preprocessing import ( + AxisMapping, + Projection, + RunInputs, + convert_experiment, + run_experiment, +) +from morphobricks_preprocessing.conversion_record import RECORD_FILENAME +from morphobricks_preprocessing.manifest import ManifestFormatError, read_manifest +from morphobricks_preprocessing.projection_run import MASKS_WARNING, RunProjection +from morphobricks_preprocessing.run_inputs_error import RunInputsError +from morphobricks_preprocessing.testing import _confirmed_design +from test_projection_convert import ( + NAMES, + ZCYX, + codes, + stack_experiment, + write_mask, + write_stack, +) + +CYX = AxisMapping("CYX", tuple("CYX")) +pytestmark = pytest.mark.filterwarnings("ignore:No experiment description:UserWarning") + + +def projected_run(tmp_path, *, method="max", z_range=None, masks=False): + """Project two stacks, run on the copies; return (copies, output, masks folder, result).""" + exp, copies, out = tmp_path / "exp", tmp_path / "copies", tmp_path / "out" + stack_experiment(exp) + folder = tmp_path / "masks" + if masks: + for name in NAMES: + write_mask(folder / "Nuclei" / name, (8, 9), "YX") + done = convert_experiment(exp, ZCYX, copies, projection=Projection(method, z_range), + masks_folder=folder if masks else None) # fmt: skip + assert done.written, done.issues + run = run_experiment(copies, CYX, output_folder=out, design=_confirmed_design(copies), + masks_folder=folder if masks else None) # fmt: skip + assert run.written, run.issues + return copies, out, folder, run + + +def manifest_of(out): + return json.loads((out / "run_manifest.json").read_text(encoding="utf-8")) + + +def test_a_run_on_projected_copies_records_and_exposes_the_projection(tmp_path): + copies, out, _, run = projected_run(tmp_path, method="mean", z_range=(1, 4)) + record = json.loads((copies / RECORD_FILENAME).read_text(encoding="utf-8")) + data = manifest_of(out)["projection"] + assert data["method"] == "mean" and data["z_range"] == [1, 4] + assert data["conversion_record"]["sha256"] == manifest_of(out)["conversion_record_sha256"] + assert [i["relative_path"] for i in data["images"]] == sorted(NAMES) + assert {tuple(i["z_range"]) for i in data["images"]} == {(1, 4)} + entry = record["files"][0]["projection"] + image = next( + i for i in data["images"] if i["relative_path"] == record["files"][0]["relative_path"] + ) + for key in ("source_content_sha256", "dtype_rule", "rounding", "source_dtype", "output_dtype"): + assert image[key] == entry[key] + assert data["masks_paired"] == [] and data["warnings"] == [] + inputs = RunInputs.open(out) + assert isinstance(inputs.projection, RunProjection) and inputs.projection.method == "mean" + assert inputs.projection.image(NAMES[0]).z_range == (1, 4) + assert inputs.projection.to_dict() == data + assert "mean projections of slices 1:4 (2 images)" in (out / "discovery_report.txt").read_text() + assert "projection" in json.loads((out / "discovery_report.json").read_text())["conversion"] + assert run.issues == () or MASKS_WARNING not in codes(run.issues) + + +def test_a_run_without_projection_has_no_projection_key(tmp_path): + """Plain images, and named-axis copies that are not projections, record no projection.""" + exp = tmp_path / "exp" + for name in NAMES: + data = np.arange(3 * 8 * 9, dtype=np.uint16).reshape(3, 8, 9) + write_stack(exp / name, data, "CYX") + for label, root in (("plain", exp), ("copies", tmp_path / "copies")): + if root != exp: + done = convert_experiment(exp, AxisMapping("CYX", tuple("CYX")), root) + assert done.written, done.issues + out = tmp_path / f"out_{label}" + run = run_experiment(root, CYX, output_folder=out, design=_confirmed_design(root)) + assert run.written, run.issues + assert "projection" not in manifest_of(out) + assert RunInputs.open(out).projection is None + + +def test_2d_masks_on_projected_images_record_the_warning(tmp_path): + _, out, _, run = projected_run(tmp_path, masks=True) + warned = [i for i in run.issues if i.code == MASKS_WARNING] + assert len(warned) == 1 and not warned[0].is_error + assert "max projections" in warned[0].message and len(warned[0].paths) == 2 + data = manifest_of(out)["projection"] + assert data["warnings"] == [MASKS_WARNING] and len(data["masks_paired"]) == 2 + assert RunInputs.open(out).projection.warnings == (MASKS_WARNING,) + text = (out / "discovery_report.txt").read_text(encoding="utf-8") + assert MASKS_WARNING in text and "masks are never projected" in text + + +def test_3d_masks_next_to_projected_copies_stop_the_run(tmp_path): + copies, out, masks, _ = projected_run(tmp_path) + write_mask(masks / "Nuclei" / NAMES[0], (5, 8, 9), "ZYX") + stopped = run_experiment(copies, CYX, output_folder=tmp_path / "out2", masks_folder=masks, + design=_confirmed_design(copies)) # fmt: skip + assert not stopped.written and "mask_planes_mismatch" in codes(stopped.issues) + + +def edit(path, change): + data = json.loads(path.read_text(encoding="utf-8")) + change(data) + path.write_text(json.dumps(data, indent=2), encoding="utf-8") + + +def test_a_manifest_that_disagrees_with_the_record_is_refused(tmp_path): + _, out, _, _ = projected_run(tmp_path) + manifest = out / "run_manifest.json" + original = manifest.read_bytes() + for change in ( + lambda d: d["projection"].__setitem__("method", "sum"), + lambda d: d["projection"]["images"][0].__setitem__("z_range", [0, 4]), + lambda d: d["projection"]["images"][0].__setitem__("source_content_sha256", "0" * 64), + lambda d: d["projection"]["images"].pop(), + lambda d: d.pop("projection"), # projected copies read as plain images + ): + edit(manifest, change) + with pytest.raises(RunInputsError): # (the table is checked first: same file bytes) + RunInputs.open(out) + manifest.write_bytes(original) + assert RunInputs.open(out).projection.method == "max" + + +def test_a_projection_without_its_record_is_refused(tmp_path): + _, out, _, _ = projected_run(tmp_path) + edit(out / "run_manifest.json", lambda d: d.__setitem__("conversion_record_sha256", None)) + with pytest.raises(RunInputsError, match="no conversion record"): + RunInputs.open(out) + + +def test_a_missing_or_changed_record_is_refused(tmp_path): + copies, out, _, _ = projected_run(tmp_path) + record = copies / RECORD_FILENAME + original = record.read_bytes() # bytes: write_text would turn LF into CRLF on Windows + edit(record, lambda d: d["projection"].__setitem__("method", "sum")) + with pytest.raises(RunInputsError, match="not the one the run read"): + RunInputs.open(out) + record.unlink() + with pytest.raises(RunInputsError, match="not the one the run read"): + RunInputs.open(out) + record.write_bytes(original) + assert RunInputs.open(out).projection is not None + + +@pytest.mark.parametrize( + "change", + [ + lambda d: d["projection"].__setitem__("method", "median"), + lambda d: d["projection"].__setitem__("extra", 1), + lambda d: d["projection"].__setitem__("warnings", [MASKS_WARNING]), + lambda d: d["projection"]["images"][0].__setitem__("source_content_sha256", "abc"), + ], +) +def test_a_malformed_projection_in_the_manifest_is_a_format_error(tmp_path, change): + _, out, _, _ = projected_run(tmp_path) + edit(out / "run_manifest.json", change) + with pytest.raises(ManifestFormatError): + read_manifest(out / "run_manifest.json")