diff --git a/AGENTS.md b/AGENTS.md index 0af91cd..46339ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -176,7 +176,22 @@ environment.yaml CellPatcher-specific conventions: -* `image_axes`/`mask_axes` are explicit and axis-aware (see `axes.py`). +* **File runs read through a clean preprocessing run** (v0.2.0, CP-2a; design + v0.4 §6): `run_patching_pipeline`, `run_patching_condition_repeat_tree` and + `run_patching_from_folders` need `common.preprocessing_run` (else + `no_preprocessing_run` with the preprocessing command). `cellpatcher.file_runs` + opens it with `RunInputs.open` with the image folder/root and masks folder + given (a moved or copied experiment is checked and read where it is now), + patches the run's FOVs with their recorded keys, and reads images and masks + with `RunInputs.read_plane`/`read_mask` (`cellpatcher.run_reading`) in one + canonical layout per run (T, Z, C, then Y, X). No CellPatcher-side axis, + pairing or calibration logic is used for them; `common.image_axes`/`mask_axes` + other than `"YX"`, `mask_source="ngff_labels"`, `mask_label_name` and labels + other than `"single"` are refused with a migration message (CP-2b removes + them for file runs). The run is opened once per call and kept for that call + only (`cellpatcher.run_session`). +* `image_axes`/`mask_axes` are explicit and axis-aware (see `axes.py`) for + in-memory arrays and lazy delivery from a path. Image/mask pairing is the preprocessing brick's `find_masks`, called by `io.pair_masks` (issue #45, owner decision 2026-10-30): TIFF masks named `_.tif` or `/.tif`; `matching.mask_type` picks @@ -229,7 +244,15 @@ in notebooks where practical. ### Inputs this repo consumes -CellPatcher reads image files, and (for `cell` mode) matching label/mask +Since v0.2.0 (CP-2a) folder and tree runs read files only through a clean +preprocessing run (`common.preprocessing_run`, opened with `RunInputs`; see §4 and +`docs/configuration_reference.md`, "File runs read through the preprocessing +run"): the images, their labels and `position_id`, their roles and their masks +are the run's. A folder mask and a label image of one type for one image are +refused by the preprocessing run (`duplicate_mask`, rc1). The rest of this +paragraph describes discovery and pairing as `validate_image_and_mask_folders` +and lazy delivery from paths still do them (until CP-2b), and as file runs did +before v0.2.0. CellPatcher reads image files, and (for `cell` mode) matching label/mask files, directly from user-provided folders. Images are found with the preprocessing brick's `discover_experiment` (`io.discover_images`; owner decision of 2026-10-30, issue #45): TIFF files and OME-Zarr images. PNG, JPEG, @@ -263,11 +286,10 @@ guessed, and `common.mask_axes` with a `C` is refused for mask-folder masks Two ways to point CellPatcher at input data: -- `run_patching_pipeline(image_folder_path, ...)` — a single flat folder of - images (optionally with a matching mask folder). `condition`/ - `biological_repeat` must be set explicitly via - `CellPatcherConfig.common.condition`/`.biological_repeat` when using this - entry point; they are not parsed from the folder path. +- `run_patching_pipeline(image_folder_path, ...)` — the images of a + preprocessing run of one condition and biological repeat (optionally with + its masks folder). Since v0.2.0 their labels are the run's; before, they + were set with `CellPatcherConfig.common.condition`/`.biological_repeat`. - `run_patching_condition_repeat_tree(image_root_path, ...)` — the `/[/]/` layout from §6, walked diff --git a/docs/configuration_reference.md b/docs/configuration_reference.md index a146b0b..9d088f7 100644 --- a/docs/configuration_reference.md +++ b/docs/configuration_reference.md @@ -14,26 +14,44 @@ This document is the complete runtime configuration reference for `CellPatcherCo --- +## File runs read through the preprocessing run (v0.2.0, CP-2a) + +`run_patching_pipeline`, `run_patching_condition_repeat_tree` and `run_patching_from_folders` read files only through a clean preprocessing run (v0.4), `common.preprocessing_run`, with the preprocessing brick's `RunInputs` (design v0.4 §6). Without it they stop with `no_preprocessing_run` and the preprocessing command (`morphobricks-preprocessing check --masks `, then `morphobricks-preprocessing run --axes --masks `). + +- **Where the experiment is now.** The run is opened with `RunInputs.open(run, experiment_root=, masks_root=)`; a masks folder left out is where the run recorded it. An experiment moved or copied since its run is checked and read where CellPatcher is given it: every image and mask must have the size and modification time the run recorded (copy with the times kept, e.g. `cp -p` or `shutil.copytree`), or the run is refused (`preprocessing_run_refused`, naming the files). +- **Images.** The run's images (`RunInputs.fovs()`), sorted by path, read with `RunInputs.read_plane` in the roles the run recorded. Their arrays are in one canonical layout per run: T and Z when an image of the run has more than one plane along them, C when an image has a channel role, in that order, then Y and X (`ZCYX`, `TYX`, `CYX`, `YX`, ...). Nothing is read transposed; an image without one of these axes is its one plane along it. With plane policies only the chosen planes are read. An image added to the experiment after the run is not part of it, so not patched (a brick calls no `discover_experiment` for files); run preprocessing again to include it. **Known limitation:** nothing says so yet. Preprocessing will give bricks a check for images added after the run (a later preprocessing PR, not in v0.4.0rc1); CellPatcher will call it then, rather than discover images itself. +- **Labels and `position_id`.** Each image's `condition`, `biological_repeat`, `technical_repeat` and `position_id` are the ones the run recorded, never parsed. `run_patching_pipeline` patches a run of one condition and biological repeat (any technical repeats); use the tree run for more. +- **Masks.** Those the run paired (masks-folder TIFFs and the label images of OME-Zarr images; a label image's type is its name), read with `RunInputs.read_mask` (the brick checks their labels). `matching.mask_type` chooses the type; `None` takes the run's only type. `cell` mode always reads masks; `random` mode when a masks folder or `matching.mask_type` is given. Every image needs a mask of the type. The mask arrays follow one layout per run too (the T and Z any mask of the run has, then Y and X); a mask without one of them (a 2-D mask of a one-plane image, or a stated shared 2-D mask) has length 1 along it. A mask the run recorded without axes (rc1 records one whose header it could not use, e.g. X before Y) is refused. A folder mask and a label image of the same type for one image are refused by the preprocessing run (`duplicate_mask`), so no clean run has them. +- **Normalization.** The CSV scopes read the run's own table (`normalization.plane_histograms` may be left unset, or must name it: `not_the_run_output`), design and image type. +- **`run_config.json`** records `common.preprocessing_run` and, as `common.image_axes`/`mask_axes`, the layout the patches were read in. +- **Migration.** For file runs, `common.image_axes`/`mask_axes` other than `"YX"`, `common.mask_source="ngff_labels"`, `common.mask_label_name`, and `common.condition`/`biological_repeat`/`technical_repeat` other than `"single"` are refused with a message saying what replaces them (CP-2b removes them for file runs). The axes and labels still describe in-memory arrays given to `run_patching`; `mask_source` and `mask_label_name` are not used there. +- **Within one call** (`run_patching`, a folder or tree run) the run is opened once; nothing is kept beyond the call, so the next call checks every image and mask again (`cellpatcher.run_session`). + +Still read by CellPatcher until CP-2b: lazy delivery from a file path (`patch_array`/`iter_patch_arrays` with paths, using `common.image_axes`), and `validate_image_and_mask_folders` (`pair_masks`). + +--- + ## `config.common` (`CommonConfig`) Shared settings that apply to every run, regardless of mode. | Field | Type | Default | Notes | |---|---|---|---| -| `image_axes` | `str` | `"YX"` | Axes string for image arrays (e.g. `"YX"`, `"CYX"`, `"ZYX"`). Must include `Y` and `X`. A TIFF image file's header is checked against it before any pixel is read, with the preprocessing brick's axis check (`review_axes`: `check_mapping` and `header_axis_issue`, design v0.4 §3): it may name what the header leaves open (a placeholder axis; the positional Y and X of a plain stack of pages, so a plain `QYX` stack may be declared `YXC`), never contradict an axis the metadata names. A Y/X swap (header `ZYX` declared `ZXY`, `axes_yx_swapped`) and a plain stack's Y and X declared X before Y (`axes_x_before_y`) are refused too: nothing is ever read transposed (since v0.2.0; v0.1.2 read a swapped image transposed). With a CSV normalization scope (`image`, `biological_repeat`), the axes must also give the roles the preprocessing run recorded (`check_sources`, `axis_mapping_mismatch`). Anything else (another order, another number of axes, an unreadable header) is refused naming the file. OME-Zarr images are matched by role instead (see "OME-Zarr images"). | -| `mask_axes` | `str` | `"YX"` | Axes string for mask arrays. Must include `Y` and `X`. Masks read from a mask folder (`mask_source="tiff_tree"`) cannot have a `C` axis: a `C` here is refused for folder and tree runs, since it would read a mask's other axes as label channels (owner decision of 2026-10-30). Save each label channel as its own mask type instead (`_cells.tif`, `_nuclei.tif`) and choose one with `matching.mask_type`. A TIFF mask file's header is checked as for `image_axes` (a Y/X swap is refused). CellPatcher takes a label plane's two dimensions as Y and X in their stored order, so any `mask_axes` with X before Y (`"XY"`, `"ZXY"`) is refused, for in-memory masks and OME-Zarr labels too (before v0.1.2 it was ignored for in-memory masks). The refusal says how to fix the masks: save each one with Y before X (its last two axes swapped, e.g. `numpy.swapaxes(mask, -1, -2)`) and declare those axes. Nothing is ever read transposed. | +| `image_axes` | `str` | `"YX"` | Axes string for image arrays given to `run_patching` (e.g. `"YX"`, `"CYX"`, `"ZYX"`). Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read the roles the preprocessing run recorded, refuse any other value than `"YX"`, and record the layout they read in `run_config.json`; the rest of this row is about lazy delivery from a file path. A TIFF image file's header is checked against it before any pixel is read, with the preprocessing brick's axis check (`review_axes`: `check_mapping` and `header_axis_issue`, design v0.4 §3): it may name what the header leaves open (a placeholder axis; the positional Y and X of a plain stack of pages, so a plain `QYX` stack may be declared `YXC`), never contradict an axis the metadata names. A Y/X swap (header `ZYX` declared `ZXY`, `axes_yx_swapped`) and a plain stack's Y and X declared X before Y (`axes_x_before_y`) are refused too: nothing is ever read transposed (since v0.2.0; v0.1.2 read a swapped image transposed). With a CSV normalization scope (`image`, `biological_repeat`), the axes must also give the roles the preprocessing run recorded (`check_sources`, `axis_mapping_mismatch`). Anything else (another order, another number of axes, an unreadable header) is refused naming the file. OME-Zarr images are matched by role instead (see "OME-Zarr images"). | +| `mask_axes` | `str` | `"YX"` | Axes string for mask arrays given to `run_patching`. Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read masks in the roles the preprocessing run recorded and refuse any other value than `"YX"`. Masks read from a mask folder (`mask_source="tiff_tree"`) cannot have a `C` axis: a `C` here is refused for folder and tree runs, since it would read a mask's other axes as label channels (owner decision of 2026-10-30). Save each label channel as its own mask type instead (`_cells.tif`, `_nuclei.tif`) and choose one with `matching.mask_type`. A TIFF mask file's header is checked as for `image_axes` (a Y/X swap is refused). CellPatcher takes a label plane's two dimensions as Y and X in their stored order, so any `mask_axes` with X before Y (`"XY"`, `"ZXY"`) is refused, for in-memory masks and OME-Zarr labels too (before v0.1.2 it was ignored for in-memory masks). The refusal says how to fix the masks: save each one with Y before X (its last two axes swapped, e.g. `numpy.swapaxes(mask, -1, -2)`) and declare those axes. Nothing is ever read transposed. | +| `preprocessing_run` | `str \| None` | `None` | The output folder of the clean preprocessing run (v0.4) that folder and tree runs read their images, masks and labels through (see "File runs read through the preprocessing run"). Required for them; not used by in-memory `run_patching` (its CSV normalization reads `normalization.plane_histograms`). Since v0.2.0. | | `output_dir` | `str \| None` | `None` | Output folder path. Required when `export.enabled=True`; keep `None` for in-memory-only runs. | | `extract_arrays` | `bool` | `False` | When `True`, extracted patch arrays are returned in `PatchingResult.patch_arrays`. | -| `condition` | `str \| None` | `"single"` | Experimental group label, stamped onto every `PatchRegion`/CSV row for this run. Part of the MorphoBricks shared CSV contract (see `AGENTS.md` §5). Set it explicitly for a single-folder `run_patching_pipeline` call, or let `run_patching_condition_repeat_tree` derive it per leaf from a `//` folder tree (see "Condition/biological-repeat folder trees" below). | -| `biological_repeat` | `str \| None` | `"single"` | Biological replicate identifier (e.g. `"YYYY_MM_DD"`, or `cellpatcher.SINGLE_BIOLOGICAL_REPEAT_LABEL`/`"single"` for a condition folder with no biological-repeat subfolder), stamped onto every `PatchRegion`/CSV row for this run. Same status as `condition`. Formerly `repeat`, which still works as a deprecated alias (see "Renamed: `repeat` → `biological_repeat`" below). | -| `technical_repeat` | `str \| None` | `"single"` | Technical replicate within the biological repeat: a well, a dish or a coverslip, never a biological repeat. Stamped onto every `PatchRegion`/CSV row for this run; a per-image `image_metadata=[{"technical_repeat": ...}]` entry overrides it for that image. `run_patching_condition_repeat_tree` derives it per leaf from an optional `///` level, and stamps `cellpatcher.SINGLE_TECHNICAL_REPEAT_LABEL` (`"single"`) when a biological-repeat folder holds its images directly. | +| `condition` | `str \| None` | `"single"` | In-memory `run_patching` only: since v0.2.0 (CP-2a) folder and tree runs take each image's labels from the preprocessing run and refuse any other value than `"single"`. Experimental group label, stamped onto every `PatchRegion`/CSV row for this run. Part of the MorphoBricks shared CSV contract (see `AGENTS.md` §5). Set it explicitly for a single-folder `run_patching_pipeline` call, or let `run_patching_condition_repeat_tree` derive it per leaf from a `//` folder tree (see "Condition/biological-repeat folder trees" below). | +| `biological_repeat` | `str \| None` | `"single"` | In-memory `run_patching` only: since v0.2.0 (CP-2a) folder and tree runs take each image's labels from the preprocessing run and refuse any other value than `"single"`. Biological replicate identifier (e.g. `"YYYY_MM_DD"`, or `cellpatcher.SINGLE_BIOLOGICAL_REPEAT_LABEL`/`"single"` for a condition folder with no biological-repeat subfolder), stamped onto every `PatchRegion`/CSV row for this run. Same status as `condition`. Formerly `repeat`, which still works as a deprecated alias (see "Renamed: `repeat` → `biological_repeat`" below). | +| `technical_repeat` | `str \| None` | `"single"` | In-memory `run_patching` only: since v0.2.0 (CP-2a) folder and tree runs take each image's labels from the preprocessing run and refuse any other value than `"single"`. Technical replicate within the biological repeat: a well, a dish or a coverslip, never a biological repeat. Stamped onto every `PatchRegion`/CSV row for this run; a per-image `image_metadata=[{"technical_repeat": ...}]` entry overrides it for that image. `run_patching_condition_repeat_tree` derives it per leaf from an optional `///` level, and stamps `cellpatcher.SINGLE_TECHNICAL_REPEAT_LABEL` (`"single"`) when a biological-repeat folder holds its images directly. | | `z_policy` | `str` | `"keep"` | How a Z axis of the images becomes 2D planes before patching. `"keep"`: every patch keeps all Z planes (the behaviour before this option). `"all"`: one patch per Z plane, with `z0`/`z1` set and a `z` level in `patch_id` (e.g. `/z002/grid-y0-x8-h8-w8`; see "Patch identity" below). `"first"`/`"select"` (with `z_index`): one plane, with `z0`/`z1` set. `"max"`/`"mean"`/`"sum"`: a projection (no `z0`). Images without a Z axis are not affected. | | `z_index` | `int \| None` | `None` | Z plane for `z_policy="select"`. | | `t_policy` | `str` | `"keep"` | As `z_policy` for a T axis: `"keep"`, `"all"` (sets `t`, a `t` level of `patch_id`), `"first"` or `"select"`; no projections. With both split, planes come in T-then-Z order (`.../t0001/z002/...`). | | `t_index` | `int \| None` | `None` | Time point for `t_policy="select"`. | -| `mask_source` | `str` | `"tiff_tree"` | Where folder and tree runs read masks. `"tiff_tree"`: a mask folder (or tree) mirroring the images, paired with the preprocessing brick's rule (`config.matching`). `"ngff_labels"`: the label image of each OME-Zarr image's own `labels` group (no mask folder; every image must be OME-Zarr). See "OME-Zarr images" below. | -| `mask_label_name` | `str \| None` | `None` | With `mask_source="ngff_labels"`, the label image to use (e.g. `"cells"`); may stay `None` when every image has exactly one. | +| `mask_source` | `str` | `"tiff_tree"` | Before v0.2.0, where folder and tree runs read masks; since CP-2a they read the masks the preprocessing run paired, and `"ngff_labels"` is refused with a migration message (a label image's type is its name: choose it with `matching.mask_type`). Former meaning: `"tiff_tree"`: a mask folder (or tree) mirroring the images, paired with the preprocessing brick's rule (`config.matching`). `"ngff_labels"`: the label image of each OME-Zarr image's own `labels` group (no mask folder; every image must be OME-Zarr). See "OME-Zarr images" below. | +| `mask_label_name` | `str \| None` | `None` | Refused for folder and tree runs since CP-2a (use `matching.mask_type`). Former meaning: with `mask_source="ngff_labels"`, the label image to use (e.g. `"cells"`); may stay `None` when every image has exactly one. | | `object_set` | `str \| None` | `None` | Cell mode: the name of the object set (mask) written to the `object_set` column and to `patch_id`. `None`: named after the mask (see "Patch identity" below), `"mask"` when the mask gives no name. Letters, digits, `.`, `_`, `-` only. | **Masks with split planes:** a mask's Z or T axis with the image's length follows the image's planes (one label plane per image plane); a mask without that axis, or with length 1, is shared by every plane. Any other length is an error, and so is a Z projection with masks that follow the Z planes. Image ids stay one per source image, and every plane and channel patch of an image shares one `position_id`. @@ -180,7 +198,7 @@ The image's folders are its path below the image folder (folder run: none) or th | Field | Type | Default | Notes | |---|---|---|---| -| `mask_type` | `str \| None` | `None` | The mask type to use (`"cells"` for `img_cells.tif` or `cells/img.tif`). `None` uses the only type in the mask folder and refuses a folder holding several (a type is never chosen for you). The types are the mask folder's: since v0.2.0 (preprocessing v0.4, which pairs an OME-Zarr image's own label images too) a label image is read only with `common.mask_source="ngff_labels"`, and a label image and a folder mask of the same type for one image are refused (`duplicate_mask`). | +| `mask_type` | `str \| None` | `None` | The mask type to use (`"cells"` for `img_cells.tif` or `cells/img.tif`, or a label image's name). Folder and tree runs (CP-2a): one of the types the preprocessing run recorded; `None` takes the run's only type and refuses a run with several. For `validate_image_and_mask_folders`: The types are the mask folder's: since v0.2.0 (preprocessing v0.4, which pairs an OME-Zarr image's own label images too) a label image is read only with `common.mask_source="ngff_labels"`, and a label image and a folder mask of the same type for one image are refused (`duplicate_mask`). | Refused, never resolved by a guess or silently skipped: @@ -370,7 +388,7 @@ Wherever `patches.csv` is written (`export.enabled=True` with `save_metadata_csv | `reason` | Empty for `processed`. For `no_patches`, the step that left none: `the patch mode generated no region in this image` (e.g. no labeled object in its mask, every object excluded by border or area), `the filters removed all region(s)`, or `normalization discarded all patch(es): they cannot be normalized from their own pixels` (decision D8). For `skipped`: `the image has no plane to patch along the axes split by z_policy/t_policy` (a Z or T axis of length 0 with `"all"`). | | `position_id` | The `position_id` every patch of the image carries in `patches.csv`, also for an image without patches: supplied via `image_metadata`, or the default of "Patch identity" above (one file is one position). Never empty since 2026-10-26. | | `n_regions`, `n_kept`, `n_normalized` | Regions the patch mode generated, those left by the filters, and those left by normalization (equal to `n_kept` without it), counted before `split_channels`; they tell the three `no_patches` reasons apart without reading `reason`. `0` for a skipped image. | -| `image_read_transposed` | `true` when `common.image_axes` declared the image file's header Y and X swapped, so the image was read transposed (v0.1.2 only: since v0.2.0 a swap is refused, so it is always `false` for a TIFF file); empty when not checked (an in-memory array, an OME-Zarr image). Since v0.1.2. | +| `image_read_transposed` | `true` when `common.image_axes` declared the image file's header Y and X swapped, so the image was read transposed (v0.1.2 only: since v0.2.0 a swap is refused, and folder and tree runs read every image through the preprocessing run, so it is always `false` for them, OME-Zarr images included); empty for an in-memory array. CP-2b removes the column. Since v0.1.2. | `position_id`, `n_regions`, `n_kept`, `n_normalized` and `image_read_transposed` were added after the first release of the table, as its last columns, so earlier columns keep their positions. @@ -380,9 +398,9 @@ There is no `failed` status: an image that cannot be read, matched to a mask or ### OME-Zarr images -With the `zarr` extra (`pip install "cellpatcher[zarr]"`), folders whose names end in `.zarr` (usually `.ome.zarr`) are images, in folder runs, tree runs and `read_array`. CellPatcher reads level 0 of the store's only image with the preprocessing brick's reader; a store with several images (positions, series or plate wells) is an error. The axes come from the OME-NGFF metadata and are matched to `common.image_axes` (or `mask_axes`) by role: a length-1 axis that the configured axes lack is dropped, a configured axis the store lacks is added with length 1, and the dimensions are reordered; any other mismatch is an error. Discovery never enters a store, and `img.ome.zarr` pairs with masks named like `img_cells.tif` or `cells/img.tif`. Without the extra, reading a store stops with the install command. OME-Zarr copies written by the preprocessing brick are not read: CellPatcher reads the source images. +Since v0.2.0 (CP-2a) folder and tree runs read OME-Zarr images and their label images through the preprocessing run like any other image (see "File runs read through the preprocessing run"); this section describes `read_array`, `read_image` and lazy delivery from a path. With the `zarr` extra (`pip install "cellpatcher[zarr]"`), folders whose names end in `.zarr` (usually `.ome.zarr`) are images for `read_array` and lazy delivery (folder and tree runs take them from the preprocessing run). CellPatcher reads level 0 of the store's only image with the preprocessing brick's reader; a store with several images (positions, series or plate wells) is an error. The axes come from the OME-NGFF metadata and are matched to `common.image_axes` (or `mask_axes`) by role: a length-1 axis that the configured axes lack is dropped, a configured axis the store lacks is added with length 1, and the dimensions are reordered; any other mismatch is an error. Discovery never enters a store, and `img.ome.zarr` pairs with masks named like `img_cells.tif` or `cells/img.tif`. Without the extra, reading a store stops with the install command. OME-Zarr copies written by the preprocessing brick are not read: CellPatcher reads the source images. -**Masks from OME-NGFF labels:** with `common.mask_source="ngff_labels"`, each OME-Zarr image's mask is its label image `common.mask_label_name` (from its `labels` group, OME-NGFF 0.4 and 0.5; the name may stay `None` when there is only one). Level 0 is read; its values must be integers, each dimension must equal the image's or have length 1, and a channel dimension must have length 1. It is then put into `common.mask_axes` by role, so a label Z or T axis with the image's length follows the image's planes and one of length 1 is shared (`z_policy`/`t_policy`). `mask_file` is written as `.ome.zarr/labels/`. A mask folder, or an image that is not OME-Zarr, is an error with this source. +**Masks from OME-NGFF labels (before v0.2.0; folder and tree runs now read the label images the preprocessing run paired, by type):** with `common.mask_source="ngff_labels"`, each OME-Zarr image's mask is its label image `common.mask_label_name` (from its `labels` group, OME-NGFF 0.4 and 0.5; the name may stay `None` when there is only one). Level 0 is read; its values must be integers, each dimension must equal the image's or have length 1, and a channel dimension must have length 1. It is then put into `common.mask_axes` by role, so a label Z or T axis with the image's length follows the image's planes and one of length 1 is shared (`z_policy`/`t_policy`). `mask_file` is written as `.ome.zarr/labels/`. A mask folder, or an image that is not OME-Zarr, is an error with this source. ### Condition/biological-repeat folder trees diff --git a/docs/normalization.md b/docs/normalization.md index 0a3f692..0ea2013 100644 --- a/docs/normalization.md +++ b/docs/normalization.md @@ -169,10 +169,10 @@ planes and what to do: | Message | Meaning and fix | |---|---| -| *The preprocessing run in ... cannot be used* (`preprocessing_run_refused`) | The preprocessing brick's `RunInputs` refuses the run (since v0.2.0): no `run_manifest.json` next to the table, a run written before preprocessing v0.4.0, a table or `experiment_design.csv` changed since the run, an image or mask changed or missing since the run, or an experiment moved since the run. Run preprocessing v0.4 on the experiment where it is now. | +| *The preprocessing run in ... cannot be used* (`preprocessing_run_refused`) | The preprocessing brick's `RunInputs` refuses the run (since v0.2.0): no `run_manifest.json` next to the table, a run written before preprocessing v0.4.0, a table or `experiment_design.csv` changed since the run, or an image or mask changed or missing since the run where CellPatcher reads it. Run preprocessing v0.4 on the experiment as it is now; an experiment only moved or copied (times kept) since its run is read where it is now when CellPatcher is given that folder and its masks folder (since CP-2a). | | *normalization.plane_histograms/manifest/experiment_design names ...* (`not_the_run_output`) | Only the run's own files, unrenamed in its output folder, are read. Give that folder's `plane_histograms.csv` (and leave `manifest` and `experiment_design` empty). | | *... the preprocessing run recorded no design* (`missing_experiment_design`) | The `biological_repeat` scope needs a run whose experiment design was confirmed (step 1 of the preprocessing brick). | -| *... image(s) found by CellPatcher have no rows in plane_histograms.csv* | Images were excluded or skipped by the preprocessing brick, or added after it ran. Run the preprocessing again on the same folder, remove those images, or use the patch scope. | +| *... image(s) found by CellPatcher have no rows in plane_histograms.csv* | In-memory runs: images whose `source_relative_path` the run has no rows for. Folder and tree runs patch only the run's images (since CP-2a; an image added after the run is not seen). Run the preprocessing again, or use the patch scope. | | *... changed after the preprocessing brick read them* / *... the images changed after the preprocessing run* | An image's size, time or pixel values differ from the statistics. Run the preprocessing again. | | *... other condition, biological repeat or technical repeat labels* | CellPatcher and the preprocessing read different folder levels: give both the same experiment folder. | | *... another position_id than their rows* | A `position_id` given in `image_metadata` is not the one the preprocessing recorded (the image name). Leave it to the image name, or give the recorded one. | diff --git a/docs/normalization_contract.md b/docs/normalization_contract.md index 27e86cb..1586b76 100644 --- a/docs/normalization_contract.md +++ b/docs/normalization_contract.md @@ -60,8 +60,8 @@ brick's | Rule | Detail | |---|---| -| Discovery | CellPatcher always discovers images with its own folder rules (`discover_condition_repeat_folders`), whether or not preprocessing was run. The manifest never decides which images are patched. | -| Experiment root | The folder given to CellPatcher as image root must be the folder given to the preprocessing brick, so relative paths match. | +| Discovery | Since v0.2.0 (CP-2a) folder and tree runs patch the preprocessing run's images (`RunInputs.fovs()`), with their FOV keys as the run recorded them; CellPatcher discovers nothing for them (design v0.4 §6). An image added after the run is not patched, and nothing says so yet: a known limitation, closed when CellPatcher calls the check for added images that preprocessing will give bricks (not in v0.4.0rc1). Before, CellPatcher discovered images with its own folder rules and the manifest never decided which were patched. | +| Experiment root | The folder given to CellPatcher as image root is where the preprocessed experiment is now: the run is opened with it (`RunInputs.open(experiment_root=...)`, and the masks folder with `masks_root`), so an experiment moved or copied since the run is checked and read there. Relative paths are the run's. | | Image types | TIFF files (`.tif`, `.tiff`, any case) and, with the extra, OME-Zarr images: folders whose names end in `.zarr`, read from level 0 of their first image. A store with several images is an error, as in preprocessing. | | OME-Zarr copies | Copies written by the preprocessing brick (`/ome_zarr/...`) are not read; CellPatcher reads the source images. | | `source_relative_path` | The image's path relative to the experiment root, with `/` separators, from its real location in the tree (`control/img.tif` for an image directly in `control/`; a linked image or folder keeps the link's path under the root, as the brick records it when it walks the tree). Never built from `ConditionRepeatFolder.relative_folder`, which inserts `single` for missing levels. | @@ -132,19 +132,23 @@ Checks before any patch is normalized; every failure blocks the run: - the plane's exact finite minimum and maximum equal `pixel_min` and `pixel_max`, and its finite pixel count equals `valid_pixel_count`. -An image CellPatcher found without any row in the CSV (excluded or skipped in preprocessing, -or added later) is reported with all such images in one message. +An image without any row in the CSV is reported with all such images in one message (in a +file run every image is the run's, so this means a row the run did not write). **The run (since v0.2.0):** the table's folder is opened with the preprocessing brick's `RunInputs.open`, and any refusal stops the run (`preprocessing_run_refused`): no manifest, a manifest before v0.4, a table or design whose SHA-256 is not the manifest's, an image or -mask of the run changed or missing (which includes an experiment moved since the run). The +mask of the run changed or missing, where CellPatcher was given the experiment (since CP-2a +a file run opens the run with its image root and masks folder; in-memory runs and +`load_plane_statistics` without `experiment_root` open it where the run recorded it). The CSV and manifest must be the run's own files, unrenamed (`not_the_run_output`). Then: -- each patched image's `manifest.source_fingerprint()` (size and modification time) equals its - manifest entry; -- CellPatcher's image axes give the same role for every axis as the manifest's - `axis_mapping`; +- a file run reads every image and mask through `RunInputs` (`read_plane`, `read_mask`), which + checks each file's fingerprint again on every read, in the roles the run recorded (before + CP-2a: the patched images' size, time and declared axes were checked with `check_sources`); +- the run is opened once per CellPatcher call and kept for that call only + (`cellpatcher.run_session`): the next call checks every image and mask again, patched or + not (before CP-2a an opened run was cached for the session, keyed on the table and manifest); - the manifest's `image_type`, when it records one, equals `normalization.image_type` (otherwise `image_type_differs`, checked before any image is read). A run that recorded no image type leaves the choice to the configuration (normalization brick decision N34). diff --git a/notebooks/CellPatcher/CHANGELOG.md b/notebooks/CellPatcher/CHANGELOG.md index b858c72..f8dd0ac 100644 --- a/notebooks/CellPatcher/CHANGELOG.md +++ b/notebooks/CellPatcher/CHANGELOG.md @@ -4,6 +4,34 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). +## [Unreleased] - File runs read through the preprocessing run (CP-2a) + +### Changed +- **Breaking.** `run_patching_pipeline`, `run_patching_condition_repeat_tree` and + `run_patching_from_folders` read files only through a clean preprocessing run (v0.4): + `common.preprocessing_run`, its output folder, is required in every mode, normalization on or + off (`no_preprocessing_run`, with the preprocessing command). The images patched are the + run's, with their labels and `position_id` as it recorded them; images and masks are read + with `RunInputs.read_plane`/`read_mask` in the roles it recorded, in one canonical layout per + run (T, Z, C, then Y, X), recorded in `run_config.json` as `common.image_axes`/`mask_axes`. + An image added to the experiment after the run is not patched, and nothing says so yet (known + limitation, closed when CellPatcher calls the preprocessing check for added images, which is + not in v0.4.0rc1). +- The run is opened with the image folder/root and the masks folder given, so an experiment + moved or copied since its run (times kept) is checked and read where it is now. A mask + changed in a copy is refused (before, only the original was checked). +- For file runs, `common.image_axes`/`mask_axes` other than `"YX"`, `mask_source="ngff_labels"`, + `mask_label_name`, and `condition`/`biological_repeat`/`technical_repeat` other than + `"single"` are refused with a migration message. `matching.mask_type` chooses among the run's + mask types (a label image's type is its name); `random` mode reads masks when a masks folder + or `matching.mask_type` is given. `run_patching_pipeline` patches a run of one condition and + biological repeat. +- The statistics cache is gone: a run is opened once per CellPatcher call and kept for that + call only, so a mask or an unpatched image changed between two calls is refused. +- `load_plane_statistics`, `recorded_image_type` and `cached_statistics` take + `experiment_root=`/`masks_root=`; they and `open_run` moved to `cellpatcher.run_open` + (still importable from `cellpatcher.statistics`). + ## [Unreleased] - The preprocessing run is read through RunInputs (CP-1, preprocessing v0.4.0rc1) ### Changed diff --git a/src/cellpatcher/config.py b/src/cellpatcher/config.py index bd0962a..dfb0f39 100644 --- a/src/cellpatcher/config.py +++ b/src/cellpatcher/config.py @@ -19,22 +19,22 @@ @accept_deprecated_repeat @dataclass class CommonConfig: + # The axes of in-memory arrays given to run_patching (canonical, Y before X). File runs + # read each image and mask in the roles the preprocessing run recorded and refuse other + # values than "YX" (CP-2a; removed for them in CP-2b). image_axes: str = "YX" mask_axes: str = "YX" output_dir: str | None = None extract_arrays: bool = False - # Stamped onto every PatchRegion/metadata-CSV row for this run. These are - # the MorphoBricks shared-CSV-contract `condition`/`biological_repeat`/ - # `technical_repeat` columns (see AGENTS.md §5). A label that is not - # given is "single", as for a missing folder level; None is stamped as - # "single" too. run_patching()/run_patching_pipeline() never infer them - # from paths; set them explicitly, or use - # orchestration.run_patching_condition_repeat_tree to derive them per - # leaf from a /[/]/ - # folder tree. technical_repeat is the well/dish/coverslip within the - # biological repeat; per-image image_metadata["technical_repeat"] - # overrides it. `repeat` stays accepted as a deprecated alias of - # biological_repeat (DeprecationWarning). + # The output folder of the clean preprocessing run (v0.4) a folder or tree run reads its + # images, masks and labels through (RunInputs, design v0.4 §6); required for file runs. + preprocessing_run: str | None = None + # The MorphoBricks shared-CSV-contract `condition`/`biological_repeat`/`technical_repeat` + # columns (AGENTS.md §5) of in-memory run_patching; a label not given (or None) is + # "single", as for a missing folder level. Folder and tree runs take each image's labels + # from the preprocessing run and refuse other values. technical_repeat is the well/dish/ + # coverslip; per-image image_metadata["technical_repeat"] overrides it. `repeat` stays + # accepted as a deprecated alias of biological_repeat (DeprecationWarning). condition: str | None = SINGLE_CONDITION_LABEL biological_repeat: str | None = SINGLE_BIOLOGICAL_REPEAT_LABEL technical_repeat: str | None = SINGLE_TECHNICAL_REPEAT_LABEL @@ -48,18 +48,16 @@ class CommonConfig: z_index: int | None = None t_policy: str = "keep" t_index: int | None = None - # Where folder and tree runs read masks (docs/normalization_contract.md - # §4): "tiff_tree", a mask folder (tree) mirroring the images and matched - # by file name with `matching`; or "ngff_labels", the label image named - # mask_label_name in each OME-Zarr image's own labels group (the name may - # be left None when every image has exactly one). + # Where folder and tree runs read masks before CP-2a. They now read the masks the + # preprocessing run paired (choose the type with matching.mask_type) and refuse other + # values than these defaults. mask_source: str = "tiff_tree" mask_label_name: str | None = None # The object set of cell-mode patches (the `object_set` column and a # level of patch_id, cellpatcher.ids). None: folder and tree runs name it - # after the mask (its OME-NGFF label name, or its file name's suffix after - # the image's stem), and "mask" when the mask gives no name (in-memory - # run_patching, or a mask named like its image). Set it to name it + # after the mask type the preprocessing run recorded (a folder name, or an + # OME-NGFF label name), and "mask" when the mask gives no name (in-memory + # run_patching). Set it to name it # explicitly; image_metadata["object_set"] overrides it per image. object_set: str | None = None diff --git a/src/cellpatcher/file_runs.py b/src/cellpatcher/file_runs.py new file mode 100644 index 0000000..ad51c03 --- /dev/null +++ b/src/cellpatcher/file_runs.py @@ -0,0 +1,388 @@ +"""Folder and tree runs, read through a clean preprocessing run (layer: orchestration). + +Design v0.4 §6 (CP-2a): a file run (:func:`cellpatcher.run_patching_pipeline`, +:func:`cellpatcher.run_patching_condition_repeat_tree`) needs a clean preprocessing run, +``common.preprocessing_run``, opened with ``RunInputs.open`` with the experiment root and +masks folder CellPatcher is given (so an experiment moved or copied since its run is checked +and read where it is now). Then: + +- the images patched are the run's, with their FOV key (condition, biological and technical + repeat, ``position_id``) as the run recorded it, never parsed from a name; +- each image is read with ``RunInputs.read_plane`` in the roles the run recorded, and each mask + with ``RunInputs.read_mask`` by the run's pairing (:mod:`cellpatcher.run_reading`); +- the CSV normalization scopes read the run's own table, design and image type. + +The settings that told CellPatcher how to read files (``common.image_axes``, ``mask_axes``, +``mask_source``, ``mask_label_name`` and the labels) are refused for file runs with a +migration message; CP-2b removes them. In-memory ``run_patching`` keeps them. +""" + +from __future__ import annotations + +import os +from collections.abc import Iterator, Mapping, Sequence +from dataclasses import dataclass, replace +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING + +from cellpatcher.ids import object_set_from_mask_type, validate_object_set +from cellpatcher.io import SINGLE_CONDITION_LABEL, ConditionRepeatFolder +from cellpatcher.normalization import ( + CSV_SCOPES, + check_image_type, + load_experiment_design, + validate_normalization, +) +from cellpatcher.pipeline import ( + PatchingResult, + _patch_planes, + check_config_identity, + run_patching, +) +from cellpatcher.planes import splits_planes +from cellpatcher.run_open import _refuse_other_file, open_run, statistics_of +from cellpatcher.run_reading import ( + iter_fov_planes, + mask_file_name, + mask_record, + read_fov, + read_fov_mask, + run_layout, + run_mask_layout, +) +from cellpatcher.run_session import keep_in_session +from cellpatcher.statistics import ( + PlaneStatistics, + PreprocessingCheckError, + check_images, +) + +if TYPE_CHECKING: + import numpy as np + from morphobricks_preprocessing.run_inputs import Fov, RunInputs + + from cellpatcher.config import CellPatcherConfig, NormalizationConfig + +PREPROCESSING_COMMAND = ( + "morphobricks-preprocessing check --masks , then " + "morphobricks-preprocessing run --axes --masks " + "(leave out --masks without masks)" +) + +_DEFAULT_AXES = "YX" +_LABELS = ("condition", "biological_repeat", "technical_repeat") + + +@dataclass(frozen=True) +class FileRun: + """A clean preprocessing run opened for a folder or tree run. + + Attributes + ---------- + inputs + The run, opened with ``RunInputs.open``. + layout + The canonical layout of every image's array (:func:`cellpatcher.run_reading.run_layout`). + mask_type + The mask type read, or None without masks. + mask_axes + The canonical layout of every mask's array (``"YX"`` without masks; + :func:`cellpatcher.run_reading.run_mask_layout`). + statistics + The run's table for a CSV normalization scope, else None. + """ + + inputs: RunInputs + layout: str + mask_type: str | None + mask_axes: str + statistics: PlaneStatistics | None + + @property + def fovs(self) -> tuple[Fov, ...]: + """Every image of the run, sorted by path.""" + return self.inputs.fovs() + + +def refuse_file_run_settings(config: CellPatcherConfig, where: str) -> None: + """Refuse a file run without a preprocessing run, or with a setting it no longer reads. + + Raises + ------ + PreprocessingCheckError + ``no_preprocessing_run`` when ``common.preprocessing_run`` is not set. + ValueError + Naming every setting a file run reads from the preprocessing run instead, and what + to do (the migration message). + """ + common = config.common + problems = [ + f"common.{name}={getattr(common, name)!r}: each image's (and mask's) axes are the roles the " + "preprocessing run recorded for it; leave it at its default 'YX'" + for name in ("image_axes", "mask_axes") + if getattr(common, name) != _DEFAULT_AXES + ] + if common.mask_source != "tiff_tree" or common.mask_label_name is not None: + problems.append( + f"common.mask_source={common.mask_source!r}, common.mask_label_name={common.mask_label_name!r}: the " + "masks are those the preprocessing run paired (masks-folder TIFFs and OME-Zarr label images); " + "choose the type with matching.mask_type (a label image's type is its name)" + ) + problems += [ + f"common.{name}={getattr(common, name)!r}: each image's labels are its folders, as the preprocessing " + "run recorded them; leave it unset" + for name in _LABELS + if getattr(common, name) not in (None, SINGLE_CONDITION_LABEL) + ] + if problems: + raise ValueError( + f"{where} reads the images, masks and labels through the preprocessing run since CellPatcher " + "v0.2.0 (design v0.4 §6), so these settings are not used for files: " + "; ".join(problems) + ". " + "The axes and labels still describe in-memory arrays given to run_patching; mask_source and " + "mask_label_name are not used there." + ) + if common.preprocessing_run is None: + _refuse_without_run(where) + + +def _refuse_without_run(where: str) -> None: + """Raise ``no_preprocessing_run``, with the preprocessing command.""" + from morphobricks_preprocessing.validation import error + + raise PreprocessingCheckError([ + error( + "no_preprocessing_run", + f"{where} reads files only through a clean preprocessing run (v0.4), and " + "common.preprocessing_run is not set.", + [], + f"Run preprocessing on the experiment first: {PREPROCESSING_COMMAND}. Then set " + "common.preprocessing_run to its output folder (by default _MorphoBricks_Preprocessing, " + "beside the experiment).", + ) + ]) + + +def masks_used(config: CellPatcherConfig, masks_root: str | os.PathLike | None) -> bool: + """Return whether the run reads masks: always in ``cell`` mode, in ``random`` mode when asked. + + ``random`` mode reads them when a masks folder or ``matching.mask_type`` is given. + + Raises + ------ + ValueError + For a masks folder in ``grid`` or ``whole_fov`` mode, which use no masks. + """ + if config.mode in {"grid", "whole_fov"}: + if masks_root is not None: + raise ValueError( + f"mask_folder_path is not supported for mode='{config.mode}'. " + "Remove it, or switch to 'cell' or 'random' mode." + ) + return False + return config.mode == "cell" or masks_root is not None or config.matching.mask_type is not None + + +def open_file_run( + config: CellPatcherConfig, + experiment_root: str | os.PathLike, + masks_root: str | os.PathLike | None, + *, + use_masks: bool, + where: str, +) -> FileRun: + """Open ``common.preprocessing_run`` with where the experiment and masks are now, before any read. + + *use_masks* (:func:`masks_used`) says whether masks are read; *masks_root* is where the + run's masks folder is now (None: where the run recorded it) either way. Must be called + inside :func:`cellpatcher.run_session.statistics_session`: the run's table is kept there + for the call's normalization. + + Raises + ------ + PreprocessingCheckError + Without a preprocessing run, when ``RunInputs.open`` refuses it, or when a check of + the normalization inputs fails. + ValueError + For a setting a file run does not read, or masks that cannot be used. + """ + refuse_file_run_settings(config, where) + check_config_identity(config) + inputs = open_run(config.common.preprocessing_run, experiment_root=experiment_root, masks_root=masks_root) + fovs = inputs.fovs() + if not fovs: + raise ValueError(f"The preprocessing run in {inputs.output_folder} has no images.") + layout = run_layout(fovs) + mask_type = _mask_type(config, inputs) if use_masks else None + mask_axes = _DEFAULT_AXES + if mask_type is not None: + mask_axes = run_mask_layout([_record(inputs, fov, mask_type) for fov in fovs]) + _check_object_set(config, mask_type) + statistics = _csv_statistics(config, inputs, layout) + return FileRun(inputs, layout, mask_type, mask_axes, statistics) + + +def _mask_type(config: CellPatcherConfig, inputs: RunInputs) -> str: + """Return the mask type the run reads: ``matching.mask_type``, or the run's only type.""" + types = inputs.mask_types() + wanted = config.matching.mask_type + if not types: + raise ValueError( + f"mode={config.mode!r} reads masks, but the preprocessing run in {inputs.output_folder} recorded none. " + f"Run preprocessing with the masks folder ({PREPROCESSING_COMMAND})." + ) + if wanted is None and len(types) > 1: + raise ValueError( + f"The preprocessing run recorded {len(types)} mask types ({', '.join(types)}); set " + "matching.mask_type to the one to use." + ) + if wanted is not None and wanted not in types: + raise ValueError( + f"matching.mask_type={wanted!r} is not a mask type of the preprocessing run, which recorded " + f"{', '.join(map(repr, types))}." + ) + return wanted if wanted is not None else types[0] + + +def _record(inputs: RunInputs, fov: Fov, mask_type: str) -> Mapping: + """Return an image's recorded mask of *mask_type*, refusing an image the run recorded none for.""" + record = mask_record(inputs, fov, mask_type) + if record is None: + raise ValueError( + f"The preprocessing run recorded no {mask_type!r} mask for {fov.relative_path}; every image needs one. " + "Add its mask and run preprocessing again." + ) + if not record.get("roles"): + # rc1 writes a run whose mask header was refused (mask_layout) with no axes for it. + raise ValueError( + f"The preprocessing run recorded no axes for the mask {record['path']}: its header was not usable (the " + "run's report says why, for example X before Y). Fix the mask and run preprocessing again." + ) + return record + + +def _check_object_set(config: CellPatcherConfig, mask_type: str) -> None: + """Refuse an object set the mask type gives that cannot be one, before any read (issue #42).""" + if config.mode == "cell" and config.common.object_set is None: + validate_object_set(object_set_from_mask_type(mask_type), name=f"The object set from mask type {mask_type!r}") + + +def _csv_statistics(config: CellPatcherConfig, inputs: RunInputs, layout: str) -> PlaneStatistics | None: + """Check a CSV normalization scope against the opened run, before any image is read. + + The run's own table, image type and (for ``biological_repeat``) design; every image must + have rows in the table. + """ + settings = normalization_settings(config, inputs) + if not settings.enabled or settings.scope not in CSV_SCOPES: + return None + validate_normalization(replace(config, normalization=settings)) + statistics = statistics_of(inputs) + keep_in_session(statistics) + check_image_type(settings, statistics) + if settings.scope == "biological_repeat": + load_experiment_design(settings, statistics) + rows_only = PlaneStatistics(statistics.plane_histograms, statistics.records, statistics.csv_sha256) + check_images(rows_only, {fov.relative_path: layout for fov in inputs.fovs()}) + return statistics + + +def normalization_settings(config: CellPatcherConfig, inputs: RunInputs) -> NormalizationConfig: + """Return ``config.normalization`` reading the run's own table at a CSV scope. + + ``normalization.plane_histograms`` may be left unset (the run's table is used) or name + the run's own table. + + Raises + ------ + PreprocessingCheckError + ``not_the_run_output`` when it names another table. + """ + from morphobricks_normalization import OUTPUT_FILENAME + + settings = config.normalization + if not settings.enabled or settings.scope not in CSV_SCOPES: + return settings + table = inputs.output_folder / OUTPUT_FILENAME + if settings.plane_histograms is not None: + _refuse_other_file("normalization.plane_histograms", Path(settings.plane_histograms).expanduser(), table) + return replace(settings, plane_histograms=str(table)) + + +def leaves(run: FileRun) -> list[tuple[ConditionRepeatFolder, list[Fov]]]: + """Group the run's images by condition, biological and technical repeat, sorted by those labels. + + Each group's folder is where its images are now (the run's folders, under the experiment + root it was opened with). + """ + groups: dict[tuple[str, str, str], list[Fov]] = {} + for fov in run.fovs: + groups.setdefault(fov.key[:3], []).append(fov) + result = [] + for (condition, biological_repeat, technical_repeat), fovs in sorted(groups.items()): + folder = run.inputs.experiment_root / PurePosixPath(fovs[0].relative_path).parent + leaf = ConditionRepeatFolder(condition, biological_repeat, folder, technical_repeat=technical_repeat) + result.append((leaf, fovs)) + return result + + +def run_config_for(config: CellPatcherConfig, run: FileRun, labels: Sequence[str]) -> CellPatcherConfig: + """Return the config the patching core runs with: the run's layouts, labels and table.""" + common = replace( + config.common, + image_axes=run.layout, + mask_axes=run.mask_axes, + **dict(zip(_LABELS, labels)), + ) + return replace(config, common=common, normalization=normalization_settings(config, run.inputs)) + + +def patch_fovs(run: FileRun, fovs: Sequence[Fov], config: CellPatcherConfig) -> PatchingResult: + """Patch images of the run with *config* (from :func:`run_config_for`), reading them through ``RunInputs``. + + With ``z_policy``/``t_policy`` splitting planes, one image is read at a time and only its + chosen planes; otherwise every image is read whole, as before. + """ + common = config.common + metadata = [fov_metadata(run, fov, config) for fov in fovs] + if splits_planes(common): + return _patch_planes(config, _plane_sources(run, fovs, metadata, config)) + images = [read_fov(run.inputs, fov, run.layout)[0] for fov in fovs] + masks = None if run.mask_type is None else [_read_mask(run, fov) for fov in fovs] + return run_patching(images=images, masks=masks, config=config, image_metadata=metadata) + + +def _plane_sources(run: FileRun, fovs: Sequence[Fov], metadata: Sequence[dict], config: CellPatcherConfig) -> Iterator: + """Yield ``(metadata, planes)`` per image, reading one image (and its mask) at a time.""" + common = config.common + for fov, meta in zip(fovs, metadata): + mask = None if run.mask_type is None else _read_mask(run, fov) + planes = iter_fov_planes( + run.inputs, fov, run.layout, mask, run.mask_axes, common, single_planes=config.normalization.enabled + ) + yield meta, planes + + +def _read_mask(run: FileRun, fov: Fov) -> np.ndarray: + """Read an image's mask of the run's type.""" + return read_fov_mask(run.inputs, fov, _record(run.inputs, fov, run.mask_type), run.mask_axes) + + +def fov_metadata(run: FileRun, fov: Fov, config: CellPatcherConfig) -> dict: + """Return an image's ``run_patching(image_metadata=...)`` entry, from the run's records. + + The FOV key is the run's (``position_id`` and ``technical_repeat`` included); the mask + file and, in ``cell`` mode without ``common.object_set``, the object set come from the + recorded mask and its type. Nothing is read transposed (``image_read_transposed`` False). + """ + meta = { + "image_file": PurePosixPath(fov.relative_path).name, + "source_relative_path": fov.relative_path, + "position_id": fov.key[3], + "technical_repeat": fov.key[2], + "image_read_transposed": False, + } + if run.mask_type is not None: + meta["mask_file"] = mask_file_name(_record(run.inputs, fov, run.mask_type)) + if config.mode == "cell" and config.common.object_set is None: + meta["object_set"] = object_set_from_mask_type(run.mask_type) + return meta diff --git a/src/cellpatcher/normalization.py b/src/cellpatcher/normalization.py index 9c19dba..7b1e851 100644 --- a/src/cellpatcher/normalization.py +++ b/src/cellpatcher/normalization.py @@ -44,12 +44,13 @@ from cellpatcher.axes import normalize_axes from cellpatcher.masking import _label_to_2d +from cellpatcher.run_open import _refuse_other_file +from cellpatcher.run_session import session_statistics from cellpatcher.statistics import ( PlaneStatistics, PreprocessingCheckError, SourcePlane, _preprocessing, - _refuse_other_file, check_images, check_planes, load_plane_statistics, @@ -64,7 +65,6 @@ from cellpatcher.config import CellPatcherConfig, NormalizationConfig CSV_SCOPES = ("biological_repeat", "image") -_STATISTICS_CACHE: dict[tuple, PlaneStatistics] = {} class NormalizationWarning(UserWarning): @@ -117,31 +117,27 @@ def validate_normalization(config) -> None: def cached_statistics( - plane_histograms: str | os.PathLike, manifest: str | os.PathLike | None = None + plane_histograms: str | os.PathLike, + manifest: str | os.PathLike | None = None, + *, + experiment_root: str | os.PathLike | None = None, + masks_root: str | os.PathLike | None = None, ) -> PlaneStatistics: - """:func:`~cellpatcher.statistics.load_plane_statistics`, once per unchanged table and manifest. + """:func:`~cellpatcher.statistics.load_plane_statistics`, opened once per CellPatcher call. - A tree run normalizes every leaf with the same, possibly large, table; the - key is the path, size and modification time of the table and the manifest. + Within one call (:mod:`cellpatcher.run_session`: a tree run normalizes every leaf with + the same, possibly large, table) the run is opened once; outside a call, or in the next + one, it is opened again, so ``RunInputs.open`` checks every image and mask of the run each + time. Nothing is reused across calls. """ csv_path = Path(plane_histograms).expanduser() - if not csv_path.is_file(): - return load_plane_statistics(csv_path, manifest) # raises the plain-language error - key = (*_file_key(csv_path), *_file_key(csv_path.parent / _preprocessing().MANIFEST_FILENAME), str(manifest)) - if key not in _STATISTICS_CACHE: - if len(_STATISTICS_CACHE) >= 4: - _STATISTICS_CACHE.pop(next(iter(_STATISTICS_CACHE))) - _STATISTICS_CACHE[key] = load_plane_statistics(csv_path, manifest) - return _STATISTICS_CACHE[key] - - -def _file_key(path: Path) -> tuple[str, int | None, int | None]: - """Return a file's resolved path, size and modification time (None, None when missing).""" - resolved = path.resolve() - if not resolved.is_file(): - return str(resolved), None, None - stat = resolved.stat() - return str(resolved), stat.st_size, stat.st_mtime_ns + statistics = session_statistics( + csv_path, + lambda: load_plane_statistics(csv_path, manifest, experiment_root=experiment_root, masks_root=masks_root), + ) + if manifest is not None: + _refuse_other_file("normalization.manifest", Path(manifest).expanduser(), statistics.manifest_path) + return statistics def experiment_design_path(settings: NormalizationConfig) -> Path: diff --git a/src/cellpatcher/orchestration.py b/src/cellpatcher/orchestration.py index 797e14c..e8107f3 100644 --- a/src/cellpatcher/orchestration.py +++ b/src/cellpatcher/orchestration.py @@ -1,11 +1,11 @@ from __future__ import annotations import copy -import functools +import os +from collections.abc import Callable from dataclasses import asdict, dataclass, replace from pathlib import Path import json -import warnings from typing import Any from cellpatcher._deprecation import accept_deprecated_repeat @@ -22,43 +22,11 @@ RandomConfig, WholeFOVConfig, ) -from cellpatcher.io import ( - SINGLE_TECHNICAL_REPEAT_LABEL, - ConditionRepeatFolder, - discover_condition_repeat_folders, - discover_folder_images, - list_supported_files, - is_ome_zarr, - read_image, - refuse_removed_matching, - pair_masks, - refuse_image_header_axes, - refuse_mask_header_axes, - refuse_transposed_mask_axes, - read_ngff_mask, - refuse_mask_channel_axis, - source_relative_path, -) +from cellpatcher.file_runs import FileRun, leaves, masks_used, open_file_run, patch_fovs, run_config_for +from cellpatcher.io import SINGLE_TECHNICAL_REPEAT_LABEL, ConditionRepeatFolder, refuse_removed_matching from cellpatcher.export import IMAGE_TABLE_FILENAME, save_image_table_csv, save_metadata_csv -from cellpatcher.ids import object_set_from_label_path, object_set_from_mask_type, validate_object_set -from cellpatcher.normalization import ( - CSV_SCOPES, - cached_statistics, - check_image_type, - load_experiment_design, - validate_normalization, -) -from cellpatcher.pipeline import ( - RUN_CONFIG_FILENAME, - PatchingResult, - _patch_planes, - check_config_identity, - check_source_identities, - run_config_data, - run_patching, -) -from cellpatcher.statistics import check_images -from cellpatcher.planes import iter_file_planes, splits_planes +from cellpatcher.pipeline import RUN_CONFIG_FILENAME, PatchingResult, run_config_data +from cellpatcher.run_session import statistics_session # Sections always written regardless of mode _COMMON_SECTIONS = ("common", "filters", "export", "extraction", "normalization") @@ -310,187 +278,24 @@ def _config_from_dict(data: dict) -> CellPatcherConfig: ) -def _mask_loaders(experiment_root, mask_folder_path, image_files, config, pairing=None): - """Resolve the mask of every image, without reading any: one loader per image, or None. - - Returns ``(loaders, mask_files, mask_type)``: each loader returns - ``(mask, mask_file name)``; *mask_files* holds each mask's file name - when the pairing gives it without a read (a mask folder), else None (an - OME-NGFF label image, whose name may need its metadata); *mask_type* is - the mask type of a mask folder, else None. All are None without masks. - Masks come from a mask folder paired with the preprocessing brick's rule - (``common.mask_source='tiff_tree'``; :func:`~cellpatcher.io.pair_masks` - with ``config.matching.mask_type``, or the given *pairing* of a tree - run), or from each OME-Zarr image's own label image - (``'ngff_labels'``). Returns Nones when the run uses no masks: modes - ``grid``/``whole_fov``, or ``random`` without a mask folder. - - Raises - ------ - ValueError - If the mode and the mask inputs do not fit, or images and masks do - not pair one to one. - """ - if config.mode in {"grid", "whole_fov"}: - if mask_folder_path is not None: - raise ValueError( - f"mask_folder_path is not supported for mode='{config.mode}'. " - "Remove it, or switch to 'cell' or 'random' mode." - ) - return None, None, None - if config.common.mask_source == "ngff_labels": - if mask_folder_path is not None: - raise ValueError( - "common.mask_source='ngff_labels' reads the masks from the OME-Zarr images' own labels; " - "remove mask_folder_path (or mask_root_path), or use common.mask_source='tiff_tree'." - ) - not_zarr = [image.name for image in image_files if not is_ome_zarr(image)] - if not_zarr: - raise ValueError( - "common.mask_source='ngff_labels' needs OME-Zarr images, which hold their labels, but " - f"{len(not_zarr)} image(s) are not OME-Zarr: {', '.join(not_zarr[:10])}. Use " - "common.mask_source='tiff_tree' with a mask folder for them." - ) - refuse_transposed_mask_axes(config.common.mask_axes) # read by role into mask_axes, never transposed - loaders = [ - functools.partial(read_ngff_mask, image, config.common.mask_label_name, config.common.mask_axes) - for image in image_files - ] - return loaders, [None] * len(image_files), None - if mask_folder_path is None: - if config.mode == "cell": - raise ValueError("mask_folder_path is required when config.mode='cell'.") - return None, None, None # random mode: masks are optional - refuse_mask_channel_axis(config.common.mask_axes) - if pairing is None: - pairing = pair_masks(experiment_root, image_files, mask_folder_path, mask_type=config.matching.mask_type) - selected = pairing.masks - refuse_mask_header_axes([selected[image] for image in image_files], config.common.mask_axes) - loaders = [functools.partial(_read_mask_file, selected[image], config.common.mask_axes) for image in image_files] - return loaders, [selected[image].name for image in image_files], pairing.mask_type - - -def _warn_transposed(paths: set[str | Path]) -> None: - """Warn that the images in *paths* are read with their header's Y and X swapped, as declared. - - Since preprocessing v0.4 a swap is refused (:func:`cellpatcher.io.header_axes_disagreement`), - so *paths* is always empty; this goes with ``image_read_transposed``. - """ - if not paths: - return - names = sorted(Path(path).name for path in paths) - warnings.warn( - f"common.image_axes declares Y and X swapped relative to the header of {len(names)} image file(s) " - f"({', '.join(names[:10])}{', ...' if len(names) > 10 else ''}): they are read transposed, and " - "images.csv records it (image_read_transposed).", - UserWarning, - stacklevel=3, - ) - - -def _read_mask_file(path: Path, axes: str): - return read_image(path, axes), path.name - - -def _check_preprocessing_inputs(config: CellPatcherConfig, experiment_root, image_files) -> None: - """Check the images against the preprocessing outputs before any patching (contract §5). - - Only with normalization at a CSV scope: every image must have rows in - ``plane_histograms.csv`` (all images without rows in one message) and, - with a manifest, the size, modification time and axes recorded for it, - and the image type it records. - The planes themselves are checked as they are patched. - """ - settings = config.normalization - if not settings.enabled or settings.scope not in CSV_SCOPES: - return - validate_normalization(config) - statistics = cached_statistics(settings.plane_histograms, settings.manifest) - check_image_type(settings, statistics) - if settings.scope == "biological_repeat": - load_experiment_design(settings, statistics) # a design problem stops the run before any image is read - image_axes = {_source_relative_path(path, experiment_root): config.common.image_axes for path in image_files} - check_images(statistics, image_axes, experiment_root=Path(experiment_root).expanduser()) - - -def _source_relative_path(image_path: Path, experiment_root) -> str: - """Path of *image_path* relative to *experiment_root* (:func:`cellpatcher.io.source_relative_path`).""" - return source_relative_path(image_path, experiment_root) - - -def _mask_object_set(config: CellPatcherConfig, mask_file: str | None, mask_type: str | None) -> str | None: - """The object set a cell-mode mask gives, or None (``common.object_set`` or ``"mask"`` then). - - Only when ``common.object_set`` is not set: an OME-NGFF mask gives its - label image name; a mask file its mask type from the preprocessing - pairing (``img_cells.tif`` or ``cells/img.tif`` -> ``"cells"``). - """ - if config.mode != "cell" or mask_file is None or config.common.object_set is not None: - return None - if config.common.mask_source == "ngff_labels": - return object_set_from_label_path(mask_file) - return None if mask_type is None else object_set_from_mask_type(mask_type) - - -def _check_mask_object_sets(config: CellPatcherConfig, image_files, mask_files, mask_type) -> None: - """Refuse an invalid ``object_set`` taken from a paired mask before any image or mask is read. - - *mask_files* are the paired mask names (None where only the read gives - the name, an OME-NGFF label image); the object set is derived exactly as - :func:`_image_metadata` derives it later (issue #42). - """ - for image_path, mask_file in zip(image_files, mask_files): - object_set = _mask_object_set(config, mask_file, mask_type) - validate_object_set(object_set, name=f"The object set from mask {mask_file!r} of image {Path(image_path).name!r}") - - -def _image_metadata( - image_path: Path, - mask_file: str | None, - experiment_root, - config: CellPatcherConfig, - mask_type: str | None = None, - *, - image_transposed: bool | None = None, -) -> dict: - """Build one image's ``run_patching(image_metadata=...)`` entry. - - Carries the source file names (and, with *experiment_root*, the image's - ``source_relative_path``) through so they land on every ``PatchRegion`` - (and the exported metadata CSV) *before* ``export_patches`` writes it — - passing this into ``run_patching`` rather than stamping the result - afterwards is what makes that ordering guarantee hold. The image file - name also gives the image's default ``position_id``, and the mask its - ``object_set`` (:func:`_mask_object_set`). Whether the image is read - transposed (its header's Y and X declared swapped) goes to - ``images.csv``; None where it was not checked. - """ - meta = {"image_file": image_path.name} - if image_transposed is not None: - meta["image_read_transposed"] = image_transposed - if mask_file is not None: - meta["mask_file"] = mask_file - if experiment_root is not None: - meta["source_relative_path"] = _source_relative_path(image_path, experiment_root) - object_set = _mask_object_set(config, mask_file, mask_type) - if object_set is not None: - meta["object_set"] = object_set - return meta - - def run_patching_pipeline( - image_folder_path, + image_folder_path: str | os.PathLike, *, config: CellPatcherConfig | None = None, - mask_folder_path=None, + mask_folder_path: str | os.PathLike | None = None, overrides: dict[str, Any] | None = None, - **removed_options, -): - """Orchestrate a folder-based patching run end-to-end. + **removed_options: object, +) -> PatchingResult: + """Patch the images of a clean preprocessing run of one condition and biological repeat. - Reads all supported image files from *image_folder_path*, optionally - matches and loads mask files, then calls :func:`run_patching` with the - supplied config. + The run is ``config.common.preprocessing_run`` (its output folder), opened + with ``RunInputs.open`` with *image_folder_path* as the experiment root + (:mod:`cellpatcher.file_runs`, design v0.4 §6): the images patched are the + run's, read with the roles it recorded, with their labels and + ``position_id`` as it recorded them, and their masks are those it paired. + Without a run, a ``PreprocessingCheckError`` (``no_preprocessing_run``) + gives the preprocessing command. A run with several conditions or + biological repeats is refused: use :func:`run_patching_condition_repeat_tree`. **Typical usage — direct attribute editing (recommended for interactive use)** @@ -500,6 +305,7 @@ def run_patching_pipeline( from cellpatcher import CellPatcherConfig, run_patching_pipeline cfg = CellPatcherConfig(mode="grid") + cfg.common.preprocessing_run = "./images_MorphoBricks_Preprocessing" cfg.grid.tile_size = 64 cfg.grid.overlap = 0.1 cfg.common.output_dir = "./patches" @@ -546,27 +352,28 @@ def run_patching_pipeline( Parameters ---------- image_folder_path: - Path to the folder containing image files. + The experiment root: where the preprocessed experiment is now (it + may have been moved or copied since the run; its images must be + unchanged). config: A :class:`~cellpatcher.CellPatcherConfig` instance. Defaults to ``CellPatcherConfig()`` (cell mode with all defaults). mask_folder_path: - Path to the folder containing mask files. - - * **Required** when ``config.mode='cell'``, unless - ``config.common.mask_source='ngff_labels'``, which reads each - OME-Zarr image's own label image instead (then it must not be - given). - * **Optional** when ``config.mode='random'`` — masks restrict the - spatial region from which patches are sampled. + Where the run's masks folder is now; None: where the run recorded + it. The masks are those the run paired (masks-folder TIFFs and + OME-Zarr label images), of the type ``config.matching.mask_type`` + (None: the run's only type). + + * ``'cell'`` mode always reads masks. + * ``'random'`` mode reads them when this or ``matching.mask_type`` + is given — masks restrict where patches are sampled. * **Must not be provided** for ``'grid'`` or ``'whole_fov'`` modes (those modes do not consume masks; a ``ValueError`` is raised if this argument is set for those modes). strategy, mask_suffixes, case_sensitive: Former matching options, removed in v0.1.1 and refused by name: - images and masks are paired with the preprocessing brick's rule - (:func:`~cellpatcher.io.pair_masks`); choose the mask type with - ``config.matching.mask_type``. + the masks are those the preprocessing run paired; choose the mask + type with ``config.matching.mask_type``. overrides: Optional dotted-path overrides applied on top of *config* before running. Keys follow the same ``"
."`` convention as @@ -579,96 +386,46 @@ def run_patching_pipeline( ------- PatchingResult Regions, optional patch arrays, and exported file paths. Each - patch's ``source_relative_path`` is its image's file name: the image - folder is the experiment root of this run. + patch's ``source_relative_path`` is its image's path relative to the + experiment root, as the run recorded it. + + Raises + ------ + PreprocessingCheckError + Without ``common.preprocessing_run``, or when ``RunInputs`` refuses + the run (an image or mask changed or missing, a run before v0.4). + ValueError + For ``common.image_axes``/``mask_axes`` other than ``"YX"``, + ``mask_source``/``mask_label_name``, or labels other than + ``"single"``: file runs read these from the run (the message says so). """ + config = _checked_config(config, overrides) + refuse_removed_matching("run_patching_pipeline", removed_options) + with statistics_session(): + use_masks = masks_used(config, mask_folder_path) + run = open_file_run( + config, image_folder_path, mask_folder_path, use_masks=use_masks, where="run_patching_pipeline" + ) + groups = leaves(run) + labels = {(leaf.condition, leaf.biological_repeat) for leaf, _ in groups} + if len(labels) > 1: + raise ValueError( + f"The preprocessing run has {len(labels)} conditions or biological repeats; " + "run_patching_pipeline patches one. Use run_patching_condition_repeat_tree for the whole " + "experiment." + ) + ((condition, biological_repeat),) = labels + leaf_config = run_config_for(config, run, (condition, biological_repeat, SINGLE_TECHNICAL_REPEAT_LABEL)) + return patch_fovs(run, run.fovs, leaf_config) + + +def _checked_config(config: CellPatcherConfig | None, overrides: dict[str, Any] | None) -> CellPatcherConfig: + """Return *config* (default: ``CellPatcherConfig()``) with *overrides* applied.""" if config is None: config = CellPatcherConfig() if not isinstance(config, CellPatcherConfig): raise TypeError("config must be a CellPatcherConfig instance.") - if overrides: - config = apply_overrides(config, overrides) - refuse_removed_matching("run_patching_pipeline", removed_options) - check_config_identity(config) # before any image or mask is listed or read (D85) - # The preprocessing brick's discovery (issue #45): a folder mixing images with - # subfolders, PNG/NumPy/... images and a mask folder inside it are refused here. - masks_used = mask_folder_path is not None and config.mode not in {"grid", "whole_fov"} - image_files = discover_folder_images(image_folder_path, mask_folder_path=mask_folder_path if masks_used else None) - _check_preprocessing_inputs(config, image_folder_path, image_files) - return _run_folder_patching( - image_folder_path, - config=config, - mask_folder_path=mask_folder_path, - experiment_root=image_folder_path, - ) - - -def _run_folder_patching( - image_folder_path, - *, - config: CellPatcherConfig, - mask_folder_path, - experiment_root, - pairing=None, -) -> PatchingResult: - """Body of :func:`run_patching_pipeline` for one image folder. - - *experiment_root* is the folder each image's ``source_relative_path`` is - relative to: the image folder itself for a flat run, the tree root for - :func:`run_patching_condition_repeat_tree`, which passes the *pairing* - of the whole tree. - """ - check_config_identity(config) # before any image or mask is listed or read (D85) - image_files = list_supported_files(image_folder_path) - if len(image_files) == 0: - raise ValueError("No supported image files found in image folder.") - check_source_identities(config, image_files) # from the names, before any read - # Headers only, before any pixel is read. - transposed_images = refuse_image_header_axes(image_files, config.common.image_axes) - _warn_transposed(transposed_images) - image_transposed = [ # an OME-Zarr image is matched by role, not checked against its header - None if is_ome_zarr(path) else path in transposed_images for path in image_files - ] - # image_files' sorted order fixes the image_0000, image_0001, ... - # image_id values (a run-local index); patch_id does not use them. - loaders, matched_mask_files, mask_type = _mask_loaders( - experiment_root, mask_folder_path, image_files, config, pairing - ) - if matched_mask_files is not None: - _check_mask_object_sets(config, image_files, matched_mask_files, mask_type) # before any read - common = config.common - - if splits_planes(common): - # One image at a time: its mask, then only its chosen planes - # (TIFF and OME-Zarr images read plane by plane with the preprocessing brick). - def sources(): - for index, image_path in enumerate(image_files): - mask, mask_file = loaders[index]() if loaders is not None else (None, None) - yield ( - _image_metadata( - image_path, mask_file, experiment_root, config, mask_type, - image_transposed=image_transposed[index], - ), - iter_file_planes( - image_path, common.image_axes, mask, common.mask_axes, common, - single_planes=config.normalization.enabled, - ), - ) - - return _patch_planes(config, sources()) - - masks, mask_files = None, [None] * len(image_files) - if loaders is not None: - masks, mask_files = (list(values) for values in zip(*(load() for load in loaders))) - # OME-Zarr images are put into common.image_axes by role (read_image). - images = [read_image(path, common.image_axes) for path in image_files] - image_metadata = [ - _image_metadata( - path, mask_file, experiment_root, config, mask_type, image_transposed=image_transposed[index], - ) - for index, (path, mask_file) in enumerate(zip(image_files, mask_files)) - ] - return run_patching(images=images, masks=masks, config=config, image_metadata=image_metadata) + return apply_overrides(config, overrides) if overrides else config @accept_deprecated_repeat @@ -691,45 +448,36 @@ class ConditionRepeatResult: def run_patching_condition_repeat_tree( - image_root_path, + image_root_path: str | os.PathLike, *, config: CellPatcherConfig | None = None, - mask_root_path=None, + mask_root_path: str | os.PathLike | None = None, overrides: dict[str, Any] | None = None, combine_metadata: bool = False, - progress_callback=None, - **removed_options, + progress_callback: Callable[[int, int, ConditionRepeatFolder | None], object] | None = None, + **removed_options: object, ) -> list[ConditionRepeatResult]: - """Run :func:`run_patching_pipeline` once per - ``/[/]`` leaf. + """Patch every image of a clean preprocessing run, one leaf per + ``/[/]``. (The function keeps its historical ``condition_repeat`` name; the folder level it reads below each condition is the biological repeat.) - Walks *image_root_path* with - :func:`~cellpatcher.io.discover_condition_repeat_folders`, which accepts - both the full MorphoBricks layout - (``//``) and a condition folder - holding images directly with no biological-repeat subfolder — the latter - is treated as a single implicit biological repeat, labeled - ``cellpatcher.io.SINGLE_BIOLOGICAL_REPEAT_LABEL`` (``"single"``), - consistently across input discovery, processing (the - ``config.common.biological_repeat`` value stamped on every patch from - that condition), and output (its own output subfolder, same as an - explicit biological repeat would get). A biological-repeat folder may in - turn hold one subfolder per technical repeat (a well, a dish): - ``///``; a - biological-repeat folder holding its images directly is one technical - repeat, labeled ``cellpatcher.io.SINGLE_TECHNICAL_REPEAT_LABEL`` - (``"single"``). - - For each discovered leaf, a copy of *config* is made with - ``common.condition``/``common.biological_repeat``/``common.technical_repeat`` - set to the values discovered from the folder path. When + The run is ``config.common.preprocessing_run``, opened with + ``RunInputs.open`` with *image_root_path* as the experiment root and + *mask_root_path* as its masks folder (:mod:`cellpatcher.file_runs`, + design v0.4 §6). Its images are grouped by the labels the run recorded + for them (``"single"`` for a level an experiment does not have, as the + preprocessing brick names it), read with the roles it recorded, and + their masks are those it paired; no name is parsed and no axis is + given. Without a run, a ``PreprocessingCheckError`` + (``no_preprocessing_run``) gives the preprocessing command. + + Each leaf gets a copy of *config* with its labels. When ``config.export.enabled`` and ``config.common.output_dir`` are set, each leaf's ``output_dir`` becomes ``//``, or ``///`` for a - technical-repeat subfolder, mirroring the input layout. ``image_id`` + technical repeat other than ``"single"``, mirroring the input layout. ``image_id`` restarts at ``image_0000`` in every leaf; ``patch_id`` starts with the image's FOV key (``///``, D85), so it is unique across the whole tree. @@ -737,20 +485,20 @@ def run_patching_condition_repeat_tree( Parameters ---------- image_root_path: - Root folder containing one subfolder per condition. + The experiment root: where the preprocessed experiment is now (it + may have been moved or copied since the run; its images must be + unchanged). config: Base :class:`~cellpatcher.CellPatcherConfig`, applied to every leaf - (with ``common.condition``/``.biological_repeat``/``.output_dir`` overridden - per leaf as described above). Defaults to ``CellPatcherConfig()``. + (with the labels and ``.output_dir`` set per leaf as described + above). Defaults to ``CellPatcherConfig()``. mask_root_path: - Mask folder mirroring *image_root_path*'s folders, read with the - preprocessing brick's pairing over the whole tree - (:func:`~cellpatcher.io.pair_masks`): - ``///.../.tif`` - or ``//.../_.tif``. - Required for ``mode='cell'``; optional for ``mode='random'``, where, - when given, every image of the tree must have a mask; ignored for - ``'grid'``/``'whole_fov'``. + Where the run's masks folder is now; None: where the run recorded + it. ``mode='cell'`` reads the masks of type + ``config.matching.mask_type`` (None: the run's only type); + ``mode='random'`` reads them when this or ``matching.mask_type`` is + given, and every image must then have one; ``'grid'``/``'whole_fov'`` + read none. overrides: Same as :func:`run_patching_pipeline`, applied to every leaf. The removed matching options are refused by name, as there. @@ -777,119 +525,63 @@ def run_patching_condition_repeat_tree( Returns ------- list[ConditionRepeatResult] - One entry per discovered leaf, in the order - :func:`~cellpatcher.io.discover_condition_repeat_folders` returns - them (conditions, then biological repeats, then technical repeats, - each sorted by name). + One entry per leaf: conditions, then biological repeats, then + technical repeats, each sorted by name. Raises ------ - ValueError - If no condition subfolders with images are found, or (propagated - from :func:`run_patching_pipeline`) if a leaf fails validation, e.g. - a missing required mask folder. The first failing leaf aborts the - whole run rather than silently skipping it. + PreprocessingCheckError, ValueError + As :func:`run_patching_pipeline`, before any image is read. A + failing leaf aborts the whole run rather than being skipped. """ - if config is None: - config = CellPatcherConfig() - if not isinstance(config, CellPatcherConfig): - raise TypeError("config must be a CellPatcherConfig instance.") - if overrides: - config = apply_overrides(config, overrides) + config = _checked_config(config, overrides) if combine_metadata and (not config.export.enabled or config.common.output_dir is None): raise ValueError( "combine_metadata=True requires config.export.enabled=True and config.common.output_dir to be set." ) - refuse_removed_matching("run_patching_condition_repeat_tree", removed_options) - if config.common.mask_source == "ngff_labels" and mask_root_path is not None: - raise ValueError( - "common.mask_source='ngff_labels' reads the masks from the OME-Zarr images' own labels; " - "remove mask_root_path, or use common.mask_source='tiff_tree'." + # grid/whole_fov read no masks: a masks folder given for them only says where the run's is now. + use_masks = masks_used(config, None if config.mode in {"grid", "whole_fov"} else mask_root_path) + with statistics_session(): + run = open_file_run( + config, image_root_path, mask_root_path, use_masks=use_masks, where="run_patching_condition_repeat_tree" ) + groups = leaves(run) + results = _patch_leaves(config, run, groups, combine_metadata, progress_callback) + if combine_metadata: # in the session: its normalization_inputs come from the run opened above + tree_config = run_config_for(config, run, (SINGLE_TECHNICAL_REPEAT_LABEL,) * 3) + _write_combined_metadata(tree_config, Path(config.common.output_dir), results) + if progress_callback is not None: + progress_callback(len(groups), len(groups), None) + return results - leaves = discover_condition_repeat_folders(image_root_path, mask_root_path=mask_root_path) - if not leaves: - raise ValueError(f"No condition subfolders with images found under: {image_root_path}") - # Checked once for the whole tree, so every image without rows is listed together. - _check_preprocessing_inputs( - config, image_root_path, [path for leaf in leaves for path in list_supported_files(leaf.image_folder)] - ) - base_output_dir = Path(config.common.output_dir) if config.common.output_dir is not None else None - # grid/whole_fov never accept a mask_folder_path (run_patching_pipeline - # raises if given one), so a mask folder existing on disk for those - # modes must be ignored rather than passed through. - masks_allowed = config.mode not in {"grid", "whole_fov"} +def _patch_leaves( + config: CellPatcherConfig, + run: FileRun, + groups: list[tuple[ConditionRepeatFolder, list]], + combine_metadata: bool, + progress_callback: Callable | None, +) -> list[ConditionRepeatResult]: + """Patch each leaf of a tree run with its own labels and output folder. - # With combined metadata, the per-leaf CSV and run config are replaced by - # one pair written in output_dir after all leaves have run. + Every leaf gets the base settings, its labels from the run, and, with + ``common.output_dir``, ``//[/]``. + With combined metadata the per-leaf CSV and run config are replaced by one + pair written in ``output_dir`` after all leaves have run. + """ + base_output_dir = Path(config.common.output_dir) if config.common.output_dir is not None else None leaf_export = ( replace(config.export, save_metadata_csv=False, save_run_config=False) if combine_metadata else config.export ) - - def leaf_config_of(leaf): - # Derive this leaf's config from the shared base: same settings - # everywhere, but condition/biological_repeat/technical_repeat/ - # output_dir specific to this leaf. - leaf_common = replace( - config.common, - condition=leaf.condition, - biological_repeat=leaf.biological_repeat, - technical_repeat=leaf.technical_repeat, - ) - if base_output_dir is not None: - leaf_common.output_dir = str(base_output_dir / leaf.relative_folder) - return replace(config, common=leaf_common, export=leaf_export) - - # The masks of the whole tree are paired at once with the preprocessing - # brick's rule, whose mask folder mirrors the whole image root - # (//... or /.../_.tif). - leaf_mask_folder = str(mask_root_path) if masks_allowed and mask_root_path is not None else None - - # Every leaf's identity first (D85), from names only: a refusal in a later leaf - # comes before any leaf's image or mask is read. That includes the mask - # pairing and the object set a paired mask gives (issue #42). - leaf_images = [] - for leaf in leaves: - leaf_config = leaf_config_of(leaf) # its labels are the folder names, object_set the base's - check_config_identity(leaf_config) - image_files = list_supported_files(leaf.image_folder) - check_source_identities(leaf_config, image_files) - leaf_images.append(image_files) - pairing = None - if leaf_mask_folder is not None and config.common.mask_source == "tiff_tree": - refuse_mask_channel_axis(config.common.mask_axes) - pairing = pair_masks( - image_root_path, - [image for images in leaf_images for image in images], - leaf_mask_folder, - mask_type=config.matching.mask_type, - ) - for leaf, image_files in zip(leaves, leaf_images): - leaf_config = leaf_config_of(leaf) - refuse_image_header_axes(image_files, leaf_config.common.image_axes) - _, matched_mask_files, mask_type = _mask_loaders( - image_root_path, leaf_mask_folder, image_files, leaf_config, pairing - ) - if matched_mask_files is not None: - _check_mask_object_sets(leaf_config, image_files, matched_mask_files, mask_type) - results: list[ConditionRepeatResult] = [] - for index, leaf in enumerate(leaves): + for index, (leaf, fovs) in enumerate(groups): if progress_callback is not None: - progress_callback(index, len(leaves), leaf) - leaf_config = leaf_config_of(leaf) - - # source_relative_path is relative to the tree root, as in the - # preprocessing brick, not to the leaf folder. - result = _run_folder_patching( - str(leaf.image_folder), - config=leaf_config, - mask_folder_path=leaf_mask_folder, - experiment_root=image_root_path, - pairing=pairing, - ) + progress_callback(index, len(groups), leaf) + leaf_config = run_config_for(config, run, (leaf.condition, leaf.biological_repeat, leaf.technical_repeat)) + if base_output_dir is not None: + leaf_config.common.output_dir = str(base_output_dir / leaf.relative_folder) + result = patch_fovs(run, fovs, replace(leaf_config, export=leaf_export)) results.append( ConditionRepeatResult( condition=leaf.condition, @@ -900,11 +592,6 @@ def leaf_config_of(leaf): relative_folder=leaf.relative_folder.as_posix(), ) ) - - if combine_metadata: - _write_combined_metadata(config, base_output_dir, results) - if progress_callback is not None: - progress_callback(len(leaves), len(leaves), None) return results diff --git a/src/cellpatcher/pipeline.py b/src/cellpatcher/pipeline.py index 505fc8a..0cd6b76 100644 --- a/src/cellpatcher/pipeline.py +++ b/src/cellpatcher/pipeline.py @@ -23,12 +23,8 @@ resolve_object_set, resolve_position_id, ) -from cellpatcher.io import ( - SINGLE_BIOLOGICAL_REPEAT_LABEL, - SINGLE_CONDITION_LABEL, - SINGLE_TECHNICAL_REPEAT_LABEL, - refuse_transposed_mask_axes, -) +from cellpatcher.io import SINGLE_BIOLOGICAL_REPEAT_LABEL, SINGLE_CONDITION_LABEL, SINGLE_TECHNICAL_REPEAT_LABEL +from cellpatcher.io import refuse_transposed_mask_axes from cellpatcher.masking import apply_label_masks_to_patches from cellpatcher.delivery import raw_export_array from cellpatcher.normalization import ( @@ -40,6 +36,7 @@ split_channel_normalization, validate_normalization, ) +from cellpatcher.run_session import one_session from cellpatcher.statistics import _preprocessing from cellpatcher.planes import iter_image_planes, plane_choices, splits_planes from cellpatcher.resize import normalize_output_size, resize_patches @@ -869,6 +866,7 @@ def _refuse_masking_computed_labels(config: CellPatcherConfig, masks) -> None: raise ValueError(refusal) +@one_session # the preprocessing run is opened once per call (cellpatcher.run_session) def run_patching(images=None, masks=None, config=None, image_metadata=None, fraction_masks=None) -> PatchingResult: """Generate patch regions (and optionally extract arrays) from in-memory images. diff --git a/src/cellpatcher/run_open.py b/src/cellpatcher/run_open.py new file mode 100644 index 0000000..8605eb2 --- /dev/null +++ b/src/cellpatcher/run_open.py @@ -0,0 +1,245 @@ +"""Open a clean preprocessing run through the brick's ``RunInputs`` (layer: io). + +Design v0.4 §6: CellPatcher reads a preprocessing run only through ``RunInputs.open``, which +refuses a run before v0.4.0, a table or design whose SHA-256 is not the manifest's, and an +image or mask changed or missing since the run. An experiment moved or copied since its run is +opened with where it is now (``experiment_root``, ``masks_root``). Re-exported by +:mod:`cellpatcher.statistics`, where these functions were before CP-2a. +""" + +from __future__ import annotations + +import os +from collections import defaultdict +from pathlib import Path +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from morphobricks_normalization import PlaneRecord + from morphobricks_preprocessing.run_inputs import RunInputs + + from cellpatcher.statistics import PlaneStatistics + + +def load_plane_statistics( + plane_histograms: str | os.PathLike, + manifest: str | os.PathLike | None = None, + *, + experiment_root: str | os.PathLike | None = None, + masks_root: str | os.PathLike | None = None, +) -> PlaneStatistics: + """Open the preprocessing run that wrote ``plane_histograms.csv``, through ``RunInputs``. + + The run's output folder is the table's folder, opened with :func:`open_run`. CellPatcher + reads no other table, manifest or design than the run's own. + + Parameters + ---------- + plane_histograms + Path of the ``plane_histograms.csv`` written by the preprocessing brick. + manifest + Path of its ``run_manifest.json``, or None. Only the run's own manifest, + beside the table, can be given. + experiment_root, masks_root + Where the experiment and its masks folder are now, for an experiment moved or copied + since the run; None: where the run recorded them. + + Returns + ------- + PlaneStatistics + + Raises + ------ + FileNotFoundError + If the table does not exist. + PreprocessingCheckError + If the table or manifest is not the run's own, ``RunInputs`` refuses + the run, or the table cannot be read, has no rows or duplicate planes. + """ + from morphobricks_preprocessing import MANIFEST_FILENAME + + from cellpatcher.statistics import require_normalization + + mbn = require_normalization() + + csv_path = Path(plane_histograms).expanduser() + if not csv_path.is_file(): + raise FileNotFoundError( + f"The plane histogram table {csv_path} does not exist. Give the path of the " + f"{mbn.OUTPUT_FILENAME} written by the preprocessing brick, or run the preprocessing " + "brick on this experiment first." + ) + folder = csv_path.parent + _refuse_other_file("normalization.plane_histograms", csv_path, folder / mbn.OUTPUT_FILENAME) + if manifest is not None: + _refuse_other_file("normalization.manifest", Path(manifest).expanduser(), folder / MANIFEST_FILENAME) + return statistics_of(open_run(folder, experiment_root=experiment_root, masks_root=masks_root)) + + +def statistics_of(inputs: RunInputs) -> PlaneStatistics: + """Return the :class:`PlaneStatistics` of a run opened with :func:`open_run`. + + Raises + ------ + PreprocessingCheckError + If the table has no rows, or several rows for one plane. + """ + from morphobricks_preprocessing import MANIFEST_FILENAME + + from cellpatcher.statistics import PlaneStatistics, require_normalization + + mbn = require_normalization() + + csv_path = inputs.output_folder / mbn.OUTPUT_FILENAME + records = inputs.histograms() + _refuse_table_rows(csv_path, records) + return PlaneStatistics( + plane_histograms=csv_path, + records=records, + csv_sha256=inputs.manifest.csv_sha256, + manifest=inputs.manifest, + manifest_path=inputs.output_folder / MANIFEST_FILENAME, + inputs=inputs, + ) + + +def open_run( + output_folder: str | os.PathLike, + *, + experiment_root: str | os.PathLike | None = None, + masks_root: str | os.PathLike | None = None, +) -> RunInputs: + """Open a clean preprocessing run with the brick's ``RunInputs.open``. + + ``RunInputs.open`` refuses a run written before v0.4.0, a table or design whose SHA-256 is + not the manifest's, and any image or mask of the run changed or missing (design v0.4 §6). + *experiment_root* and *masks_root* say where the experiment and its masks folder are now, + for an experiment moved or copied since the run (None: where the run recorded them); the + images and masks there are the ones checked and read. + + Raises + ------ + PreprocessingCheckError + ``preprocessing_run_refused`` with the brick's reason when ``RunInputs`` + refuses the run, or ``unreadable_plane_histograms`` when its table + cannot be read. + """ + from morphobricks_normalization.csv_io import CsvFormatError + from morphobricks_preprocessing.run_inputs import RunInputs, RunInputsError + from morphobricks_preprocessing.validation import error + + from cellpatcher.statistics import PreprocessingCheckError + + folder = Path(output_folder).expanduser() + roots = [Path(root).expanduser() if root is not None else None for root in (experiment_root, masks_root)] + try: + return RunInputs.open(folder, *roots) + except RunInputsError as exc: + raise PreprocessingCheckError([ + error( + "preprocessing_run_refused", + f"The preprocessing run in {folder} cannot be used: {exc}", + [str(folder)], + "Run preprocessing v0.4 on the experiment as it is now and give CellPatcher that run. For an " + "experiment only moved or copied since its run, give CellPatcher the experiment and masks " + "folder where they are now (the image folder or root, and mask_folder_path or mask_root_path).", + ) + ]) from exc + except CsvFormatError as exc: + raise PreprocessingCheckError([ + error( + "unreadable_plane_histograms", + f"The plane histogram table cannot be read: {exc}", + [str(folder)], + "Give the plane_histograms.csv the preprocessing brick wrote, or run it again.", + ) + ]) from exc + + +def _refuse_other_file(setting: str, given: Path, expected: Path) -> None: + """Refuse a file *setting* names that is not the preprocessing run's own *expected* file.""" + from morphobricks_preprocessing.validation import error + + from cellpatcher.statistics import PreprocessingCheckError + + if given.resolve() == expected.resolve(): + return + raise PreprocessingCheckError([ + error( + "not_the_run_output", + f"{setting} names {given}, but CellPatcher reads a preprocessing run only through its " + f"output folder, where this file is {expected.name}.", + [str(given)], + f"Give the {expected.name} the preprocessing brick wrote in its output folder, unrenamed " + "and in place.", + ) + ]) + + +def _refuse_table_rows(csv_path: Path, records: tuple[PlaneRecord, ...]) -> None: + """Refuse a table without rows, or with several rows for one plane.""" + from morphobricks_preprocessing.validation import error, plural + + from cellpatcher.statistics import PreprocessingCheckError + + issues = [] + if not records: + issues.append(error( + "empty_plane_histograms", + "The plane histogram table has no rows.", + [str(csv_path)], + "Run the preprocessing brick on this experiment again.", + )) + counts: dict[str, int] = defaultdict(int) + for record in records: + counts[record.plane_id] += 1 + duplicates = [plane_id for plane_id, count in counts.items() if count > 1] + if duplicates: + issues.append(error( + "duplicate_plane_rows", + f"The plane histogram table {csv_path.name} has several rows for " + f"{plural(len(duplicates), 'plane')}, so they cannot be linked to " + "their patches.", + duplicates, + "Use the table the preprocessing brick wrote, without combining or editing it.", + )) + if issues: + raise PreprocessingCheckError(issues) + + +def recorded_image_type( + plane_histograms: str | os.PathLike, + manifest: str | os.PathLike | None = None, + *, + experiment_root: str | os.PathLike | None = None, + masks_root: str | os.PathLike | None = None, +) -> str | None: + """Return the image type the preprocessing run recorded, read through ``RunInputs``. + + ``normalization.image_type`` must equal it (normalization brick decision + N34). The run is opened as :func:`load_plane_statistics` opens it. + + Parameters + ---------- + plane_histograms + Path of the run's ``plane_histograms.csv``. + manifest + Path of its ``run_manifest.json``, or None; only the run's own. + experiment_root, masks_root + As in :func:`load_plane_statistics`. + + Returns + ------- + str or None + ``"fluorescence"`` or ``"rgb"``; None when the run recorded no image + type, so the configuration decides. + + Raises + ------ + FileNotFoundError + If the table does not exist. + PreprocessingCheckError + As :func:`load_plane_statistics`. + """ + statistics = load_plane_statistics(plane_histograms, manifest, experiment_root=experiment_root, masks_root=masks_root) + return statistics.inputs.image_type() diff --git a/src/cellpatcher/run_reading.py b/src/cellpatcher/run_reading.py new file mode 100644 index 0000000..2a7e400 --- /dev/null +++ b/src/cellpatcher/run_reading.py @@ -0,0 +1,187 @@ +"""Read the images and masks of a file run through preprocessing's ``RunInputs`` (layer: io). + +Design v0.4 §6 (CP-2a): a folder or tree run reads every image with ``RunInputs.read_plane`` +and every mask with ``RunInputs.read_mask``, as the preprocessing run recorded them. Nothing +here names an axis of a file, pairs a mask or reads a calibration: an image's axes are the +roles the run recorded for it (:attr:`Fov.roles`), and a mask's are those the run recorded +for that mask (its ``roles``, and ``shared`` for a stated 2-D mask serving every plane). + +The arrays handed to the patching core are in one canonical layout, built from those roles: +the run's T, Z and C axes in that order (:func:`run_layout`), then Y and X, as +``RunInputs.read_image`` orders them. Nothing is read transposed or reordered +other than by role. +""" + +from __future__ import annotations + +from collections.abc import Iterator, Mapping, Sequence +from pathlib import PurePosixPath +from typing import TYPE_CHECKING + +import numpy as np + +from cellpatcher.planes import ( + ImagePlane, + PlaneChoice, + apply_plane_choice, + plane_choices, + select_mask_plane, +) + +if TYPE_CHECKING: + from morphobricks_preprocessing.run_inputs import Fov, RunInputs + + from cellpatcher.config import CommonConfig + +_STACK_ROLES = "TZC" + + +def run_layout(fovs: Sequence[Fov]) -> str: + """Return the canonical layout of every image array of a run: T, Z, C in that order, then ``YX``. + + T or Z is in it when an image has more than one plane along it, C when an image has a + channel role. An image without one of these axes has length 1 along it in its array (its + one plane), never a guessed plane. + """ + counts = {"T": lambda fov: fov.t_count, "Z": lambda fov: fov.z_count, "C": lambda fov: 2} + return "".join( + role for role in _STACK_ROLES if any(role in fov.roles and counts[role](fov) > 1 for fov in fovs) + ) + "YX" + + +def _plane_shape(fov: Fov, layout: str) -> tuple[int, ...]: + """Return an image's T, Z and C lengths in *layout*, from the run's records, with 0 for Y and X. + + Only the T and Z lengths are used, to choose the planes; Y and X are not read here. + """ + lengths = {"T": fov.t_count, "Z": fov.z_count, "C": fov.c_count, "Y": 0, "X": 0} + return tuple(lengths[axis] for axis in layout) + + +def read_fov(inputs: RunInputs, fov: Fov, layout: str, choice: PlaneChoice | None = None) -> tuple[np.ndarray, str]: + """Read an image, or the planes *choice* names, through ``RunInputs.read_plane``. + + Parameters + ---------- + inputs, fov + The opened run and one of its images. + layout + The run's canonical layout (:func:`run_layout`). + choice + The T and Z planes to read (an index, a projection over every plane, or None for every + plane); None reads the whole image. + + Returns + ------- + tuple + The pixels, as stored, and their axes: *layout* without the axes *choice* resolved to + one plane (a projection is computed after reading every plane along its axis). + + Raises + ------ + morphobricks_preprocessing.run_inputs.RunInputsError + If the image changed since the run. + """ + choice = choice or PlaneChoice(t=None, z=None, suffix="") + counts = {"T": fov.t_count, "Z": fov.z_count} + indices = {} + for axis in ("T", "Z"): + value = choice.value(axis) if axis in layout else None + indices[axis] = [value] if isinstance(value, int) else list(range(counts[axis] if axis in layout else 1)) + stack = np.stack([np.stack([_read_plane(inputs, fov, z, t) for z in indices["Z"]]) for t in indices["T"]]) + kept = "".join( + axis for axis in "TZCYX" if axis in "YX" or (axis in layout and not isinstance(choice.value(axis), int)) + ) + pixels = stack.reshape([size for axis, size in zip("TZCYX", stack.shape) if axis in kept]) + projections = PlaneChoice( + t=choice.t if isinstance(choice.t, str) else None, z=choice.z if isinstance(choice.z, str) else None, suffix="" + ) + return apply_plane_choice(pixels, kept, projections) + + +def _read_plane(inputs: RunInputs, fov: Fov, z: int, t: int) -> np.ndarray: + """Read one ``(C, Y, X)`` plane, refusing a plane too large for the brick's memory limit.""" + from morphobricks_preprocessing import PlaneMemoryError + + try: + return inputs.read_plane(fov, z=z, t=t) + except PlaneMemoryError as exc: + raise ValueError( + f"CellPatcher needs each plane whole in memory to patch it, and a plane of {fov.relative_path} is too " + f"large: {exc}" + ) from exc + + +def mask_record(inputs: RunInputs, fov: Fov, mask_type: str) -> Mapping | None: + """Return the run's record of an image's mask of *mask_type*, or None when it recorded none.""" + for record in (inputs.manifest.masks or {}).get("files", ()): + if record["image"] == fov.relative_path and record["type"] == mask_type: + return record + return None + + +def mask_layout(record: Mapping) -> str: + """Return the canonical layout of a recorded mask: ``YX`` when shared, else its T and Z roles, then ``YX``.""" + if record.get("shared"): + return "YX" + return "".join(role for role in "TZ" if role in str(record["roles"])) + "YX" + + +def read_fov_mask(inputs: RunInputs, fov: Fov, record: Mapping, layout: str) -> np.ndarray: + """Read an image's recorded mask through ``RunInputs.read_mask``, in the run's mask *layout*. + + A stated shared 2-D mask is read once; a mask with depth slices or time points is read + plane by plane, the brick checking each plane's labels. An axis of *layout* the mask does + not have is added with length 1 (one plane, shared by the image's planes along it). + + Raises + ------ + morphobricks_preprocessing.run_inputs.RunInputsError + If the mask changed since the run. + ValueError + For a label outside 0 to the int32 limit (the brick's ``check_labels``). + """ + own = mask_layout(record) + roles, shape = str(record["roles"]), list(record["shape"]) + ts = range(shape[roles.index("T")]) if "T" in own else [None] + zs = range(shape[roles.index("Z")]) if "Z" in own else [None] + stack = np.stack([np.stack([inputs.read_mask(fov, record["type"], z=z, t=t) for z in zs]) for t in ts]) + return stack.reshape([size for axis, size in zip("TZYX", stack.shape) if axis in layout]) + + +def run_mask_layout(records: Sequence[Mapping]) -> str: + """Return the canonical layout of every mask array of a run: the T and Z any mask has, then ``YX``.""" + return "".join(role for role in "TZ" if any(role in mask_layout(record) for record in records)) + "YX" + + +def mask_file_name(record: Mapping) -> str: + """Return a mask's ``mask_file``: its file name, or ``/labels/`` for a label image.""" + from morphobricks_preprocessing.masks import FOLDER + + path = PurePosixPath(str(record["path"])) + if record["source"] == FOLDER: + return path.name + return "/".join(path.parts[-3:]) + + +def iter_fov_planes( + inputs: RunInputs, + fov: Fov, + layout: str, + mask: np.ndarray | None, + mask_axes: str, + common: CommonConfig, + *, + single_planes: bool = False, +) -> Iterator[ImagePlane]: + """Yield the planes of an image chosen by the plane policies, reading only those planes. + + The planes and their mask planes equal those :func:`cellpatcher.planes.iter_image_planes` + gives for the whole image. Raises like :func:`cellpatcher.planes.plane_choices`. + """ + mask_shape = None if mask is None else mask.shape + shape = _plane_shape(fov, layout) + for choice in plane_choices(shape, layout, mask_shape, mask_axes, common, single_planes=single_planes): + pixels, axes = read_fov(inputs, fov, layout, choice) + plane_mask, plane_mask_axes = select_mask_plane(mask, mask_axes, choice) + yield ImagePlane(pixels, axes, plane_mask, plane_mask_axes, choice.index("Z"), choice.index("T"), choice.suffix) diff --git a/src/cellpatcher/run_session.py b/src/cellpatcher/run_session.py new file mode 100644 index 0000000..046a7c9 --- /dev/null +++ b/src/cellpatcher/run_session.py @@ -0,0 +1,83 @@ +"""One opened preprocessing run per CellPatcher call (layer: orchestration). + +A CellPatcher call (``run_patching``, a folder or tree run) opens the +preprocessing run it reads with ``RunInputs.open`` once, and every step of +that call (the checks, each leaf's normalization, ``run_config.json``) reuses +it. ``RunInputs.open`` checks every image and mask of the run against the +manifest, and the brick checks each file again whenever it is read, so within +a call nothing is read unchecked. + +Nothing is kept beyond the call: the next call opens the run again, so an +image or mask changed in between, patched or not, is refused then (design +v0.4 §6; CP-2a). Before CP-2a an opened run was cached for the whole session, +keyed on the table and manifest only, and a mask or an unpatched image changed +after the first open was not checked again. +""" + +from __future__ import annotations + +import functools +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from contextvars import ContextVar +from pathlib import Path +from typing import TYPE_CHECKING, ParamSpec, TypeVar + +if TYPE_CHECKING: + from cellpatcher.statistics import PlaneStatistics + +_SESSION: ContextVar[dict[str, PlaneStatistics] | None] = ContextVar("cellpatcher_run_session", default=None) +_P = ParamSpec("_P") +_R = TypeVar("_R") + + +@contextmanager +def statistics_session() -> Iterator[None]: + """Keep the runs opened inside the block for the block only; a nested block shares the outer one.""" + if _SESSION.get() is not None: + yield + return + token = _SESSION.set({}) + try: + yield + finally: + _SESSION.reset(token) + + +def one_session(function: Callable[_P, _R]) -> Callable[_P, _R]: + """Run *function* inside :func:`statistics_session`.""" + + @functools.wraps(function) + def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R: + with statistics_session(): + return function(*args, **kwargs) + + return wrapper + + +def session_statistics(csv_path: Path, load: Callable[[], PlaneStatistics]) -> PlaneStatistics: + """Return the statistics of *csv_path* opened in this call, opening them with *load* if needed. + + Outside a session nothing is kept: *load* runs on every call. + """ + session = _SESSION.get() + if session is None: + return load() + key = str(csv_path.expanduser().resolve()) + if key not in session: + session[key] = load() + return session[key] + + +def keep_in_session(statistics: PlaneStatistics) -> None: + """Make *statistics*, opened by a file run with its own roots, the ones its table gives in this call. + + Raises + ------ + RuntimeError + Outside :func:`statistics_session`. + """ + session = _SESSION.get() + if session is None: + raise RuntimeError("keep_in_session needs an open statistics_session().") + session[str(statistics.plane_histograms.expanduser().resolve())] = statistics diff --git a/src/cellpatcher/statistics.py b/src/cellpatcher/statistics.py index 62797cc..4a7894d 100644 --- a/src/cellpatcher/statistics.py +++ b/src/cellpatcher/statistics.py @@ -4,10 +4,10 @@ v0.4 §6) and runs the checks of ``docs/normalization_contract.md`` §5, which block a run before any patch is normalized: -- :func:`load_plane_statistics` opens the run that wrote - ``plane_histograms.csv`` with ``RunInputs.open``, which refuses a run - before v0.4.0, a table or design whose SHA-256 is not the manifest's, and - an image or mask changed since the run; +- :func:`load_plane_statistics` (from :mod:`cellpatcher.run_open`) opens the + run that wrote ``plane_histograms.csv`` with ``RunInputs.open``, which + refuses a run before v0.4.0, a table or design whose SHA-256 is not the + manifest's, and an image or mask changed since the run; - :func:`check_images` reports every image without rows in one message and checks each image's size, modification time and axes against the manifest; - :func:`check_planes` links each patched plane to its row and compares the @@ -22,7 +22,6 @@ from __future__ import annotations -import os from collections import defaultdict from collections.abc import Iterable, Mapping from dataclasses import dataclass, field @@ -32,9 +31,16 @@ import numpy as np +from cellpatcher import run_open as _run_open from cellpatcher.axes import normalize_axes from cellpatcher.io import SINGLE_CONDITION_LABEL as SINGLE_LABEL +# Opening the run moved to cellpatcher.run_open (CP-2a); re-exported here. +load_plane_statistics = _run_open.load_plane_statistics +open_run = _run_open.open_run +recorded_image_type = _run_open.recorded_image_type +statistics_of = _run_open.statistics_of + if TYPE_CHECKING: from morphobricks_normalization import PlaneRecord from morphobricks_preprocessing import RunManifest, ValidationIssue @@ -308,148 +314,6 @@ def channel_records(self) -> Iterable[ChannelRecord]: yield ChannelRecord(record, c) -def load_plane_statistics(plane_histograms: str | os.PathLike, manifest: str | os.PathLike | None = None) -> PlaneStatistics: - """Open the preprocessing run that wrote ``plane_histograms.csv``, through ``RunInputs``. - - The run's output folder is the table's folder. The preprocessing brick's - ``RunInputs.open`` reads the table, the design and ``run_manifest.json`` - there, and refuses a run written before v0.4.0, a table or design whose - SHA-256 is not the manifest's, and an image or mask changed or missing - since the run (design v0.4 §6). CellPatcher reads no other table, - manifest or design than the run's own. - - Parameters - ---------- - plane_histograms - Path of the ``plane_histograms.csv`` written by the preprocessing brick. - manifest - Path of its ``run_manifest.json``, or None. Only the run's own manifest, - beside the table, can be given. - - Returns - ------- - PlaneStatistics - - Raises - ------ - FileNotFoundError - If the table does not exist. - PreprocessingCheckError - If the table or manifest is not the run's own, ``RunInputs`` refuses - the run, or the table cannot be read, has no rows or duplicate planes. - """ - mbn = require_normalization() - from morphobricks_preprocessing import MANIFEST_FILENAME - - csv_path = Path(plane_histograms).expanduser() - if not csv_path.is_file(): - raise FileNotFoundError( - f"The plane histogram table {csv_path} does not exist. Give the path of the " - f"{mbn.OUTPUT_FILENAME} written by the preprocessing brick, or run the preprocessing " - "brick on this experiment first." - ) - folder = csv_path.parent - _refuse_other_file("normalization.plane_histograms", csv_path, folder / mbn.OUTPUT_FILENAME) - if manifest is not None: - _refuse_other_file("normalization.manifest", Path(manifest).expanduser(), folder / MANIFEST_FILENAME) - inputs = open_run(folder) - records = inputs.histograms() - _refuse_table_rows(csv_path, records) - return PlaneStatistics( - plane_histograms=csv_path, - records=records, - csv_sha256=inputs.manifest.csv_sha256, - manifest=inputs.manifest, - manifest_path=inputs.output_folder / MANIFEST_FILENAME, - inputs=inputs, - ) - - -def open_run(output_folder: str | os.PathLike) -> RunInputs: - """Open a clean preprocessing run with the brick's ``RunInputs.open``. - - Raises - ------ - PreprocessingCheckError - ``preprocessing_run_refused`` with the brick's reason when ``RunInputs`` - refuses the run, or ``unreadable_plane_histograms`` when its table - cannot be read. - """ - from morphobricks_normalization.csv_io import CsvFormatError - from morphobricks_preprocessing.run_inputs import RunInputs, RunInputsError - from morphobricks_preprocessing.validation import error - - folder = Path(output_folder).expanduser() - try: - return RunInputs.open(folder) - except RunInputsError as exc: - raise PreprocessingCheckError([ - error( - "preprocessing_run_refused", - f"The preprocessing run in {folder} cannot be used: {exc}", - [str(folder)], - "Run preprocessing v0.4 on the experiment where it is now, then give CellPatcher " - "the plane_histograms.csv of that run.", - ) - ]) from exc - except CsvFormatError as exc: - raise PreprocessingCheckError([ - error( - "unreadable_plane_histograms", - f"The plane histogram table cannot be read: {exc}", - [str(folder)], - "Give the plane_histograms.csv the preprocessing brick wrote, or run it again.", - ) - ]) from exc - - -def _refuse_other_file(setting: str, given: Path, expected: Path) -> None: - """Refuse a file *setting* names that is not the preprocessing run's own *expected* file.""" - from morphobricks_preprocessing.validation import error - - if given.resolve() == expected.resolve(): - return - raise PreprocessingCheckError([ - error( - "not_the_run_output", - f"{setting} names {given}, but CellPatcher reads a preprocessing run only through its " - f"output folder, where this file is {expected.name}.", - [str(given)], - f"Give the {expected.name} the preprocessing brick wrote in its output folder, unrenamed " - "and in place.", - ) - ]) - - -def _refuse_table_rows(csv_path: Path, records: tuple[PlaneRecord, ...]) -> None: - """Refuse a table without rows, or with several rows for one plane.""" - from morphobricks_preprocessing.validation import error, plural - - issues = [] - if not records: - issues.append(error( - "empty_plane_histograms", - "The plane histogram table has no rows.", - [str(csv_path)], - "Run the preprocessing brick on this experiment again.", - )) - counts: dict[str, int] = defaultdict(int) - for record in records: - counts[record.plane_id] += 1 - duplicates = [plane_id for plane_id, count in counts.items() if count > 1] - if duplicates: - issues.append(error( - "duplicate_plane_rows", - f"The plane histogram table {csv_path.name} has several rows for " - f"{plural(len(duplicates), 'plane')}, so they cannot be linked to " - "their patches.", - duplicates, - "Use the table the preprocessing brick wrote, without combining or editing it.", - )) - if issues: - raise PreprocessingCheckError(issues) - - def check_images( statistics: PlaneStatistics, image_axes: Mapping[str, str], @@ -615,35 +479,6 @@ def check_planes( return matched -def recorded_image_type(plane_histograms: str | os.PathLike, manifest: str | os.PathLike | None = None) -> str | None: - """The image type the preprocessing run recorded, read through ``RunInputs``. - - ``normalization.image_type`` must equal it (normalization brick decision - N34). The run is opened as :func:`load_plane_statistics` opens it. - - Parameters - ---------- - plane_histograms - Path of the run's ``plane_histograms.csv``. - manifest - Path of its ``run_manifest.json``, or None; only the run's own. - - Returns - ------- - str or None - ``"fluorescence"`` or ``"rgb"``; None when the run recorded no image - type, so the configuration decides. - - Raises - ------ - FileNotFoundError - If the table does not exist. - PreprocessingCheckError - As :func:`load_plane_statistics`. - """ - return load_plane_statistics(plane_histograms, manifest).inputs.image_type() - - def _labels_text(labels) -> str: return "/".join("(none)" if label is None else label for label in labels) diff --git a/tests/helpers.py b/tests/helpers.py index e03fe6e..dfd6929 100644 --- a/tests/helpers.py +++ b/tests/helpers.py @@ -93,3 +93,63 @@ def write_ngff_labels(path, labels, axes="zyx"): label_group.attrs.update({"image-label": {"version": "0.4"}}) zarr.open_group(str(path / "labels"), mode="a", **options).attrs.update({"labels": list(labels)}) return path + + +def preprocess(root, axes="YX", *, masks=None, output=None, stored=None, **options): + """Run the preprocessing brick on the experiment *root* and return its output folder. + + Folder and tree runs read files only through a clean preprocessing run + (design v0.4 §6, CP-2a). *axes* is the roles of the images' axes, as + ``morphobricks-preprocessing run --axes`` takes them (*stored*: the + file's own axes when they differ, e.g. ``"IYX"`` for a plain stack); + *masks* the masks folder. The output goes beside *root* unless *output* is given. + """ + from pathlib import Path + + import morphobricks_preprocessing as mbp + + root = Path(root) + output = Path(output) if output is not None else root.parent / f"{root.name}_preprocessing" + result = mbp.run_experiment( + root, + mbp.AxisMapping(stored or axes, axes), + output_folder=output, + masks_folder=masks, + **options, # design, image_type, mask_axes, missing_as_singleton, ... + ) + if not result.written: + raise AssertionError("preprocessing did not run: " + "; ".join(map(str, result.issues))) + return output + + +def file_run_config(config, root, axes="YX", **preprocessing): + """Preprocess *root* (:func:`preprocess`) and set ``config.common.preprocessing_run``; return *config*.""" + config.common.preprocessing_run = str(preprocess(root, axes, **preprocessing)) + return config + + +def preprocessed_tree(root, *, config, mask_root_path=None, axes="YX", **kwargs): + """Preprocess *root* (with its masks folder), then run the tree run through that run (CP-2a).""" + from cellpatcher.orchestration import run_patching_condition_repeat_tree + + file_run_config(config, root, axes, masks=mask_root_path) + return run_patching_condition_repeat_tree(root, config=config, mask_root_path=mask_root_path, **kwargs) + + +def preprocessed_folder(folder, *, config, mask_folder_path=None, axes="YX", **kwargs): + """Preprocess *folder* (with its masks folder), then run the folder run through that run (CP-2a).""" + from cellpatcher.orchestration import run_patching_pipeline + + file_run_config(config, folder, axes, masks=mask_folder_path) + return run_patching_pipeline(folder, config=config, mask_folder_path=mask_folder_path, **kwargs) + + +def as_file_run(config, run=None): + """A copy of *config* for a file run through *run* (default: the run of its table), axes left to it (CP-2a).""" + import copy + import os + + config = copy.deepcopy(config) + config.common.image_axes = config.common.mask_axes = "YX" + config.common.preprocessing_run = str(run or os.path.dirname(config.normalization.plane_histograms)) + return config diff --git a/tests/test_biological_repeat.py b/tests/test_biological_repeat.py index 4728dfb..cf9520b 100644 --- a/tests/test_biological_repeat.py +++ b/tests/test_biological_repeat.py @@ -27,12 +27,17 @@ ConditionRepeatResult, apply_overrides, load_config, - run_patching_condition_repeat_tree, - run_patching_pipeline, ) from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching from cellpatcher.types import PatchRegion from tests.helpers import named +from tests.helpers import preprocessed_tree as _tree + + +def _run(config): + """Patch one in-memory image named a.tif (config labels apply to in-memory runs; file runs + take their labels from the preprocessing run since CP-2a).""" + return run_patching(images=[np.ones((16, 16), np.float32)], config=config, image_metadata=[{"image_file": "a.tif"}]) def _write_image(path, shape=(16, 16)): @@ -69,14 +74,13 @@ def _build_tree(root): def test_pipeline_csv_has_biological_repeat_column_and_no_repeat_column(tmp_path): - _write_image(tmp_path / "images" / "a.tif") cfg = _export_config(tmp_path / "out") cfg.common.condition = "treated" cfg.common.biological_repeat = "2026_01_01" with warnings.catch_warnings(record=True) as records: warnings.simplefilter("always") - run_patching_pipeline(str(tmp_path / "images"), config=cfg) + _run(cfg) assert _deprecations(records) == [] header, rows = _read_csv(tmp_path / "out" / "patches.csv") @@ -96,7 +100,7 @@ def test_tree_per_leaf_csvs_have_biological_repeat_column(tmp_path): with warnings.catch_warnings(record=True) as records: warnings.simplefilter("always") - results = run_patching_condition_repeat_tree(str(tmp_path / "images"), config=cfg) + results = _tree(tmp_path / "images", config=cfg) assert _deprecations(records) == [] assert {(r.condition, r.biological_repeat) for r in results} == { @@ -116,7 +120,7 @@ def test_tree_combined_csv_has_biological_repeat_column(tmp_path): _build_tree(tmp_path / "images") cfg = _export_config(tmp_path / "out") - run_patching_condition_repeat_tree(str(tmp_path / "images"), config=cfg, combine_metadata=True) + _tree(tmp_path / "images", config=cfg, combine_metadata=True) header, rows = _read_csv(tmp_path / "out" / "patches.csv") assert "biological_repeat" in header and "repeat" not in header @@ -134,7 +138,7 @@ def test_tree_flat_root_is_single_condition_and_single_biological_repeat(tmp_pat _write_image(tmp_path / "images" / "a.tif") cfg = _export_config(tmp_path / "out") - run_patching_condition_repeat_tree(str(tmp_path / "images"), config=cfg, combine_metadata=True) + _tree(tmp_path / "images", config=cfg, combine_metadata=True) _, rows = _read_csv(tmp_path / "out" / "patches.csv") assert [(row["condition"], row["biological_repeat"]) for row in rows] == [("single", "single")] @@ -154,12 +158,10 @@ def test_single_label_constants(): def test_config_repeat_attribute_warns_once_and_gives_identical_output(tmp_path): - _write_image(tmp_path / "images" / "a.tif") - new_cfg = _export_config(tmp_path / "new") new_cfg.common.condition = "treated" new_cfg.common.biological_repeat = "2026_01_01" - run_patching_pipeline(str(tmp_path / "images"), config=new_cfg) + _run(new_cfg) old_cfg = _export_config(tmp_path / "old") old_cfg.common.condition = "treated" @@ -169,7 +171,7 @@ def test_config_repeat_attribute_warns_once_and_gives_identical_output(tmp_path) with warnings.catch_warnings(record=True) as run_records: warnings.simplefilter("always") - run_patching_pipeline(str(tmp_path / "images"), config=old_cfg) + _run(old_cfg) assert _deprecations(run_records) == [] assert (tmp_path / "old" / "patches.csv").read_text() == (tmp_path / "new" / "patches.csv").read_text() @@ -209,18 +211,9 @@ def test_apply_overrides_repeat_key_warns_once(): def test_pipeline_overrides_repeat_key_gives_identical_csv(tmp_path): - _write_image(tmp_path / "images" / "a.tif") - run_patching_pipeline( - str(tmp_path / "images"), - config=_export_config(tmp_path / "new"), - overrides={"common.biological_repeat": "2026_01_01"}, - ) + _run(apply_overrides(_export_config(tmp_path / "new"), {"common.biological_repeat": "2026_01_01"})) with pytest.warns(DeprecationWarning) as records: - run_patching_pipeline( - str(tmp_path / "images"), - config=_export_config(tmp_path / "old"), - overrides={"common.repeat": "2026_01_01"}, - ) + _run(apply_overrides(_export_config(tmp_path / "old"), {"common.repeat": "2026_01_01"})) assert len(_deprecations(records)) == 1 assert (tmp_path / "old" / "patches.csv").read_text() == (tmp_path / "new" / "patches.csv").read_text() @@ -269,7 +262,7 @@ def test_discovery_and_tree_results_expose_deprecated_repeat_alias(tmp_path): assert leaves[0].repeat == leaves[0].biological_repeat assert len(_deprecations(records)) == 1 - results = run_patching_condition_repeat_tree(str(tmp_path / "images"), config=CellPatcherConfig(mode="whole_fov")) + results = _tree(tmp_path / "images", config=CellPatcherConfig(mode="whole_fov")) assert isinstance(results[0], ConditionRepeatResult) with pytest.warns(DeprecationWarning, match="ConditionRepeatResult.repeat") as records: assert results[0].repeat == results[0].biological_repeat diff --git a/tests/test_condition_repeat_tree.py b/tests/test_condition_repeat_tree.py index 563ba0a..d918808 100644 --- a/tests/test_condition_repeat_tree.py +++ b/tests/test_condition_repeat_tree.py @@ -14,8 +14,8 @@ ) from cellpatcher.orchestration import load_config, run_patching_condition_repeat_tree from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching -from tests.helpers import make_synthetic_cell_image_and_labels -from tests.helpers import named +from tests.helpers import make_synthetic_cell_image_and_labels, named +from tests.helpers import preprocessed_tree as _tree def _write_image(path, shape=(16, 16)): @@ -161,7 +161,7 @@ def test_tree_whole_fov_stamps_condition_repeat_and_mirrors_output(tmp_path): cfg.export.image_format = "npy" cfg.common.output_dir = str(tmp_path / "out") - results = run_patching_condition_repeat_tree(image_root, config=cfg) + results = _tree(image_root, config=cfg) assert len(results) == 2 by_condition = {r.condition: r for r in results} @@ -191,7 +191,7 @@ def test_tree_flat_root_stamps_single_condition_and_repeat(tmp_path): cfg.export.image_format = "npy" cfg.common.output_dir = str(tmp_path / "out") - results = run_patching_condition_repeat_tree(image_root, config=cfg) + results = _tree(image_root, config=cfg) assert len(results) == 1 patch = results[0].result.patches[0] assert (patch.condition, patch.biological_repeat) == (SINGLE_CONDITION_LABEL, SINGLE_BIOLOGICAL_REPEAT_LABEL) @@ -212,7 +212,7 @@ def test_tree_cell_mode_explicit_and_single_repeat_with_mirrored_masks(tmp_path) cfg.cell.patch_size = 24 cfg.cell.exclude_border = True - results = run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + results = _tree(image_root, config=cfg, mask_root_path=mask_root) assert len(results) == 2 for r in results: assert len(r.result.patches) > 0 @@ -238,12 +238,12 @@ def test_tree_cell_mode_mask_folder_with_several_masks_per_image(tmp_path): cfg = CellPatcherConfig(mode="cell") cfg.cell.patch_size = 24 with pytest.raises(ValueError, match="4 mask types .cells, mito, nuclei, nucleoli.; set matching.mask_type"): - run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + _tree(image_root, config=cfg, mask_root_path=mask_root) cfg.matching.mask_type = "cells" cfg.export.enabled = True cfg.common.output_dir = str(tmp_path / "out") - results = run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + results = _tree(image_root, config=cfg, mask_root_path=mask_root) assert len(results) == 2 assert {patch.mask_file for r in results for patch in r.result.patches} == {"img_cells.tif"} assert {patch.object_set for r in results for patch in r.result.patches} == {"cells"} @@ -262,8 +262,8 @@ def test_tree_cell_mode_missing_mask_folder_raises_per_leaf(tmp_path): mask_root.mkdir(parents=True) cfg = CellPatcherConfig(mode="cell") - with pytest.raises(ValueError, match="No mask in .* pairs with an image"): - run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + with pytest.raises(ValueError, match="reads masks, but the preprocessing run .* recorded none"): + _tree(image_root, config=cfg, mask_root_path=mask_root) def test_tree_grid_mode_ignores_mask_root_path(tmp_path): @@ -274,7 +274,7 @@ def test_tree_grid_mode_ignores_mask_root_path(tmp_path): mask_root.mkdir(parents=True) cfg = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - results = run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + results = _tree(image_root, config=cfg, mask_root_path=mask_root) assert len(results) == 1 assert len(results[0].result.patches) > 0 @@ -294,25 +294,17 @@ def test_tree_random_mode_mask_root_needs_a_mask_for_every_image(tmp_path): cfg = CellPatcherConfig(mode="random") cfg.random.n_patches = 3 cfg.random.patch_size = 8 - with pytest.raises(ValueError, match="Images with no 'mask' mask: control/img.tif"): - run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + with pytest.raises(ValueError, match="recorded no 'mask' mask for control/img.tif; every image needs one"): + _tree(image_root, config=cfg, mask_root_path=mask_root) write_array(mask_root / "control" / "img_mask.tif", np.ones((64, 64), dtype=np.uint8)) - results = run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + results = _tree(image_root, config=cfg, mask_root_path=mask_root) assert {r.condition: len(r.result.patches) for r in results} == {"control": 3, "treated": 3} - results = run_patching_condition_repeat_tree(image_root, config=cfg) + results = _tree(image_root, config=cfg) assert {r.condition: len(r.result.patches) for r in results} == {"control": 3, "treated": 3} -def test_tree_no_condition_folders_raises(tmp_path): - root = tmp_path / "raw_data" - root.mkdir() - cfg = CellPatcherConfig(mode="whole_fov") - with pytest.raises(ValueError, match="No condition subfolders"): - run_patching_condition_repeat_tree(root, config=cfg) - - def test_tree_aggregated_patch_counts_match_per_leaf_csv_rows(tmp_path): image_root = tmp_path / "raw_data" _write_image(image_root / "a" / "2026_01_01" / "x.tif") @@ -324,7 +316,7 @@ def test_tree_aggregated_patch_counts_match_per_leaf_csv_rows(tmp_path): cfg.export.image_format = "npy" cfg.common.output_dir = str(tmp_path / "out") - results = run_patching_condition_repeat_tree(image_root, config=cfg) + results = _tree(image_root, config=cfg) assert len(results) == 3 for r in results: csv_path = tmp_path / "out" / r.condition / r.biological_repeat / "patches.csv" @@ -350,7 +342,7 @@ def test_tree_combine_metadata_writes_single_csv_and_config_at_output_root(tmp_p _write_image(image_root / "control" / "c.tif") out = tmp_path / "out" - results = run_patching_condition_repeat_tree(image_root, config=_combined_tree_config(tmp_path), combine_metadata=True) + results = _tree(image_root, config=_combined_tree_config(tmp_path), combine_metadata=True) assert (out / "patches.csv").exists() assert (out / RUN_CONFIG_FILENAME).exists() @@ -389,7 +381,7 @@ def test_tree_progress_callback_reports_each_leaf_then_completion(tmp_path): _write_image(image_root / "b" / "y.tif") calls = [] - run_patching_condition_repeat_tree( + _tree( image_root, config=CellPatcherConfig(mode="whole_fov"), progress_callback=lambda done, total, leaf: calls.append((done, total, leaf.condition if leaf else None)), @@ -459,7 +451,7 @@ def test_tree_stamps_technical_repeat_and_mirrors_it_in_the_output(tmp_path): cfg.export.image_format = "npy" cfg.common.output_dir = str(tmp_path / "out") - results = run_patching_condition_repeat_tree(image_root, config=cfg) + results = _tree(image_root, config=cfg) stamped = { (r.condition, r.biological_repeat, r.technical_repeat): {p.technical_repeat for p in r.result.patches} for r in results @@ -484,7 +476,7 @@ def test_tree_combined_csv_keeps_technical_repeats_apart(tmp_path): _write_image(image_root / "control" / "c.tif") out = tmp_path / "out" - results = run_patching_condition_repeat_tree( + results = _tree( image_root, config=_combined_tree_config(tmp_path), combine_metadata=True ) with (out / "patches.csv").open("r", encoding="utf-8") as f: @@ -514,7 +506,7 @@ def test_tree_cell_mode_finds_masks_in_mirrored_technical_repeat_folders(tmp_pat cfg = CellPatcherConfig(mode="cell") cfg.cell.patch_size = 16 - results = run_patching_condition_repeat_tree(image_root, config=cfg, mask_root_path=mask_root) + results = _tree(image_root, config=cfg, mask_root_path=mask_root) assert [r.technical_repeat for r in results] == ["B02", "B03"] for r in results: assert r.result.patches @@ -563,7 +555,7 @@ def test_tree_runs_wells_inside_a_biological_repeat_folder_named_single(tmp_path _write_image(image_root / "ctrl" / "single" / "wellA" / "img1.tif") _write_image(image_root / "ctrl" / "single" / "wellB" / "img2.tif") - results = run_patching_condition_repeat_tree(image_root, config=CellPatcherConfig(mode="whole_fov")) + results = _tree(image_root, config=CellPatcherConfig(mode="whole_fov")) stamped = { (r.condition, r.biological_repeat, r.technical_repeat): {p.image_file for p in r.result.patches} for r in results @@ -624,8 +616,6 @@ def test_a_tree_refuses_images_in_formats_no_longer_read(tmp_path, name): write_array(root / "control" / "r1" / name, np.ones((16, 16), np.uint8)) with pytest.raises(ValueError, match=rf"in a format CellPatcher no longer reads: control/r1/{name}\."): discover_condition_repeat_folders(root) - with pytest.raises(ValueError, match=rf"no longer reads: control/r1/{name}\."): - run_patching_condition_repeat_tree(root, config=CellPatcherConfig(mode="whole_fov")) def test_a_png_beside_condition_folders_is_refused_by_name(tmp_path): diff --git a/tests/test_declared_axes.py b/tests/test_declared_axes.py index 13fb390..82f5d0e 100644 --- a/tests/test_declared_axes.py +++ b/tests/test_declared_axes.py @@ -1,6 +1,9 @@ """Declared axes against file headers: the preprocessing brick's axis check (design v0.4 §3). -CellPatcher keeps no copy of the rule: the declared axes are checked as the +Since CP-2a folder and tree runs declare no axes: they read each file in the roles the +preprocessing run recorded, and refuse ``common.image_axes``/``mask_axes`` other than the +default. Lazy delivery from a file path still declares them (until CP-2b), and CellPatcher +keeps no copy of the rule: the declared axes are checked as the file's roles with the brick's ``review_axes`` (``check_mapping`` and ``header_axis_issue``). A declaration may name what a header leaves open and never contradict it. Since @@ -20,7 +23,13 @@ from cellpatcher.export import IMAGE_TABLE_FILENAME from cellpatcher.orchestration import run_patching_pipeline from cellpatcher.pipeline import run_patching -from tests.helpers import make_synthetic_cell_image_and_labels, named, write_tiff +from tests.helpers import ( + file_run_config, + make_synthetic_cell_image_and_labels, + named, + preprocessed_folder, + write_tiff, +) def _grid_config(image_axes, output_dir=None): @@ -50,7 +59,7 @@ def recording(files, mapping): monkeypatch.setattr(morphobricks_preprocessing, "review_axes", recording) write_tiff(tmp_path / "exp" / "a.tif", np.zeros((3, 16, 16), dtype=np.uint8), "ZYX") - run_patching_pipeline(str(tmp_path / "exp"), config=_grid_config("ZYX")) + cellpatcher.io.refuse_image_header_axes([tmp_path / "exp" / "a.tif"], "ZYX") # as lazy delivery checks a path assert ("a.tif", "ZYX") in calls @@ -61,33 +70,44 @@ def test_y_and_x_declared_swapped_are_refused_never_read_transposed(tmp_path): image = np.arange(3 * 16 * 24, dtype=np.uint16).reshape(3, 16, 24) # Z, Y, X in the header write_tiff(tmp_path / "exp" / "a.tif", image, "ZYX") write_tiff(tmp_path / "exp" / "b.tif", image, "ZYX") - config = _grid_config("ZXY", tmp_path / "out") - config.common.z_policy = "select" - config.common.z_index = 1 + paths = [tmp_path / "exp" / "a.tif", tmp_path / "exp" / "b.tif"] with pytest.raises(ValueError, match=r"common.image_axes='ZXY' does not fit the header of 2 image file\(s\): " r"a.tif: \[axes_yx_swapped\].* b.tif: \[axes_yx_swapped\]") as error_info: - run_patching_pipeline(str(tmp_path / "exp"), config=config) + cellpatcher.io.refuse_image_header_axes(paths, "ZXY") # lazy delivery from a path assert "nothing is read transposed" in str(error_info.value) + # A file run (CP-2a) declares nothing: the declaration is refused, and the image is read + # in the roles the preprocessing run recorded, Y before X. + config = file_run_config(_grid_config("ZXY", tmp_path / "out"), tmp_path / "exp", "ZYX") + config.common.z_policy, config.common.z_index = "select", 1 + with pytest.raises(ValueError, match=r"common.image_axes='ZXY': each image's \(and mask's\) axes are the roles"): + run_patching_pipeline(str(tmp_path / "exp"), config=config) assert not (tmp_path / "out").exists() + config.common.image_axes = "YX" + result = run_patching_pipeline(str(tmp_path / "exp"), config=config) + patch = result.patches[0] + assert np.array_equal(result.patch_arrays[0], image[1, patch.y0:patch.y1, patch.x0:patch.x1]) def test_axes_as_named_are_recorded_as_not_transposed(tmp_path): write_tiff(tmp_path / "exp" / "a.tif", np.zeros((3, 16, 24), dtype=np.uint8), "ZYX") - config = _grid_config("ZYX", tmp_path / "out") + config = _grid_config("YX", tmp_path / "out") config.common.z_policy = "all" - run_patching_pipeline(str(tmp_path / "exp"), config=config) + preprocessed_folder(tmp_path / "exp", config=config, axes="ZYX") assert _image_rows(tmp_path / "out")["a.tif"]["image_read_transposed"] == "false" def test_a_mask_declared_swapped_is_refused(tmp_path): - # A mask is checked with the same rule as an image: a Y/X swap is refused. + # A mask is checked with the same rule as an image: a Y/X swap is refused (lazy delivery); + # a file run declares no mask axes at all since CP-2a. image, labels, _ = make_synthetic_cell_image_and_labels(shape=(96, 96)) write_tiff(tmp_path / "exp" / "a.tif", image, "YX") write_tiff(tmp_path / "masks" / "a_cells.tif", np.ascontiguousarray(labels.T.astype(np.uint16)), "YX") with pytest.raises(ValueError, match=r"common.mask_axes='XY' does not fit the header of 1 mask file\(s\): " r"a_cells.tif: \[axes_yx_swapped\]"): - run_patching_pipeline(str(tmp_path / "exp"), config=_cell_config("YX", "XY"), - mask_folder_path=str(tmp_path / "masks")) + cellpatcher.io.refuse_mask_header_axes([tmp_path / "masks" / "a_cells.tif"], "XY") + config = file_run_config(_cell_config("YX", "XY"), tmp_path / "exp", masks=tmp_path / "masks") + with pytest.raises(ValueError, match=r"common.mask_axes='XY': each image's \(and mask's\) axes"): + run_patching_pipeline(str(tmp_path / "exp"), config=config, mask_folder_path=str(tmp_path / "masks")) @pytest.mark.parametrize("mode", ["cell", "random"]) @@ -110,31 +130,10 @@ def test_a_mask_file_whose_header_names_x_before_y_is_refused(tmp_path): write_tiff(tmp_path / "exp" / "a.tif", image, "YX") (tmp_path / "masks").mkdir() tifffile.imwrite(tmp_path / "masks" / "a_cells.tif", labels.astype(np.uint16), metadata={"axes": "XY"}) - with pytest.raises(ValueError, match=r"\[mask_layout\].*Masks are never transposed"): - run_patching_pipeline(str(tmp_path / "exp"), config=_cell_config("YX", "XY"), - mask_folder_path=str(tmp_path / "masks")) - - -def test_a_swap_the_preprocessing_run_did_not_declare_is_refused_with_its_statistics(tmp_path): - # With a CSV normalization scope, the preprocessing brick's check_sources compares - # common.image_axes with the roles the preprocessing run recorded for each file. A Y/X - # swap CellPatcher accepts against the header is refused there unless that run declared it - # too: the statistics were computed with the other roles (kept as is, owner decision of - # 31 October 2026). - from cellpatcher.orchestration import run_patching_condition_repeat_tree - from cellpatcher.statistics import PreprocessingCheckError - from tests.test_normalization import _config, _preprocess, _write_tiff - - root = tmp_path / "experiment" - rng = np.random.default_rng(0) - for path in ("control/bio1/a.tif", "treated/bio1/b.tif"): - _write_tiff(root / path, rng.integers(0, 200, (2, 16, 16), dtype=np.uint8), "ZYX") - csv_path = _preprocess(root, tmp_path / "preprocessing", "ZYX") - with pytest.raises(PreprocessingCheckError) as error_info: - run_patching_condition_repeat_tree(root, config=_config(csv_path, image_axes="ZXY", scope="image")) - codes = {issue.code for issue in error_info.value.issues} - assert codes == {"axis_mapping_mismatch"} - assert "given ZXY, preprocessing ZYX" in str(error_info.value) + # Since CP-2a the preprocessing run pairs it; rc1 records it without axes, and a file run refuses that. + config = file_run_config(_cell_config("YX", "YX"), tmp_path / "exp", masks=tmp_path / "masks") + with pytest.raises(ValueError, match=r"recorded no axes for the mask a_cells.tif: its header was not usable"): + run_patching_pipeline(str(tmp_path / "exp"), config=config, mask_folder_path=str(tmp_path / "masks")) def test_lazy_delivery_refuses_a_mask_declared_swapped(tmp_path): @@ -181,7 +180,7 @@ def test_an_in_memory_run_records_no_transposition(tmp_path): def test_other_disagreements_are_refused(tmp_path, header, declared, problem): write_tiff(tmp_path / "exp" / "a.tif", np.zeros((3, 16, 16), dtype=np.uint8), header) with pytest.raises(ValueError, match=rf"common.image_axes='{declared}'.*a.tif.*{problem}"): - run_patching_pipeline(str(tmp_path / "exp"), config=_grid_config(declared)) + cellpatcher.io.refuse_image_header_axes([tmp_path / "exp" / "a.tif"], declared) # lazy delivery def test_a_plain_stack_may_be_declared_with_its_channels_last(tmp_path): diff --git a/tests/test_delivery.py b/tests/test_delivery.py index 6f21afc..e83f251 100644 --- a/tests/test_delivery.py +++ b/tests/test_delivery.py @@ -8,6 +8,7 @@ import copy import importlib.util import json +from pathlib import Path import numpy as np import pytest @@ -49,7 +50,9 @@ def experiment(tmp_path_factory): labels[0, 3:9, 3:11] = 1 labels[1, 12:20, 10:17] = 2 + index _write_tiff(masks / "cells" / path, labels, "ZYX") # preprocessing layout: / - result = mbp.run_experiment(root, mbp.AxisMapping("ZCYX", "ZCYX"), output_folder=base / "preprocessing") + result = mbp.run_experiment( + root, mbp.AxisMapping("ZCYX", "ZCYX"), output_folder=base / "preprocessing", masks_folder=masks + ) assert result.written return root, masks, result.csv_path @@ -96,6 +99,11 @@ def config(csv_path): def _run(root, masks, config, **options): + # A file run reads through the preprocessing run, in the roles it recorded (CP-2a); the + # axes stay set for the lazy delivery from paths and arrays below. + config = copy.deepcopy(config) + config.common.image_axes = config.common.mask_axes = "YX" + config.common.preprocessing_run = str(Path(config.normalization.plane_histograms).parent) return run_patching_condition_repeat_tree( root, config=config, mask_root_path=masks if config.mode == "cell" else None, **options ) diff --git a/tests/test_file_runs.py b/tests/test_file_runs.py new file mode 100644 index 0000000..ede8b39 --- /dev/null +++ b/tests/test_file_runs.py @@ -0,0 +1,242 @@ +"""CP-2a: folder and tree runs read files only through a clean preprocessing run (design v0.4 §6). + +The images, masks and FOV identities come from ``RunInputs`` (``read_plane``, ``read_mask``, +``fovs``), opened with the experiment root and masks folder CellPatcher is given, so an +experiment moved or copied since its run is checked and read where it is now. The settings +that told CellPatcher how to read files are refused for file runs with a migration message. +""" + +import hashlib +import json +import shutil +from types import SimpleNamespace + +import numpy as np +import pytest +import tifffile + +from cellpatcher import CellPatcherConfig +from cellpatcher.orchestration import ( + run_patching_condition_repeat_tree, + run_patching_pipeline, +) +from cellpatcher.statistics import PreprocessingCheckError + + +def _clean_run(tmp_path, **kwargs): + from morphobricks_preprocessing.testing import make_clean_run + + return make_clean_run(tmp_path / "clean", **kwargs) + + +def _cell_config(run, **common): + config = CellPatcherConfig(mode="cell") + config.cell.patch_size = 8 + config.cell.exclude_border = False + config.common.extract_arrays = True + config.common.preprocessing_run = str(run.output) + for name, value in common.items(): + setattr(config.common, name, value) + return config + + +def test_a_tree_without_a_preprocessing_run_is_refused_with_the_command(tmp_path): + # Former code: the tree was walked and read by CellPatcher itself. + root = tmp_path / "raw_data" + (root / "control").mkdir(parents=True) + tifffile.imwrite(root / "control" / "c.tif", np.ones((8, 8), np.uint16)) + with pytest.raises(PreprocessingCheckError, match=r"(?s)preprocessing_run is not set.*morphobricks-preprocessing run") as refused: + run_patching_condition_repeat_tree(root, config=CellPatcherConfig(mode="whole_fov")) + assert [issue.code for issue in refused.value.issues] == ["no_preprocessing_run"] + + +def _patches(leaves): + return {(p.patch_id, p.mask_file): array for leaf in leaves for p, array in zip(leaf.result.patches, leaf.result.patch_arrays)} + + +def _labels_moved(path): + """Rewrite a mask with its two objects swapped in place, keeping its size: changed labels.""" + labels = tifffile.imread(path) + tifffile.imwrite(path, np.where(labels > 0, 3 - labels, 0).astype(labels.dtype), metadata={"axes": "YX"}) + + +def test_a_moved_experiment_is_opened_where_it_is_now(tmp_path): + # CP-2a: RunInputs.open with the experiment root and masks folder CellPatcher is given. + # Former code (CP-1): the run was opened where it recorded the experiment, so a moved + # experiment was refused. + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + before = _patches(run_patching_condition_repeat_tree(run.experiment, config=_cell_config(run))) + moved, moved_masks = tmp_path / "moved" / "experiment", tmp_path / "moved" / "masks" + moved.parent.mkdir() + shutil.move(run.experiment, moved) + shutil.move(run.masks, moved_masks) + after = _patches(run_patching_condition_repeat_tree(moved, mask_root_path=moved_masks, config=_cell_config(run))) + assert after.keys() == before.keys() and len(after) == 8 + assert all(np.array_equal(after[key], before[key]) for key in before) + # Without the masks folder's new place the run's masks are missing: refused, never guessed. + with pytest.raises(PreprocessingCheckError, match="4 masks changed or missing since the run"): + run_patching_condition_repeat_tree(moved, config=_cell_config(run)) + + +def test_a_copied_experiment_is_checked_where_it_is_read(tmp_path): + # CP-2a closes a CP-1 gap: with a copy of the experiment (the original in place), the run + # was opened at the original, and the copy's masks were read without any check. A mask + # changed in the copy was used. Now the copy is opened and read through RunInputs. + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + copy, copy_masks = tmp_path / "copy" / "experiment", tmp_path / "copy" / "masks" + shutil.copytree(run.experiment, copy) + shutil.copytree(run.masks, copy_masks) + config = _cell_config(run) + config.normalization.enabled = True + config.normalization.scope = "image" + config.normalization.plane_histograms = str(run.output / "plane_histograms.csv") + assert run_patching_condition_repeat_tree(copy, mask_root_path=copy_masks, config=config) + _labels_moved(copy_masks / "Cells" / "ctrl" / "rep1" / "a.tif") + with pytest.raises(PreprocessingCheckError, match=r"1 mask changed or missing since the run: Cells/ctrl/rep1/a.tif"): + run_patching_condition_repeat_tree(copy, mask_root_path=copy_masks, config=config) + + +def test_an_opened_run_is_not_reused_once_a_mask_or_another_image_changed(tmp_path): + # Former code (CP-1): cached_statistics kept the opened run for the session, keyed on the + # table and manifest; a mask, or an image not patched, changed after the first open was + # not checked again. Now every call opens the run again (cellpatcher.run_session). + from cellpatcher.normalization import cached_statistics + + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + table = run.output / "plane_histograms.csv" + assert cached_statistics(table).inputs is not None + _labels_moved(run.masks / "Cells" / "drug" / "rep1" / "b.tif") + with pytest.raises(PreprocessingCheckError, match="1 mask changed or missing since the run"): + cached_statistics(table) + + other = _clean_run(tmp_path / "other", axes="YX", shape=(16, 20)) + other_table = other.output / "plane_histograms.csv" + assert cached_statistics(other_table).inputs is not None + image = other.experiment / "drug" / "rep1" / "b.ome.tif" + tifffile.imwrite(image, np.ones((16, 20), np.uint16), ome=True, metadata={"axes": "YX"}) + with pytest.raises(PreprocessingCheckError, match="1 image changed or missing since the run"): + cached_statistics(other_table) + + +def test_within_one_call_the_run_is_opened_once(tmp_path, monkeypatch): + # A tree run normalizes every leaf with one opened run (the reason the cache existed). + from cellpatcher import file_runs + + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",), design=True) + opened = [] + real = file_runs.open_run + monkeypatch.setattr(file_runs, "open_run", lambda *args, **kwargs: opened.append(1) or real(*args, **kwargs)) + from cellpatcher import run_open + + monkeypatch.setattr(run_open, "open_run", lambda *args, **kwargs: pytest.fail("the run was opened again")) + config = _cell_config(run) + config.normalization.enabled, config.normalization.scope = True, "biological_repeat" + assert len(run_patching_condition_repeat_tree(run.experiment, config=config)) == 2 + assert opened == [1] + + +def test_file_runs_read_through_run_inputs_only(tmp_path, monkeypatch): + # No CellPatcher-side axis, pairing or file reading is used for a file run. + import cellpatcher.io + import cellpatcher.planes + + def refused(*args, **kwargs): + raise AssertionError("CellPatcher read or paired a file itself") + + for module, name in ((cellpatcher.io, "pair_masks"), (cellpatcher.io, "read_image"), (cellpatcher.io, "read_array"), + (cellpatcher.io, "refuse_image_header_axes"), (cellpatcher.io, "discover_images"), + (cellpatcher.planes, "iter_file_planes"), (cellpatcher.planes, "read_file_plane")): + monkeypatch.setattr(module, name, refused) + run = _clean_run(tmp_path, axes="ZCYX", shape=(3, 2, 16, 20), mask_types=()) + config = CellPatcherConfig(mode="grid") + config.grid.tile_size = 8 + config.common.preprocessing_run, config.common.extract_arrays = str(run.output), True + for z_policy in ("keep", "all"): + config.common.z_policy = z_policy + leaves = run_patching_condition_repeat_tree(run.experiment, config=config) + patch, array = leaves[0].result.patches[0], leaves[0].result.patch_arrays[0] + source = tifffile.imread(run.experiment / patch.source_relative_path) # Z, C, Y, X as written + expected = source[:, :, :8, :8] if z_policy == "keep" else source[patch.z0, :, :8, :8] + assert np.array_equal(array, expected) + + +@pytest.mark.parametrize(("setting", "value"), [ + ("image_axes", "ZCYX"), ("mask_axes", "ZYX"), ("mask_source", "ngff_labels"), ("mask_label_name", "Cells"), + ("condition", "ctrl"), ("biological_repeat", "rep1"), ("technical_repeat", "B02"), +]) +def test_the_former_file_settings_raise_a_migration_message(tmp_path, monkeypatch, setting, value): + from cellpatcher import file_runs + + monkeypatch.setattr(file_runs, "open_run", lambda *args, **kwargs: pytest.fail("opened before the refusal")) + config = _cell_config(SimpleNamespace(output=tmp_path / "run"), **{setting: value}) + with pytest.raises(ValueError, match=rf"(?s)since CellPatcher v0.2.0.*common.{setting}=.*in-memory arrays"): + run_patching_pipeline(tmp_path, config=config) + + +def test_a_folder_run_patches_one_condition_and_repeat(tmp_path): + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + with pytest.raises(ValueError, match="2 conditions or biological repeats.*run_patching_condition_repeat_tree"): + run_patching_pipeline(run.experiment, config=_cell_config(run)) + + +def test_the_run_config_records_the_layout_read_and_the_run(tmp_path): + from cellpatcher.orchestration import load_config + + run = _clean_run(tmp_path, axes="CYX", shape=(2, 16, 20), mask_types=("Cells",)) + config = _cell_config(run) + config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") + run_patching_condition_repeat_tree(run.experiment, config=config, combine_metadata=True) + saved = load_config(tmp_path / "out" / "run_config.json") + assert (saved.common.image_axes, saved.common.mask_axes) == ("CYX", "YX") + assert saved.common.preprocessing_run == str(run.output) + assert config.common.image_axes == "YX" # the caller's config is not changed + + +def test_another_table_than_the_run_s_is_refused(tmp_path): + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + other = _clean_run(tmp_path / "other", axes="YX", shape=(16, 20)) + config = _cell_config(run) + config.normalization.enabled, config.normalization.scope = True, "image" + config.normalization.plane_histograms = str(other.output / "plane_histograms.csv") + with pytest.raises(PreprocessingCheckError) as refused: + run_patching_condition_repeat_tree(run.experiment, config=config) + assert [issue.code for issue in refused.value.issues] == ["not_the_run_output"] + + +def test_masks_with_and_without_depth_share_the_run_s_layout(tmp_path): + # A three-plane image with a ZYX mask beside a one-plane image with a 2-D mask: the run's + # mask layout is ZYX, and the 2-D mask is its image's one plane (never a guessed plane). + from tests.helpers import file_run_config, write_tiff + + root, masks = tmp_path / "exp", tmp_path / "masks" + labels = np.zeros((3, 16, 16), np.uint16) + for z in range(3): + labels[z, 2 + 3 * z:5 + 3 * z, 2:5] = z + 1 + write_tiff(root / "ctrl" / "r1" / "a.tif", np.ones((3, 16, 16), np.uint16), "ZYX") + write_tiff(masks / "cells" / "ctrl" / "r1" / "a.tif", labels, "ZYX") + write_tiff(root / "ctrl" / "r1" / "b.tif", np.ones((16, 16), np.uint16), "YX") + write_tiff(masks / "cells" / "ctrl" / "r1" / "b.tif", labels[1], "YX") + config = CellPatcherConfig(mode="cell") + config.cell.patch_size, config.cell.exclude_border, config.common.z_policy = 6, False, "all" + file_run_config(config, root, "ZYX", masks=masks, missing_as_singleton=True) + (leaf,) = run_patching_condition_repeat_tree(root, mask_root_path=masks, config=config) + found = sorted((p.image_file, p.z0, p.label_id) for p in leaf.result.patches) + assert found == [("a.tif", 0, 1), ("a.tif", 1, 2), ("a.tif", 2, 3), ("b.tif", 0, 2)] + + +def test_a_moved_experiment_s_combined_run_config_is_written_from_the_call_s_run(tmp_path): + # Review of CP-2a: the combined run_config.json of a tree run was written after the call's + # run session had closed, so its normalization_inputs opened the run again where it had + # recorded the experiment: a moved experiment was refused after every leaf was patched. + run = _clean_run(tmp_path, axes="YX", shape=(16, 20), mask_types=("Cells",)) + moved, moved_masks = tmp_path / "moved" / "experiment", tmp_path / "moved" / "masks" + moved.parent.mkdir() + shutil.move(run.experiment, moved) + shutil.move(run.masks, moved_masks) + config = _cell_config(run) + config.normalization.enabled, config.normalization.scope = True, "image" + config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") + run_patching_condition_repeat_tree(moved, mask_root_path=moved_masks, config=config, combine_metadata=True) + saved = json.loads((tmp_path / "out" / "run_config.json").read_text()) + table_sha256 = hashlib.sha256((run.output / "plane_histograms.csv").read_bytes()).hexdigest() + assert saved["normalization_inputs"]["plane_histograms_sha256"] == table_sha256 diff --git a/tests/test_image_table.py b/tests/test_image_table.py index 2419ffa..74a900f 100644 --- a/tests/test_image_table.py +++ b/tests/test_image_table.py @@ -14,12 +14,12 @@ from cellpatcher.config import CellPatcherConfig, CommonConfig, ExportConfig, GridConfig, NormalizationConfig from cellpatcher.export import IMAGE_TABLE_FILENAME, export_patches from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree, run_patching_pipeline from cellpatcher.pipeline import run_patching from cellpatcher.types import IMAGE_TABLE_COLUMNS, ImageRecord, PatchRegion from tests.helpers import make_synthetic_cell_image_and_labels from tests.helpers import named +from tests.helpers import preprocessed_folder, preprocessed_tree # image_id tells images apart in an in-memory run, which has no file names. IDENTITY = ("condition", "biological_repeat", "technical_repeat", "image_id", "image_file") @@ -78,8 +78,8 @@ def _cell_config(output_dir, **common): def test_folder_run_lists_images_with_and_without_patches(tmp_path): images, masks = _cell_folders(tmp_path) - config = _cell_config(tmp_path / "out", condition="treated", biological_repeat="2026_01_01") - result = run_patching_pipeline(images, mask_folder_path=masks, config=config) + config = _cell_config(tmp_path / "out") # a file run's labels are the preprocessing run's (CP-2a) + result = preprocessed_folder(images, mask_folder_path=masks, config=config) out = tmp_path / "out" assert result.image_table_file == str(out / IMAGE_TABLE_FILENAME) @@ -89,7 +89,7 @@ def test_folder_run_lists_images_with_and_without_patches(tmp_path): by_file = {row["image_file"]: row for row in image_rows} assert by_file["a.tif"] == { - "condition": "treated", "biological_repeat": "2026_01_01", "technical_repeat": "single", + "condition": "single", "biological_repeat": "single", "technical_repeat": "single", "image_id": "image_0000", "image_file": "a.tif", "mask_file": "a_mask.tif", "source_relative_path": "a.tif", "image_height": "96", "image_width": "128", "n_patches": "4", "status": "processed", "reason": "", @@ -108,7 +108,7 @@ def test_folder_run_lists_images_with_and_without_patches(tmp_path): def test_every_exported_patch_file_belongs_to_a_listed_image(tmp_path): images, masks = _cell_folders(tmp_path) - run_patching_pipeline(images, mask_folder_path=masks, config=_cell_config(tmp_path / "out")) + preprocessed_folder(images, mask_folder_path=masks, config=_cell_config(tmp_path / "out")) patch_rows, image_rows = _assert_tables_join(tmp_path / "out") listed = {row["image_file"] for row in image_rows} for row in patch_rows: @@ -118,7 +118,7 @@ def test_every_exported_patch_file_belongs_to_a_listed_image(tmp_path): def test_a_run_where_no_image_gives_a_patch_still_lists_every_image(tmp_path): images, masks = _cell_folders(tmp_path, empty=("a", "b", "c")) - result = run_patching_pipeline(images, mask_folder_path=masks, config=_cell_config(tmp_path / "out")) + result = preprocessed_folder(images, mask_folder_path=masks, config=_cell_config(tmp_path / "out")) assert result.patches == [] patch_rows, image_rows = _assert_tables_join(tmp_path / "out") @@ -318,7 +318,7 @@ def _summary(rows): def test_tree_run_writes_a_table_in_every_leaf_folder(tmp_path): raw, masks = _tree(tmp_path) - results = run_patching_condition_repeat_tree(raw, mask_root_path=masks, config=_cell_config(tmp_path / "out")) + results = preprocessed_tree(raw, mask_root_path=masks, config=_cell_config(tmp_path / "out")) rows = [] for leaf in results: @@ -330,7 +330,7 @@ def test_tree_run_writes_a_table_in_every_leaf_folder(tmp_path): def test_combined_tree_run_writes_one_table_for_every_image(tmp_path): raw, masks = _tree(tmp_path) - results = run_patching_condition_repeat_tree( + results = preprocessed_tree( raw, mask_root_path=masks, config=_cell_config(tmp_path / "out"), combine_metadata=True ) diff --git a/tests/test_io.py b/tests/test_io.py index 50ea8f2..3cde6e1 100644 --- a/tests/test_io.py +++ b/tests/test_io.py @@ -9,7 +9,7 @@ validate_image_and_mask_folders, write_array, ) -from tests.helpers import write_tiff +from tests.helpers import file_run_config, write_tiff def test_list_supported_files_sorted_and_filtered(tmp_path): @@ -124,7 +124,7 @@ def test_run_patching_from_folders_grid_mode(tmp_path): img_dir.mkdir() write_array(img_dir / "a.tif", np.arange(64).reshape(8, 8)) - config = CellPatcherConfig(mode="grid") + config = file_run_config(CellPatcherConfig(mode="grid"), img_dir) config.grid.tile_size = 4 result = run_patching_from_folders(img_dir, config=config) @@ -144,7 +144,7 @@ def test_run_patching_from_folders_cell_mode(tmp_path): write_array(img_dir / "sample.tif", image) write_array(mask_dir / "sample_mask.tif", labels) - config = CellPatcherConfig(mode="cell") + config = file_run_config(CellPatcherConfig(mode="cell"), img_dir, masks=mask_dir) config.cell.patch_size = 8 result = run_patching_from_folders(img_dir, mask_folder_path=mask_dir, config=config) @@ -156,8 +156,9 @@ def test_run_patching_from_folders_cell_requires_mask_folder(tmp_path): img_dir.mkdir() write_array(img_dir / "a.tif", np.zeros((8, 8), dtype=np.float32)) - with pytest.raises(ValueError, match="mask_folder_path"): - run_patching_from_folders(img_dir, config=CellPatcherConfig(mode="cell")) + config = file_run_config(CellPatcherConfig(mode="cell"), img_dir) # a run without masks + with pytest.raises(ValueError, match="mode='cell' reads masks, but the preprocessing run .* recorded none"): + run_patching_from_folders(img_dir, config=config) from cellpatcher.orchestration import ( load_config, @@ -172,7 +173,7 @@ def test_run_patching_pipeline_alias_grid_mode(tmp_path): img_dir.mkdir() write_array(img_dir / "a.tif", np.arange(64).reshape(8, 8)) - config = CellPatcherConfig(mode="grid") + config = file_run_config(CellPatcherConfig(mode="grid"), img_dir) config.grid.tile_size = 4 result = run_patching_pipeline(img_dir, config=config) assert len(result.patches) == 4 @@ -219,7 +220,7 @@ def test_run_patching_pipeline_random_mode_with_optional_mask(tmp_path): write_array(img_dir / "a.tif", img) write_array(mask_dir / "a_mask.tif", mask) - config = CellPatcherConfig(mode="random") + config = file_run_config(CellPatcherConfig(mode="random"), img_dir, masks=mask_dir) config.random.patch_size = 4 config.random.n_patches = 5 config.random.seed = 0 @@ -233,7 +234,7 @@ def test_run_patching_pipeline_random_mode_without_mask(tmp_path): img_dir.mkdir() write_array(img_dir / "a.tif", np.zeros((32, 32), dtype=np.float32)) - config = CellPatcherConfig(mode="random") + config = file_run_config(CellPatcherConfig(mode="random"), img_dir) config.random.patch_size = 4 config.random.n_patches = 3 result = run_patching_pipeline(img_dir, config=config) @@ -273,6 +274,7 @@ def test_load_config_is_usable_for_patching(tmp_path): # Switch to grid so no mask folder is needed loaded.mode = "grid" loaded.grid.tile_size = 4 + file_run_config(loaded, img_dir) result = run_patching_pipeline(img_dir, config=loaded) assert len(result.patches) == 4 @@ -391,11 +393,7 @@ def test_images_in_formats_other_than_tiff_and_ome_zarr_are_refused_by_name(tmp_ write_array(img_dir / name, np.zeros((4, 4), np.uint8)) refusal = rf"1 image\(s\) in a format CellPatcher no longer reads: {name}\..*TIFF files .* OME-Zarr" with pytest.raises(ValueError, match=refusal): - validate_image_and_mask_folders(img_dir, mask_dir) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=4)) - with pytest.raises(ValueError, match=refusal): - run_patching_pipeline(img_dir, config=config) - + validate_image_and_mask_folders(img_dir, mask_dir) # file runs read the preprocessing run's images (CP-2a) @pytest.mark.parametrize("name", ["b.czi", "b.nd2", "b.lif", "b.LSM", "b.webp"]) @@ -407,10 +405,7 @@ def test_microscope_and_other_image_formats_are_refused_by_name_not_skipped(tmp_ (img_dir / name).write_bytes(b"not decoded") refusal = rf"1 image\(s\) in a format CellPatcher no longer reads: {name}\..*Convert them to TIFF or OME-Zarr" with pytest.raises(ValueError, match=refusal): - validate_image_and_mask_folders(img_dir, mask_dir) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=4)) - with pytest.raises(ValueError, match=refusal): - run_patching_pipeline(img_dir, config=config) + validate_image_and_mask_folders(img_dir, mask_dir) # file runs read the preprocessing run's images (CP-2a) (img_dir / name).unlink() (img_dir / "notes.txt").write_text("not an image") # other files stay ignored assert validate_image_and_mask_folders(img_dir, mask_dir) @@ -421,16 +416,13 @@ def test_a_folder_run_refuses_images_beside_subfolders(tmp_path): # patched a.tif and ignored the subfolder. img_dir, mask_dir = _folders(tmp_path, ["a.tif", "more/b.tif"], ["a_cells.tif"]) with pytest.raises(ValueError, match=r"\[tiff_outside_leaf\].*\(a.tif\)"): - validate_image_and_mask_folders(img_dir, mask_dir) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=4)) - with pytest.raises(ValueError, match=r"\[tiff_outside_leaf\].*\(a.tif\)"): - run_patching_pipeline(img_dir, config=config) + validate_image_and_mask_folders(img_dir, mask_dir) # file runs read the preprocessing run's images (CP-2a) def test_a_folder_run_refuses_a_folder_of_image_subfolders(tmp_path): img_dir, _ = _folders(tmp_path, ["ctrl/a.tif", "treated/b.tif"], []) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=4)) - with pytest.raises(ValueError, match=r"holds its images in subfolders \(ctrl, treated\).*run_patching_condition_repeat_tree"): + config = file_run_config(CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=4)), img_dir) + with pytest.raises(ValueError, match=r"run has 2 conditions or biological repeats.*run_patching_condition_repeat_tree"): run_patching_pipeline(img_dir, config=config) @@ -583,7 +575,8 @@ def test_a_refused_multi_axis_mask_says_how_to_save_it(tmp_path, axes): def test_mask_axes_cannot_read_a_named_z_stack_as_label_channels(tmp_path, mask_axes): # A mask whose header names a Z axis passes find_masks. common.mask_axes with a C axis # used to override that header and read the Z slices as label channels (with - # cell.mask_channel); a C axis is now refused for mask files (owner decision 2026-10-30). + # cell.mask_channel). Since CP-2a file runs read masks in the roles the preprocessing run + # recorded, and refuse any common.mask_axes but the default with a migration message. from cellpatcher import CellPatcherConfig img_dir, mask_dir = _folders(tmp_path, ["a.tif"], []) @@ -595,6 +588,5 @@ def test_mask_axes_cannot_read_a_named_z_stack_as_label_channels(tmp_path, mask_ config.common.mask_axes = mask_axes config.cell.mask_channel = 0 config.cell.exclude_border = False - with pytest.raises(ValueError, match=r"common\.mask_axes=.*channel \(C\) axis") as refused: + with pytest.raises(ValueError, match=rf"common\.mask_axes='{mask_axes}': each image's \(and mask's\) axes are the roles"): run_patching_pipeline(img_dir, config=config, mask_folder_path=mask_dir) - assert re.search(_MASK_CHANNEL_ADVICE, str(refused.value)) diff --git a/tests/test_morphobricks_contract.py b/tests/test_morphobricks_contract.py index a8ccf74..141ff1d 100644 --- a/tests/test_morphobricks_contract.py +++ b/tests/test_morphobricks_contract.py @@ -15,7 +15,7 @@ from cellpatcher.config import CellPatcherConfig from cellpatcher.io import SINGLE_TECHNICAL_REPEAT_LABEL, write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree +from tests.helpers import preprocessed_tree SHARED_COLUMNS = ( "condition", @@ -77,7 +77,7 @@ def _config(tmp_path: Path) -> CellPatcherConfig: def test_combined_tree_export_meets_the_shared_csv_contract(tmp_path): - run_patching_condition_repeat_tree(_tree(tmp_path), config=_config(tmp_path), combine_metadata=True) + preprocessed_tree(_tree(tmp_path), config=_config(tmp_path), combine_metadata=True) rows = _assert_contract(tmp_path / "out" / "patches.csv") wells = {row["technical_repeat"] for row in rows if row["condition"] == "treated"} assert wells == {"B02", "B03"} @@ -87,7 +87,7 @@ def test_combined_tree_export_meets_the_shared_csv_contract(tmp_path): def test_per_leaf_tree_export_meets_the_shared_csv_contract(tmp_path): - run_patching_condition_repeat_tree(_tree(tmp_path), config=_config(tmp_path)) + preprocessed_tree(_tree(tmp_path), config=_config(tmp_path)) csvs = sorted((tmp_path / "out").rglob("patches.csv")) assert [p.parent.relative_to(tmp_path / "out").as_posix() for p in csvs] == [ "control/2026_01_01", diff --git a/tests/test_ngff_labels.py b/tests/test_ngff_labels.py index f9ba874..c924fa7 100644 --- a/tests/test_ngff_labels.py +++ b/tests/test_ngff_labels.py @@ -1,7 +1,9 @@ -"""``common.mask_source='ngff_labels'``: masks from the OME-NGFF labels of OME-Zarr images. +"""Masks from the OME-NGFF label images of OME-Zarr images, read through the preprocessing run. -See docs/normalization_contract.md §4. Needs the ``preprocessing`` extra with -``zarr``; the tests skip without it. +Since CP-2a a file run reads the masks the preprocessing run paired, a label image's type +being its name (``matching.mask_type``), with ``RunInputs.read_mask``; the former +``common.mask_source='ngff_labels'`` and ``mask_label_name`` are refused. Needs the ``zarr`` +extra; the tests skip without it. """ import importlib.util @@ -11,8 +13,8 @@ from cellpatcher.config import CellPatcherConfig, CommonConfig from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree, run_patching_pipeline -from tests.helpers import write_ngff, write_ngff_labels +from cellpatcher.orchestration import run_patching_condition_repeat_tree +from tests.helpers import file_run_config, preprocess, write_ngff, write_ngff_labels pytestmark = pytest.mark.skipif( importlib.util.find_spec("morphobricks_preprocessing") is None or importlib.util.find_spec("zarr") is None, @@ -32,12 +34,15 @@ def _cell_config(**common): config = CellPatcherConfig(mode="cell") config.cell.patch_size = 6 config.cell.exclude_border = False - config.common.mask_source = "ngff_labels" for name, value in common.items(): setattr(config.common, name, value) return config +def _tree(root, config, axes="YX", **preprocessing): + return run_patching_condition_repeat_tree(root, config=file_run_config(config, root, axes, **preprocessing)) + + def _image(root, name="a.ome.zarr", shape=(16, 16), axes="yx", labels=None, label_axes=None): path = write_ngff(root / "control" / name, np.ones(shape, dtype=np.uint16), axes) if labels: @@ -54,73 +59,73 @@ def test_mask_source_is_validated(): def test_cell_mode_reads_the_only_label_image(tmp_path): _image(tmp_path, labels={"cells": _labels((16, 16), [(1, 2, 2), (2, 9, 9)])}) - (leaf,) = run_patching_condition_repeat_tree(tmp_path, config=_cell_config()) - assert [(p.label_id, p.mask_file) for p in leaf.result.patches] == [ - (1, "a.ome.zarr/labels/cells"), - (2, "a.ome.zarr/labels/cells"), + (leaf,) = _tree(tmp_path, _cell_config()) + assert [(p.label_id, p.mask_file, p.object_set) for p in leaf.result.patches] == [ + (1, "a.ome.zarr/labels/cells", "cells"), + (2, "a.ome.zarr/labels/cells", "cells"), ] -def test_the_label_image_is_chosen_by_name(tmp_path): +def test_the_label_image_is_chosen_by_its_type(tmp_path): _image(tmp_path, labels={"cells": _labels((16, 16), [(1, 2, 2)]), "nuclei": _labels((16, 16), [(7, 9, 9)])}) - - (leaf,) = run_patching_condition_repeat_tree(tmp_path, config=_cell_config(mask_label_name="nuclei")) + config = _cell_config() + config.matching.mask_type = "nuclei" + (leaf,) = _tree(tmp_path, config) assert [(p.label_id, p.mask_file) for p in leaf.result.patches] == [(7, "a.ome.zarr/labels/nuclei")] - with pytest.raises(ValueError, match="several label images \\(cells, nuclei\\)"): - run_patching_condition_repeat_tree(tmp_path, config=_cell_config()) - with pytest.raises(ValueError, match="no label image 'membranes'"): - run_patching_condition_repeat_tree(tmp_path, config=_cell_config(mask_label_name="membranes")) + config.matching.mask_type = None + with pytest.raises(ValueError, match=r"recorded 2 mask types \(cells, nuclei\); set matching.mask_type"): + run_patching_condition_repeat_tree(tmp_path, config=config) + config.matching.mask_type = "membranes" + with pytest.raises(ValueError, match="'membranes' is not a mask type of the preprocessing run"): + run_patching_condition_repeat_tree(tmp_path, config=config) def test_label_planes_follow_or_are_shared_by_the_image_planes(tmp_path): per_plane = np.stack([_labels((16, 16), [(z + 1, 2 + 4 * z, 2 + 4 * z)]) for z in range(3)]) _image(tmp_path, shape=(3, 16, 16), axes="zyx", labels={"cells": per_plane}) - config = _cell_config(image_axes="ZYX", mask_axes="ZYX", z_policy="all") - (leaf,) = run_patching_condition_repeat_tree(tmp_path, config=config) + (leaf,) = _tree(tmp_path, _cell_config(z_policy="all"), "ZYX") assert [(p.z0, p.label_id) for p in leaf.result.patches] == [(0, 1), (1, 2), (2, 3)] + # One 2-D label plane serves every plane only when stated shared for its type (preprocessing). shared_root = tmp_path / "shared" _image(shared_root, shape=(3, 16, 16), axes="zyx", labels={"cells": per_plane[:1]}) - (leaf,) = run_patching_condition_repeat_tree(shared_root, config=config) + with pytest.raises(AssertionError, match="does not have the depth slices and time points of its image"): + preprocess(shared_root, "ZYX") + (leaf,) = _tree(shared_root, _cell_config(z_policy="all"), "ZYX", mask_axes={"cells": "YX"}) assert [(p.z0, p.label_id) for p in leaf.result.patches] == [(0, 1), (1, 1), (2, 1)] -def test_a_label_image_that_does_not_fit_is_an_error(tmp_path): +def test_a_label_image_that_does_not_fit_has_no_clean_run(tmp_path): + # Former code: CellPatcher's own reader refused these; now the preprocessing run does, so no + # file run reads them. _image(tmp_path, labels={"cells": _labels((12, 16), [(1, 2, 2)])}) - with pytest.raises(ValueError, match="its Y axis has length 12, the image's 16"): - run_patching_condition_repeat_tree(tmp_path, config=_cell_config()) - - channels = tmp_path / "channels" - _image(channels, shape=(2, 16, 16), axes="cyx", labels={"cells": _labels((2, 16, 16), [(1, 2, 2)])}) - with pytest.raises(ValueError, match="its channel axis has length 2"): - run_patching_condition_repeat_tree(channels, config=_cell_config(image_axes="CYX")) - + with pytest.raises(AssertionError, match="do not have the size"): + preprocess(tmp_path) floats = tmp_path / "floats" _image(floats, labels={"cells": _labels((16, 16), [(1, 2, 2)]).astype(np.float32)}) - with pytest.raises(ValueError, match="hold integers"): - run_patching_condition_repeat_tree(floats, config=_cell_config()) + with pytest.raises(AssertionError, match="do not hold whole numbers"): + preprocess(floats) -def test_ngff_labels_needs_ome_zarr_images_and_no_mask_folder(tmp_path): +def test_the_former_label_settings_are_refused_for_file_runs(tmp_path): _image(tmp_path, labels={"cells": _labels((16, 16), [(1, 2, 2)])}) write_array(tmp_path / "control" / "b.tif", np.ones((16, 16), dtype=np.uint16)) - with pytest.raises(ValueError, match="1 image\\(s\\) are not OME-Zarr: b.tif"): - run_patching_condition_repeat_tree(tmp_path, config=_cell_config()) - - with pytest.raises(ValueError, match="remove mask_root_path"): - run_patching_condition_repeat_tree(tmp_path, config=_cell_config(), mask_root_path=tmp_path) - with pytest.raises(ValueError, match="remove mask_folder_path"): - run_patching_pipeline(tmp_path / "control", config=_cell_config(), mask_folder_path=tmp_path) + for setting in ({"mask_source": "ngff_labels"}, {"mask_label_name": "cells"}): + with pytest.raises(ValueError, match="common.mask_source=.*choose the type with matching.mask_type"): + run_patching_condition_repeat_tree(tmp_path, config=_cell_config(preprocessing_run="run", **setting)) + # Every image needs a mask of the type: the TIFF has no label image. + with pytest.raises(ValueError, match="recorded no 'cells' mask for control/b.tif"): + _tree(tmp_path, _cell_config()) def test_random_mode_samples_inside_the_labels(tmp_path): _image(tmp_path, shape=(32, 32), labels={"cells": _labels((32, 32), [(1, 20, 20)])}) config = CellPatcherConfig(mode="random") config.random.patch_size, config.random.n_patches = 3, 5 - config.common.mask_source = "ngff_labels" + config.matching.mask_type = "cells" # random mode reads masks when asked - (leaf,) = run_patching_condition_repeat_tree(tmp_path, config=config) + (leaf,) = _tree(tmp_path, config) assert leaf.result.patches assert {p.mask_file for p in leaf.result.patches} == {"a.ome.zarr/labels/cells"} assert all(17 <= p.y0 <= 23 and 17 <= p.x0 <= 23 for p in leaf.result.patches) diff --git a/tests/test_normalization.py b/tests/test_normalization.py index 726f9ed..8965b3a 100644 --- a/tests/test_normalization.py +++ b/tests/test_normalization.py @@ -18,7 +18,7 @@ from cellpatcher.orchestration import load_config, run_patching_condition_repeat_tree, run_patching_pipeline from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching from cellpatcher.statistics import PreprocessingCheckError -from tests.helpers import named +from tests.helpers import named, write_tiff as _write_tiff pytestmark = [ pytest.mark.skipif( @@ -37,13 +37,6 @@ ] -def _write_tiff(path, data, axes): - import tifffile - - path.parent.mkdir(parents=True, exist_ok=True) - tifffile.imwrite(path, data, imagej=True, metadata={"axes": axes}) - - def _confirmed_design(root, pairs=None): """The preprocessing brick's experiment design of *root*, every repeat group confirmed. @@ -85,6 +78,14 @@ def _config(csv_path, *, mode="grid", image_axes="ZCYX", scope="image", method=" return config +def _file(config): # a file run reads through the preprocessing run that wrote the table (CP-2a) + config.common.image_axes, config.common.preprocessing_run = "YX", os.path.dirname(config.normalization.plane_histograms) + return config + + +_tree = lambda root, config, **kwargs: run_patching_condition_repeat_tree(root, config=_file(config), **kwargs) + + def _read_csv(path): with open(path, newline="", encoding="utf-8") as f: reader = csv.DictReader(f) @@ -130,7 +131,7 @@ def test_a_tree_of_links_matches_the_brick_statistics(experiment, tmp_path): (linked / "treated" / "bio1").symlink_to(stored / "treated" / "bio1", target_is_directory=True) csv_path = _preprocess(linked, tmp_path / "linked_preprocessing", "ZCYX") - results = run_patching_condition_repeat_tree(linked, config=_config(csv_path)) + results = _tree(linked, _config(csv_path)) patches = [patch for leaf in results for patch in leaf.result.patches] assert {patch.source_relative_path for patch in patches} == set(IMAGES) @@ -197,7 +198,7 @@ def test_image_scope_pools_every_z_plane_of_the_image_per_channel(experiment, tm config = _config(csv_path, scope="image") config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") - results = run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + results = _tree(root, config, combine_metadata=True) for leaf in results: for patch, array in zip(leaf.result.patches, leaf.result.patch_arrays): @@ -240,7 +241,7 @@ def test_image_scope_pools_every_z_plane_of_the_image_per_channel(experiment, tm def test_biological_repeat_pools_the_repeat_across_conditions_per_channel(experiment): root, stacks, csv_path = experiment config = _config(csv_path, scope="biological_repeat", method="zscore") - results = run_patching_condition_repeat_tree(root, config=config) + results = _tree(root, config) patches = _patches_by_image(results) pooled = np.concatenate([stacks[m].transpose(1, 0, 2, 3).reshape(2, -1) for m in IMAGES], axis=1) @@ -263,7 +264,7 @@ def test_per_time_point_gives_each_time_point_its_own_parameters(tmp_path): config = _config(csv_path, image_axes="TYX", per_time_point=True) config.common.t_policy = "all" - (leaf,) = run_patching_condition_repeat_tree(root, config=config) + (leaf,) = _tree(root, config) by_t = {patch.t: patch for patch in leaf.result.patches} assert (by_t[0].normalization_parameters.offset, by_t[1].normalization_parameters.offset) == ((0.0,), (100.0,)) assert by_t[1].normalization_group == "control/a.tif::t=1" @@ -288,7 +289,7 @@ def test_split_channels_keep_their_own_parameters(experiment): root, _, csv_path = experiment config = _config(csv_path) config.extraction.split_channels = True - results = run_patching_condition_repeat_tree(root, config=config) + results = _tree(root, config) patches = [p for leaf in results for p in leaf.result.patches if p.source_relative_path == "control/bio1/a.tif"] offsets = {(p.c, p.normalization_parameters.channels, p.normalization_parameters.offset) for p in patches} assert len(offsets) == 2 and all(channels == (c,) and len(offset) == 1 for c, channels, offset in offsets) @@ -397,14 +398,14 @@ def test_keep_and_projections_are_refused_under_normalization(experiment): config = _config(csv_path) config.common.z_policy = policy with pytest.raises(ValueError, match="Normalization needs one plane per patch"): - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) def test_normalization_off_leaves_arrays_and_columns_unchanged(experiment): root, _, csv_path = experiment config = _config(csv_path) config.normalization.enabled = False - results = run_patching_condition_repeat_tree(root, config=config) + results = _tree(root, config) assert results[0].result.patch_arrays[0].dtype == np.uint8 columns = [*NORMALIZATION_COLUMNS, "normalization_parameters"] assert {getattr(p, column) for leaf in results for p in leaf.result.patches for column in columns} == {None} @@ -415,14 +416,13 @@ def test_normalization_off_leaves_arrays_and_columns_unchanged(experiment): # --------------------------------------------------------------------------- -def test_images_without_rows_are_reported_together(experiment): +def test_images_added_after_the_run_are_not_part_of_it(experiment): + # CP-2a: a file run patches the run's images (RunInputs.fovs), with no discover_experiment (design v0.4 + # §6), so an image added since is not seen. Former code: found and refused (images_without_rows). root, stacks, csv_path = experiment _write_tiff(root / "control" / "bio1" / "new.tif", stacks["control/bio1/a.tif"], "ZCYX") - _write_tiff(root / "treated" / "bio1" / "later.tif", stacks["control/bio1/a.tif"], "ZCYX") - with pytest.raises(PreprocessingCheckError) as error_info: - run_patching_condition_repeat_tree(root, config=_config(csv_path)) - (issue,) = error_info.value.issues - assert issue.paths == ("control/bio1/new.tif", "treated/bio1/later.tif") + patched = {patch.source_relative_path for leaf in _tree(root, _config(csv_path)) for patch in leaf.result.patches} + assert patched == set(IMAGES) def test_a_changed_image_is_detected_by_its_fingerprint_and_by_its_pixels(experiment): @@ -433,12 +433,12 @@ def test_a_changed_image_is_detected_by_its_fingerprint_and_by_its_pixels(experi stat = path.stat() _write_tiff(path, changed, "ZCYX") with pytest.raises(PreprocessingCheckError, match="changed or missing since the run: treated/bio1/c.tif"): - run_patching_condition_repeat_tree(root, config=_config(csv_path)) + _tree(root, _config(csv_path)) # Same size and time as the run recorded, but other pixel values: check_planes. os.utime(path, ns=(stat.st_atime_ns, stat.st_mtime_ns)) with pytest.raises(PreprocessingCheckError, match="the images changed after the preprocessing run"): - run_patching_condition_repeat_tree(root, config=_config(csv_path)) + _tree(root, _config(csv_path)) def test_in_memory_runs_need_source_relative_path_for_csv_scopes(experiment): @@ -472,7 +472,7 @@ def test_warnings_explain_the_scope(experiment): def messages(config): with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) return " ".join(str(w.message) for w in caught) assert "background" in messages(_config(csv_path, scope="image")) @@ -492,7 +492,7 @@ def test_a_flat_folder_run_normalizes_against_its_own_preprocessing(tmp_path): _write_tiff(folder / "a.tif", image, "YX") csv_path = _preprocess(folder, tmp_path / "preprocessing", "YX") - result = run_patching_pipeline(folder, config=_config(csv_path, image_axes="YX")) + result = run_patching_pipeline(folder, config=_file(_config(csv_path, image_axes="YX"))) found = { (p.source_relative_path, p.normalization_parameters.offset, p.normalization_parameters.scale) for p in result.patches @@ -514,7 +514,7 @@ def test_a_table_with_the_robust_histogram_range_normalizes(tmp_path): ) config = _config(robust, image_axes="YX", method="percentile") - result = run_patching_pipeline(folder, config=config) # the checks of every plane pass + result = run_patching_pipeline(folder, config=_file(config)) # the checks of every plane pass parameters = {p.normalization_parameters for p in result.patches} assert len(parameters) == 1 (only,) = parameters @@ -525,7 +525,7 @@ def test_a_table_with_the_robust_histogram_range_normalizes(tmp_path): # With the default range the same image gives no intensity spread to normalize by. minmax = _preprocess(folder, tmp_path / "minmax", "YX") with pytest.raises(ValueError, match="zero intensity spread"): - run_patching_pipeline(folder, config=_config(minmax, image_axes="YX", method="percentile")) + run_patching_pipeline(folder, config=_file(_config(minmax, image_axes="YX", method="percentile"))) def test_clip_comes_after_normalizing_and_before_resizing(): @@ -556,7 +556,7 @@ def test_split_channel_rows_fill_only_their_own_channel_pair(experiment, tmp_pat config = _config(csv_path) config.extraction.split_channels = True config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") - run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + _tree(root, config, combine_metadata=True) header, rows = _read_csv(tmp_path / "out" / "patches.csv") assert header[header.index("normalization_exact") + 1 :][:4] == CHANNEL_COLUMNS @@ -573,7 +573,7 @@ def test_a_selected_channel_row_fills_only_its_channel_pair(experiment, tmp_path config = _config(csv_path) config.extraction.split_channels = [1] config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") - run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + _tree(root, config, combine_metadata=True) header, rows = _read_csv(tmp_path / "out" / "patches.csv") assert rows and {row["c"] for row in rows} == {"1"} @@ -596,7 +596,7 @@ def test_the_brick_gives_the_parameters_and_arrays_of_the_earlier_code(experimen config = _config( csv_path, scope="biological_repeat", method="percentile", percentile=(2.0, 98.0), clip=(0.0, 1.0) ) - results = run_patching_condition_repeat_tree(root, config=config) + results = _tree(root, config) records = load_plane_statistics(csv_path).records for leaf in results: for patch, array in zip(leaf.result.patches, leaf.result.patch_arrays): @@ -669,7 +669,7 @@ def test_run_config_records_the_brick_settings_and_the_design(experiment, tmp_pa for scope in ("image", "biological_repeat"): config = _config(csv_path, scope=scope, pooling="conditions") config.export.enabled, config.common.output_dir = True, str(tmp_path / scope) - run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + _tree(root, config, combine_metadata=True) saved = json.loads((tmp_path / scope / RUN_CONFIG_FILENAME).read_text(encoding="utf-8")) assert saved["normalization_settings"]["scope"] == scope inputs = saved["normalization_inputs"] @@ -684,7 +684,7 @@ def test_run_config_records_the_brick_settings_and_the_design(experiment, tmp_pa def test_the_old_offset_and_scale_attributes_are_deprecated(experiment): root, _, csv_path = experiment - (leaf, *_) = run_patching_condition_repeat_tree(root, config=_config(csv_path)) + (leaf, *_) = _tree(root, _config(csv_path)) patch = leaf.result.patches[0] with pytest.warns(DeprecationWarning, match="normalization_parameters"): offset = patch.normalization_offset @@ -712,12 +712,12 @@ def _unpaired_experiment(tmp_path, pairs=None, design="confirmed"): def test_the_design_pairs_repeats_with_different_names(tmp_path): root, images, csv_path = _unpaired_experiment(tmp_path) config = _config(csv_path, image_axes="YX", scope="biological_repeat") - patches = _patches_by_image(run_patching_condition_repeat_tree(root, config=config)) + patches = _patches_by_image(_tree(root, config)) assert {p.normalization_group for p in patches.values()} == {"rep_a", "rep_b"} # unpaired: two groups root, images, csv_path = _unpaired_experiment(tmp_path / "paired", pairs={("treated", "rep_b"): "rep_a"}) config = _config(csv_path, image_axes="YX", scope="biological_repeat") - patches = _patches_by_image(run_patching_condition_repeat_tree(root, config=config)) + patches = _patches_by_image(_tree(root, config)) every = np.concatenate([image.ravel() for image in images.values()]) assert {p.normalization_group for p in patches.values()} == {"rep_a"} assert {p.normalization_parameters.offset for p in patches.values()} == {(float(every.min()),)} @@ -735,7 +735,7 @@ def test_pooling_by_conditions_weighs_each_condition_equally(tmp_path): def offsets(pooling): config = _config(csv_path, image_axes="YX", scope="biological_repeat", method="zscore", pooling=pooling) config.grid.tile_size = 4 - patches = _patches_by_image(run_patching_condition_repeat_tree(root, config=config)) + patches = _patches_by_image(_tree(root, config)) return patches["control/rep1/a.tif"].normalization_parameters.offset[0] by_pixels = np.concatenate([small.ravel(), large.ravel()]).astype(np.float64).mean() @@ -754,7 +754,7 @@ def no_reading(*args, **kwargs): monkeypatch.setattr(cellpatcher.io, "read_image", no_reading) config = _config(csv_path, image_axes="YX", scope="biological_repeat") with pytest.raises(PreprocessingCheckError) as missing: - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) assert [issue.code for issue in missing.value.issues] == ["missing_experiment_design"] assert "step 1 of the preprocessing brick" in str(missing.value) @@ -765,7 +765,7 @@ def no_reading(*args, **kwargs): mbp.write_experiment_design(unconfirmed, tmp_path / "unconfirmed.csv") config.normalization.experiment_design = str(tmp_path / "unconfirmed.csv") with pytest.raises(PreprocessingCheckError) as elsewhere: - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) assert [issue.code for issue in elsewhere.value.issues] == ["not_the_run_output"] @@ -778,7 +778,7 @@ def test_a_design_changed_after_preprocessing_is_refused(tmp_path): mbp.write_experiment_design(changed, design_path) config = _config(csv_path, image_axes="YX", scope="biological_repeat") with pytest.raises(PreprocessingCheckError, match="design experiment_design.csv is not the one the run wrote") as caught: - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) assert [issue.code for issue in caught.value.issues] == ["preprocessing_run_refused"] @@ -811,7 +811,7 @@ def test_the_design_is_read_with_the_shared_strict_reader(tmp_path, edit, match) root, csv_path = _design_edited(tmp_path, edit) config = _config(csv_path, image_axes="YX", scope="biological_repeat") with pytest.raises(PreprocessingCheckError, match=match) as caught: - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) assert [issue.code for issue in caught.value.issues] == ["unreadable_experiment_design"] @@ -828,14 +828,14 @@ def recording(path, *, expected_sha256=None): monkeypatch.setattr(design_module, "read_confirmed_design", recording) root, _, csv_path = _unpaired_experiment(tmp_path) manifest = json.loads((csv_path.parent / "run_manifest.json").read_text(encoding="utf-8")) - run_patching_condition_repeat_tree(root, config=_config(csv_path, image_axes="YX", scope="biological_repeat")) + _tree(root, _config(csv_path, image_axes="YX", scope="biological_repeat")) assert calls and set(calls) == {manifest["experiment_design_sha256"]} def test_a_run_without_a_design_is_refused_at_the_repeat_scope(tmp_path): root, _, csv_path = _unpaired_experiment(tmp_path, design=None) with pytest.raises(PreprocessingCheckError) as caught: - run_patching_condition_repeat_tree(root, config=_config(csv_path, image_axes="YX", scope="biological_repeat")) + _tree(root, _config(csv_path, image_axes="YX", scope="biological_repeat")) (issue,) = caught.value.issues assert issue.code == "missing_experiment_design" assert "the preprocessing run recorded no design" in issue.message @@ -849,7 +849,7 @@ def test_repeat_groups_missing_a_condition_are_reported(tmp_path): csv_path = _preprocess(root, tmp_path / "incomplete_out", "YX") config = _config(csv_path, image_axes="YX", scope="biological_repeat") with pytest.warns(UserWarning, match="do not have images from every condition"): - run_patching_condition_repeat_tree(root, config=config) + _tree(root, config) def test_raw_patches_of_a_removed_scope_still_normalize_on_load(experiment, tmp_path): @@ -859,7 +859,7 @@ def test_raw_patches_of_a_removed_scope_still_normalize_on_load(experiment, tmp_ config = _config(csv_path) config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") config.normalization.export_values = "raw" - results = run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + results = _tree(root, config, combine_metadata=True) delivered = {p.patch_id: a for leaf in results for p, a in zip(leaf.result.patches, leaf.result.patch_arrays)} # The same files as a run at the removed condition scope would have left them. @@ -886,8 +886,8 @@ def test_csv_scopes_refuse_a_table_without_its_manifest(experiment): # Issue #46 warned and went on without run_manifest.json; CP-1 reads the run through # RunInputs, which needs it, so the table, size/time and axis checks always run. root, _, csv_path = experiment - with_manifest = _messages(lambda: run_patching_condition_repeat_tree( - root, config=_config(csv_path, scope="biological_repeat") + with_manifest = _messages(lambda: _tree( + root, _config(csv_path, scope="biological_repeat") )) assert "run_manifest.json" not in with_manifest assert "experiment_design_sha256" not in with_manifest @@ -898,7 +898,7 @@ def test_csv_scopes_refuse_a_table_without_its_manifest(experiment): (alone / name).write_bytes((csv_path.parent / name).read_bytes()) for scope in ("image", "biological_repeat"): with pytest.raises(PreprocessingCheckError, match="holds no run_manifest.json") as caught: - run_patching_condition_repeat_tree(root, config=_config(alone / csv_path.name, scope=scope)) + _tree(root, _config(alone / csv_path.name, scope=scope)) assert [issue.code for issue in caught.value.issues] == ["preprocessing_run_refused"] @@ -911,7 +911,7 @@ def test_a_run_that_recorded_no_design_is_refused_at_the_repeat_scope(experiment manifest["experiment_design_sha256"] = None manifest_path.write_text(json.dumps(manifest), encoding="utf-8") with pytest.raises(PreprocessingCheckError) as caught: - run_patching_condition_repeat_tree(root, config=_config(csv_path, scope="biological_repeat")) + _tree(root, _config(csv_path, scope="biological_repeat")) assert [issue.code for issue in caught.value.issues] == ["missing_experiment_design"] diff --git a/tests/test_ome_zarr_images.py b/tests/test_ome_zarr_images.py index 536ed37..eac3191 100644 --- a/tests/test_ome_zarr_images.py +++ b/tests/test_ome_zarr_images.py @@ -22,7 +22,13 @@ ) from cellpatcher.orchestration import run_patching_condition_repeat_tree from cellpatcher.statistics import PreprocessingUnavailableError -from tests.helpers import write_ngff, write_ngff_labels +from tests.helpers import ( + preprocess, + preprocessed_folder, + preprocessed_tree, + write_ngff, + write_ngff_labels, +) needs_zarr = pytest.mark.skipif( importlib.util.find_spec("morphobricks_preprocessing") is None or importlib.util.find_spec("zarr") is None, @@ -148,7 +154,7 @@ def test_tree_run_patches_ome_zarr_and_tiff_images_alike(tmp_path): config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) config.common.extract_arrays = True - (leaf,) = run_patching_condition_repeat_tree(root, config=config) + (leaf,) = preprocessed_tree(root, config=config) by_file = {} for patch, array in zip(leaf.result.patches, leaf.result.patch_arrays): by_file.setdefault(patch.image_file, []).append(array) @@ -169,7 +175,7 @@ def test_cell_mode_matches_tiff_masks_to_stores(tmp_path): config = CellPatcherConfig(mode="cell") config.cell.patch_size = 8 - (leaf,) = run_patching_condition_repeat_tree(images, config=config, mask_root_path=masks) + (leaf,) = preprocessed_tree(images, config=config, mask_root_path=masks) assert [(p.image_file, p.mask_file, p.label_id) for p in leaf.result.patches] == [ ("a.ome.zarr", "a_mask.tif", 1) ] @@ -177,18 +183,16 @@ def test_cell_mode_matches_tiff_masks_to_stores(tmp_path): @needs_zarr @pytest.mark.parametrize("label_name, mask_type, outcome", [ - ("cells", None, "a_mask.tif"), + ("cells", None, r"recorded 2 mask types \(cells, mask\); set matching.mask_type"), ("cells", "mask", "a_mask.tif"), - ("cells", "cells", "No mask of type 'cells' pairs.*mask_source='ngff_labels'"), - ("mask", None, r"\[duplicate_mask\]"), + ("cells", "cells", "a.ome.zarr/labels/cells"), + ("mask", None, "duplicate"), ]) def test_tiff_masks_are_paired_beside_the_stores_own_label_images(tmp_path, label_name, mask_type, outcome): - # Since preprocessing v0.4, find_masks pairs an OME-Zarr image's own label images too - # (design v0.4 §2). A mask folder run reads only the folder's files: a label image is not - # a mask type of the folder (before this fix: "the mask folder holds 2 mask types"), and - # choosing it is refused instead of looking for the label inside the mask folder. A - # label image and a folder mask of the same type for one image are preprocessing's - # duplicate_mask. + # Since CP-2a a file run reads the masks the preprocessing run paired: a store's own label + # images (their type is their name) beside the masks folder's files. A label image and a + # folder mask of the same type for one image are refused by preprocessing (duplicate_mask, + # rc1), so no clean run has them. images, masks = tmp_path / "images", tmp_path / "masks" labels = np.zeros((16, 16), dtype=np.uint16) labels[4:9, 4:9] = 1 @@ -198,19 +202,27 @@ def test_tiff_masks_are_paired_beside_the_stores_own_label_images(tmp_path, labe config = CellPatcherConfig(mode="cell") config.cell.patch_size = 8 config.matching.mask_type = mask_type - if outcome.endswith(".tif"): - (leaf,) = run_patching_condition_repeat_tree(images, config=config, mask_root_path=masks) + if outcome == "duplicate": + with pytest.raises(AssertionError, match="two masks of the same type \\(for example a file in the masks folder and a label image"): + preprocess(images, masks=masks) + elif outcome.startswith("a"): + (leaf,) = preprocessed_tree(images, config=config, mask_root_path=masks) assert [(p.mask_file, p.label_id) for p in leaf.result.patches] == [(outcome, 1)] else: with pytest.raises(ValueError, match=outcome): - run_patching_condition_repeat_tree(images, config=config, mask_root_path=masks) + preprocessed_tree(images, config=config, mask_root_path=masks) @needs_zarr def test_ome_zarr_planes_link_to_preprocessing_rows(tmp_path): import morphobricks_preprocessing as mbp - from cellpatcher.statistics import SourcePlane, check_images, check_planes, load_plane_statistics + from cellpatcher.statistics import ( + SourcePlane, + check_images, + check_planes, + load_plane_statistics, + ) root = tmp_path / "experiment" data = np.random.default_rng(0).integers(0, 255, (2, 3, 16, 16), dtype=np.uint8) @@ -220,7 +232,7 @@ def test_ome_zarr_planes_link_to_preprocessing_rows(tmp_path): statistics = load_plane_statistics(result.csv_path) config = CellPatcherConfig(mode="whole_fov") - config.common.image_axes, config.common.z_policy = "CZYX", "all" + config.common.z_policy, config.common.preprocessing_run = "all", str(tmp_path / "pre") config.extraction.split_channels = True (leaf,) = run_patching_condition_repeat_tree(root, config=config) patches = leaf.result.patches @@ -240,14 +252,11 @@ def test_ome_zarr_planes_link_to_preprocessing_rows(tmp_path): @needs_zarr def test_an_ome_zarr_image_is_matched_by_role_not_by_header_position(tmp_path): - # OME-Zarr axes are put into common.image_axes by role (conform_axes), so its stored order - # (ZCYX with Z of length 1) is not compared position by position with the declared CYX. - from cellpatcher.orchestration import run_patching_pipeline - + # Read in the roles the preprocessing run recorded (CP-2a), in canonical order: its stored + # ZCYX with Z of length 1 gives CYX patches. data = np.arange(2 * 16 * 16, dtype=np.uint16).reshape(1, 2, 16, 16) write_ngff(tmp_path / "exp" / "a.zarr", data, axes="zcyx") config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.image_axes = "CYX" config.common.extract_arrays = True - result = run_patching_pipeline(str(tmp_path / "exp"), config=config) + result = preprocessed_folder(tmp_path / "exp", config=config, axes="ZCYX") assert result.patch_arrays[0].shape == (2, 8, 8) diff --git a/tests/test_patch_geometry.py b/tests/test_patch_geometry.py index 653d58d..e1f409e 100644 --- a/tests/test_patch_geometry.py +++ b/tests/test_patch_geometry.py @@ -7,13 +7,12 @@ from cellpatcher.config import CellConfig, CellPatcherConfig, CommonConfig, ExportConfig, ExtractionConfig, GridConfig from cellpatcher.export import save_metadata_csv from cellpatcher.extract import extract_patch -from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree from cellpatcher.pipeline import _padding, run_patching from cellpatcher.types import PatchRegion from tests.helpers import make_synthetic_cell_image_and_labels -from tests.helpers import named +from tests.helpers import named, write_tiff +from tests.helpers import preprocessed_tree GEOMETRY_COLUMNS = [ "image_height", "image_width", "pad_top", "pad_bottom", "pad_left", "pad_right", "resized", "output_axes", @@ -161,14 +160,14 @@ def test_padding_matches_what_extraction_pads(pad_mode): def test_combined_tree_csv_records_each_image_size(tmp_path): for well, shape in (("A01", (32, 32)), ("A02", (32, 48))): - write_array(tmp_path / "raw" / "control" / "2024_01_01" / well / "img.tif", _stack(shape=shape)) + write_tiff(tmp_path / "raw" / "control" / "2024_01_01" / well / "img.tif", _stack(shape=shape), "CYX") config = CellPatcherConfig( mode="grid", - common=CommonConfig(image_axes="CYX", output_dir=str(tmp_path / "out")), + common=CommonConfig(output_dir=str(tmp_path / "out")), grid=GridConfig(tile_size=16), export=ExportConfig(enabled=True), ) - run_patching_condition_repeat_tree(str(tmp_path / "raw"), config=config, combine_metadata=True) + preprocessed_tree(str(tmp_path / "raw"), config=config, axes="CYX", combine_metadata=True) with (tmp_path / "out" / "patches.csv").open(newline="", encoding="utf-8") as f: rows = list(csv.DictReader(f)) diff --git a/tests/test_patch_identity.py b/tests/test_patch_identity.py index 8e91167..95dc2fc 100644 --- a/tests/test_patch_identity.py +++ b/tests/test_patch_identity.py @@ -14,9 +14,10 @@ from cellpatcher.config import CellPatcherConfig, CommonConfig, ExtractionConfig, GridConfig from cellpatcher.ids import DEFAULT_OBJECT_SET from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree, run_patching_pipeline +from cellpatcher.orchestration import run_patching_pipeline from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching -from tests.helpers import write_ngff, write_ngff_labels +from tests.helpers import preprocess, write_ngff, write_ngff_labels +from tests.helpers import preprocessed_folder as _folder, preprocessed_tree as _tree needs_preprocessing = pytest.mark.skipif( importlib.util.find_spec("morphobricks_preprocessing") is None, reason="needs morphobricks-preprocessing (a core dependency)" @@ -69,7 +70,7 @@ def _ids_of(result, image_file): def test_same_file_alone_in_ten_files_and_after_an_earlier_file_has_the_same_patch_ids(tmp_path): alone, alone_masks = tmp_path / "alone" / "images", tmp_path / "alone" / "masks" _write_pair(alone, alone_masks, "m_target", 7) - alone_result = run_patching_pipeline(alone, mask_folder_path=alone_masks, config=_cell_config()) + alone_result = _folder(alone, mask_folder_path=alone_masks, config=_cell_config()) expected = _ids_of(alone_result, "m_target.tif") assert expected == ["single/single/single/m_target/cells/label10", "single/single/single/m_target/cells/label47"] @@ -77,12 +78,12 @@ def test_same_file_alone_in_ten_files_and_after_an_earlier_file_has_the_same_pat for index in range(9): _write_pair(ten, ten_masks, f"n{index}", index) _write_pair(ten, ten_masks, "m_target", 7) - result = run_patching_pipeline(ten, mask_folder_path=ten_masks, config=_cell_config()) + result = _folder(ten, mask_folder_path=ten_masks, config=_cell_config()) assert _ids_of(result, "m_target.tif") == expected # A file that sorts first shifts every image_id, but no patch_id. _write_pair(ten, ten_masks, "a_first", 11) - shifted = run_patching_pipeline(ten, mask_folder_path=ten_masks, config=_cell_config()) + shifted = _folder(ten, mask_folder_path=ten_masks, config=_cell_config()) target_image_ids = {p.image_id for p in shifted.patches if p.image_file == "m_target.tif"} assert target_image_ids != {p.image_id for p in result.patches if p.image_file == "m_target.tif"} assert _ids_of(shifted, "m_target.tif") == expected @@ -93,10 +94,10 @@ def test_same_file_alone_in_ten_files_and_after_an_earlier_file_has_the_same_pat def test_grid_ids_do_not_depend_on_the_other_files(tmp_path): config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=16)) write_array(tmp_path / "one" / "img.tif", _image(1)) - alone = run_patching_pipeline(tmp_path / "one", config=config) + alone = _folder(tmp_path / "one", config=config) for name, seed in (("aaa.tif", 2), ("img.tif", 1), ("zzz.tif", 3)): write_array(tmp_path / "three" / name, _image(seed)) - three = run_patching_pipeline(tmp_path / "three", config=config) + three = _folder(tmp_path / "three", config=config) assert _ids_of(three, "img.tif") == _ids_of(alone, "img.tif") == [ f"single/single/single/img/grid-y{y}-x{x}-h16-w16" for y in (0, 16) for x in (0, 16) ] @@ -106,7 +107,7 @@ def test_two_wells_with_the_same_file_names_get_unique_ids_in_a_tree_run(tmp_pat images, masks = tmp_path / "images", tmp_path / "masks" for well in ("B02", "B03"): _write_pair(images / "control" / "2026_01_01" / well, masks / "control" / "2026_01_01" / well, "img", 1) - leaves = run_patching_condition_repeat_tree( + leaves = _tree( images, mask_root_path=masks, config=_cell_config(tmp_path / "out"), combine_metadata=True ) @@ -128,7 +129,7 @@ def test_a_rerun_writes_a_byte_identical_patches_csv(tmp_path): for index in range(3): _write_pair(images, masks, f"img{index}", index) for out in ("first", "second"): - run_patching_pipeline(images, mask_folder_path=masks, config=_cell_config(tmp_path / out)) + _folder(images, mask_folder_path=masks, config=_cell_config(tmp_path / out)) for table in ("patches.csv", "images.csv"): assert (tmp_path / "first" / table).read_bytes() == (tmp_path / "second" / table).read_bytes() @@ -195,9 +196,9 @@ def test_whole_fov_and_supplied_position_id(): def test_folder_runs_fill_position_id_without_split_channels(tmp_path): - write_array(tmp_path / "img.ome.tif", _image(0)) - config = CellPatcherConfig(mode="whole_fov", common=CommonConfig(condition="treated", biological_repeat="r1")) - result = run_patching_pipeline(tmp_path, config=config) + # Since CP-2a the labels and position_id are those the preprocessing run recorded. + write_array(tmp_path / "exp" / "treated" / "r1" / "img.ome.tif", _image(0)) + result = _folder(tmp_path / "exp", config=CellPatcherConfig(mode="whole_fov")) # D85: the image name without its whole extension (preprocessing's rule: .ome.tif -> img). assert [patch.position_id for patch in result.patches] == ["img"] assert [record.position_id for record in result.image_records] == ["img"] @@ -231,19 +232,19 @@ def _rectangles(result, image_file): def test_random_rectangles_do_not_depend_on_the_place_in_the_run(tmp_path): write_array(tmp_path / "alone" / "m_target.tif", _image(7)) - alone = _rectangles(run_patching_pipeline(tmp_path / "alone", config=_random_config()), "m_target.tif") + alone = _rectangles(_folder(tmp_path / "alone", config=_random_config()), "m_target.tif") assert len(alone) == 5 ten = tmp_path / "ten" for index in range(9): write_array(ten / f"n{index}.tif", _image(index)) write_array(ten / "m_target.tif", _image(7)) - assert _rectangles(run_patching_pipeline(ten, config=_random_config()), "m_target.tif") == alone + assert _rectangles(_folder(ten, config=_random_config()), "m_target.tif") == alone write_array(ten / "a_first.tif", _image(11)) - assert _rectangles(run_patching_pipeline(ten, config=_random_config()), "m_target.tif") == alone + assert _rectangles(_folder(ten, config=_random_config()), "m_target.tif") == alone # Another seed draws other rectangles; so does another position. - reseeded = run_patching_pipeline(tmp_path / "alone", config=_random_config(seed=1)) + reseeded = _folder(tmp_path / "alone", config=_random_config(seed=1)) assert _rectangles(reseeded, "m_target.tif") != alone other = run_patching(images=_image(7), config=_random_config(), image_metadata=[{"image_file": "other.tif"}]) assert [(p.y0, p.x0) for p in other.patches] != [(y0, x0) for _, y0, x0 in alone] @@ -281,7 +282,7 @@ def test_object_set_precedence(tmp_path): write_array(images / "img.tif", _image(0)) # Every paired mask has a type (issue #45: the preprocessing pairing), and it is the object set. write_array(masks / "img_cells.tif", _labels(0)) - result = run_patching_pipeline(images, mask_folder_path=masks, config=_cell_config()) + result = _folder(images, mask_folder_path=masks, config=_cell_config()) assert {patch.object_set for patch in result.patches} == {"cells"} # Without a name of its own (in memory), a mask is the default object set. @@ -290,7 +291,7 @@ def test_object_set_precedence(tmp_path): config = _cell_config() config.common.object_set = "cytoplasm" - result = run_patching_pipeline(images, mask_folder_path=masks, config=config) + result = _folder(images, mask_folder_path=masks, config=config) assert {patch.object_set for patch in result.patches} == {"cytoplasm"} # In memory, image_metadata wins over the config; other modes have none. @@ -311,13 +312,13 @@ def test_object_set_must_be_one_safe_level(value): @needs_zarr def test_object_set_is_the_ngff_label_name(tmp_path): - path = write_ngff(tmp_path / "control" / "a.ome.zarr", np.ones((16, 16), dtype=np.uint16), "yx") + path = write_ngff(tmp_path / "exp" / "control" / "a.ome.zarr", np.ones((16, 16), dtype=np.uint16), "yx") labels = np.zeros((16, 16), dtype=np.uint16) labels[3:7, 3:7] = 9 write_ngff_labels(path, {"nuclei": labels}, "yx") config = _cell_config() - config.common.mask_source = "ngff_labels" - (leaf,) = run_patching_condition_repeat_tree(tmp_path, config=config) + config.matching.mask_type = None # the run's only type, the label image's name + (leaf,) = _tree(tmp_path / "exp", config=config) assert [patch.patch_id for patch in leaf.result.patches] == ["control/single/single/a/nuclei/label9"] @@ -336,40 +337,40 @@ def test_the_connected_components_opt_in_is_recorded_in_run_config(tmp_path): assert saved["cell"]["binary_mask_policy"] == "connected_components" -@pytest.mark.parametrize("z_policy", ["keep", "max"]) -def test_a_multi_series_tiff_read_whole_is_refused(tmp_path, z_policy): +def test_a_multi_series_tiff_never_reaches_a_file_run(tmp_path): + # File runs read only a clean preprocessing run's images (CP-2a), and the run refuses a + # file of several series. Former code: CellPatcher's own reader refused it. import tifffile path = tmp_path / "images" / "plate.tif" path.parent.mkdir() tifffile.imwrite(path, np.zeros((2, 16, 16), dtype=np.uint16)) tifffile.imwrite(path, np.ones((2, 16, 16), dtype=np.uint16), append=True) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.image_axes, config.common.z_policy = "ZYX", z_policy - with pytest.raises(ValueError, match="plate.tif holds 2 image series"): - run_patching_pipeline(path.parent, config=config) + with pytest.raises(AssertionError, match="series"): + preprocess(path.parent, "ZYX") @needs_preprocessing def test_a_multi_series_tiff_is_refused_when_reading_planes(tmp_path): + # Lazy delivery from a path still reads planes with planes.read_file_plane (until CP-2b). import tifffile + from cellpatcher.planes import PlaneChoice, read_file_plane + path = tmp_path / "plate.tif" tifffile.imwrite(path, np.zeros((2, 16, 16), dtype=np.uint16)) tifffile.imwrite(path, np.ones((2, 16, 16), dtype=np.uint16), append=True) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.image_axes, config.common.z_policy = "ZYX", "all" with pytest.raises(ValueError, match="plate.tif holds 2 images"): - run_patching_pipeline(tmp_path, config=config) + read_file_plane(path, "ZYX", PlaneChoice(t=None, z=0, suffix=""), CommonConfig(image_axes="ZYX", z_policy="all")) @needs_zarr def test_a_store_with_several_images_is_refused_when_reading_planes(tmp_path): - write_ngff(tmp_path / "plate.ome.zarr", np.zeros((2, 16, 16), dtype=np.uint16), "zyx", images=2) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.image_axes, config.common.z_policy = "ZYX", "all" + from cellpatcher.planes import PlaneChoice, read_file_plane + + path = write_ngff(tmp_path / "plate.ome.zarr", np.zeros((2, 16, 16), dtype=np.uint16), "zyx", images=2) with pytest.raises(ValueError, match="plate.ome.zarr holds 2 images"): - run_patching_pipeline(tmp_path, config=config) + read_file_plane(path, "ZYX", PlaneChoice(t=None, z=0, suffix=""), CommonConfig(image_axes="ZYX", z_policy="all")) def test_positions_differing_only_in_case_are_refused_in_one_repeat(): @@ -408,14 +409,13 @@ def test_a_duplicate_position_is_refused_whatever_tokens_its_patches_get(): image_metadata=[{"position_id": "img"}, {"position_id": "img"}]) -def test_a_duplicate_position_is_refused_in_a_folder_run(tmp_path): - """The lazily read folder route (one source at a time) checks it too, before the image's planes.""" - write_array(tmp_path / "Img.tif", _image(0)) - write_array(tmp_path / "img.tiff", _image(1)) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.z_policy = "all" - with pytest.raises(ValueError, match=r"\[duplicate_position_id\].*Img.tif, img.tiff"): - run_patching_pipeline(tmp_path, config=config) +def test_a_duplicate_position_never_reaches_a_file_run(tmp_path): + """File runs take each FOV key from a clean preprocessing run (CP-2a), whose discovery + refuses Img.tif and img.tiff before CellPatcher reads anything (D85).""" + write_array(tmp_path / "images" / "Img.tif", _image(0)) + write_array(tmp_path / "images" / "img.tiff", _image(1)) + with pytest.raises(AssertionError, match=r"same name apart from the file extension or upper/lower case"): + preprocess(tmp_path / "images") def test_names_differing_only_in_case_clash_in_the_run(): @@ -546,115 +546,47 @@ def test_config_fov_labels_must_be_text(field): @pytest.mark.parametrize("split", [False, True]) def test_a_folder_run_refuses_config_identity_before_reading(tmp_path, monkeypatch, field, split): """Re-audit of #41: an invalid config label (also from a reloaded JSON config) is refused - before any image or mask is listed or read, on the eager and the plane-split routes.""" - from cellpatcher import orchestration + before the preprocessing run is opened or any file read, on the eager and the plane-split + routes. Since CP-2a a file run takes its labels from the run and refuses any it is given.""" + from cellpatcher import file_runs from cellpatcher.orchestration import load_config, save_config_for_mode def no_read(*args, **kwargs): raise AssertionError("a file was read before the identity refusal") - images, masks = tmp_path / "images", tmp_path / "masks" - write_array(images / "img.tif", _image(0)) - write_array(masks / "img.tif", _labels(0)) - monkeypatch.setattr(orchestration, "read_image", no_read) - monkeypatch.setattr(orchestration, "list_supported_files", no_read) - monkeypatch.setattr(orchestration, "_mask_loaders", no_read) + monkeypatch.setattr(file_runs, "open_run", no_read) path = save_config_for_mode("cell", tmp_path / "config.json") data = json.loads(path.read_text()) data["common"][field] = 7 + data["common"]["preprocessing_run"] = str(tmp_path / "run") if split: data["common"]["z_policy"] = "all" path.write_text(json.dumps(data)) - with pytest.raises(ValueError, match=f"{field} must be"): - run_patching_pipeline(images, mask_folder_path=masks, config=load_config(path)) + refusal = "object_set must be" if field == "object_set" else f"common.{field}=7: each image's labels are its folders" + with pytest.raises(ValueError, match=refusal): + run_patching_pipeline(tmp_path / "images", mask_folder_path=tmp_path / "masks", config=load_config(path)) @pytest.mark.parametrize("planes", ["keep", "all"]) -def test_colliding_positions_are_refused_before_any_read(tmp_path, monkeypatch, planes): - """Re-audit of #41: Img.tif and img.tiff collide (D85); the refusal comes from the names, - before any image or mask is read, on the eager and the plane-split routes.""" - from cellpatcher import orchestration - - def no_read(*args, **kwargs): - raise AssertionError("a file was read before the identity refusal") - - images, masks = tmp_path / "images", tmp_path / "masks" - for name in ("Img.tif", "img.tiff"): - write_array(images / name, _image(0)) - write_array(masks / name, _labels(0)) - monkeypatch.setattr(orchestration, "read_image", no_read) - monkeypatch.setattr(orchestration, "_mask_loaders", no_read) - config = _cell_config() - config.common.z_policy = planes - with pytest.raises(ValueError, match=r"\[duplicate_position_id\].*Img.tif, img.tiff"): - run_patching_pipeline(images, mask_folder_path=masks, config=config) - - -def test_a_tree_refuses_a_later_leaf_before_reading_any_leaf(tmp_path, monkeypatch): - """Every leaf's identity is checked before the first leaf is read.""" - from cellpatcher import orchestration - - def no_read(*args, **kwargs): - raise AssertionError("a leaf was read before the identity refusal") - - write_array(tmp_path / "ctrl" / "r1" / "a.tif", _image(0)) - for name in ("Img.tif", "img.tiff"): - write_array(tmp_path / "ctrl" / "r2" / name, _image(1)) - monkeypatch.setattr(orchestration, "read_image", no_read) - monkeypatch.setattr(orchestration, "_run_folder_patching", no_read) - config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - with pytest.raises(ValueError, match=r"\[duplicate_position_id\].*ctrl/r2/Img.tif, ctrl/r2/img.tiff"): - run_patching_condition_repeat_tree(tmp_path, config=config) - - -def _refuse_mask_object_set(monkeypatch): - """Make every derived mask object set invalid, and every image or mask read fail loudly. - - A mask type always gives a valid name today (object_set_from_mask_type - replaces other characters), so the refusal is forced to show *when* it is checked. - """ - from cellpatcher import orchestration +def test_a_mask_type_object_set_is_refused_before_any_read(tmp_path, monkeypatch, planes): + """Issue #42: the object set from the run's mask type is checked before any image or mask + is read. A mask type always gives a valid name today (object_set_from_mask_type replaces + other characters), so the refusal is forced to show *when* it is checked.""" + from cellpatcher import file_runs def no_read(*args, **kwargs): raise AssertionError("a file was read before the object_set refusal") - monkeypatch.setattr(orchestration, "object_set_from_mask_type", lambda mask_type: "bad/name") - monkeypatch.setattr(orchestration, "read_image", no_read) - monkeypatch.setattr(orchestration, "_read_mask_file", no_read) - - -@pytest.mark.parametrize("planes", ["keep", "all"]) -def test_a_mask_named_object_set_is_refused_before_any_read(tmp_path, monkeypatch, planes): - """Issue #42: the object set from a matched mask's name is checked from the names alone.""" images, masks = tmp_path / "images", tmp_path / "masks" write_array(images / "img.tif", _image(0)) write_array(masks / "img_cells.tif", _labels(0)) - _refuse_mask_object_set(monkeypatch) + monkeypatch.setattr(file_runs, "object_set_from_mask_type", lambda mask_type: "bad/name") + monkeypatch.setattr(file_runs, "read_fov", no_read) + monkeypatch.setattr(file_runs, "read_fov_mask", no_read) config = _cell_config() config.common.z_policy = planes - with pytest.raises(ValueError, match="object set from mask 'img_cells.tif'"): - run_patching_pipeline(images, mask_folder_path=masks, config=config) - - -def test_a_tree_refuses_a_later_leaf_mask_object_set_before_reading_any_leaf(tmp_path, monkeypatch): - """Issue #42: a later leaf's mask-derived object set is refused before the first leaf is read.""" - images, masks = tmp_path / "images", tmp_path / "masks" - for leaf in ("r1", "r2"): - write_array(images / "ctrl" / leaf / "img.tif", _image(0)) - write_array(masks / "ctrl" / leaf / "img_cells.tif", _labels(0)) - from cellpatcher import orchestration - - real = orchestration.object_set_from_mask_type - _refuse_mask_object_set(monkeypatch) - calls = [] - - def bad_in_second_leaf(mask_type): - calls.append(mask_type) - return "bad/name" if len(calls) > 1 else real(mask_type) - - monkeypatch.setattr(orchestration, "object_set_from_mask_type", bad_in_second_leaf) - with pytest.raises(ValueError, match="object set from mask 'img_cells.tif'"): - run_patching_condition_repeat_tree(images, config=_cell_config(), mask_root_path=masks) + with pytest.raises(ValueError, match="object set from mask type 'cells'"): + _folder(images, mask_folder_path=masks, config=config) @pytest.mark.parametrize("value", ["NA", "nan", "None", "NULL", "", " ctrl", "ctrl "]) diff --git a/tests/test_plane_policies.py b/tests/test_plane_policies.py index 93eb756..f563d61 100644 --- a/tests/test_plane_policies.py +++ b/tests/test_plane_policies.py @@ -7,9 +7,18 @@ import pytest from cellpatcher.config import CellPatcherConfig, CommonConfig, GridConfig -from cellpatcher.orchestration import run_patching_pipeline +from cellpatcher.orchestration import ( + run_patching_condition_repeat_tree, + run_patching_pipeline, +) from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching -from tests.helpers import named, write_tiff +from tests.helpers import ( + file_run_config, + named, + preprocess, + preprocessed_folder, + write_tiff, +) def _stack(shape=(3, 16, 16)): @@ -217,11 +226,11 @@ def test_folder_run_exports_one_file_and_row_per_plane(tmp_path): # The header names the axes: a three-plane stack without them could be read as RGB. for name in ("a.tif", "b.tif"): write_tiff(tmp_path / "images" / name, _stack(), "ZYX") - config = _grid_config(z_policy="all") + config = _grid_config("YX", z_policy="all") # a file run reads the roles the run recorded (CP-2a) config.export.enabled = True config.common.output_dir = str(tmp_path / "out") - result = run_patching_pipeline(tmp_path / "images", config=config) + result = preprocessed_folder(tmp_path / "images", config=config, axes="ZYX") assert len(result.exported_files) == 2 * 3 * 4 with open(tmp_path / "out" / "patches.csv", newline="", encoding="utf-8") as f: @@ -236,18 +245,17 @@ def test_folder_run_exports_one_file_and_row_per_plane(tmp_path): assert saved["common"]["z_policy"] == "all" -def test_mask_axes_that_disagree_with_the_mask_header_are_refused(tmp_path): - # A ZYX mask (its ImageJ header names Z) declared TYX would have its depth slices read as - # time points: a guessed axis, refused naming both the declared and the header axes. +def test_a_mask_whose_header_disagrees_with_its_image_has_no_clean_run(tmp_path): + # A ZYX mask (its ImageJ header names Z) for a TYX image would have its depth slices read + # as time points. Since CP-2a masks are read in the roles the preprocessing run recorded, + # and the run refuses this pair, so no file run reads it. image_folder, mask_folder = tmp_path / "exp", tmp_path / "masks" write_tiff(image_folder / "a.tif", np.random.default_rng(0).integers(0, 100, (3, 32, 32), dtype=np.uint16), "TYX") mask = np.zeros((3, 32, 32), dtype=np.uint16) mask[:, 5:15, 5:15] = 1 write_tiff(mask_folder / "a_cells.tif", mask, "ZYX") - config = _cell_config(image_axes="TYX", mask_axes="TYX", t_policy="all") - - with pytest.raises(ValueError, match=r"a_cells.tif: \[axes_contradict_header\].*header ZYX, confirmed TYX"): - run_patching_pipeline(str(image_folder), config=config, mask_folder_path=str(mask_folder)) + with pytest.raises(AssertionError, match="does not have the depth slices and time points of its image"): + preprocess(image_folder, "TYX", masks=mask_folder) def test_mask_axes_that_agree_with_the_mask_header_are_accepted(tmp_path): @@ -256,32 +264,32 @@ def test_mask_axes_that_agree_with_the_mask_header_are_accepted(tmp_path): mask = np.zeros((3, 32, 32), dtype=np.uint16) mask[:, 5:15, 5:15] = 1 write_tiff(mask_folder / "a_cells.tif", mask, "TYX") - config = _cell_config(image_axes="TYX", mask_axes="TYX", t_policy="all") + config = _cell_config(image_axes="YX", mask_axes="YX", t_policy="all") - result = run_patching_pipeline(str(image_folder), config=config, mask_folder_path=str(mask_folder)) + result = preprocessed_folder(image_folder, config=config, mask_folder_path=mask_folder, axes="TYX") assert [p.t for p in result.patches] == [0, 1, 2] -def test_image_axes_that_disagree_with_the_image_header_are_refused(tmp_path): +def test_a_file_run_refuses_declared_image_axes(tmp_path): # A ZYX image (its ImageJ header names Z) declared TYX would have its depth slices read as - # time points; declared CYX, as channels. Refused naming both the declared and the header axes. + # time points; declared CYX, as channels. Since CP-2a a file run declares nothing: it reads + # the roles the preprocessing run recorded and refuses the setting (migration message). write_tiff(tmp_path / "exp" / "a.tif", _stack(), "ZYX") for declared, policy in (("TYX", {"t_policy": "all"}), ("CYX", {})): - with pytest.raises(ValueError, match=rf"common.image_axes='{declared}'.*a.tif: \[axes_contradict_header\].*header ZYX, confirmed {declared}"): - run_patching_pipeline(str(tmp_path / "exp"), config=_grid_config(image_axes=declared, **policy)) - + config = file_run_config(_grid_config(image_axes=declared, **policy), tmp_path / "exp", "ZYX") + with pytest.raises(ValueError, match=rf"common.image_axes='{declared}': each image's \(and mask's\) axes"): + run_patching_pipeline(str(tmp_path / "exp"), config=config) -def test_image_axes_are_checked_for_every_tree_leaf_before_any_leaf_runs(tmp_path): - from cellpatcher.orchestration import run_patching_condition_repeat_tree +def test_a_run_mixing_stacks_and_single_planes_patches_both(tmp_path): + # One layout per run (CP-2a): Z is there because one image has three planes; the 2-D image + # is its one plane, z=0. Former code: both read with common.image_axes="ZYX". root = tmp_path / "raw" - write_tiff(root / "a_cond" / "r1" / "a.tif", _stack(), "TYX") - write_tiff(root / "b_cond" / "r1" / "b.tif", _stack(), "ZYX") # a later leaf disagrees - config = _grid_config(image_axes="TYX", t_policy="all") - config.common.output_dir = str(tmp_path / "out") - with pytest.raises(ValueError, match=r"b.tif: \[axes_contradict_header\].*header ZYX, confirmed TYX"): - run_patching_condition_repeat_tree(str(root), config=config) - assert not (tmp_path / "out").exists() # no leaf ran + write_tiff(root / "a_cond" / "r1" / "a.tif", _stack(), "ZYX") + write_tiff(root / "b_cond" / "r1" / "b.tif", _stack((1, 16, 16)), "ZYX") + config = file_run_config(_grid_config("YX", z_policy="all"), root, "ZYX", missing_as_singleton=True) + leaves = run_patching_condition_repeat_tree(str(root), config=config) + assert [sorted({p.z0 for p in leaf.result.patches}) for leaf in leaves] == [[0, 1, 2], [0]] def test_a_plain_stack_without_axis_metadata_keeps_its_declared_axes(tmp_path): @@ -292,7 +300,8 @@ def test_a_plain_stack_without_axis_metadata_keeps_its_declared_axes(tmp_path): (tmp_path / "exp").mkdir() tifffile.imwrite(tmp_path / "exp" / "a.tif", _stack().astype(np.uint16), photometric="minisblack", metadata=None) assert tifffile.TiffFile(tmp_path / "exp" / "a.tif").series[0].axes == "IYX" - result = run_patching_pipeline(str(tmp_path / "exp"), config=_grid_config(image_axes="TYX", t_policy="all")) + config = file_run_config(_grid_config("YX", t_policy="all"), tmp_path / "exp", "TYX", stored="IYX") + result = run_patching_pipeline(str(tmp_path / "exp"), config=config) assert sorted({p.t for p in result.patches}) == [0, 1, 2] @@ -300,7 +309,7 @@ def test_lazy_delivery_refuses_image_and_mask_files_whose_header_disagrees(tmp_p from cellpatcher.delivery import patch_array write_tiff(tmp_path / "exp" / "a.tif", _stack(), "TYX") - result = run_patching_pipeline(str(tmp_path / "exp"), config=_grid_config(image_axes="TYX", t_policy="all")) + result = preprocessed_folder(tmp_path / "exp", config=_grid_config("YX", t_policy="all"), axes="TYX") patch = result.patches[0] write_tiff(tmp_path / "z.tif", _stack(), "ZYX") config = _grid_config(image_axes="TYX", t_policy="all") diff --git a/tests/test_plane_reading.py b/tests/test_plane_reading.py index cdef38d..ac37035 100644 --- a/tests/test_plane_reading.py +++ b/tests/test_plane_reading.py @@ -1,10 +1,12 @@ """Folder runs that split planes read one plane at a time (docs/normalization_contract.md §3). -With the ``preprocessing`` extra, only the chosen planes of a TIFF or -OME-Zarr image are decoded, with the preprocessing brick's ``read_plane``; -the patches equal those of ``run_patching`` on the whole arrays. +Since CP-2a a file run reads each image through the preprocessing run with +``RunInputs.read_plane``, one ``(z, t)`` plane (every channel) per call: with +plane policies only the chosen planes are read, and the patches equal those of +``run_patching`` on the whole arrays. """ +import copy import importlib.util import numpy as np @@ -13,7 +15,7 @@ from cellpatcher.config import CellPatcherConfig, GridConfig from cellpatcher.orchestration import run_patching_pipeline from cellpatcher.pipeline import run_patching -from tests.helpers import write_ngff +from tests.helpers import file_run_config, write_ngff needs_preprocessing = pytest.mark.skipif( importlib.util.find_spec("morphobricks_preprocessing") is None, reason="needs morphobricks-preprocessing (a core dependency)" @@ -36,6 +38,7 @@ def _write_tiff(path, data, axes): def _config(image_axes, mode="grid", **common): + """An in-memory config in *image_axes*, the layout a file run reads the same images in.""" config = CellPatcherConfig(mode=mode, grid=GridConfig(tile_size=8)) config.common.image_axes = image_axes config.common.extract_arrays = True @@ -52,19 +55,26 @@ def _same_result(folder_result, memory_result): np.testing.assert_array_equal(folder_array, memory_array) +def _folder(folder, config, axes): + """Preprocess *folder*, then run the folder run with the file-run form of *config* (CP-2a).""" + file_config = copy.deepcopy(config) + file_config.common.image_axes = "YX" + return run_patching_pipeline(folder, config=file_run_config(file_config, folder, axes)) + + @pytest.fixture def read_planes(monkeypatch): - """Record every plane the preprocessing reader decodes, as ``(z, c, t)``.""" - import morphobricks_preprocessing as mbp + """Record every plane a file run reads through ``RunInputs.read_plane``, as ``(z, t)``.""" + from morphobricks_preprocessing.run_inputs import RunInputs calls = [] - original = mbp.read_plane + original = RunInputs.read_plane - def spy(path, mapping, *, z=None, c=None, t=None, **kwargs): - calls.append((z, c, t)) - return original(path, mapping, z=z, c=c, t=t, **kwargs) + def spy(self, fov, z=0, t=0, **kwargs): + calls.append((z, t)) + return original(self, fov, z, t, **kwargs) - monkeypatch.setattr(mbp, "read_plane", spy) + monkeypatch.setattr(RunInputs, "read_plane", spy) return calls @@ -73,89 +83,76 @@ def spy(path, mapping, *, z=None, c=None, t=None, **kwargs): def test_tiff_planes_equal_the_in_memory_run(tmp_path, read_planes, policies): stacks = [_stack((3, 2, 16, 16), seed) for seed in range(2)] # Z, C, Y, X for name, stack in zip(("a.tif", "b.tif"), stacks): - _write_tiff(tmp_path / name, stack, "ZCYX") + _write_tiff(tmp_path / "exp" / name, stack, "ZCYX") config = _config("ZCYX", **policies) - folder = run_patching_pipeline(tmp_path, config=config) + folder = _folder(tmp_path / "exp", config, "ZCYX") memory = run_patching(images=stacks, config=config, image_metadata=[{"image_file": n} for n in ("a.tif", "b.tif")]) _same_result(folder, memory) - # Only the chosen planes were decoded, one channel at a time. + # Only the chosen planes were read, every channel at once. planes = 3 if policies["z_policy"] == "all" else 1 - assert len(read_planes) == 2 * planes * 2 + assert len(read_planes) == 2 * planes @needs_preprocessing def test_t_and_z_split_from_a_tiff_with_a_kept_channel(tmp_path, read_planes): stack = _stack((2, 3, 2, 8, 8)) # T, Z, C, Y, X - _write_tiff(tmp_path / "a.tif", stack, "TZCYX") + _write_tiff(tmp_path / "exp" / "a.tif", stack, "TZCYX") config = _config("TZCYX", z_policy="all", t_policy="first") - folder = run_patching_pipeline(tmp_path, config=config) + folder = _folder(tmp_path / "exp", config, "TZCYX") _same_result(folder, run_patching(images=stack, config=config, image_metadata=[{"image_file": "a.tif"}])) - assert sorted(read_planes) == [(z, c, 0) for z in range(3) for c in range(2)] + assert sorted(read_planes) == [(z, 0) for z in range(3)] @needs_preprocessing def test_random_planes_match_between_folder_and_memory_runs(tmp_path, read_planes): stacks = [_stack((2, 32, 32), seed) for seed in range(3)] for index, stack in enumerate(stacks): - _write_tiff(tmp_path / f"{index}.tif", stack, "ZYX") + _write_tiff(tmp_path / "exp" / f"{index}.tif", stack, "ZYX") config = _config("ZYX", mode="random", z_policy="all") config.random.patch_size, config.random.n_patches = 4, 2 - folder = run_patching_pipeline(tmp_path, config=config) + folder = _folder(tmp_path / "exp", config, "ZYX") memory = run_patching(images=stacks, config=config, image_metadata=[{"image_file": f"{i}.tif"} for i in range(3)]) _same_result(folder, memory) @needs_preprocessing -def test_keep_and_projections_read_whole_images(tmp_path, read_planes): +def test_keep_and_projections_read_every_plane(tmp_path, read_planes): stack = _stack((3, 16, 16)) - _write_tiff(tmp_path / "a.tif", stack, "ZYX") + _write_tiff(tmp_path / "exp" / "a.tif", stack, "ZYX") for policy in ("keep", "max"): config = _config("ZYX", z_policy=policy) _same_result( - run_patching_pipeline(tmp_path, config=config), + _folder(tmp_path / "exp", config, "ZYX"), run_patching(images=stack, config=config, image_metadata=[{"image_file": "a.tif"}]), ) - assert read_planes == [] + assert read_planes == [(z, 0) for z in range(3)] * 2 @needs_preprocessing def test_a_plane_over_the_memory_limit_is_an_error(tmp_path, monkeypatch): import morphobricks_preprocessing as mbp + from morphobricks_preprocessing.run_inputs import RunInputs def too_large(*args, **kwargs): raise mbp.PlaneMemoryError("Reading one plane of a.tif whole needs about 2 GiB of working memory.") - _write_tiff(tmp_path / "a.tif", _stack((3, 16, 16)), "ZYX") - monkeypatch.setattr(mbp, "read_plane", too_large) + _write_tiff(tmp_path / "exp" / "a.tif", _stack((3, 16, 16)), "ZYX") + config = file_run_config(_config("YX", z_policy="all"), tmp_path / "exp", "ZYX") + monkeypatch.setattr(RunInputs, "read_plane", too_large) with pytest.raises(ValueError, match="needs each plane whole in memory.*2 GiB"): - run_patching_pipeline(tmp_path, config=_config("ZYX", z_policy="all")) - - -def test_without_the_plane_reader_planes_come_from_the_whole_image(tmp_path, monkeypatch): - # preprocessing is a core dependency (its header axes are checked before any read); only - # its plane-by-plane reader is taken away here. - import cellpatcher.planes - - monkeypatch.setattr(cellpatcher.planes, "_plane_reader", lambda path, common: None) - stack = _stack((3, 16, 16)) - _write_tiff(tmp_path / "a.tif", stack, "ZYX") - config = _config("ZYX", z_policy="all") - _same_result( - run_patching_pipeline(tmp_path, config=config), - run_patching(images=stack, config=config, image_metadata=[{"image_file": "a.tif"}]), - ) + run_patching_pipeline(tmp_path / "exp", config=config) @needs_zarr -def test_ome_zarr_planes_are_read_in_the_configured_axis_order(tmp_path, read_planes): - data = _stack((2, 3, 16, 16)) # stored C, Z, Y, X - write_ngff(tmp_path / "a.ome.zarr", data, "czyx") +def test_ome_zarr_planes_are_read_in_the_canonical_axis_order(tmp_path, read_planes): + data = _stack((2, 3, 16, 16)) # stored C, Z, Y, X; read Z, C, Y, X by role (CP-2a) + write_ngff(tmp_path / "exp" / "a.ome.zarr", data, "czyx") config = _config("ZCYX", z_policy="all") - folder = run_patching_pipeline(tmp_path, config=config) + folder = _folder(tmp_path / "exp", config, "CZYX") as_zcyx = np.transpose(data, (1, 0, 2, 3)) _same_result(folder, run_patching(images=as_zcyx, config=config, image_metadata=[{"image_file": "a.ome.zarr"}])) - assert len(read_planes) == 3 * 2 + assert len(read_planes) == 3 diff --git a/tests/test_provenance_metadata.py b/tests/test_provenance_metadata.py index a5b0071..02a7928 100644 --- a/tests/test_provenance_metadata.py +++ b/tests/test_provenance_metadata.py @@ -7,10 +7,10 @@ from cellpatcher.config import CellPatcherConfig, GridConfig from cellpatcher.io import write_array from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching -from cellpatcher.orchestration import run_patching_pipeline from cellpatcher.types import PatchRegion from tests.helpers import make_synthetic_cell_image_and_labels from tests.helpers import named +from tests.helpers import preprocessed_folder def test_to_dict_includes_height_and_width(): @@ -169,7 +169,7 @@ def test_run_patching_pipeline_stamps_image_and_mask_file_before_export(tmp_path cfg.export.image_format = "npy" cfg.common.output_dir = str(tmp_path / "out") - result = run_patching_pipeline(str(img_dir), mask_folder_path=str(mask_dir), config=cfg) + result = preprocessed_folder(str(img_dir), mask_folder_path=str(mask_dir), config=cfg) assert len(result.patches) > 0 for patch in result.patches: assert patch.image_file == "sample001.tif" @@ -190,7 +190,7 @@ def test_run_patching_pipeline_grid_mode_stamps_image_file_only(tmp_path): write_array(img_dir / "tile_source.tif", np.ones((16, 16), dtype=np.float32)) cfg = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - result = run_patching_pipeline(str(img_dir), config=cfg) + result = preprocessed_folder(str(img_dir), config=cfg) assert len(result.patches) > 0 assert all(p.image_file == "tile_source.tif" for p in result.patches) assert all(p.mask_file is None for p in result.patches) diff --git a/tests/test_read_patch_table.py b/tests/test_read_patch_table.py index 9fbb1ee..77b7d02 100644 --- a/tests/test_read_patch_table.py +++ b/tests/test_read_patch_table.py @@ -25,10 +25,9 @@ ) from cellpatcher.export import save_metadata_csv from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching from cellpatcher.types import PatchRegion -from tests.helpers import make_synthetic_cell_image_and_labels, named +from tests.helpers import make_synthetic_cell_image_and_labels, named, preprocessed_tree def _exported_cell_run(tmp_path): @@ -67,7 +66,7 @@ def test_labels_stay_text_in_a_tree_run(tmp_path): write_array(root / relative, np.arange(256, dtype=np.uint16).reshape(16, 16)) config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8), export=ExportConfig(enabled=True)) config.common.output_dir = str(tmp_path / "out") - leaves = run_patching_condition_repeat_tree(root, config=config, combine_metadata=True) + leaves = preprocessed_tree(root, config=config, combine_metadata=True) patches, run_config = read_patch_table(tmp_path / "out" / "patches.csv") assert run_config is not None assert {(p.biological_repeat, p.technical_repeat, p.position_id) for p in patches} == { diff --git a/tests/test_rgb_normalization.py b/tests/test_rgb_normalization.py index 1c1bfbb..28fc8dd 100644 --- a/tests/test_rgb_normalization.py +++ b/tests/test_rgb_normalization.py @@ -16,6 +16,7 @@ from cellpatcher.orchestration import load_config, run_patching_condition_repeat_tree, run_patching_pipeline from cellpatcher.pipeline import RUN_CONFIG_FILENAME, run_patching from cellpatcher.statistics import PreprocessingCheckError, recorded_image_type +from tests.helpers import as_file_run pytestmark = [ pytest.mark.skipif( @@ -195,7 +196,7 @@ def test_image_scope_pools_r_g_and_b_of_the_whole_image(tmp_path): _write_tiff(folder / f"img{index}.tif", image) csv_path = _preprocess(folder, tmp_path / "preprocessing") - result = run_patching_pipeline(folder, config=_config(csv_path, scope="image")) + result = run_patching_pipeline(folder, config=as_file_run(_config(csv_path, scope="image"))) by_image = {} for patch, array in zip(result.patches, result.patch_arrays): @@ -219,7 +220,7 @@ def test_biological_repeat_scope_pools_rgb_across_conditions(tmp_path): _write_tiff(root / path, image) csv_path = _preprocess(root, tmp_path / "preprocessing") - results = run_patching_condition_repeat_tree(root, config=_config(csv_path, scope="biological_repeat")) + results = run_patching_condition_repeat_tree(root, config=as_file_run(_config(csv_path, scope="biological_repeat"))) pooled = np.concatenate([image.ravel() for image in images.values()]) low, high = np.percentile(pooled, (1.0, 99.8)) @@ -237,12 +238,12 @@ def test_the_manifest_image_type_must_match_the_configuration(tmp_path): fluorescence = _config(rgb_csv, scope="image") fluorescence.normalization.image_type = "fluorescence" with pytest.raises(PreprocessingCheckError, match="confirmed for 'rgb' images") as raised: - run_patching_pipeline(folder, config=fluorescence) + run_patching_pipeline(folder, config=as_file_run(fluorescence)) assert raised.value.issues[0].code == "image_type_differs" fluorescence_csv = _preprocess(folder, tmp_path / "fluorescence", image_type="fluorescence") with pytest.raises(PreprocessingCheckError, match="confirmed for 'fluorescence' images"): - run_patching_pipeline(folder, config=_config(fluorescence_csv, scope="image")) + run_patching_pipeline(folder, config=as_file_run(_config(fluorescence_csv, scope="image"))) def test_recorded_image_type_is_read_through_run_inputs(tmp_path): @@ -270,5 +271,5 @@ def test_without_an_image_type_in_the_manifest_the_configuration_decides(tmp_pat _write_tiff(folder / "img0.tif", _rgb_images(1)[0]) csv_path = _preprocess(folder, tmp_path / "preprocessing", image_type=None) - result = run_patching_pipeline(folder, config=_config(csv_path, scope="image")) + result = run_patching_pipeline(folder, config=as_file_run(_config(csv_path, scope="image"))) assert {patch.normalization_parameters.method for patch in result.patches} == {"rgb percentile 1-99.8"} diff --git a/tests/test_run_inputs.py b/tests/test_run_inputs.py index d06a28f..61129fa 100644 --- a/tests/test_run_inputs.py +++ b/tests/test_run_inputs.py @@ -154,12 +154,12 @@ def test_a_clean_run_normalizes_a_tree_run_through_run_inputs(tmp_path): run = make_clean_run(tmp_path / "clean", design=True) config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) - config.common.image_axes = "CYX" config.normalization = NormalizationConfig( enabled=True, plane_histograms=str(run.output / "plane_histograms.csv"), scope="biological_repeat", ) statistics = load_plane_statistics(config.normalization.plane_histograms) assert statistics.manifest.manifest_version == "2" + config.common.preprocessing_run = str(run.output) # a file run reads through the run (CP-2a) results = run_patching_condition_repeat_tree(run.experiment, config=config) patches = [patch for leaf in results for patch in leaf.result.patches] assert patches diff --git a/tests/test_source_relative_path.py b/tests/test_source_relative_path.py index 2601c66..a605945 100644 --- a/tests/test_source_relative_path.py +++ b/tests/test_source_relative_path.py @@ -11,10 +11,11 @@ from cellpatcher.config import CellPatcherConfig, GridConfig from cellpatcher.io import write_array -from cellpatcher.orchestration import run_patching_condition_repeat_tree, run_patching_pipeline from cellpatcher.pipeline import run_patching from cellpatcher.types import PatchRegion -from tests.helpers import named +from tests.helpers import named, preprocess, write_tiff +from tests.helpers import preprocessed_folder as run_folder +from tests.helpers import preprocessed_tree as run_tree # patches.csv header before source_relative_path was added; existing columns # must keep these names and this order. @@ -53,7 +54,7 @@ def test_field_is_appended_after_all_existing_fields(): def test_csv_appends_the_column_and_keeps_earlier_columns(tmp_path): _write_image(tmp_path / "images" / "a.tif") - run_patching_pipeline(tmp_path / "images", config=_grid_config(tmp_path / "out")) + run_folder(tmp_path / "images", config=_grid_config(tmp_path / "out")) header, rows = _read_csv(tmp_path / "out" / "patches.csv") assert header[: len(_EARLIER_HEADER) + 1] == [*_EARLIER_HEADER, "source_relative_path"] @@ -64,7 +65,7 @@ def test_flat_folder_run_uses_the_file_name(tmp_path): _write_image(tmp_path / "images" / "b.tif") _write_image(tmp_path / "images" / "a.tif") - result = run_patching_pipeline(tmp_path / "images", config=_grid_config()) + result = run_folder(tmp_path / "images", config=_grid_config()) assert {patch.source_relative_path for patch in result.patches} == {"a.tif", "b.tif"} for patch in result.patches: assert patch.source_relative_path == patch.image_file @@ -77,7 +78,7 @@ def test_tree_run_uses_the_real_location_not_the_output_folder(tmp_path): _write_image(root / "treated" / "2026_01_02" / "B02" / "field.tif") _write_image(root / "treated" / "2026_01_02" / "B03" / "field.tif") - results = run_patching_condition_repeat_tree(root, config=_grid_config()) + results = run_tree(root, config=_grid_config()) by_folder = { leaf.relative_folder: {patch.source_relative_path for patch in leaf.result.patches} @@ -97,7 +98,7 @@ def test_flat_root_tree_run_uses_the_file_name(tmp_path): root = tmp_path / "experiment" _write_image(root / "a.tif") - results = run_patching_condition_repeat_tree(root, config=_grid_config()) + results = run_tree(root, config=_grid_config()) assert {patch.source_relative_path for patch in results[0].result.patches} == {"a.tif"} @@ -106,7 +107,7 @@ def test_combined_tree_csv_records_the_column(tmp_path): _write_image(root / "control" / "img.tif") _write_image(root / "treated" / "2026_01_01" / "img.tif") - run_patching_condition_repeat_tree(root, config=_grid_config(tmp_path / "out"), combine_metadata=True) + run_tree(root, config=_grid_config(tmp_path / "out"), combine_metadata=True) _, rows = _read_csv(tmp_path / "out" / "patches.csv") assert {row["source_relative_path"] for row in rows} == {"control/img.tif", "treated/2026_01_01/img.tif"} @@ -129,12 +130,11 @@ def test_run_patching_without_metadata_leaves_it_empty(): def test_split_channel_patches_keep_it(tmp_path): root = tmp_path / "experiment" - write_array(root / "control" / "rgb.tif", np.ones((2, 16, 16), dtype=np.float32)) + write_tiff(root / "control" / "rgb.tif", np.ones((2, 16, 16), dtype=np.float32), "CYX") config = _grid_config() - config.common.image_axes = "CYX" config.extraction.split_channels = True - results = run_patching_condition_repeat_tree(root, config=config) + results = run_tree(root, config=config, axes="CYX") patches = results[0].result.patches assert {patch.c for patch in patches} == {0, 1} assert {patch.source_relative_path for patch in patches} == {"control/rgb.tif"} @@ -161,7 +161,7 @@ def test_tree_of_links_records_the_path_under_the_root(tmp_path): # is the path the preprocessing brick records for the same tree. root = _linked_tree(tmp_path) - results = run_patching_condition_repeat_tree(root, config=_grid_config(tmp_path / "out"), combine_metadata=True) + results = run_tree(root, config=_grid_config(tmp_path / "out"), combine_metadata=True) by_leaf = { leaf.relative_folder: sorted({patch.source_relative_path for patch in leaf.result.patches}) @@ -180,21 +180,19 @@ def test_tree_of_links_records_the_path_under_the_root(tmp_path): # The pixels are the linked files' own. config = _grid_config() config.common.extract_arrays = True - for leaf in run_patching_condition_repeat_tree(root, config=config): + for leaf in run_tree(root, config=config): for patch, array in zip(leaf.result.patches, leaf.result.patch_arrays): assert np.all(array == {"a.tif": 1.0, "b.tif": 2.0, "c.tif": 3.0}[patch.image_file]) def test_folder_of_links_uses_the_file_names(tmp_path): root = _linked_tree(tmp_path) - result = run_patching_pipeline(root / "control" / "2026_01_01", config=_grid_config()) + result = run_folder(root / "control" / "2026_01_01", config=_grid_config()) assert {patch.source_relative_path for patch in result.patches} == {"a.tif", "b.tif"} def test_a_broken_image_link_is_refused_not_skipped(tmp_path): root = _linked_tree(tmp_path) (root / "control" / "2026_01_01" / "lost.tif").symlink_to(tmp_path / "store" / "missing.tif") - with pytest.raises(ValueError, match="lost.tif"): - run_patching_condition_repeat_tree(root, config=_grid_config()) - with pytest.raises(ValueError, match="lost.tif"): - run_patching_pipeline(root / "control" / "2026_01_01", config=_grid_config()) + with pytest.raises(AssertionError, match="lost.tif"): # the preprocessing run refuses it (CP-2a) + preprocess(root) diff --git a/tests/test_split_channels.py b/tests/test_split_channels.py index e7e4bff..910fac8 100644 --- a/tests/test_split_channels.py +++ b/tests/test_split_channels.py @@ -5,9 +5,9 @@ from cellpatcher.config import CellPatcherConfig, CommonConfig, ExtractionConfig, GridConfig from cellpatcher.io import write_array -from cellpatcher.orchestration import load_config, run_patching_condition_repeat_tree, run_patching_pipeline +from cellpatcher.orchestration import load_config from cellpatcher.pipeline import run_patching -from tests.helpers import named, write_tiff +from tests.helpers import named, preprocessed_folder, preprocessed_tree, write_tiff def _channel_image(n_channels=3, size=16, channels_last=False): @@ -183,10 +183,10 @@ def test_a_folder_run_with_a_missing_channel_writes_nothing(tmp_path): folder.mkdir() for name in ("a.tif", "b.tif"): write_tiff(folder / name, _channel_image(n_channels=2), "CYX") - config = _grid_config(split_channels=[2]) + config = _grid_config("YX", split_channels=[2]) # file runs read the run's roles (CP-2a) config.export.enabled, config.common.output_dir = True, str(tmp_path / "out") with pytest.raises(ValueError, match="has 2 channels"): - run_patching_pipeline(folder, config=config) + preprocessed_folder(folder, config=config, axes="CYX") assert not (tmp_path / "out" / "patches.csv").exists() @@ -245,11 +245,11 @@ def test_split_channels_in_combined_tree_run(tmp_path): for condition in ("control", "treated"): write_tiff(image_root / condition / "img0.tif", _channel_image(n_channels=2), "CYX") - config = _grid_config() + config = _grid_config("YX") config.export.enabled = True config.export.image_format = "npy" config.common.output_dir = str(tmp_path / "out") - run_patching_condition_repeat_tree(image_root, config=config, combine_metadata=True) + preprocessed_tree(image_root, config=config, axes="CYX", combine_metadata=True) with (tmp_path / "out" / "patches.csv").open("r", encoding="utf-8") as f: rows = list(csv.DictReader(f)) @@ -265,11 +265,11 @@ def test_same_file_name_in_two_wells_gives_two_positions(tmp_path): for well in ("B02", "B03"): write_tiff(image_root / "control" / "2026_01_01" / well / "img0.tif", _channel_image(n_channels=2), "CYX") - config = _grid_config() + config = _grid_config("YX") config.export.enabled = True config.export.image_format = "npy" config.common.output_dir = str(tmp_path / "out") - run_patching_condition_repeat_tree(image_root, config=config, combine_metadata=True) + preprocessed_tree(image_root, config=config, axes="CYX", combine_metadata=True) with (tmp_path / "out" / "patches.csv").open("r", encoding="utf-8") as f: rows = list(csv.DictReader(f)) diff --git a/tests/test_statistics.py b/tests/test_statistics.py index 870852a..b772d6b 100644 --- a/tests/test_statistics.py +++ b/tests/test_statistics.py @@ -430,6 +430,7 @@ def test_cellpatcher_patches_link_to_the_preprocessing_rows(tmp_path): statistics = load_plane_statistics(output / mbn.OUTPUT_FILENAME) config = CellPatcherConfig(mode="grid", grid=GridConfig(tile_size=8)) + config.common.preprocessing_run = str(output) # a file run reads through the run (CP-2a) patches = [patch for leaf in run_patching_condition_repeat_tree(root, config=config) for patch in leaf.result.patches] paths = {patch.source_relative_path for patch in patches} assert paths == {"0012/img.tif", "treated/001/wellA/field.tif", "treated/001/wellB/field.tif"}