diff --git a/AGENTS.md b/AGENTS.md index 46339ed..88db218 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,16 +186,28 @@ CellPatcher-specific conventions: with `RunInputs.read_plane`/`read_mask` (`cellpatcher.run_reading`) in one canonical layout per run (T, Z, C, then Y, X). No CellPatcher-side axis, pairing or calibration logic is used for them; `common.image_axes`/`mask_axes` - other than `"YX"`, `mask_source="ngff_labels"`, `mask_label_name` and labels - other than `"single"` are refused with a migration message (CP-2b removes - them for file runs). The run is opened once per call and kept for that call - only (`cellpatcher.run_session`). + other than `"YX"`, labels other than `"single"` and + `normalization.experiment_root`/`masks_root` are refused with a migration + message. The run is opened once per call and kept for that call only + (`cellpatcher.run_session`). Since CP-2b no file is read or paired by + CellPatcher itself: `validate_image_and_mask_folders(..., preprocessing_run=)` + returns the run's pairing, lazy delivery of a file run's patches is + `iter_run_patch_arrays` (`cellpatcher.run_delivery`, through `RunInputs`), + and `patch_array`/`iter_patch_arrays` take arrays only. `run_config.json` is + version 2 (`cellpatcher.run_config_file`): `run_config_version` and a + `preprocessing` section (the run, its manifest's SHA-256, a file run's array + layout); a file run records no `image_axes`/`mask_axes`/labels; readers + refuse other versions. Grid/whole-FOV folder runs accept `mask_folder_path` + (where the run's masks folder is now), and `normalization.experiment_root`/ + `masks_root` say where a moved experiment is for `run_patching` at a CSV + scope (owner decisions of 2026-11-02). * `image_axes`/`mask_axes` are explicit and axis-aware (see `axes.py`) for - in-memory arrays and lazy delivery from a path. - Image/mask pairing is the preprocessing brick's `find_masks`, called by - `io.pair_masks` (issue #45, owner decision 2026-10-30): TIFF masks named - `_.tif` or `/.tif`; `matching.mask_type` picks - the type. CellPatcher has no pairing rule of its own; the former matching + in-memory arrays only (canonical, Y before X; a mask with X first is refused). + Image/mask pairing is the preprocessing run's (its `find_masks`; issue #45, + owner decision 2026-10-30): TIFF masks named + `_.tif` or `/.tif`, or an OME-Zarr image's label + image; `matching.mask_type` picks the type. CellPatcher has no pairing rule + of its own (`io.pair_masks` was removed in CP-2b); the former matching options (`strategy`, `mask_suffixes`, `case_sensitive`, `ignore_unrequested_masks`) and matchers (`find_candidate_matches`, ...) were removed in v0.1.1 (D86, owner decision 2026-10-31): the options are @@ -249,40 +261,23 @@ preprocessing run (`common.preprocessing_run`, opened with `RunInputs`; see §4 `docs/configuration_reference.md`, "File runs read through the preprocessing run"): the images, their labels and `position_id`, their roles and their masks are the run's. A folder mask and a label image of one type for one image are -refused by the preprocessing run (`duplicate_mask`, rc1). The rest of this -paragraph describes discovery and pairing as `validate_image_and_mask_folders` -and lazy delivery from paths still do them (until CP-2b), and as file runs did -before v0.2.0. CellPatcher reads image files, and (for `cell` mode) matching label/mask -files, directly from user-provided folders. Images are found with the -preprocessing brick's `discover_experiment` (`io.discover_images`; owner -decision of 2026-10-30, issue #45): TIFF files and OME-Zarr images. PNG, JPEG, -BMP, GIF, NumPy and microscope-format images (`.czi`, `.nd2`, `.lif`, ...; -`io.UNSUPPORTED_IMAGE_EXTENSIONS`) are refused by name, never skipped (exported -`.npy` patches are still written and read back). OME-Zarr images are folders ending -in `.zarr`, read from level 0 of their only image, with their axes from the -OME-NGFF metadata put into `common.image_axes` by role (`io.read_image`); -discovery never enters them, and `img.ome.zarr` pairs with masks named like -`img_cells.tif` or `cells/img.tif`. With `common.mask_source="ngff_labels"`, the masks of OME-Zarr -images are their own OME-NGFF label images (`common.mask_label_name`, -`io.read_ngff_mask`, contract §4) instead of a mask folder. Matching is -paired by the preprocessing brick's `find_masks` (`io.pair_masks`, used by -`validate_image_and_mask_folders` and both run entry points; issue #45): a -mask is a TIFF file `//.tif` or -`/_.tif` in a mask folder outside the -image root, the image name being its `position_id` (D85); -`config.matching.mask_type` picks the type (None: the only one). Its header -checks (preprocessing v0.3.2) are refused for the type used and ignored for the -other types; CellPatcher also checks masks with its own axes on reading, and -refuses `common.mask_axes` that disagree with the axes a mask's header names -(`io.refuse_mask_header_axes`, read with preprocessing `inspect_image`; a ZYX mask -declared TYX would read Z slices as time points). `common.image_axes` is checked -the same way against each TIFF image's header (`io.refuse_image_header_axes`) in -folder and tree runs (every leaf before any runs) and in lazy delivery from files, -which also checks mask files; OME-Zarr images are matched by role instead. Mask -files with a channel (C) axis or without axis metadata are refused, never -guessed, and `common.mask_axes` with a `C` is refused for mask-folder masks -(owner decision of 2026-10-30): users save one mask type per file -(`_cells.tif`, `_nuclei.tif`) and name a stack's Z/T axes. +refused by the preprocessing run (`duplicate_mask`, rc1). Since CP-2b CellPatcher +discovers, pairs and reads no file itself for any run: `validate_image_and_mask_folders` +returns the run's recorded pairing and lazy delivery reads through the run +(`iter_run_patch_arrays`). What the preprocessing run applied: images are found with +its `discover_experiment` (TIFF files and OME-Zarr images; `io.discover_images` still +wraps it for `discover_condition_repeat_folders`, refusing PNG, JPEG, BMP, GIF, NumPy +and microscope-format images by name, `io.UNSUPPORTED_IMAGE_EXTENSIONS`), and masks are +paired by its `find_masks`: a TIFF file `//.tif` +or `/_.tif` in a masks folder outside the +experiment, the image name being its `position_id` (D85), or an OME-Zarr image's own +label image (its type is its name); `config.matching.mask_type` picks the type (None: +the only one). The run checks every image's and mask's axes against its header (design +v0.4 §3; nothing is read transposed) and refuses masks with a channel axis or unnamed +extra axes (one mask type per file, named Z/T axes). Removed in CP-2b: +`common.mask_source`/`mask_label_name` (a config file holding their defaults still +loads), `io.read_image`/`conform_axes`/`read_ngff_mask`, the header-axis checks +(`io.refuse_image_header_axes`, `refuse_mask_header_axes`) and `planes.read_file_plane`. Two ways to point CellPatcher at input data: @@ -326,7 +321,12 @@ Two ways to point CellPatcher at input data: (`cellpatcher.RUN_CONFIG_FILENAME`) with the full run config — every CSV row's `config_file` column names it — so a row plus that file fully determine how to reproduce the patch (coordinates, strategy parameters - including `random.seed`, extraction/resize settings, axes). + including `random.seed`, extraction/resize settings, axes). Version 2 since + v0.2.0 (CP-2b): `run_config_version: 2` and, when a preprocessing run was read, + `preprocessing` = {`run`, `manifest_sha256`, and for a folder or tree run + `layout` (`image`, `mask`)}; a folder or tree run records no + `common.image_axes`/`mask_axes`/labels (MorphoRelate's `common.image_axes` + check then reports "not checked"). Readers refuse versions other than 1 and 2. - For `run_patching_condition_repeat_tree`, one `patches.csv` + `run_config.json` pair per `/[/]/` output folder by @@ -349,9 +349,8 @@ Two ways to point CellPatcher at input data: later, `position_id` (the one its patches carry, also for an image without patches; always filled since decision D45) and `n_regions`/`n_kept`/`n_normalized` (counts after the patch mode, the - filters and normalization) and `image_read_transposed` (the image's - header Y and X declared swapped, read transposed; v0.1.2 only, as - a swap is refused since v0.2.0). There is no + filters and normalization) (v0.1.2's last column `image_read_transposed` + was removed in v0.2.0, CP-2b). There is no `failed` status: a failing image stops the run with an error before any table is written. Every `patches.csv` row joins exactly one `images.csv` row on `condition`/`biological_repeat`/`technical_repeat`/`image_id`/ @@ -404,7 +403,7 @@ bricks (core dependencies, pinned tags in `setup.py`), CellPatcher: `run_manifest.json`, all required; `cellpatcher.statistics`) and runs the checks of contract §5; - splits Z/T stacks into planes (`common.z_policy`/`t_policy`) and reads - OME-Zarr images and OME-NGFF label masks (`common.mask_source`) (see + OME-Zarr images and OME-NGFF label masks through the preprocessing run (see "Inputs" above); - normalizes patches (`config.normalization`, `cellpatcher.normalization`) at the biological-repeat scope (across conditions, from the preprocessing diff --git a/docs/api_examples.md b/docs/api_examples.md index 982e6ab..742abd5 100644 --- a/docs/api_examples.md +++ b/docs/api_examples.md @@ -128,14 +128,15 @@ print(len(result.patches)) # 1 ## Example 5 — Folder matching and loading -Use this pattern when images and masks live in separate folders and need reliable pairing. The pairing is the preprocessing brick's (`morphobricks_preprocessing.find_masks`): masks are TIFF files named `_.tif` or `/.tif` (see [`config.matching`](configuration_reference.md)). +Use this pattern to list the image/mask pairs of a preprocessing run. The pairing is the preprocessing run's (its `find_masks`; since v0.2.0 CellPatcher pairs nothing itself): masks are TIFF files named `_.tif` or `/.tif` (see [`config.matching`](configuration_reference.md)). ```python # ./masks holds img_01_cells.tif, img_01_nuclei.tif, ...: choose the type. -# pairs is a dict[Path, Path]: {image_path: mask_path} for every image +# pairs is a dict[Path, Path]: {image_path: mask_path} for every image of the run pairs = validate_image_and_mask_folders( image_folder_path="./images", mask_folder_path="./masks", + preprocessing_run="./images_MorphoBricks_Preprocessing", mask_type="cells", ) diff --git a/docs/configuration_reference.md b/docs/configuration_reference.md index 9d088f7..9c3e6b7 100644 --- a/docs/configuration_reference.md +++ b/docs/configuration_reference.md @@ -14,20 +14,21 @@ This document is the complete runtime configuration reference for `CellPatcherCo --- -## File runs read through the preprocessing run (v0.2.0, CP-2a) +## File runs read through the preprocessing run (v0.2.0, CP-2a and CP-2b) `run_patching_pipeline`, `run_patching_condition_repeat_tree` and `run_patching_from_folders` read files only through a clean preprocessing run (v0.4), `common.preprocessing_run`, with the preprocessing brick's `RunInputs` (design v0.4 §6). Without it they stop with `no_preprocessing_run` and the preprocessing command (`morphobricks-preprocessing check --masks `, then `morphobricks-preprocessing run --axes --masks `). - **Where the experiment is now.** The run is opened with `RunInputs.open(run, experiment_root=, masks_root=)`; a masks folder left out is where the run recorded it. An experiment moved or copied since its run is checked and read where CellPatcher is given it: every image and mask must have the size and modification time the run recorded (copy with the times kept, e.g. `cp -p` or `shutil.copytree`), or the run is refused (`preprocessing_run_refused`, naming the files). - **Images.** The run's images (`RunInputs.fovs()`), sorted by path, read with `RunInputs.read_plane` in the roles the run recorded. Their arrays are in one canonical layout per run: T and Z when an image of the run has more than one plane along them, C when an image has a channel role, in that order, then Y and X (`ZCYX`, `TYX`, `CYX`, `YX`, ...). Nothing is read transposed; an image without one of these axes is its one plane along it. With plane policies only the chosen planes are read. An image added to the experiment after the run is not part of it, so not patched (a brick calls no `discover_experiment` for files); run preprocessing again to include it. **Known limitation:** nothing says so yet. Preprocessing will give bricks a check for images added after the run (a later preprocessing PR, not in v0.4.0rc1); CellPatcher will call it then, rather than discover images itself. - **Labels and `position_id`.** Each image's `condition`, `biological_repeat`, `technical_repeat` and `position_id` are the ones the run recorded, never parsed. `run_patching_pipeline` patches a run of one condition and biological repeat (any technical repeats); use the tree run for more. +- **Masks folder in `grid`/`whole_fov` mode.** These modes read no masks; `mask_folder_path` (or `mask_root_path`) given with them only says where the run's masks folder is now, and its masks are checked there (owner decision of 2026-11-02; before CP-2b the folder run refused it). - **Masks.** Those the run paired (masks-folder TIFFs and the label images of OME-Zarr images; a label image's type is its name), read with `RunInputs.read_mask` (the brick checks their labels). `matching.mask_type` chooses the type; `None` takes the run's only type. `cell` mode always reads masks; `random` mode when a masks folder or `matching.mask_type` is given. Every image needs a mask of the type. The mask arrays follow one layout per run too (the T and Z any mask of the run has, then Y and X); a mask without one of them (a 2-D mask of a one-plane image, or a stated shared 2-D mask) has length 1 along it. A mask the run recorded without axes (rc1 records one whose header it could not use, e.g. X before Y) is refused. A folder mask and a label image of the same type for one image are refused by the preprocessing run (`duplicate_mask`), so no clean run has them. - **Normalization.** The CSV scopes read the run's own table (`normalization.plane_histograms` may be left unset, or must name it: `not_the_run_output`), design and image type. -- **`run_config.json`** records `common.preprocessing_run` and, as `common.image_axes`/`mask_axes`, the layout the patches were read in. -- **Migration.** For file runs, `common.image_axes`/`mask_axes` other than `"YX"`, `common.mask_source="ngff_labels"`, `common.mask_label_name`, and `common.condition`/`biological_repeat`/`technical_repeat` other than `"single"` are refused with a message saying what replaces them (CP-2b removes them for file runs). The axes and labels still describe in-memory arrays given to `run_patching`; `mask_source` and `mask_label_name` are not used there. +- **`run_config.json`** (version 2, CP-2b; see "Deterministic reconstruction" below) records `run_config_version: 2` and a `preprocessing` section: the run (`run`), the SHA-256 of its `run_manifest.json` when the call opened it (`manifest_sha256`), and the `layout` (`image`, `mask`) the patches were read in. It does not record `common.image_axes`, `mask_axes` or the labels (the run's, not settings), so it loads back with `load_config` as a config a file run accepts. +- **Migration.** For file runs, `common.image_axes`/`mask_axes` other than `"YX"`, `common.condition`/`biological_repeat`/`technical_repeat` other than `"single"` and `normalization.experiment_root`/`masks_root` are refused with a message saying what replaces them. The axes and labels still describe in-memory arrays given to `run_patching`. `common.mask_source` and `mask_label_name` were removed in CP-2b: `CommonConfig(mask_source=...)` is a `TypeError`; a config file holding their defaults still loads, one with another value is refused. - **Within one call** (`run_patching`, a folder or tree run) the run is opened once; nothing is kept beyond the call, so the next call checks every image and mask again (`cellpatcher.run_session`). -Still read by CellPatcher until CP-2b: lazy delivery from a file path (`patch_array`/`iter_patch_arrays` with paths, using `common.image_axes`), and `validate_image_and_mask_folders` (`pair_masks`). +**No file is read or paired by CellPatcher itself** (CP-2b). `validate_image_and_mask_folders(image_folder, mask_folder, *, preprocessing_run=, mask_type=)` returns the pairing the run recorded, opened where the folders are now. Lazy delivery of a file run's patches is `iter_run_patch_arrays(patches, config, *, experiment_root=, masks_root=)`, which reads each patch's plane with `RunInputs.read_plane` (its mask with `read_mask`) from the run's image at the patch's `source_relative_path`, and refuses a patch whose image or FOV key is not the run's; `patch_array`, `iter_patch_arrays` and `PatchTorchDataset` take arrays only (a path is a `TypeError`). --- @@ -37,8 +38,8 @@ Shared settings that apply to every run, regardless of mode. | Field | Type | Default | Notes | |---|---|---|---| -| `image_axes` | `str` | `"YX"` | Axes string for image arrays given to `run_patching` (e.g. `"YX"`, `"CYX"`, `"ZYX"`). Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read the roles the preprocessing run recorded, refuse any other value than `"YX"`, and record the layout they read in `run_config.json`; the rest of this row is about lazy delivery from a file path. A TIFF image file's header is checked against it before any pixel is read, with the preprocessing brick's axis check (`review_axes`: `check_mapping` and `header_axis_issue`, design v0.4 §3): it may name what the header leaves open (a placeholder axis; the positional Y and X of a plain stack of pages, so a plain `QYX` stack may be declared `YXC`), never contradict an axis the metadata names. A Y/X swap (header `ZYX` declared `ZXY`, `axes_yx_swapped`) and a plain stack's Y and X declared X before Y (`axes_x_before_y`) are refused too: nothing is ever read transposed (since v0.2.0; v0.1.2 read a swapped image transposed). With a CSV normalization scope (`image`, `biological_repeat`), the axes must also give the roles the preprocessing run recorded (`check_sources`, `axis_mapping_mismatch`). Anything else (another order, another number of axes, an unreadable header) is refused naming the file. OME-Zarr images are matched by role instead (see "OME-Zarr images"). | -| `mask_axes` | `str` | `"YX"` | Axes string for mask arrays given to `run_patching`. Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read masks in the roles the preprocessing run recorded and refuse any other value than `"YX"`. Masks read from a mask folder (`mask_source="tiff_tree"`) cannot have a `C` axis: a `C` here is refused for folder and tree runs, since it would read a mask's other axes as label channels (owner decision of 2026-10-30). Save each label channel as its own mask type instead (`_cells.tif`, `_nuclei.tif`) and choose one with `matching.mask_type`. A TIFF mask file's header is checked as for `image_axes` (a Y/X swap is refused). CellPatcher takes a label plane's two dimensions as Y and X in their stored order, so any `mask_axes` with X before Y (`"XY"`, `"ZXY"`) is refused, for in-memory masks and OME-Zarr labels too (before v0.1.2 it was ignored for in-memory masks). The refusal says how to fix the masks: save each one with Y before X (its last two axes swapped, e.g. `numpy.swapaxes(mask, -1, -2)`) and declare those axes. Nothing is ever read transposed. | +| `image_axes` | `str` | `"YX"` | Axes string for image arrays given to `run_patching` (e.g. `"YX"`, `"CYX"`, `"ZYX"`). Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read the roles the preprocessing run recorded, refuse any other value than `"YX"`, and record the layout they read in `run_config.json`'s `preprocessing.layout` (CP-2b). Since CP-2b no file is read with these axes (lazy delivery takes arrays; see "File runs read through the preprocessing run"), so CellPatcher checks no file header against them: the preprocessing run checks each file's axes (design v0.4 §3; nothing is ever read transposed). | +| `mask_axes` | `str` | `"YX"` | Axes string for mask arrays given to `run_patching`. Must include `Y` and `X`. Since v0.2.0 (CP-2a) folder and tree runs read masks in the roles the preprocessing run recorded and refuse any other value than `"YX"`. CellPatcher takes a label plane's two dimensions as Y and X in their stored order, so any `mask_axes` with X before Y (`"XY"`, `"ZXY"`) is refused (before v0.1.2 it was ignored for in-memory masks). The refusal says how to fix the masks: give each one with Y before X (its last two axes swapped, e.g. `numpy.swapaxes(mask, -1, -2)`) and declare those axes. Nothing is ever read transposed. | | `preprocessing_run` | `str \| None` | `None` | The output folder of the clean preprocessing run (v0.4) that folder and tree runs read their images, masks and labels through (see "File runs read through the preprocessing run"). Required for them; not used by in-memory `run_patching` (its CSV normalization reads `normalization.plane_histograms`). Since v0.2.0. | | `output_dir` | `str \| None` | `None` | Output folder path. Required when `export.enabled=True`; keep `None` for in-memory-only runs. | | `extract_arrays` | `bool` | `False` | When `True`, extracted patch arrays are returned in `PatchingResult.patch_arrays`. | @@ -50,8 +51,6 @@ Shared settings that apply to every run, regardless of mode. | `z_index` | `int \| None` | `None` | Z plane for `z_policy="select"`. | | `t_policy` | `str` | `"keep"` | As `z_policy` for a T axis: `"keep"`, `"all"` (sets `t`, a `t` level of `patch_id`), `"first"` or `"select"`; no projections. With both split, planes come in T-then-Z order (`.../t0001/z002/...`). | | `t_index` | `int \| None` | `None` | Time point for `t_policy="select"`. | -| `mask_source` | `str` | `"tiff_tree"` | Before v0.2.0, where folder and tree runs read masks; since CP-2a they read the masks the preprocessing run paired, and `"ngff_labels"` is refused with a migration message (a label image's type is its name: choose it with `matching.mask_type`). Former meaning: `"tiff_tree"`: a mask folder (or tree) mirroring the images, paired with the preprocessing brick's rule (`config.matching`). `"ngff_labels"`: the label image of each OME-Zarr image's own `labels` group (no mask folder; every image must be OME-Zarr). See "OME-Zarr images" below. | -| `mask_label_name` | `str \| None` | `None` | Refused for folder and tree runs since CP-2a (use `matching.mask_type`). Former meaning: with `mask_source="ngff_labels"`, the label image to use (e.g. `"cells"`); may stay `None` when every image has exactly one. | | `object_set` | `str \| None` | `None` | Cell mode: the name of the object set (mask) written to the `object_set` column and to `patch_id`. `None`: named after the mask (see "Patch identity" below), `"mask"` when the mask gives no name. Letters, digits, `.`, `_`, `-` only. | **Masks with split planes:** a mask's Z or T axis with the image's length follows the image's planes (one label plane per image plane); a mask without that axis, or with length 1, is shared by every plane. Any other length is an error, and so is a Z projection with masks that follow the Z planes. Image ids stay one per source image, and every plane and channel patch of an image shares one `position_id`. @@ -171,7 +170,7 @@ Controls whether and how patch arrays are written to disk. ## Finding images -Images are found with the **MorphoBricks preprocessing brick's `discover_experiment`** (through `cellpatcher.io.discover_images`; issue #45, owner decision of 2026-10-30), in folder runs, tree runs and `validate_image_and_mask_folders`. CellPatcher has no folder walk of its own, so it patches the images the preprocessing brick measures and normalizes, under the same labels. +Images are found with the **MorphoBricks preprocessing brick's `discover_experiment`** (through `cellpatcher.io.discover_images`; issue #45, owner decision of 2026-10-30) in `discover_condition_repeat_folders` and `list_supported_files`. Since v0.2.0 folder and tree runs patch the images their preprocessing run recorded and discover nothing (design v0.4 §6); the rules below are the brick's, which that run applied. - **Images are TIFF files** (`.tif`, `.tiff`, any case, optional `.ome`) **and OME-Zarr folders** (`.zarr`, usually `.ome.zarr`). Nothing else is an image. - **PNG, JPEG, BMP, GIF and NumPy (`.npy`) images are no longer read**, and microscope formats (`.czi`, `.nd2`, `.lif`, `.lsm`, `.ims`, `.vsi`, ...; `cellpatcher.io.UNSUPPORTED_IMAGE_EXTENSIONS`) never were. Such a file where images are found is refused with an error naming it, never skipped: convert it to TIFF or OME-Zarr, or move it out of the image folder. Other files (`.txt`, `.csv`, ...) are ignored, as before. Exported patches may still be written as `.npy` (`export.image_format`) and are read back by `read_array` and the Torch dataset. @@ -185,7 +184,7 @@ Images are found with the **MorphoBricks preprocessing brick's `discover_experim ## `config.matching` (`MatchConfig`, which masks to pair) -Images and masks are paired with the **MorphoBricks preprocessing brick's rule**: CellPatcher calls `morphobricks_preprocessing.find_masks` (through `cellpatcher.io.pair_masks`) and has no pairing rule of its own, so CellPatcher, the preprocessing brick and the bricks that use its discovery pair the same files (issue #45, owner decision of 2026-10-30). +Images and masks are paired by the **preprocessing run** (its `find_masks`; issue #45, owner decision of 2026-10-30). Since v0.2.0 CellPatcher reads the pairing the run recorded (`RunInputs.read_mask`; `validate_image_and_mask_folders` returns it) and has no pairing of its own (`pair_masks` was removed in CP-2b), so every brick reads the same pairs. The layouts below are the brick's. A mask is a **TIFF file** (`.tif`, `.tiff`, any case, optional `.ome`) in the mask folder, in either layout (they may be mixed): @@ -198,16 +197,9 @@ The image's folders are its path below the image folder (folder run: none) or th | Field | Type | Default | Notes | |---|---|---|---| -| `mask_type` | `str \| None` | `None` | The mask type to use (`"cells"` for `img_cells.tif` or `cells/img.tif`, or a label image's name). Folder and tree runs (CP-2a): one of the types the preprocessing run recorded; `None` takes the run's only type and refuses a run with several. For `validate_image_and_mask_folders`: The types are the mask folder's: since v0.2.0 (preprocessing v0.4, which pairs an OME-Zarr image's own label images too) a label image is read only with `common.mask_source="ngff_labels"`, and a label image and a folder mask of the same type for one image are refused (`duplicate_mask`). | +| `mask_type` | `str \| None` | `None` | The mask type to use (`"cells"` for `img_cells.tif` or `cells/img.tif`, or a label image's name). Folder and tree runs (CP-2a): one of the types the preprocessing run recorded; `None` takes the run's only type and refuses a run with several. `validate_image_and_mask_folders(..., mask_type=)` chooses the same way. A label image and a folder mask of the same type for one image are refused by the preprocessing run (`duplicate_mask`). | -Refused, never resolved by a guess or silently skipped: - -- a mask that could belong to two images or two types (an image name that starts another image's name: `a_seg_mask.tif` with images `a` and `a_seg`); -- a file in the mask folder that pairs with no image (including a mask named exactly like its image, `masks/img.tif`, and a non-TIFF mask such as `img_mask.npy`); -- an image with no mask of the type used (in `cell` mode, and in `random` mode when a mask folder is given); -- two mask types whose names differ only in case, or two masks of one image and type. - -`find_masks`'s header checks are refused for the masks of the type used and ignored for the other types, which are never read (owner decision, preprocessing v0.3.2): an unreadable mask (`unreadable_mask`), a mask with axes other than Y, X and Z/T axes its metadata names (`mask_not_2d`: a stack written without axes, or a channel axis, is refused; no axis is guessed. Save each label channel as its own mask type, for example `_cells.tif` and `_nuclei.tif`, and save a Z or T stack with its Z/T axes named, as an ImageJ hyperstack or an OME-TIFF. `common.mask_axes` with a `C` axis is refused for these masks), another Y/X size than its image (`mask_size_mismatch`), non-integer pixels (`mask_pixel_type`), and masks of the type used with different depth slices or time points for one image (`mask_planes_differ`; the same difference between two types, one of them unused, is ignored). CellPatcher also checks each mask against its image when it reads it, with `common.mask_axes`. System files (`._img.tif`, other names starting with `.`, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are skipped in image and mask folders, as in the preprocessing brick. +The preprocessing run refuses an ambiguous mask (one that could belong to two images or two types), a mask whose header it cannot use (no Y/X of its image, non-integer pixels, axes other than Y, X and named Z/T: save each label channel as its own mask type), and two masks of one type for one image; a file in the masks folder that pairs with no image is a warning in its report. CellPatcher refuses an image the run recorded no mask of the type for (in `cell` mode, and in `random` mode when masks are read), and reads each mask's labels through the brick's `check_labels`. **Former options (removed in v0.1.1).** `strategy`, `mask_suffixes`, `case_sensitive` and `ignore_unrequested_masks` (in `MatchConfig`, a `run_config.json`'s `matching` section, and as arguments of `validate_image_and_mask_folders`, `run_patching_pipeline`, `run_patching_condition_repeat_tree` and `run_patching_from_folders`) are refused by name with a `ValueError` saying to use `matching.mask_type`, whatever their value: an old `run_config.json` that holds them fails until they are removed from its `matching` section. `find_candidate_matches`, `filter_shared_matches`, `select_requested_masks`, `LEGACY_MATCHING_DEFAULTS` and `cellpatcher.io.image_stem` were removed with them (D86). @@ -230,6 +222,7 @@ Normalization with the statistics of the preprocessing brick, following the [nor | `percentile` | `(float, float)` | `(1.0, 99.8)` | Low and high percentile for `"percentile"`, `0 ≤ low < high ≤ 100`. | | `scope` | `str` | `"image"` | Pixels the parameters come from: `"biological_repeat"` (one repeat group: the same biological repeat in every condition, technical repeats merged, as paired in the preprocessing brick's `experiment_design.csv`), `"image"` (all Z and T planes of the source image) or `"patch"` (the patch's own pixels). Per channel, except for `image_type="rgb"`. `"condition"` and `"technical_repeat"` were removed (October 2026) and are refused. | | `per_time_point` | `bool` | `False` | Normalize each time point separately (`t` joins the group). | +| `experiment_root`, `masks_root` | `str \| None` | `None` | For `run_patching` at a CSV scope: where the run's experiment and masks folder are now, when moved since the run (owner decision of 2026-11-02); `None`: where the run recorded them. Folder and tree runs refuse them: their image folder/root and `mask_folder_path`/`mask_root_path` say it. Since v0.2.0 (CP-2b). | | `experiment_design` | `str \| None` | `None` | For `scope="biological_repeat"`: the preprocessing run's `experiment_design.csv`, which must be the one next to `plane_histograms` (`not_the_run_output` otherwise); `None` uses it. Every repeat group must be confirmed, the file must be the one the run's manifest records, and a run that recorded no design is refused (`missing_experiment_design`). | | `pooling` | `str` | `"pixels"` | For `scope="biological_repeat"`: `"pixels"` (each pixel counts once) or `"conditions"` (each condition of a repeat group weighs the same). | | `image_type` | `str` | `"fluorescence"` | `"fluorescence"` (each channel normalized separately) or `"rgb"`: colour images of three 8- or 16-bit channels, normalized with StarDist's percentile method, one offset and scale from the R, G and B pixels together (written for `C0`, `C1` and `C2`). `"rgb"` needs `method="percentile"` and `extraction.split_channels=False`. When the preprocessing manifest records an image type, this must be the same. | @@ -245,15 +238,15 @@ Normalization with the statistics of the preprocessing brick, following the [nor | Normalized files (default) | `export.enabled=True` | float32 `.tif`/`.npy` | | Raw files plus parameters | `export.enabled=True`, `normalization.export_values="raw"`; read back with `cellpatcher.iter_exported_patches(csv)`/`load_exported_patch` or `ExportedPatchTorchDataset`, which normalize on load with each row's `Normalization_Offset_C`/`Normalization_Scale_C` columns (never recomputed; the `;`-separated `normalization_offset`/`normalization_scale` of earlier tables are read too) | Source dtype when the patch is only cropped; float32 raw values when resized or masked | | In memory, eager | `common.extract_arrays=True` | float32 arrays in `PatchingResult.patch_arrays`, within `memory_limit` | -| In memory, lazy | Run with `extract_arrays=False` (the patches still record their normalization), then `cellpatcher.iter_patch_arrays(patches, images, config, masks=...)` or `patch_array(...)`; `PatchTorchDataset(images, patches, config=config)` is built on them | float32, one patch at a time | +| In memory, lazy | Run with `extract_arrays=False` (the patches still record their normalization), then, for `run_patching`'s arrays, `cellpatcher.iter_patch_arrays(patches, images, config, masks=...)` or `patch_array(...)` (`PatchTorchDataset(images, patches, config=config)` is built on them), and for a folder or tree run `cellpatcher.iter_run_patch_arrays(patches, config)`, which reads one plane at a time through the preprocessing run | float32, one patch at a time | ```python -from cellpatcher import iter_patch_arrays, list_supported_files, run_patching_condition_repeat_tree +from cellpatcher import iter_run_patch_arrays, run_patching_condition_repeat_tree -for leaf in run_patching_condition_repeat_tree(root, config=config): # extract_arrays=False - images = list_supported_files(leaf.image_folder) # paths: read one plane at a time - for patch, array in iter_patch_arrays(leaf.result.patches, images, config): - ... # one normalized patch at a time +leaves = run_patching_condition_repeat_tree(root, config=config) # extract_arrays=False +patches = [patch for leaf in leaves for patch in leaf.result.patches] +for patch, array in iter_run_patch_arrays(patches, config): # experiment_root=, masks_root= if moved + ... # one normalized patch at a time ``` **What happens** (contract §6–§9): @@ -268,9 +261,9 @@ for leaf in run_patching_condition_repeat_tree(root, config=config): # extract_ **Recorded:** written after `source_relative_path` in `patches.csv`: `normalization_method` (e.g. `percentile 1-99.8`), `normalization_scope`, `normalization_group` (the repeat group label, ``, or `::` for the patch scope; `::t=` with `per_time_point`) and `normalization_exact` (`true`/`false`), then one `Normalization_Offset_C`/`Normalization_Scale_C` pair per channel, grouped by channel (full-precision numbers from the normalization brick's `Parameters.to_columns()`; a split-channel patch fills only its own channel's pair). In memory, `PatchRegion.normalization_parameters` is the brick's `Parameters` (`offset`, `scale`, `channels`); `normalization_offset`/`normalization_scale` are deprecated read-only properties giving the former `;`-separated text. `run_config.json` gains the `normalization` section, a top-level `normalization_inputs` object (`plane_histograms_sha256`, `schema_version`, `manifest_used`, `manifest_version`, `preprocessing_version`, and for `biological_repeat` `experiment_design_file`/`experiment_design_sha256`) and `normalization_settings` (the brick's `NormalizationSettings.to_dict()`); a `run_config.json` without `normalization` loads with normalization off. With normalization, the images of a run must have the same number of channels. -**In-memory runs:** `run_patching` normalizes too; for the CSV scopes each image needs `image_metadata=[{"source_relative_path": ...}]` and the right `common` labels (folder and tree runs fill both). The manifest's file checks run in folder and tree runs. +**In-memory runs:** `run_patching` normalizes too; for the CSV scopes each image needs `image_metadata=[{"source_relative_path": ...}]` and the right `common` labels (folder and tree runs fill both). The run is opened where it recorded the experiment and masks folder, or where `normalization.experiment_root`/`masks_root` say they are now (an experiment moved since the run; owner decision of 2026-11-02). The manifest's file checks run in folder and tree runs. -The `common.mask_source` and `common.mask_label_name` fields for OME-Zarr label masks (contract §4) exist since Milestone 2, and `cellpatcher.statistics` (loading `plane_histograms.csv`/`run_manifest.json`, contract §5 checks) since Milestone 1. +`cellpatcher.statistics` (loading `plane_histograms.csv`/`run_manifest.json`, contract §5 checks) exists since Milestone 1. --- @@ -388,9 +381,7 @@ Wherever `patches.csv` is written (`export.enabled=True` with `save_metadata_csv | `reason` | Empty for `processed`. For `no_patches`, the step that left none: `the patch mode generated no region in this image` (e.g. no labeled object in its mask, every object excluded by border or area), `the filters removed all region(s)`, or `normalization discarded all patch(es): they cannot be normalized from their own pixels` (decision D8). For `skipped`: `the image has no plane to patch along the axes split by z_policy/t_policy` (a Z or T axis of length 0 with `"all"`). | | `position_id` | The `position_id` every patch of the image carries in `patches.csv`, also for an image without patches: supplied via `image_metadata`, or the default of "Patch identity" above (one file is one position). Never empty since 2026-10-26. | | `n_regions`, `n_kept`, `n_normalized` | Regions the patch mode generated, those left by the filters, and those left by normalization (equal to `n_kept` without it), counted before `split_channels`; they tell the three `no_patches` reasons apart without reading `reason`. `0` for a skipped image. | -| `image_read_transposed` | `true` when `common.image_axes` declared the image file's header Y and X swapped, so the image was read transposed (v0.1.2 only: since v0.2.0 a swap is refused, and folder and tree runs read every image through the preprocessing run, so it is always `false` for them, OME-Zarr images included); empty for an in-memory array. CP-2b removes the column. Since v0.1.2. | - -`position_id`, `n_regions`, `n_kept`, `n_normalized` and `image_read_transposed` were added after the first release of the table, as its last columns, so earlier columns keep their positions. +`position_id`, `n_regions`, `n_kept`, `n_normalized` were added after the first release of the table, as its last columns, so earlier columns keep their positions. The last column of v0.1.2, `image_read_transposed`, was removed in v0.2.0 (CP-2b): since preprocessing v0.4 refuses every Y/X swap it was always `false` or empty. There is no `failed` status: an image that cannot be read, matched to a mask or patched stops the whole run with an error before any table is written, as before, so no image is ever left out silently. @@ -398,9 +389,9 @@ There is no `failed` status: an image that cannot be read, matched to a mask or ### OME-Zarr images -Since v0.2.0 (CP-2a) folder and tree runs read OME-Zarr images and their label images through the preprocessing run like any other image (see "File runs read through the preprocessing run"); this section describes `read_array`, `read_image` and lazy delivery from a path. With the `zarr` extra (`pip install "cellpatcher[zarr]"`), folders whose names end in `.zarr` (usually `.ome.zarr`) are images for `read_array` and lazy delivery (folder and tree runs take them from the preprocessing run). CellPatcher reads level 0 of the store's only image with the preprocessing brick's reader; a store with several images (positions, series or plate wells) is an error. The axes come from the OME-NGFF metadata and are matched to `common.image_axes` (or `mask_axes`) by role: a length-1 axis that the configured axes lack is dropped, a configured axis the store lacks is added with length 1, and the dimensions are reordered; any other mismatch is an error. Discovery never enters a store, and `img.ome.zarr` pairs with masks named like `img_cells.tif` or `cells/img.tif`. Without the extra, reading a store stops with the install command. OME-Zarr copies written by the preprocessing brick are not read: CellPatcher reads the source images. +Since v0.2.0 (CP-2a) folder and tree runs read OME-Zarr images and their label images through the preprocessing run like any other image (see "File runs read through the preprocessing run"); this section describes `read_array` and discovery. With the `zarr` extra (`pip install "cellpatcher[zarr]"`), folders whose names end in `.zarr` (usually `.ome.zarr`) are images for `read_array`, which reads level 0 of the store's only image, in its stored axis order, with the preprocessing brick's reader; a store with several images (positions, series or plate wells) is an error. (`read_image`, which put a store into `common.image_axes` by role, was removed in CP-2b with lazy delivery from a path.) Discovery never enters a store, and `img.ome.zarr` pairs with masks named like `img_cells.tif` or `cells/img.tif`. Without the extra, reading a store stops with the install command. OME-Zarr copies written by the preprocessing brick are not read: CellPatcher reads the source images. -**Masks from OME-NGFF labels (before v0.2.0; folder and tree runs now read the label images the preprocessing run paired, by type):** with `common.mask_source="ngff_labels"`, each OME-Zarr image's mask is its label image `common.mask_label_name` (from its `labels` group, OME-NGFF 0.4 and 0.5; the name may stay `None` when there is only one). Level 0 is read; its values must be integers, each dimension must equal the image's or have length 1, and a channel dimension must have length 1. It is then put into `common.mask_axes` by role, so a label Z or T axis with the image's length follows the image's planes and one of length 1 is shared (`z_policy`/`t_policy`). `mask_file` is written as `.ome.zarr/labels/`. A mask folder, or an image that is not OME-Zarr, is an error with this source. +**Masks from OME-NGFF labels:** a folder or tree run reads the label images the preprocessing run paired, a label image's type being its name (`matching.mask_type`); `mask_file` is `.ome.zarr/labels/`. The former `common.mask_source="ngff_labels"`/`mask_label_name` were removed in v0.2.0 (CP-2b). ### Condition/biological-repeat folder trees @@ -438,7 +429,8 @@ Output is identical whichever name is used; files are always written with `biolo Each exported `patches.csv` row, together with the `run_config.json` file it references via `config_file` (written alongside it whenever `export.enabled=True`; see `cellpatcher.RUN_CONFIG_FILENAME`), is enough to reconstruct the patch without re-deriving anything from code: - The row's own coordinates (`y0/y1/x0/x1`, `z0/z1`, `t`, `c`) and `output_height`/`output_width` fully describe the patch geometry and final exported array shape; `image_height`/`image_width`, the `pad_*` columns, `resized` and `output_axes` say, without the source image or the config, where the rectangle lies in its image, which pixels of the array are padding, whether the array was resampled and what its axes are. -- `run_config.json` is the complete, serialized `CellPatcherConfig` for the run — strategy parameters (including `random.seed`), extraction/resize settings, matching strategy, and axes — everything needed to reproduce the same coordinates deterministically from the same inputs. +- `run_config.json` is the complete, serialized `CellPatcherConfig` for the run — strategy parameters (including `random.seed`), extraction/resize settings, the mask type, and the axes of in-memory arrays — everything needed to reproduce the same coordinates deterministically from the same inputs. +- Since v0.2.0 (CP-2b) it is **version 2**: `run_config_version: 2`, and when a preprocessing run was read (a folder or tree run, or a CSV normalization scope) a `preprocessing` section with the run (`run`), the SHA-256 of its `run_manifest.json` when the call opened it (`manifest_sha256`), which names the run's images, masks, roles and table exactly (a manifest that changes during the call is refused, and no `run_config.json` is written), and for a folder or tree run the `layout` (`image`, `mask`) of the arrays it built from the run's recorded roles (T, Z, C, then Y, X). A folder or tree run does not record `common.image_axes`, `mask_axes` or the labels (each patch's labels are in `patches.csv`). `load_config`, `read_patch_table` and the exported-patch readers (`iter_exported_patches`, `load_exported_patch`, `ExportedPatchTorchDataset`) read versions 1 (no version: written before v0.2.0, or a config file) and 2, and refuse any other naming the version. ## Typed refusals diff --git a/docs/normalization.md b/docs/normalization.md index 0ea2013..5543f43 100644 --- a/docs/normalization.md +++ b/docs/normalization.md @@ -123,8 +123,8 @@ channels never share parameters (except the three channels of an RGB image). selected channels. A single-channel patch keeps the parameters of its channel in the image's group, so selecting channels never changes their values. - **OME-Zarr images** (folders ending in `.zarr`) are read like TIFF files; their masks can be a - mask folder or the label images inside them (`common.mask_source="ngff_labels"`, with - `common.mask_label_name`). + mask folder or the label images inside them, as the preprocessing run paired them (a label + image's type is its name: `matching.mask_type`). ## What is recorded @@ -151,7 +151,7 @@ own settings (`normalization_settings`). | Normalized files (default) | `export.enabled=True`: float32 `.tif`/`.npy` files | | Raw files plus parameters (smaller, reversible) | `normalization.export_values="raw"`; read them back normalized with `cellpatcher.iter_exported_patches("patches.csv")` or `ExportedPatchTorchDataset`, which apply each row's offset and scale | | Arrays in memory | `common.extract_arrays=True` (stops above `normalization.memory_limit`, 4 GiB by default) | -| One patch at a time (large datasets) | Run with `extract_arrays=False`, then `cellpatcher.iter_patch_arrays(patches, images, config)`; `PatchTorchDataset(images, patches, config=config)` for PyTorch | +| One patch at a time (large datasets) | Run with `extract_arrays=False`, then `cellpatcher.iter_run_patch_arrays(patches, config)` for a folder or tree run (read through its preprocessing run), or `cellpatcher.iter_patch_arrays(patches, images, config)` for `run_patching`'s arrays; `PatchTorchDataset(images, patches, config=config)` for PyTorch (arrays) | All of these give the same arrays for the same settings. diff --git a/docs/normalization_contract.md b/docs/normalization_contract.md index 1586b76..a0f1cb6 100644 --- a/docs/normalization_contract.md +++ b/docs/normalization_contract.md @@ -96,7 +96,12 @@ memory limit is an error, even if the preprocessing brick read it in blocks. ## 4. Masks -`common.mask_source` selects where masks come from: +Since v0.2.0 folder and tree runs read the masks their preprocessing run paired (a label +image's type is its name, chosen with `matching.mask_type`); `common.mask_source` and +`mask_label_name`, which selected the source below before v0.2.0, were removed in CP-2b. The +rules below are the ones the preprocessing run applies. + +Before v0.2.0, `common.mask_source` selected where masks come from: | Value | Masks | Matching | |---|---|---| @@ -293,7 +298,7 @@ In memory, `PatchRegion.normalization_parameters` holds the brick's `Parameters` | Normalized files (default) | `export.enabled=True`, `normalization.export_values="normalized"` | float32 `.tif`/`.npy` | | Raw files plus parameters | `export.enabled=True`, `normalization.export_values="raw"` | Source dtype; the columns of §10 hold the parameters, and loaders normalize on load | | In memory, eager | `common.extract_arrays=True` | float32 arrays in `PatchingResult.patch_arrays` | -| In memory, lazy | A NumPy iterator over patches (core) and `PatchTorchDataset` (torch extra) | float32, one patch at a time | +| In memory, lazy | A NumPy iterator over patches (core: `iter_patch_arrays` for arrays, `iter_run_patch_arrays` for a folder or tree run, read through its preprocessing run) and `PatchTorchDataset` (torch extra, arrays) | float32, one patch at a time | - All modes give identical arrays for the same configuration. - Eager delivery estimates its memory first and stops above `normalization.memory_limit`, @@ -311,6 +316,7 @@ In memory, `PatchRegion.normalization_parameters` holds the brick's `Parameters` | `enabled` | `bool` | `False` | Off keeps current behaviour exactly | | `plane_histograms` | `str \| None` | `None` | Path of `plane_histograms.csv` (§5) | | `manifest` | `str \| None` | `None` | The run's `run_manifest.json`, next to the CSV; `None` uses it, and another file is refused (§5) | +| `experiment_root`, `masks_root` | `str \| None` | `None` | `run_patching` only: where the run's experiment and masks folder are now, if moved since the run; refused by folder and tree runs (since v0.2.0) | | `method` | `str` | `"percentile"` | §6 | | `percentile` | `tuple[float, float]` | `(1.0, 99.8)` | Low and high percentiles, `0 ≤ low < high ≤ 100` | | `clip` | `tuple[float, float] \| None` | `None` | Limits applied right after normalizing (by the brick's `NormalizationParameters.apply`), before masking and resizing; the same in eager, lazy, exported and raw-on-load delivery | @@ -323,8 +329,8 @@ In memory, `PatchRegion.normalization_parameters` holds the brick's `Parameters` | `pooling` | `str` | `"pixels"` | §7: `"pixels"` or `"conditions"` | | `image_type` | `str` | `"fluorescence"` | §6: `"fluorescence"` (each channel separately) or `"rgb"` (R, G and B together, `"percentile"` only, `extraction.split_channels=False`); must equal the manifest's image type when it records one (§5) | -`config.common` gains `mask_source` (`"tiff_tree"` or `"ngff_labels"`, default `"tiff_tree"`) -and `mask_label_name` (`str | None`, default `None`) (§4), and `z_policy`/`t_policy` +`config.common` gained `mask_source` and `mask_label_name` (§4; removed in v0.2.0, CP-2b), and +`z_policy`/`t_policy` (`"keep"` by default, or `"all"`, `"first"`, `"select"`; Z also `"max"`, `"mean"`, `"sum"`) with `z_index`/`t_index` (`int | None`), which split Z and T in folder, tree and `run_patching` runs (§3; Milestone 2, decided October 2026). diff --git a/notebooks/CellPatcher/CHANGELOG.md b/notebooks/CellPatcher/CHANGELOG.md index f8dd0ac..1bce3cd 100644 --- a/notebooks/CellPatcher/CHANGELOG.md +++ b/notebooks/CellPatcher/CHANGELOG.md @@ -4,6 +4,72 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). +## [Unreleased] - File-run removals and run_config.json version 2 (CP-2b) + +### Changed +- `run_config.json` is version 2 (`run_config_version`). When a preprocessing run was read + it has a `preprocessing` section: the run's folder (`run`), the SHA-256 of its + `run_manifest.json` taken when the call opened it (`manifest_sha256`), and for a folder or + tree run the array `layout` (`image`, `mask`) built from the run's recorded roles. A manifest + that changes during the call (while it is opened, between two opens, or before + `run_config.json` is written) is refused, and no `run_config.json` is left. A folder + or tree run no longer records `common.image_axes`, `mask_axes` or its labels there (they are + the run's, not settings), so its `run_config.json` loads back as a config a file run accepts. + `load_config`, `read_patch_table` and the exported-patch readers refuse any version but 1 (no + version: written before v0.2.0, or a plain config file) and 2. +- `validate_image_and_mask_folders(image_folder, mask_folder, *, preprocessing_run=, mask_type=)` + returns the pairing the preprocessing run recorded (opened where the folders are now); it + pairs nothing itself, and refuses without a run (`no_preprocessing_run`). It moved to + `cellpatcher.file_runs` (still `cellpatcher.validate_image_and_mask_folders`). +- New `iter_run_patch_arrays(patches, config, *, experiment_root=, masks_root=)`: lazy + delivery of a folder or tree run's patches, each plane read with `RunInputs.read_plane` + (masks with `read_mask`) from the image the run recorded at the patch's + `source_relative_path`, refusing another image or FOV key. +- `run_patching_pipeline` (and `run_patching_from_folders`) accept `mask_folder_path` in `grid` + and `whole_fov` mode (owner decision of 2026-11-02): it says where the run's masks folder is + now; those masks are checked there, never read. Before, it was refused. +- `normalization.experiment_root`/`masks_root` (owner decision of 2026-11-02): where the + experiment and masks folder of a moved experiment are now, for `run_patching` at the `image` + or `biological_repeat` scope. Folder and tree runs refuse them (their arguments say it). + +### Removed (breaking) +- `patch_array`, `iter_patch_arrays` and `PatchTorchDataset` take arrays only: a file path is a + `TypeError` naming `iter_run_patch_arrays` (`PatchTorchDataset` refuses it when it is built; + before, CellPatcher read it with `common.image_axes` and its own header check). +- `images.csv` loses its last column `image_read_transposed` (always `false` or empty since + preprocessing v0.4), and `ImageRecord` its field. +- `common.mask_source` and `common.mask_label_name`: `CommonConfig(mask_source=...)` is a + `TypeError`; a config or `run_config.json` with their defaults still loads, one with another + value is refused naming `matching.mask_type`. +- `cellpatcher.io`: `pair_masks`, `MaskPairing`, `read_image`, `conform_axes`, + `read_ngff_mask`, `refuse_image_header_axes`, `refuse_mask_header_axes`, + `refuse_header_axes`, `header_axes_disagreement`, `refuse_mask_channel_axis`, + `source_relative_path`, `discover_folder_images` and `validate_image_and_mask_folders` (moved, + above); `cellpatcher.planes`: `iter_file_planes`, `read_file_plane`; `cellpatcher.ids`: + `object_set_from_label_path`; `cellpatcher.pipeline`: `check_source_identities`, + `run_config_data` (now `cellpatcher.run_config_file`). `cached_statistics` moved from + `cellpatcher.normalization` to `cellpatcher.run_open`. + +### Known breakage until CP-3 +- **CellPatcher's own notebook (`notebooks/CellPatcher/CellPatcher.ipynb`) does not work after + this change.** Its import cell fails: it still imports `pair_masks`, `read_ngff_mask` and + `cellpatcher.normalization.cached_statistics`, and it offers the removed settings. It is + rewritten in CP-3, the next PR; no compatibility shim is added meanwhile (owner decision of + 2026-11-02). Use the Python API until then. +- Not in this change: CellPatcher does not yet call preprocessing's check for images added to + the experiment after its run. It is wired in once preprocessing v0.4.0rc2 provides it. + +### Downstream: for the brick migrations +- **MorphoRelate** (`io/cellpatcher.py`): `check_image_axes` reads `common.image_axes` from + `run_config.json`; a folder or tree run's version 2 file has none, so it reports "not + checked" (no error). Its migration should take the axes from the preprocessing run (or + `preprocessing.layout`). `read_patch_table` now refuses an unknown `run_config_version`. +- **MorphoEmbed** (`provider_crops.py`): `PatchTorchDataset([image], ...)` is fine with an + array; a path is now a `TypeError` when the dataset is built. +- **fiberfeatures**: `data/experiment_001/reference/patch_config.json` still holds + `mask_source`/`mask_label_name` at their defaults. It loads with `load_config`, but + `CommonConfig(**common)` on it is a `TypeError`; a regenerated reference loses both keys. + ## [Unreleased] - File runs read through the preprocessing run (CP-2a) ### Changed @@ -13,7 +79,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). off (`no_preprocessing_run`, with the preprocessing command). The images patched are the run's, with their labels and `position_id` as it recorded them; images and masks are read with `RunInputs.read_plane`/`read_mask` in the roles it recorded, in one canonical layout per - run (T, Z, C, then Y, X), recorded in `run_config.json` as `common.image_axes`/`mask_axes`. + run (T, Z, C, then Y, X), recorded in `run_config.json` as `common.image_axes`/`mask_axes` + (in its `preprocessing.layout` since CP-2b). An image added to the experiment after the run is not patched, and nothing says so yet (known limitation, closed when CellPatcher calls the preprocessing check for added images, which is not in v0.4.0rc1). diff --git a/src/cellpatcher/__init__.py b/src/cellpatcher/__init__.py index 505a194..3dd5c6b 100644 --- a/src/cellpatcher/__init__.py +++ b/src/cellpatcher/__init__.py @@ -24,7 +24,9 @@ WholeFOVConfig, ) from cellpatcher.delivery import iter_exported_patches, iter_patch_arrays, load_exported_patch, patch_array +from cellpatcher.run_delivery import iter_run_patch_arrays from cellpatcher.errors import CellPatcherRefusal, LabelRangeError, SamplingExhaustedError +from cellpatcher.file_runs import validate_image_and_mask_folders from cellpatcher.export import ( PatchTableError, export_patches, @@ -45,7 +47,6 @@ list_supported_files, read_array, read_array_shape, - validate_image_and_mask_folders, write_array, run_patching_from_folders, ) @@ -132,6 +133,7 @@ "save_patch_array", "export_patches", "iter_patch_arrays", + "iter_run_patch_arrays", "patch_array", "iter_exported_patches", "load_exported_patch", diff --git a/src/cellpatcher/config.py b/src/cellpatcher/config.py index dfb0f39..31951a3 100644 --- a/src/cellpatcher/config.py +++ b/src/cellpatcher/config.py @@ -13,15 +13,12 @@ from cellpatcher.ids import validate_object_set from cellpatcher.planes import validate_plane_policies -MASK_SOURCES = ("tiff_tree", "ngff_labels") - - @accept_deprecated_repeat @dataclass class CommonConfig: - # The axes of in-memory arrays given to run_patching (canonical, Y before X). File runs - # read each image and mask in the roles the preprocessing run recorded and refuse other - # values than "YX" (CP-2a; removed for them in CP-2b). + # The axes of in-memory arrays given to run_patching (canonical, Y before X). Folder and + # tree runs read each image and mask in the roles the preprocessing run recorded, refuse + # other values than "YX" and do not record them in run_config.json (design v0.4 §6). image_axes: str = "YX" mask_axes: str = "YX" output_dir: str | None = None @@ -29,12 +26,10 @@ class CommonConfig: # The output folder of the clean preprocessing run (v0.4) a folder or tree run reads its # images, masks and labels through (RunInputs, design v0.4 §6); required for file runs. preprocessing_run: str | None = None - # The MorphoBricks shared-CSV-contract `condition`/`biological_repeat`/`technical_repeat` - # columns (AGENTS.md §5) of in-memory run_patching; a label not given (or None) is - # "single", as for a missing folder level. Folder and tree runs take each image's labels - # from the preprocessing run and refuse other values. technical_repeat is the well/dish/ - # coverslip; per-image image_metadata["technical_repeat"] overrides it. `repeat` stays - # accepted as a deprecated alias of biological_repeat (DeprecationWarning). + # The shared-CSV-contract labels (AGENTS.md §5) of in-memory run_patching; None is "single", + # as for a missing folder level. Folder and tree runs take each image's from the run and + # refuse other values. technical_repeat (well/dish/coverslip) is overridden per image by + # image_metadata["technical_repeat"]; `repeat` is a deprecated alias of biological_repeat. condition: str | None = SINGLE_CONDITION_LABEL biological_repeat: str | None = SINGLE_BIOLOGICAL_REPEAT_LABEL technical_repeat: str | None = SINGLE_TECHNICAL_REPEAT_LABEL @@ -48,25 +43,14 @@ class CommonConfig: z_index: int | None = None t_policy: str = "keep" t_index: int | None = None - # Where folder and tree runs read masks before CP-2a. They now read the masks the - # preprocessing run paired (choose the type with matching.mask_type) and refuse other - # values than these defaults. - mask_source: str = "tiff_tree" - mask_label_name: str | None = None - # The object set of cell-mode patches (the `object_set` column and a - # level of patch_id, cellpatcher.ids). None: folder and tree runs name it - # after the mask type the preprocessing run recorded (a folder name, or an - # OME-NGFF label name), and "mask" when the mask gives no name (in-memory - # run_patching). Set it to name it - # explicitly; image_metadata["object_set"] overrides it per image. + # The object set of cell-mode patches (the `object_set` column and a level of patch_id, + # cellpatcher.ids). None: folder and tree runs name it after the mask type the run recorded + # (a folder name, or an OME-NGFF label name), in-memory run_patching "mask". Set it to name + # it explicitly; image_metadata["object_set"] overrides it per image. object_set: str | None = None def __post_init__(self) -> None: validate_plane_policies(self) - if self.mask_source not in MASK_SOURCES: - raise ValueError(f"mask_source must be one of {list(MASK_SOURCES)}; got {self.mask_source!r}.") - if self.mask_label_name is not None and not isinstance(self.mask_label_name, str): - raise ValueError("mask_label_name must be a string or None.") validate_object_set(self.object_set) @@ -372,6 +356,9 @@ class NormalizationConfig: manifest The run's ``run_manifest.json``: None, or the one in the table's folder (another file is refused). + experiment_root, masks_root + For ``run_patching`` at a CSV scope: where the run's experiment and masks + folder are now, if moved since the run (None: where it recorded them). method ``"minmax"``, ``"percentile"`` (default), ``"zscore"`` or ``"robust_zscore"`` (MAD x 1.4826); only ``"percentile"`` for @@ -427,6 +414,8 @@ class NormalizationConfig: enabled: bool = False plane_histograms: str | None = None manifest: str | None = None + experiment_root: str | None = None + masks_root: str | None = None method: str = "percentile" percentile: tuple[float, float] = (1.0, 99.8) clip: tuple[float, float] | None = None @@ -500,16 +489,15 @@ def __post_init__(self) -> None: class MatchConfig: """Which masks to pair with the images. - Images and masks are paired with the MorphoBricks preprocessing brick's - rule (``morphobricks_preprocessing.find_masks``, issue #45), so every - brick pairs the same files: a mask is a TIFF file + Images and masks are paired by the preprocessing run (its ``find_masks``, + design v0.4 §6), so every brick reads the same pairs: a mask is a TIFF file ``//.tif`` or - ``/_.tif`` in the mask folder - (:func:`~cellpatcher.io.pair_masks`). + ``/_.tif`` in the mask folder, or + an OME-Zarr image's label image (its type is its name). ``mask_type`` names the mask type to use (``"cells"`` for - ``img_cells.tif`` or ``cells/img.tif``). None (default) uses the only - type in the mask folder, and refuses a folder holding several. + ``img_cells.tif`` or ``cells/img.tif``). None (default) uses the run's only + type, and refuses a run holding several. The former options ``strategy``, ``mask_suffixes``, ``case_sensitive`` and ``ignore_unrequested_masks`` were removed in v0.1.1: given here, set diff --git a/src/cellpatcher/delivery.py b/src/cellpatcher/delivery.py index 165848c..1fb73a1 100644 --- a/src/cellpatcher/delivery.py +++ b/src/cellpatcher/delivery.py @@ -7,7 +7,9 @@ patch): each patch is rebuilt from its source image with the pipeline's own steps (plane, extract, normalize with the patch's recorded parameters, mask, resize, channel), so only one patch, and one image or plane, is in memory at - a time; ``cellpatcher.torch_dataset.PatchTorchDataset`` is built on it; + a time; ``cellpatcher.torch_dataset.PatchTorchDataset`` is built on it. The + images are arrays (``run_patching``); a folder or tree run's patches are read + through their preprocessing run with :func:`cellpatcher.iter_run_patch_arrays`; - **from exported files**, with :func:`iter_exported_patches` (or :func:`load_exported_patch`): files exported with ``normalization.export_values="raw"`` are normalized on load with each row's @@ -24,25 +26,26 @@ import csv import json import os -from collections.abc import Iterator, Mapping, Sequence +from collections.abc import Iterable, Iterator, Mapping, Sequence from pathlib import Path from types import SimpleNamespace +from typing import TYPE_CHECKING import numpy as np from cellpatcher.axes import normalize_axes from cellpatcher.extract import extract_patch -from cellpatcher.io import ( - read_array, - read_image, - refuse_image_header_axes, - refuse_mask_header_axes, - refuse_transposed_mask_axes, -) +from cellpatcher.io import read_array, refuse_transposed_mask_axes from cellpatcher.masking import apply_label_mask_to_patch from cellpatcher.normalization import PatchNormalization, parameters_from_row -from cellpatcher.planes import apply_plane_choice, patch_plane_choice, read_file_plane, select_mask_plane +from cellpatcher.planes import apply_plane_choice, patch_plane_choice, select_mask_plane from cellpatcher.resize import resize_patch +from cellpatcher.run_config_file import delivery_config, run_config_version + +if TYPE_CHECKING: + from cellpatcher.config import CellPatcherConfig + from cellpatcher.planes import PlaneChoice + from cellpatcher.types import PatchRegion def patch_axes(config, patch) -> str: @@ -55,7 +58,9 @@ def patch_axes(config, patch) -> str: return axes -def patch_array(patch, image, config, *, mask=None) -> np.ndarray: +def patch_array( + patch: PatchRegion, image: np.ndarray, config: CellPatcherConfig, *, mask: np.ndarray | None = None +) -> np.ndarray: """Rebuild one patch's array from its source image, as the pipeline delivers it. Parameters @@ -64,9 +69,9 @@ def patch_array(patch, image, config, *, mask=None) -> np.ndarray: A ``PatchRegion`` from a run with *config* (its ``z0``/``t``, ``c``, ``label_id`` and ``normalization_*`` fields are used). image - Its source image, in ``config.common.image_axes``: an array, or a - file path (for a TIFF or OME-Zarr image, only the patch's plane is - read). + Its source image, in ``config.common.image_axes``, as an array. A file + path is refused: the patches of a folder or tree run are delivered with + :func:`cellpatcher.iter_run_patch_arrays`, through their preprocessing run. config The run's ``CellPatcherConfig``. mask @@ -85,7 +90,13 @@ def patch_array(patch, image, config, *, mask=None) -> np.ndarray: return _finish_patch(patch, plane, axes, mask_plane, mask_axes, config) -def iter_patch_arrays(patches, images: Sequence, config, *, masks: Sequence | None = None) -> Iterator[tuple]: +def iter_patch_arrays( + patches: Iterable[PatchRegion], + images: Sequence, + config: CellPatcherConfig, + *, + masks: Sequence | None = None, +) -> Iterator[tuple]: """Yield ``(patch, array)`` for every patch, one at a time (lazy delivery). Parameters @@ -96,8 +107,9 @@ def iter_patch_arrays(patches, images: Sequence, config, *, masks: Sequence | No *images*. A run with ``extract_arrays=False`` and no export gives them, with their normalization recorded, without holding any array. images - The run's source images, in image order: arrays or file paths (for a - folder run, ``cellpatcher.list_supported_files(folder)``). + The run's source images, in image order, as arrays (a file path is + refused: see :func:`cellpatcher.iter_run_patch_arrays` for folder and + tree runs). config The run's ``CellPatcherConfig``. masks @@ -161,8 +173,10 @@ def iter_exported_patches(metadata_file, *, root_dir=None) -> Iterator[tuple[dic yield row, load_exported_patch(row, root_dir=root, run_config=run_configs[name]) -def _normalize_on_load(row: Mapping, array, run_config: Mapping | None) -> np.ndarray: +def _normalize_on_load(row: Mapping, array: np.ndarray, run_config: Mapping | None) -> np.ndarray: """Normalize a loaded patch file when its run exported raw values (else return it as stored).""" + if run_config is not None: + run_config_version(run_config, "run_config.json") settings = (run_config or {}).get("normalization") or {} if settings.get("export_values") != "raw": return array @@ -172,7 +186,7 @@ def _normalize_on_load(row: Mapping, array, run_config: Mapping | None) -> np.nd patch = _row_patch(row) clip = settings.get("clip") return PatchNormalization(parameters).apply( - array, patch_axes(_run_config(run_config), patch), clip=tuple(clip) if clip else None + array, patch_axes(delivery_config(run_config, "run_config.json"), patch), clip=tuple(clip) if clip else None ) @@ -202,25 +216,32 @@ def _splits_channel(config, patch, axes: str) -> bool: return bool(config.extraction.split_channels and patch.c is not None and "C" in axes) -def _image_plane(image, choice, config): - common = config.common - if isinstance(image, (str, os.PathLike)): - refuse_image_header_axes([image], common.image_axes) - return read_file_plane(image, common.image_axes, choice, common) - return apply_plane_choice(image, common.image_axes, choice) +def _image_plane(image: np.ndarray, choice: PlaneChoice, config: CellPatcherConfig) -> tuple[np.ndarray, str]: + """Return the plane *choice* of an in-memory image, refusing a file path.""" + _refuse_file(image, "image") + return apply_plane_choice(image, config.common.image_axes, choice) -def _mask_plane(mask, choice, config): +def _mask_plane( + mask: np.ndarray | None, choice: PlaneChoice, config: CellPatcherConfig +) -> tuple[np.ndarray | None, str | None]: + """Return the mask plane for *choice* of an in-memory mask, refusing a file path or X before Y.""" if mask is None: return None, None - common = config.common - if isinstance(mask, (str, os.PathLike)): - refuse_mask_header_axes([mask], common.mask_axes) - mask = read_image(mask, common.mask_axes) - else: - refuse_transposed_mask_axes(common.mask_axes) - mask = np.asarray(mask) - return select_mask_plane(mask, normalize_axes(common.mask_axes, mask.ndim), choice) + _refuse_file(mask, "mask") + refuse_transposed_mask_axes(config.common.mask_axes) + mask = np.asarray(mask) + return select_mask_plane(mask, normalize_axes(config.common.mask_axes, mask.ndim), choice) + + +def _refuse_file(source: object, what: str) -> None: + """Refuse a file path: since v0.2.0 files are read only through the preprocessing run (CP-2b).""" + if isinstance(source, (str, os.PathLike)): + raise TypeError( + f"Lazy delivery takes the {what}s of run_patching as arrays; got the path {source}. Since CellPatcher " + "v0.2.0 an image or mask file is read only through its preprocessing run (design v0.4 §6): deliver " + "the patches of a folder or tree run with cellpatcher.iter_run_patch_arrays(patches, config)." + ) def _finish_patch(patch, plane, axes, mask_plane, mask_axes, config) -> np.ndarray: @@ -268,20 +289,3 @@ def number(name): t=number("t"), c=number("c"), ) - - -def _run_config(data: Mapping): - """The config of a ``run_config.json``, for the axes of its patches. - - A run normalized at a scope removed since (``condition``, - ``technical_repeat``; normalization brick N29) still loads here: its - parameters are in ``patches.csv``, and only the patch axes are needed. - """ - from cellpatcher.config import REMOVED_NORMALIZATION_SCOPES - from cellpatcher.orchestration import _config_from_dict - - data = dict(data) - normalization = data.get("normalization") - if isinstance(normalization, Mapping) and normalization.get("scope") in REMOVED_NORMALIZATION_SCOPES: - data["normalization"] = {**normalization, "scope": "image"} - return _config_from_dict(data) diff --git a/src/cellpatcher/export.py b/src/cellpatcher/export.py index 6940d62..1049837 100644 --- a/src/cellpatcher/export.py +++ b/src/cellpatcher/export.py @@ -85,10 +85,7 @@ def save_image_table_csv(image_records, csv_path): writer = csv.DictWriter(f, fieldnames=list(IMAGE_TABLE_COLUMNS)) writer.writeheader() for record in image_records: - row = record.to_dict() - if isinstance(row["image_read_transposed"], bool): - row["image_read_transposed"] = "true" if row["image_read_transposed"] else "false" - writer.writerow(row) + writer.writerow(record.to_dict()) def image_records_from_patches(patches) -> list[ImageRecord]: @@ -257,7 +254,7 @@ def _typed(text: str, kind: type, optional: bool, column: str): raise TypeError(f"no reader for {column} ({kind})") -def _read_run_config(csv_path: Path, names: set[str], problems: list[str]): +def _read_run_config(csv_path: Path, names: set[str], problems: list[str]) -> dict | None: """The ``run_config.json`` the rows' ``config_file`` names (relative to the CSV's folder), or None.""" import json @@ -282,6 +279,13 @@ def _read_run_config(csv_path: Path, names: set[str], problems: list[str]): if not isinstance(run_config, dict): problems.append(f"config file {path} holds a {type(run_config).__name__}, not the JSON object CellPatcher writes") return None + from cellpatcher.run_config_file import run_config_version + + try: + run_config_version(run_config, f"config file {path}") + except ValueError as exc: + problems.append(str(exc)) + return None return run_config diff --git a/src/cellpatcher/file_runs.py b/src/cellpatcher/file_runs.py index ad51c03..9bb667b 100644 --- a/src/cellpatcher/file_runs.py +++ b/src/cellpatcher/file_runs.py @@ -12,9 +12,9 @@ with ``RunInputs.read_mask`` by the run's pairing (:mod:`cellpatcher.run_reading`); - the CSV normalization scopes read the run's own table, design and image type. -The settings that told CellPatcher how to read files (``common.image_axes``, ``mask_axes``, -``mask_source``, ``mask_label_name`` and the labels) are refused for file runs with a -migration message; CP-2b removes them. In-memory ``run_patching`` keeps them. +The settings that told CellPatcher how to read files (``common.image_axes``, ``mask_axes`` and +the labels) are refused for file runs with a migration message; in-memory ``run_patching`` +keeps them. ``mask_source`` and ``mask_label_name`` were removed in v0.2.0 (CP-2b). """ from __future__ import annotations @@ -26,7 +26,11 @@ from typing import TYPE_CHECKING from cellpatcher.ids import object_set_from_mask_type, validate_object_set -from cellpatcher.io import SINGLE_CONDITION_LABEL, ConditionRepeatFolder +from cellpatcher.io import ( + SINGLE_CONDITION_LABEL, + ConditionRepeatFolder, + refuse_removed_matching, +) from cellpatcher.normalization import ( CSV_SCOPES, check_image_type, @@ -50,7 +54,7 @@ run_layout, run_mask_layout, ) -from cellpatcher.run_session import keep_in_session +from cellpatcher.run_session import keep_in_session, remember from cellpatcher.statistics import ( PlaneStatistics, PreprocessingCheckError, @@ -122,12 +126,12 @@ def refuse_file_run_settings(config: CellPatcherConfig, where: str) -> None: for name in ("image_axes", "mask_axes") if getattr(common, name) != _DEFAULT_AXES ] - if common.mask_source != "tiff_tree" or common.mask_label_name is not None: - problems.append( - f"common.mask_source={common.mask_source!r}, common.mask_label_name={common.mask_label_name!r}: the " - "masks are those the preprocessing run paired (masks-folder TIFFs and OME-Zarr label images); " - "choose the type with matching.mask_type (a label image's type is its name)" - ) + problems += [ + f"normalization.{name}={getattr(config.normalization, name)!r}: where the experiment and masks are now " + "is the image folder or root and mask_folder_path or mask_root_path of the run; leave it unset" + for name in ("experiment_root", "masks_root") + if getattr(config.normalization, name) is not None + ] problems += [ f"common.{name}={getattr(common, name)!r}: each image's labels are its folders, as the preprocessing " "run recorded them; leave it unset" @@ -138,25 +142,23 @@ def refuse_file_run_settings(config: CellPatcherConfig, where: str) -> None: raise ValueError( f"{where} reads the images, masks and labels through the preprocessing run since CellPatcher " "v0.2.0 (design v0.4 §6), so these settings are not used for files: " + "; ".join(problems) + ". " - "The axes and labels still describe in-memory arrays given to run_patching; mask_source and " - "mask_label_name are not used there." + "The axes and labels still describe in-memory arrays given to run_patching." ) if common.preprocessing_run is None: _refuse_without_run(where) -def _refuse_without_run(where: str) -> None: - """Raise ``no_preprocessing_run``, with the preprocessing command.""" +def _refuse_without_run(where: str, setting: str = "common.preprocessing_run") -> None: + """Raise ``no_preprocessing_run`` (*setting* not set), with the preprocessing command.""" from morphobricks_preprocessing.validation import error raise PreprocessingCheckError([ error( "no_preprocessing_run", - f"{where} reads files only through a clean preprocessing run (v0.4), and " - "common.preprocessing_run is not set.", + f"{where} reads files only through a clean preprocessing run (v0.4), and {setting} is not set.", [], f"Run preprocessing on the experiment first: {PREPROCESSING_COMMAND}. Then set " - "common.preprocessing_run to its output folder (by default _MorphoBricks_Preprocessing, " + f"{setting} to its output folder (by default _MorphoBricks_Preprocessing, " "beside the experiment).", ) ]) @@ -165,19 +167,11 @@ def _refuse_without_run(where: str) -> None: def masks_used(config: CellPatcherConfig, masks_root: str | os.PathLike | None) -> bool: """Return whether the run reads masks: always in ``cell`` mode, in ``random`` mode when asked. - ``random`` mode reads them when a masks folder or ``matching.mask_type`` is given. - - Raises - ------ - ValueError - For a masks folder in ``grid`` or ``whole_fov`` mode, which use no masks. + ``random`` mode reads them when a masks folder or ``matching.mask_type`` is given. ``grid`` + and ``whole_fov`` read none: a masks folder given for them only says where the run's masks + folder is now (owner decision of 2026-11-02), and the run checks its masks there. """ if config.mode in {"grid", "whole_fov"}: - if masks_root is not None: - raise ValueError( - f"mask_folder_path is not supported for mode='{config.mode}'. " - "Remove it, or switch to 'cell' or 'random' mode." - ) return False return config.mode == "cell" or masks_root is not None or config.matching.mask_type is not None @@ -208,11 +202,12 @@ def open_file_run( refuse_file_run_settings(config, where) check_config_identity(config) inputs = open_run(config.common.preprocessing_run, experiment_root=experiment_root, masks_root=masks_root) + remember("file_run", config.common.preprocessing_run, "folder or tree run") fovs = inputs.fovs() if not fovs: raise ValueError(f"The preprocessing run in {inputs.output_folder} has no images.") layout = run_layout(fovs) - mask_type = _mask_type(config, inputs) if use_masks else None + mask_type = _mask_type(config.matching.mask_type, inputs, f"mode={config.mode!r}") if use_masks else None mask_axes = _DEFAULT_AXES if mask_type is not None: mask_axes = run_mask_layout([_record(inputs, fov, mask_type) for fov in fovs]) @@ -221,13 +216,12 @@ def open_file_run( return FileRun(inputs, layout, mask_type, mask_axes, statistics) -def _mask_type(config: CellPatcherConfig, inputs: RunInputs) -> str: - """Return the mask type the run reads: ``matching.mask_type``, or the run's only type.""" +def _mask_type(wanted: str | None, inputs: RunInputs, reader: str) -> str: + """Return the mask type *reader* reads: *wanted* (``matching.mask_type``), or the run's only type.""" types = inputs.mask_types() - wanted = config.matching.mask_type if not types: raise ValueError( - f"mode={config.mode!r} reads masks, but the preprocessing run in {inputs.output_folder} recorded none. " + f"{reader} reads masks, but the preprocessing run in {inputs.output_folder} recorded none. " f"Run preprocessing with the masks folder ({PREPROCESSING_COMMAND})." ) if wanted is None and len(types) > 1: @@ -372,17 +366,71 @@ def fov_metadata(run: FileRun, fov: Fov, config: CellPatcherConfig) -> dict: The FOV key is the run's (``position_id`` and ``technical_repeat`` included); the mask file and, in ``cell`` mode without ``common.object_set``, the object set come from the - recorded mask and its type. Nothing is read transposed (``image_read_transposed`` False). + recorded mask and its type. """ meta = { "image_file": PurePosixPath(fov.relative_path).name, "source_relative_path": fov.relative_path, "position_id": fov.key[3], "technical_repeat": fov.key[2], - "image_read_transposed": False, } if run.mask_type is not None: meta["mask_file"] = mask_file_name(_record(run.inputs, fov, run.mask_type)) if config.mode == "cell" and config.common.object_set is None: meta["object_set"] = object_set_from_mask_type(run.mask_type) return meta + + +def validate_image_and_mask_folders( + image_folder_path: str | os.PathLike, + mask_folder_path: str | os.PathLike, + *, + preprocessing_run: str | os.PathLike | None = None, + mask_type: str | None = None, + **removed_options: object, +) -> dict[Path, Path]: + """Return the mask of every image of a clean preprocessing run, as the run paired them. + + Since v0.2.0 (design v0.4 §6) CellPatcher pairs no file itself: the run is opened with + ``RunInputs.open`` with *image_folder_path* as the experiment root and *mask_folder_path* as + its masks folder (where they are now), so every image and mask is checked against the run. + + Parameters + ---------- + image_folder_path + The experiment root, where the preprocessed experiment is now. + mask_folder_path + Where the run's masks folder is now. + preprocessing_run + The run's output folder (``common.preprocessing_run`` of a file run). + mask_type + The mask type (``matching.mask_type``); None: the run's only type. + **removed_options + The matching options removed in v0.1.1, refused by name. + + Returns + ------- + dict + Each image's path mapped to its mask's path (a label image's is inside its OME-Zarr image). + + Raises + ------ + PreprocessingCheckError + ``no_preprocessing_run`` without *preprocessing_run*, or when ``RunInputs`` refuses the run. + ValueError + For a mask type the run did not record, several types without *mask_type*, or an image + the run recorded no mask of the type for. + """ + from morphobricks_preprocessing.masks import FOLDER + + refuse_removed_matching("validate_image_and_mask_folders", removed_options) + if preprocessing_run is None: + _refuse_without_run("validate_image_and_mask_folders", "preprocessing_run=") + inputs = open_run(preprocessing_run, experiment_root=image_folder_path, masks_root=mask_folder_path) + chosen = _mask_type(mask_type, inputs, "validate_image_and_mask_folders") + pairs = {} + for fov in inputs.fovs(): + record = _record(inputs, fov, chosen) + root = inputs.masks_root if record["source"] == FOLDER else inputs.experiment_root + pairs[inputs.experiment_root / fov.relative_path] = root / str(record["path"]) + return pairs diff --git a/src/cellpatcher/ids.py b/src/cellpatcher/ids.py index f76dada..2f49e42 100644 --- a/src/cellpatcher/ids.py +++ b/src/cellpatcher/ids.py @@ -100,12 +100,6 @@ def object_set_from_mask_type(mask_type) -> str | None: return re.sub(r"[^A-Za-z0-9._-]", "_", str(mask_type)).strip("_") or None -def object_set_from_label_path(mask_file) -> str: - """The label image name of an OME-NGFF mask ``/labels/`` (``io.read_ngff_mask``).""" - name = str(mask_file).rsplit("/", 1)[-1] - return re.sub(r"[^A-Za-z0-9._-]", "_", name).strip("_") or DEFAULT_OBJECT_SET - - def resolve_object_set(meta, common) -> str: """The ``object_set`` of a cell-mode image. diff --git a/src/cellpatcher/io.py b/src/cellpatcher/io.py index 70aeffa..afe5676 100644 --- a/src/cellpatcher/io.py +++ b/src/cellpatcher/io.py @@ -1,11 +1,8 @@ from __future__ import annotations -import os import warnings -from collections.abc import Iterable from dataclasses import dataclass from pathlib import Path -from typing import TYPE_CHECKING import numpy as np from morphobricks_normalization.records import SINGLE @@ -13,9 +10,6 @@ from cellpatcher._deprecation import accept_deprecated_repeat from cellpatcher.axes import normalize_axes -if TYPE_CHECKING: - from morphobricks_preprocessing import TiffMetadata - # ".zarr" is an OME-Zarr image: a folder, read with the preprocessing brick's # reader and the ``zarr`` extra (docs/normalization_contract.md §2); every other # extension is a file. @@ -271,26 +265,6 @@ def discover_images(root_path, *, mask_folder_path=None): return result -def discover_folder_images(image_folder_path, *, mask_folder_path=None) -> list[Path]: - """The images of a single image folder (a folder run), found by :func:`discover_images`. - - The folder must hold its images directly. A folder holding subfolders - of images is a tree: run it with - :func:`~cellpatcher.orchestration.run_patching_condition_repeat_tree`. - Returns an empty list when the folder holds no image. - """ - folder = Path(image_folder_path).expanduser() - result = discover_images(folder, mask_folder_path=mask_folder_path) - nested = [leaf.relative_path for leaf in result.leaves if leaf.relative_path != "."] - if nested: - raise ValueError( - f"The image folder '{folder}' holds its images in subfolders ({', '.join(nested[:10])}" - f"{', ...' if len(nested) > 10 else ''}). A folder run reads the images directly in one " - "folder; run a tree of condition and repeat folders with run_patching_condition_repeat_tree." - ) - return sorted(folder / path for path in result.images) - - def discover_condition_repeat_folders( root_path, *, @@ -326,7 +300,6 @@ def discover_condition_repeat_folders( When *mask_root_path* is given, the same leaf path is mirrored under it for each leaf's ``mask_folder``; its existence is not checked here. - The pairing itself reads the whole mask root (:func:`pair_masks`). *supported_extensions* is no longer supported: the images are the brick's (TIFF and OME-Zarr). Any value other than None is refused. @@ -499,123 +472,6 @@ def read_ome_zarr(path) -> tuple[np.ndarray, str]: return pixels, image.axes -def read_ngff_mask(path, label_name: str | None, axes: str) -> tuple[np.ndarray, str]: - """Read an OME-Zarr image's label image as its mask, put into *axes* by role. - - Follows ``docs/normalization_contract.md`` §4: the label image comes from - the image's ``labels`` group (*label_name* may be None when it has exactly - one), level 0 is read, its values must be integers, and each of its - dimensions must equal the image's or have length 1 (a channel axis, length - 1). A label Z or T axis with the image's length then follows the image's - planes, and one of length 1 is shared (``common.z_policy``/``t_policy``). - - Returns - ------- - tuple - The mask in *axes*, and its ``mask_file`` name, - ``/labels/