[guijacquemet]

CP-2b of the CellPatcher v0.2.0 plan (docs/restart/preprocessing-design-v0.4.md §6, §8 and §9 on MorphoFeature claude/restart-doc). It builds on CP-2a (#69, merged as 6a35426). It stays pinned to morphobricks-preprocessing@v0.4.0rc1 (bd4a5dc). After this PR, CellPatcher discovers, pairs and reads no file itself, for any run.

Behaviour changes

• run_config.json version 2 (cellpatcher.run_config_file):
  • It has run_config_version: 2.
  • When a preprocessing run was read (a folder or tree run, or a CSV normalization scope), it also has a preprocessing section:
    • run: the run's folder;
    • manifest_sha256: the SHA-256 of run_manifest.json, taken when this call opened the run. A manifest that changes during the call is refused: while it is opened, between two opens in one call, or before run_config.json is written (then no run_config.json is left; review fix);
    • for a folder or tree run, layout (image, mask): the order CellPatcher built the arrays in from the run's recorded roles.
  • A file run no longer records common.image_axes, mask_axes or its labels. They are the run's values, not settings, and each patch's labels are in patches.csv.
  • So a file run's run_config.json now loads back with load_config as a config a file run accepts. On main, the layout recorded as image_axes (for example 'CYX') and the leaf labels made the migration check refuse it.
  • load_config, read_patch_table and the exported-patch readers (iter_exported_patches, load_exported_patch, ExportedPatchTorchDataset) refuse any run_config_version other than 1 and 2. Version 1 means no version: a file written before v0.2.0, or a plain config file.
• validate_image_and_mask_folders(image_folder, mask_folder, *, preprocessing_run=, mask_type=):
  • It returns the pairing the run recorded. The run is opened with RunInputs.open where the two folders are now.
  • Without a run it is refused (no_preprocessing_run, with the command).
  • Also refused:
    • an image without a recorded mask of the type;
    • a type the run did not record, and several types with no mask_type;
    • a changed mask.
  • It moved to cellpatcher.file_runs and is still cellpatcher.validate_image_and_mask_folders.
• Lazy delivery:
  • New cellpatcher.iter_run_patch_arrays(patches, config, *, experiment_root=, masks_root=) delivers a folder or tree run's patches.
    • Each patch's image is the run's image at its source_relative_path. A patch whose image is not in the run, or whose FOV key differs from the recorded one, is refused.
    • Only the patch's plane is read, with RunInputs.read_plane; its mask is read with read_mask.
  • patch_array, iter_patch_arrays and PatchTorchDataset take arrays only. A path raises a TypeError naming iter_run_patch_arrays (PatchTorchDataset when it is built; review fix). On main, CellPatcher read the path itself, with common.image_axes and its own header check.
• Owner decision 3: run_patching_pipeline and run_patching_from_folders accept mask_folder_path in grid and whole_fov mode, as the tree run already did. It only says where the run's masks folder is now: the masks are checked there and never read. On main this was refused.
• Owner decision 4: new normalization.experiment_root/masks_root say where a moved experiment and its masks folder are now, for run_patching at the image or biological_repeat scope.
  • The run is opened there.
  • Folder and tree runs refuse these settings with the migration message: their arguments say where the experiment is.
• Removed (breaking, listed in the CHANGELOG):
  • images.csv loses its last column, image_read_transposed, and ImageRecord loses the field. Since preprocessing v0.4 it was always false or empty.
  • common.mask_source and common.mask_label_name are removed:
    • CommonConfig(mask_source=...) is a TypeError;
    • a config or run_config.json that holds their defaults (every file written before v0.2.0) still loads;
    • a file with any other value is refused, and the message names matching.mask_type.
  • Removed from cellpatcher.io:
    • pair_masks, MaskPairing and the find_masks header filtering;
    • read_image, conform_axes and read_ngff_mask;
    • refuse_image_header_axes, refuse_mask_header_axes, refuse_header_axes and header_axes_disagreement;
    • refuse_mask_channel_axis, source_relative_path and discover_folder_images.
  • Removed elsewhere:
    • cellpatcher.planes: iter_file_planes and read_file_plane, with the last transposition plumbing (plane.T for a file whose X came before Y);
    • cellpatcher.ids: object_set_from_label_path;
    • cellpatcher.pipeline: check_source_identities.
  • Moved:
    • run_config_data: from pipeline to cellpatcher.run_config_file;
    • cached_statistics: from cellpatcher.normalization to cellpatcher.run_open.
• Kept, because in-memory runs need them:
  • common.image_axes/mask_axes. Masks with X before Y are still refused, with refuse_transposed_mask_axes.
  • The labels.
  • ids.py: patch identity, and position_id from image_file. That uses preprocessing's position_id and only for in-memory images; file runs give the recorded position_id.
  • read_array.
  • discover_condition_repeat_folders/list_supported_files, which the notebook uses; no run uses them. discover_condition_repeat_folders wraps preprocessing's discover_experiment; list_supported_files lists one folder's entries itself (iterdir, with preprocessing's is_system_name), a listing helper for in-memory use.
• Not in this PR: wiring preprocessing's check for images that are not in the run. It is being added in preprocessing PR claude/pp-8b-brick-checks, which is not merged and not in rc1. It is wired once a preprocessing release candidate includes it. The known-limitation docs are unchanged.

Tests

• New or rewritten regression tests:
  • test_file_runs.py:
    • run_config v2 content, and loading the file back and running it again;
    • an unknown version refused by load_config, iter_exported_patches and read_patch_table (3 cases);
    • lazy delivery through the run for a moved experiment, with a foreign image and a foreign FOV key refused, and the path refusal;
    • normalization.experiment_root/masks_root for a moved experiment, and their refusal in a tree run.
  • test_io.py:
    • validate_image_and_mask_folders needs a run;
    • the run's pairing (3 cases);
    • a moved experiment, then a changed mask refused;
    • choosing a mask type, and an image without a mask;
    • a mask the run did not pair is refused, not dropped;
    • grid and whole_fov runs with a moved masks folder (2 cases).
  • Elsewhere:
    • test_types_and_config.py: the mask_source settings are removed, and old files still load;
    • test_declared_axes.py: no transposition column; a mask path refused;
    • test_plane_policies.py: a file's planes delivered through the run;
    • test_delivery.py: equality of the lazy arrays now via iter_run_patch_arrays.
• On old main: I ran this branch's tests against origin/main source (ce0379c).
  • 17 tests fail, plus 3 modules that cannot be imported (test_io, test_delivery, test_ome_zarr_images).
    • 3 of the 17 are test_dependency_pins artefacts of the PYTHONPATH swap. They pass normally.
    • test_an_opened_run_is_not_reused... fails only because of the moved cached_statistics import.
  • With the imports pointed back to main's locations, test_io/test_ome_zarr_images fail 11 more tests:
    • grid/whole_fov with a masks folder: main raises "not supported";
    • validate: main returns its own pairing, or rejects preprocessing_run= as an unknown option.
  • Example: on main, test_the_run_config_records... finds image_axes recorded and no version, and no reader checks a version.
• Removed, because their premise is gone:
  • CellPatcher's own pairing and discovery checks through validate_image_and_mask_folders. These are now the preprocessing run's checks:
    • empty image or mask folder;
    • an ambiguous name;
    • a mask named like its image;
    • a non-TIFF mask;
    • PNG/JPEG/microscope formats (12 parametrized cases);
    • images beside subfolders;
    • a masks folder inside the images;
    • macOS resource forks;
    • the three find_masks header-code tests, the deprecated-name test and the multi-axis mask advice (2 cases).
  • CellPatcher's own header-axis check for lazy delivery from a path: test_the_shared_rule_is_the_preprocessing_brick_s and test_other_disagreements_are_refused (3 cases).
  • Plain-stack YXC, conform_axes (2), the read_file_plane multi-series/multi-image tests (2) and test_mask_source_is_validated.
  • Where the behaviour is kept, no assertion was weakened. The file-run refusals of declared axes stay.

Gate (Python 3.11, uv venvs, rc1 bd4a5dc; review head e0137b1)

• pytest with .[test,zarr]: 988 passed, 7 skipped. All 7 skips need torch. The torch-gated modules (test_torch_dataset, test_delivery, test_biological_repeat) were also run with CPU torch 2.15: 54 passed.
• pytest with .[test]: 967 passed, 28 skipped: 21 need the zarr extra, 7 need torch.
• CI on e0137b1 gives the same counts and is green.
• ruff check src tests: 58 findings, against 62 on main. None is new; 4 of main's are gone.
• # noqa count: 1, the same as main.
• Author's figures on e61dd6e: 983 passed / 7 skipped and 962 / 28.

Review fixes (independent review, 2026-11-02)

• 001f059, ca77cff: run_config.json refuses a run_manifest.json that changed during the call. Before, only a change while RunInputs.open ran was refused, and that branch had no test. Now the SHA-256 is checked again when run_config.json is written; the data is built before the file is created, so a refusal leaves no file and no export. A second open in one call also refuses a manifest changed since the first. 3 regression tests, which fail on main.
• 00d5c4b: PatchTorchDataset refuses a path when it is built, not in a DataLoader worker. Regression test (torch).
• 35874de: the in-memory half of the removed test_lazy_delivery_refuses_a_mask_declared_swapped: an array mask declared XY in lazy delivery. Nothing covered it any more.
• e0137b1: a test of the version 2 run_config.json of an in-memory run (a CSV scope: run and SHA, no layout, axes kept; the patch scope: no section).
• 01390bb: annotations on the functions this PR changed (§9 ANN).
• 44e6c74: the CHANGELOG says plainly that the notebook is broken until CP-3, with no shim. It notes that the added-images check is wired after preprocessing v0.4.0rc2, and it names the downstream items for the brick migrations.

§9 figures (measured)

• Tools: ruff --select C901,PLR0913,PLR0915,ANN with limits 15, 8 and 50, plus an AST count of executable lines.
• New modules: run_config_file.py (182 lines) and run_delivery.py (104 lines).
  • Each declares its layer (orchestration) and has from __future__ import annotations.
  • Public names have NumPy docstrings.
• The rule check passes, ANN included, on:
  • run_config_file, run_delivery, run_session, run_open and file_runs;
  • the rewritten delivery._image_plane/_mask_plane/_refuse_file.
• Largest values among new and rewritten functions:
  • complexity 6 (run_config_data);
  • parameters 7 (run_patching_condition_repeat_tree, unchanged);
  • statements 16 (open_run, open_file_run, iter_run_patch_arrays, run_patching_condition_repeat_tree);
  • executable lines 33 (open_run, each line of a wrapped call counted).
• Files:
  • Files over 600 lines all shrank: io.py 1090→582, pipeline.py 1066→1030, normalization.py 832→804, orchestration.py 625→591, config.py 611→599, test_patch_identity.py 609→583.
  • Other changes: planes.py 389→243, test_io.py 592→406, file_runs.py 388→436, run_open.py 245→301, test_file_runs.py 242→344. No file grows past 600.
• No duplication:
  • The manifest SHA uses preprocessing's file_sha256 and MANIFEST_FILENAME.
  • Pairing, reading and label checks are all RunInputs calls.

Downstream (inspected, not edited)

• MorphoRelate (ea19bb0), io/cellpatcher.py:
  • check_image_axes reads common.image_axes from CellPatcher's run_config.json. A file run's v2 file no longer has it, so MorphoRelate reports "not checked" (axes_check_state); there is no error. Its migration PR should take the axes from the preprocessing run.
  • read_patch_table now refuses an unknown run_config_version.
• MorphoEmbed (a8e88df): provider_crops.py builds PatchTorchDataset([self.image], ...). That is fine if self.image is an array; a path would now be a TypeError.
• fiberfeatures (362f1ba):
  • Its notebook reads only mode and grid from data/experiment_001/reference/patch_config.json.
  • That file still holds mask_source/mask_label_name at their defaults. It loads with load_config, but CommonConfig(**common) would be a TypeError.
  • A regenerated reference (docs/make_example_results.py, asdict) would lose the two keys.
• MorphoFeature (4e6970f): its image_read_transposed test is MorphoFeature's own inputs.read_transposed. It uses no CellPatcher name that changed (border_labels, object_sides).
• MorphoObjects (ca548cd): it uses check_patch_bounds/check_region_mask only. It is unaffected.
• CellPatcher notebook: it still imports pair_masks/read_ngff_mask and cellpatcher.normalization.cached_statistics, and offers the removed settings, so its import cell fails until CP-3. It already could not run a file run without preprocessing_run since CP-2a.

Owner decisions (2026-11-02)

1. normalization.masks_root stays beside experiment_root, because a moved masks folder needs it.
2. CellPatcher's notebook may be broken on main until CP-3, the next PR. No shim is added; the CHANGELOG says so.
3. The check for unrecorded images is wired after preprocessing v0.4.0rc2, not here.

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

---

Generated by Claude Code

[guijacquemet]

Independent review: merged as 93394e6 (merge commit)

Checked against design v0.4 §6, §8 and §9 (MorphoFeature claude/restart-doc), with preprocessing pinned at v0.4.0rc1 (bd4a5dc).

What was verified

• No file discovered, paired or read by CellPatcher for a run. grep in src for tifffile, zarr, imread, glob, os.walk and iterdir. What remains:
  • read_array/write_array: exported patches and user arrays;
  • read_ome_zarr, via preprocessing's open_ngff, behind read_array;
  • discover_images/discover_condition_repeat_folders, which wrap discover_experiment;
  • list_supported_files: CellPatcher's own iterdir listing. No run uses it; it is kept for the notebook until CP-3. The PR text said it wraps discover_experiment; that is now corrected.
  • check_images: its manifest branch calls preprocessing's inspect_image/check_sources. Every caller passes table-only statistics, so no run reaches it.
• No transposition code left. No .T, and no swapaxes outside the advice text. The remaining np.transpose/moveaxis calls only put C, Y, X first; none swaps Y and X.
• run_config versions. v1 and v2 are read, and anything else is refused, by load_config, read_patch_table, iter_exported_patches, load_exported_patch and ExportedPatchTorchDataset. Old files with the default mask_source/mask_label_name load; other values are refused, naming matching.mask_type.
• mask_folder_path in grid/whole_fov is only passed to RunInputs.open (use_masks=False).
• iter_run_patch_arrays refuses a foreign image and a foreign FOV key.
• Removed tests: each of the 34 removed test names was checked against its stated premise. The moved tests keep their assertions.
• Regression tests fail on old main: I ran this branch's tests on main's source and got 17 failures and 3 modules that cannot be imported. My new torch test also fails on main.
• Downstream list checked against MorphoRelate ea19bb0, MorphoEmbed a8e88df, fiberfeatures 362f1ba, MorphoObjects ca548cd and MorphoFeature 4e6970f. It is accurate, and it is now in the CHANGELOG for the brick migrations.

Findings fixed (on the branch)

1. 001f059/ca77cff: only a manifest change during RunInputs.open was refused, and that path had no test. Now a change before run_config.json is written, or between two opens in one call, is refused too, and no file is left. 3 tests.
2. 00d5c4b: PatchTorchDataset gave the path TypeError only when an item was loaded, inside a DataLoader worker. It now refuses at construction. Test added.
3. 35874de: lazy delivery's refusal of an in-memory XY mask had lost its only test. Restored.
4. e0137b1: the v2 run_config.json of an in-memory run (CSV scope and patch scope) was untested. Test added.
5. 01390bb: annotations on the changed functions.
6. 44e6c74: the CHANGELOG now says plainly that the notebook is broken until CP-3, with no shim. It also covers the rc2 wiring and the downstream items.

Gate (head e0137b1)

• .[test,zarr]: 988 passed, 7 skipped (all need torch). The torch-gated modules were also run with CPU torch: 54 passed.
• .[test]: 967 passed, 28 skipped (21 need zarr, 7 need torch).
• CI: green with the same counts.
• ruff: 58 findings, against 62 on main, none new. # noqa: 1, the same as main.

§9 figures (measured)

Method: ruff --isolated --select C901,PLR0913,PLR0915,ANN (limits 15/8/50) on the changed src files, plus AST counts.

• No new finding. Against main: 314 → 257 findings, ANN001 271 → 225, C901 3 → 3, PLR0915 3 → 3.
• Largest among new and rewritten functions:
  • complexity 6;
  • parameters 7 (run_patching_condition_repeat_tree, unchanged);
  • executable lines 31 (open_run).
• One pre-existing exception: PatchTorchDataset.__init__ (10 parameters plus self, 39 statements, a public signature) was touched by fix 2 but did not grow.
• Files:
  • largest new file: run_config_file.py, 187 lines;
  • files over 600 lines all shrank or stayed the same: pipeline.py 1066→1031, normalization.py 832→804, io.py 1090→582, orchestration.py 625→592, config.py 611→599, test_normalization.py 951→951;
  • other changes: run_open.py 245→332, file_runs.py 388→436, test_file_runs.py 242→430.

Follow-up (not in this PR's scope)

• In-memory image_axes with X before Y. run_patching still accepts in-memory common.image_axes with X before Y (e.g. 'XY'). Only masks are refused. §6 asks for a canonical-only layout argument for in-memory arrays. This was already the case on main; suggested for CP-3 or an issue.
• For the brick migrations:
  • MorphoRelate's check_image_axes now reports "not checked" for file runs;
  • MorphoEmbed's PatchTorchDataset takes arrays only, which it already passes;
  • fiberfeatures' reference patch_config.json holds mask_source.

---

Generated by Claude Code