Skip to content

CP-1: read the preprocessing run through RunInputs (preprocessing v0.4.0rc1) - #68

Merged
guijacquemet merged 8 commits into
mainfrom
claude/cp-1-run-inputs
Nov 2, 2026
Merged

guijacquemet merged 8 commits into
mainfrom
claude/cp-1-run-inputs

Conversation

@guijacquemet

@guijacquemet guijacquemet commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

CP-1 of the CellPatcher v0.2.0 plan (docs/restart/preprocessing-design-v0.4.md §6 and §8 on MorphoFeature claude/restart-doc). CellPatcher now reads the preprocessing run's table, design and manifest through the brick's RunInputs, and is pinned to morphobricks-preprocessing@v0.4.0rc1 (bd4a5dc). Normalization stays at v0.3.2, the tag rc1 pins. This PR does not depend on preprocessing PR #113 (PP-7).

Behaviour changes

  • The run is opened with RunInputs.open. load_plane_statistics, and with it cached_statistics, the normalizer and recorded_image_type, opens the table's output folder this way. The following are refused with preprocessing_run_refused, quoting the brick's reason:
    • a run written before v0.4.0 (v1 manifest);
    • a table or design whose SHA-256 is not the one in the manifest;
    • any image or mask of the run that has changed or gone missing.
  • Removed from CellPatcher:
    • its own version pin (SUPPORTED_MANIFEST_VERSION = "1", SUPPORTED_SCHEMA_VERSION, the check in _preprocessing());
    • its table SHA check and manifest reader;
    • _warn_unchecked_inputs.
  • The manifest is required. A table without run_manifest.json is now refused. It used to run with a warning.
  • Only the run's own files are read. If normalization.plane_histograms, normalization.manifest or normalization.experiment_design names any file other than the run's own in its output folder, it is refused with not_the_run_output. Naming the run's own file still works.
  • The biological_repeat scope needs a recorded design. A run that recorded no design is refused with missing_experiment_design. It used to read whatever design was next to the table, with a warning. The design is still read with the brick's strict read_confirmed_design, against the SHA-256 in the manifest.
  • run_inputs() is replaced by normalization_inputs(), built from the opened run. The run_config.json keys are unchanged, so manifest_version is now "2".
  • Forced by the rc1 pin:
    • rc1 removed check_declared_axes/declared_axes_transposed. io.header_axes_disagreement now checks declared axes with the brick's review_axes (check_mapping + header_axis_issue).
    • A Y/X swap is therefore refused (axes_yx_swapped) and never read transposed.
    • image_read_transposed is always false for TIFFs until CP-2b removes it.
    • pair_masks calls the new find_masks (rc1 dropped ome_zarr_labels and always pairs an OME-Zarr image's own label images). A mask folder run still chooses among the mask folder's types only; a label-only type is refused, pointing to mask_source="ngff_labels". A label image and a folder mask of one type for one image are preprocessing's duplicate_mask (new refusal, documented).
    • Discovery now refuses an image named like a mask of another image (image_named_as_mask, an error since v0.4).
  • Pin test. tests/test_dependency_pins.py now checks the git repository and tag each brick was installed from (PEP 610 direct_url.json), because the rc1 tag still reports version 0.3.4. A non-git install (no direct_url.json, or a local folder, editable or not) fails and names the reason. It never falls back to the version number.
  • Docs. The statements this PR made false are corrected in the configuration_reference.md rows, AGENTS.md §5, the config docstrings, docs/normalization.md (the removed warnings and the new refusals), docs/normalization_contract.md (§1, §5, §11, §13, §14) and the notebook CHANGELOG.

Tests

New or rewritten regression tests. 14 of them fail on old main (source checked out from origin/main, run against rc1):

  • test_statistics.py:
    • test_cellpatcher_pins_no_table_or_manifest_version_of_its_own
    • test_load_opens_the_run_through_run_inputs
    • test_load_refuses_a_table_without_its_manifest
    • test_load_refuses_a_manifest_other_than_the_run_s
    • test_load_refuses_a_renamed_table
    • test_load_refuses_a_run_before_v040
    • test_load_refuses_a_run_whose_image_changed_since
    • test_a_clean_run_normalizes_a_tree_run_through_run_inputs, which uses morphobricks_preprocessing.testing.make_clean_run
  • test_normalization.py:
    • test_a_changed_image_is_detected_by_its_fingerprint_and_by_its_pixels
    • test_a_run_without_a_design_is_refused_at_the_repeat_scope
    • test_csv_scopes_refuse_a_table_without_its_manifest
    • test_a_run_that_recorded_no_design_is_refused_at_the_repeat_scope
  • test_rgb_normalization.py::test_recorded_image_type_is_read_through_run_inputs
  • test_declared_axes.py::test_y_and_x_declared_swapped_are_refused_never_read_transposed

Other tests were updated for the rc1 messages and codes.

The RunInputs tests now live in tests/test_run_inputs.py, which keeps test_statistics.py under 600 lines. New in review: test_ome_zarr_images.py::test_tiff_masks_are_paired_beside_the_stores_own_label_images, where 2 of 4 cases fail on 56310c3, and the pin test's failure-reason cases.

Gate after review (Python 3.11, uv venv, rc1):

  • pytest with .[test,zarr]: 1003 passed, 7 skipped. All 7 skips are torch tests. CI is the same.
  • pytest with .[test] (CI): 980 passed, 30 skipped.
  • ruff check src tests: 67 findings, all in main's set (one I001 fewer). CellPatcher has no ruff config or tools/check.py.
  • §9 rules (ANN, C901, PLR0913, PLR0915, function length) are clean on every new and rewritten function.
  • Over-limit files did not grow: io.py 1091→1090, statistics.py 721→689, normalization.py 837→836, orchestration.py 939→938, config.py 613→613. No test file crossed 600 lines.

Left for later PRs

  • CP-2a: file runs read images and masks through RunInputs (read_plane/read_mask, fovs()), with the experiment root as the run records it, or given for a moved experiment. The old arguments raise with a migration message.
    • Until then, RunInputs.open checks the images where the run recorded them, so an experiment moved since the run is refused.
  • CP-2b:
    • remove image_axes/mask_axes for file runs, image_read_transposed (with _warn_transposed and the transposed plumbing), pair_masks/find_masks and ids.py;
    • a new run-config version that records the manifest SHA-256 in place of image_axes.
  • CP-3:
    • the notebooks: CellPatcher.ipynb still offers the manifest/design paths and words them as optional;
    • plan.md (historical) still describes the optional manifest;
    • the construct.yaml/environment.yaml installers.
  • Known gap until CP-2a:
    • cached_statistics reuses an opened RunInputs within a process (keyed on the table and manifest). A mask, or an image that is not patched, changed after the first open in the same session is not re-checked. Patched images are re-checked by check_sources on every run.
    • When CellPatcher's experiment root is a copy of the run's (the original still in place), the patched images are checked by size, time and axes (check_sources) and per-plane extrema (check_planes), not through RunInputs.
  • Code rules. tools/code_rules.py and its baseline were not adopted, because this PR did not need them. CellPatcher still has no tools/check.py or ruff config.

Generated by Claude Code

…4.0rc1)

Pin morphobricks-preprocessing to v0.4.0rc1 (normalization stays at v0.3.2,
the tag rc1 pins) and open the run that wrote plane_histograms.csv with the
brick's RunInputs (design v0.4 §6):

- load_plane_statistics opens the table's output folder with RunInputs.open:
  a run before v0.4.0, a table or design not the manifest's, and an image or
  mask changed since the run are refused (preprocessing_run_refused).
  CellPatcher's own version pin (manifest "1" only) and its table SHA check
  are gone; a table without its manifest is refused, no longer warned about.
- normalization.manifest / experiment_design may name only the run's own
  files (not_the_run_output); the biological_repeat scope refuses a run that
  recorded no design (missing_experiment_design), instead of a warning.
- run_inputs() is replaced by normalization_inputs(), built from the opened
  run (run_config.json keys unchanged; the manifest version is now "2").
- recorded_image_type reads the image type through RunInputs.

Forced by the rc1 pin (its removed check_declared_axes and find_masks'
ome_zarr_labels): io.header_axes_disagreement checks declared axes with the
brick's check_mapping/header_axis_issue, so a Y/X swap is refused like any
other contradiction and nothing is read transposed; pair_masks calls the
new find_masks.

The pin test checks the tag each brick was installed from (PEP 610
direct_url.json), as the rc1 tag still reports version 0.3.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
@guijacquemet
guijacquemet marked this pull request as ready for review November 2, 2026 06:01
Move the RunInputs tests added by CP-1 from test_statistics.py (624 lines,
559 on main) to a new tests/test_run_inputs.py, and shorten the comment
that took test_io.py to 602 lines (design v0.4 §9: files at most 600 lines).
No test changes behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
…pository and tag

The pin test fell back to comparing the reported version with the tag when an
install recorded no git revision. For a local or editable install of a checkout
reporting the tag's number this was a false pass, and for the v0.4.0rc1 tag
(which reports 0.3.4) a failure without its reason. It now fails naming the
install (no direct_url.json, a local folder, editable or not), and also checks
the repository URL, so a fork's tag of the same name does not pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
_declared_axes_problem rebuilt review_axes for one file: check_mapping, then
header_axis_issue, then axis_check_issues (a name preprocessing does not
export). Call review_axes itself and keep its errors, so CellPatcher holds no
composition of the brick's axis rule. Same refusals and messages; the
unreadable-header and dimension-count refusals stay, as review_axes leaves an
unreadable file out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
refuse_header_axes, refuse_image_header_axes and _warn_transposed were
rewritten or redocumented by CP-1 without annotations (design v0.4 §9).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
Preprocessing v0.4's find_masks lost ome_zarr_labels=False and always pairs
an OME-Zarr image's own label images (design v0.4 §2). pair_masks took all of
them as mask-folder types: a folder run with one TIFF mask type next to a
store holding a label image was refused ('the mask folder holds 2 mask
types'), and choosing the label's type joined the label's path to the mask
folder and failed on a file that does not exist with an unrelated message.

The types to choose from are now the mask folder's files; a label-only type
is refused naming mask_source='ngff_labels'. A label image and a folder mask
of the same type for one image stay preprocessing's duplicate_mask refusal,
now documented. Regression test fails on 56310c3 for the first two cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
… warnings

docs/normalization.md still listed the two warnings CP-1 removed (a CSV scope
without run_manifest.json; a manifest without experiment_design_sha256) and
had no entry for the new refusals. The normalization contract still said the
manifest is never required, looked up any design or manifest, and read
manifest version 1. Both now describe reading the run through RunInputs
(preprocessing_run_refused, not_the_run_output, missing_experiment_design),
and the configuration reference's plane_histograms row no longer names the
v0.3.0 table as the one read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
@guijacquemet
guijacquemet merged commit 27fa182 into main Nov 2, 2026
2 checks passed

Copy link
Copy Markdown
Contributor Author

Independent review, then merged with a merge commit (27fa182). The branch is kept.

What I checked

  • Science rules. The run is opened only through RunInputs.open, and every refusal stops the run. CellPatcher no longer has its own manifest reader, SHA check or version pin. Axes are checked with the brick's review_axes, so nothing is read transposed. patch_id is not parsed, and no silent fill or fallback was introduced.
  • Warnings that became refusals: a missing manifest, a design not recorded at the repeat scope, and a manifest or design other than the run's own. Each one is intentional, matches the owner decisions of 2026-11-02, and is now documented.
  • Regression tests. I ran the PR's tests against old main with its own deps (preprocessing v0.3.4): 29 fail, all of them CP-1 behaviour.
  • Renamed or removed names (run_inputs, SUPPORTED_*, check_declared_axes): nothing left in CellPatcher src, notebooks or docs calls them. The notebook uses recorded_image_type, which still exists, and catches PreprocessingCheckError. The local checkouts of MorphoFeature, MorphoRelate, MorphoEmbed, MorphoObjects and fiberfeatures don't use any of the removed names.

Fixes pushed to the branch

  • 67ff375: test_statistics.py had reached 624 lines and test_io.py 602. The RunInputs tests moved to tests/test_run_inputs.py.

  • 7b4321d: the pin test could pass falsely. It fell back to comparing version numbers for a non-git or editable install. It now requires a git install of the pinned repository and tag, and otherwise fails with the reason.

  • 5b9ca8a: the axis check rebuilt review_axes from its parts, including the unexported axis_check_issues. It now calls review_axes itself.

  • c7a1a87: added type annotations to the axis-check functions CP-1 rewrote (§9).

  • cc0e116: since rc1, find_masks always pairs OME-Zarr label images. Two mask-folder cases broke:

    • A folder run with one TIFF mask type next to a store that has a label image was refused: "the mask folder holds 2 mask types".
    • Choosing the label's type built a path that does not exist inside the mask folder.

    The mask types are now the mask folder's own, and a type that is only a label image is refused, pointing to mask_source="ngff_labels". A regression test covers this.

  • 2ef7818, 82d858f: normalization.md, normalization_contract.md (§1, §5, §11, §13, §14) and the configuration_reference.md row for plane_histograms now describe reading the run through RunInputs. Added a CHANGELOG entry.

Gate (Python 3.11, uv venv, rc1)

  • .[test,zarr]: 1003 passed, 7 skipped (all torch). CI: the same.
  • .[test] in CI: 980 passed, 30 skipped.
  • ruff on src and tests: 67 findings, all already on main.
  • No file over 600 lines grew.

Left for later PRs (listed in the PR description)

  • The statistics cache reuses an opened RunInputs within one session.
  • An experiment root that is a copy of the run's is checked with check_sources/check_planes rather than through RunInputs.
  • Both are covered by CP-2a's reads through RunInputs.
  • A label image and a folder mask of the same type for one image are now refused (duplicate_mask), as preprocessing v0.4 pairs them.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants