Repository navigation
CP-1: read the preprocessing run through RunInputs (preprocessing v0.4.0rc1) - #68
Conversation
…4.0rc1) Pin morphobricks-preprocessing to v0.4.0rc1 (normalization stays at v0.3.2, the tag rc1 pins) and open the run that wrote plane_histograms.csv with the brick's RunInputs (design v0.4 §6): - load_plane_statistics opens the table's output folder with RunInputs.open: a run before v0.4.0, a table or design not the manifest's, and an image or mask changed since the run are refused (preprocessing_run_refused). CellPatcher's own version pin (manifest "1" only) and its table SHA check are gone; a table without its manifest is refused, no longer warned about. - normalization.manifest / experiment_design may name only the run's own files (not_the_run_output); the biological_repeat scope refuses a run that recorded no design (missing_experiment_design), instead of a warning. - run_inputs() is replaced by normalization_inputs(), built from the opened run (run_config.json keys unchanged; the manifest version is now "2"). - recorded_image_type reads the image type through RunInputs. Forced by the rc1 pin (its removed check_declared_axes and find_masks' ome_zarr_labels): io.header_axes_disagreement checks declared axes with the brick's check_mapping/header_axis_issue, so a Y/X swap is refused like any other contradiction and nothing is read transposed; pair_masks calls the new find_masks. The pin test checks the tag each brick was installed from (PEP 610 direct_url.json), as the rc1 tag still reports version 0.3.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
Move the RunInputs tests added by CP-1 from test_statistics.py (624 lines, 559 on main) to a new tests/test_run_inputs.py, and shorten the comment that took test_io.py to 602 lines (design v0.4 §9: files at most 600 lines). No test changes behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
…pository and tag The pin test fell back to comparing the reported version with the tag when an install recorded no git revision. For a local or editable install of a checkout reporting the tag's number this was a false pass, and for the v0.4.0rc1 tag (which reports 0.3.4) a failure without its reason. It now fails naming the install (no direct_url.json, a local folder, editable or not), and also checks the repository URL, so a fork's tag of the same name does not pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
_declared_axes_problem rebuilt review_axes for one file: check_mapping, then header_axis_issue, then axis_check_issues (a name preprocessing does not export). Call review_axes itself and keep its errors, so CellPatcher holds no composition of the brick's axis rule. Same refusals and messages; the unreadable-header and dimension-count refusals stay, as review_axes leaves an unreadable file out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
refuse_header_axes, refuse_image_header_axes and _warn_transposed were rewritten or redocumented by CP-1 without annotations (design v0.4 §9). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
Preprocessing v0.4's find_masks lost ome_zarr_labels=False and always pairs
an OME-Zarr image's own label images (design v0.4 §2). pair_masks took all of
them as mask-folder types: a folder run with one TIFF mask type next to a
store holding a label image was refused ('the mask folder holds 2 mask
types'), and choosing the label's type joined the label's path to the mask
folder and failed on a file that does not exist with an unrelated message.
The types to choose from are now the mask folder's files; a label-only type
is refused naming mask_source='ngff_labels'. A label image and a folder mask
of the same type for one image stay preprocessing's duplicate_mask refusal,
now documented. Regression test fails on 56310c3 for the first two cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
… warnings docs/normalization.md still listed the two warnings CP-1 removed (a CSV scope without run_manifest.json; a manifest without experiment_design_sha256) and had no entry for the new refusals. The normalization contract still said the manifest is never required, looked up any design or manifest, and read manifest version 1. Both now describe reading the run through RunInputs (preprocessing_run_refused, not_the_run_output, missing_experiment_design), and the configuration reference's plane_histograms row no longer names the v0.3.0 table as the one read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc
|
Independent review, then merged with a merge commit (27fa182). The branch is kept. What I checked
Fixes pushed to the branch
Gate (Python 3.11, uv venv, rc1)
Left for later PRs (listed in the PR description)
Generated by Claude Code |
CP-1 of the CellPatcher v0.2.0 plan (
docs/restart/preprocessing-design-v0.4.md§6 and §8 on MorphoFeatureclaude/restart-doc). CellPatcher now reads the preprocessing run's table, design and manifest through the brick'sRunInputs, and is pinned tomorphobricks-preprocessing@v0.4.0rc1(bd4a5dc). Normalization stays atv0.3.2, the tag rc1 pins. This PR does not depend on preprocessing PR #113 (PP-7).Behaviour changes
RunInputs.open.load_plane_statistics, and with itcached_statistics, the normalizer andrecorded_image_type, opens the table's output folder this way. The following are refused withpreprocessing_run_refused, quoting the brick's reason:SUPPORTED_MANIFEST_VERSION = "1",SUPPORTED_SCHEMA_VERSION, the check in_preprocessing());_warn_unchecked_inputs.run_manifest.jsonis now refused. It used to run with a warning.normalization.plane_histograms,normalization.manifestornormalization.experiment_designnames any file other than the run's own in its output folder, it is refused withnot_the_run_output. Naming the run's own file still works.biological_repeatscope needs a recorded design. A run that recorded no design is refused withmissing_experiment_design. It used to read whatever design was next to the table, with a warning. The design is still read with the brick's strictread_confirmed_design, against the SHA-256 in the manifest.run_inputs()is replaced bynormalization_inputs(), built from the opened run. Therun_config.jsonkeys are unchanged, somanifest_versionis now"2".check_declared_axes/declared_axes_transposed.io.header_axes_disagreementnow checks declared axes with the brick'sreview_axes(check_mapping+header_axis_issue).axes_yx_swapped) and never read transposed.image_read_transposedis alwaysfalsefor TIFFs until CP-2b removes it.pair_maskscalls the newfind_masks(rc1 droppedome_zarr_labelsand always pairs an OME-Zarr image's own label images). A mask folder run still chooses among the mask folder's types only; a label-only type is refused, pointing tomask_source="ngff_labels". A label image and a folder mask of one type for one image are preprocessing'sduplicate_mask(new refusal, documented).image_named_as_mask, an error since v0.4).tests/test_dependency_pins.pynow checks the git repository and tag each brick was installed from (PEP 610direct_url.json), because the rc1 tag still reports version0.3.4. A non-git install (nodirect_url.json, or a local folder, editable or not) fails and names the reason. It never falls back to the version number.configuration_reference.mdrows, AGENTS.md §5, the config docstrings,docs/normalization.md(the removed warnings and the new refusals),docs/normalization_contract.md(§1, §5, §11, §13, §14) and the notebook CHANGELOG.Tests
New or rewritten regression tests. 14 of them fail on old
main(source checked out fromorigin/main, run against rc1):test_statistics.py:test_cellpatcher_pins_no_table_or_manifest_version_of_its_owntest_load_opens_the_run_through_run_inputstest_load_refuses_a_table_without_its_manifesttest_load_refuses_a_manifest_other_than_the_run_stest_load_refuses_a_renamed_tabletest_load_refuses_a_run_before_v040test_load_refuses_a_run_whose_image_changed_sincetest_a_clean_run_normalizes_a_tree_run_through_run_inputs, which usesmorphobricks_preprocessing.testing.make_clean_runtest_normalization.py:test_a_changed_image_is_detected_by_its_fingerprint_and_by_its_pixelstest_a_run_without_a_design_is_refused_at_the_repeat_scopetest_csv_scopes_refuse_a_table_without_its_manifesttest_a_run_that_recorded_no_design_is_refused_at_the_repeat_scopetest_rgb_normalization.py::test_recorded_image_type_is_read_through_run_inputstest_declared_axes.py::test_y_and_x_declared_swapped_are_refused_never_read_transposedOther tests were updated for the rc1 messages and codes.
The RunInputs tests now live in
tests/test_run_inputs.py, which keepstest_statistics.pyunder 600 lines. New in review:test_ome_zarr_images.py::test_tiff_masks_are_paired_beside_the_stores_own_label_images, where 2 of 4 cases fail on 56310c3, and the pin test's failure-reason cases.Gate after review (Python 3.11, uv venv, rc1):
pytestwith.[test,zarr]: 1003 passed, 7 skipped. All 7 skips are torch tests. CI is the same.pytestwith.[test](CI): 980 passed, 30 skipped.ruff check src tests: 67 findings, all inmain's set (one I001 fewer). CellPatcher has no ruff config ortools/check.py.io.py1091→1090,statistics.py721→689,normalization.py837→836,orchestration.py939→938,config.py613→613. No test file crossed 600 lines.Left for later PRs
RunInputs(read_plane/read_mask,fovs()), with the experiment root as the run records it, or given for a moved experiment. The old arguments raise with a migration message.RunInputs.openchecks the images where the run recorded them, so an experiment moved since the run is refused.image_axes/mask_axesfor file runs,image_read_transposed(with_warn_transposedand the transposed plumbing),pair_masks/find_masksandids.py;image_axes.plan.md(historical) still describes the optional manifest;construct.yaml/environment.yamlinstallers.cached_statisticsreuses an openedRunInputswithin a process (keyed on the table and manifest). A mask, or an image that is not patched, changed after the first open in the same session is not re-checked. Patched images are re-checked bycheck_sourceson every run.check_sources) and per-plane extrema (check_planes), not throughRunInputs.tools/code_rules.pyand its baseline were not adopted, because this PR did not need them. CellPatcher still has notools/check.pyor ruff config.Generated by Claude Code