From 3b616a0a8791a0a93d5059e96c4cf311f717724d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 12:37:37 +0000 Subject: [PATCH 01/13] Prepare v0.5.0: version 0.5.0, notebook and README install v0.5.0 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- CHANGELOG.md | 10 +++++++++- README.md | 8 ++++---- notebooks/MorphoBricks_Preprocessing.ipynb | 4 ++-- notebooks/notebook_latest_versions.yaml | 2 +- pyproject.toml | 2 +- tests/test_notebook_setup.py | 12 ++++++------ 6 files changed, 23 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index be71a37a..73be657c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,15 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature, on b are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the [migration guide](docs/migration-0.3-to-0.4.md). -## Unreleased +## 0.5.0 - 2026-11-09 + +Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`): +the reader, the canonical writer, publication, reconciliation, the described run, semantic +`RunInputs` and the notebook that writes `morphobricks.toml`. Runs without a description keep +the v0.4.0 behaviour (with a warning). The release step: the version is 0.5.0, the notebook +installs `v0.5.0` (its `RELEASE_REF` is the release-preparation commit), and the Gate G1 record +of this release candidate is in `docs/conformance.md`. The normalization brick stays pinned at +v0.3.4. - **Notebook: folder pictures with a 1:1 crop, "Tag like the saved group", colour mask evidence (PR 17c).** *Images found* adds a picture of each folder's first image to the labels table, diff --git a/README.md b/README.md index 65948ac8..44c18fd3 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ from there. ## Status -Pre-alpha; the current release is v0.4.0 (what is planned next is in +Pre-alpha; the current release is v0.5.0 (what is planned next is in [docs/roadmap.md](docs/roadmap.md)). Implemented and tested: - Folder discovery and validation, TIFF and OME-Zarr header inspection without reading pixels @@ -54,7 +54,7 @@ and the [CHANGELOG](CHANGELOG.md). ## Installation ```bash -pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.4.0" +pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0" ``` For development, from a clone: @@ -64,7 +64,7 @@ pip install -e ".[dev]" ``` The optional `notebook` extra installs the widget dependencies for the notebook -(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.4.0"`). +(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0"`). The notebook's single runnable cell can set this up automatically: it uses a compatible nearby checkout or installed package, and installs missing requirements into the active kernel. A standalone download installs from GitHub if needed (internet access and Git required). @@ -216,7 +216,7 @@ ruff format . pytest ``` -Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.4.0 is in [docs/roadmap.md](docs/roadmap.md). +Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.5.0 is in [docs/roadmap.md](docs/roadmap.md). ## License diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 20fcccfa..60fdb5ef 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -62,11 +62,11 @@ "from pathlib import Path\n", "\n", "# LabConstrictor version identifiers.\n", - "current_version = \"0.4.0\"\n", + "current_version = \"0.5.0\"\n", "notebook_name = \"MorphoBricks_Preprocessing\"\n", "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", - "RELEASE_TAG = \"v0.4.0\"\n", + "RELEASE_TAG = \"v0.5.0\"\n", "RELEASE_REF = \"4b8a32e58985d0753c2bfb7e2998893a84185533\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", diff --git a/notebooks/notebook_latest_versions.yaml b/notebooks/notebook_latest_versions.yaml index 3d433554..5c6fb1b8 100644 --- a/notebooks/notebook_latest_versions.yaml +++ b/notebooks/notebook_latest_versions.yaml @@ -1,3 +1,3 @@ # Latest version of each notebook, read by LabConstrictor's version check by notebook_name. # Equal to the notebook's current_version and its RELEASE_TAG without the "v" (README, "Notebook"). -"MorphoBricks_Preprocessing": "0.4.0" +"MorphoBricks_Preprocessing": "0.5.0" diff --git a/pyproject.toml b/pyproject.toml index 19ee218a..98a4c84a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "morphobricks-preprocessing" -version = "0.4.0" +version = "0.5.0" description = "MorphoBricks preprocessing brick: per-plane intensity histograms and normalization methods for microscopy TIFFs." readme = "README.md" license = "MIT" diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index 74cdef8c..96fff613 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -70,7 +70,7 @@ def setup_environment(tmp_path, monkeypatch): installs=[], imports=[], versions={ - "morphobricks-preprocessing": "0.4.0", + "morphobricks-preprocessing": "0.5.0", "morphobricks-normalization": "0.3.4", "ipywidgets": "7.7.1", }, @@ -96,7 +96,7 @@ def install(command, *, check): assert check is True state.installs.append(command) state.available.update({*REQUIRED, ZARR}) - state.versions["morphobricks-preprocessing"] = "0.4.0" + state.versions["morphobricks-preprocessing"] = "0.5.0" monkeypatch.setattr(subprocess, "run", install) @@ -135,7 +135,7 @@ def checkout_at(path): package.mkdir(parents=True) (package / "__init__.py").write_text("", encoding="utf-8") (path / "pyproject.toml").write_text( - '[project]\nname = "morphobricks-preprocessing"\nversion = "0.4.0"\n', + '[project]\nname = "morphobricks-preprocessing"\nversion = "0.5.0"\n', encoding="utf-8", ) return path @@ -150,7 +150,7 @@ def test_setup_reuses_the_matching_installed_release(setup_environment, capsys): "morphobricks_preprocessing", ] out = capsys.readouterr().out - assert "Ready: preprocessing 0.4.0" in out + assert "Ready: preprocessing 0.5.0" in out assert "ipywidgets 7.7.1" in out @@ -165,7 +165,7 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) assert namespace["RELEASE_REF"] == "4b8a32e58985d0753c2bfb7e2998893a84185533" - assert "pinned v0.4.0 candidate" in capsys.readouterr().out + assert "pinned v0.5.0 candidate" in capsys.readouterr().out def test_setup_replaces_a_mismatched_installed_release(setup_environment): @@ -256,7 +256,7 @@ def test_an_explicit_development_override_uses_the_checkout_and_says_so( assert sys.path[0] == str(checkout / "src") assert f"commit {OTHER_COMMIT}, uncommitted changes" in out assert "DEVELOPMENT OVERRIDE is on" in out - assert "DEVELOPMENT OVERRIDE on" in out.split("Notebook 0.4.0")[1] + assert "DEVELOPMENT OVERRIDE on" in out.split("Notebook 0.5.0")[1] record = namespace["SETUP_RECORD"] assert record["checkout_commit"] == OTHER_COMMIT assert record["checkout_has_uncommitted_changes"] is True From c74d717e4f4f0a72be8b991435413adf4a2a1d37 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 12:37:42 +0000 Subject: [PATCH 02/13] Point the notebook at the v0.5.0 preparation commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- notebooks/MorphoBricks_Preprocessing.ipynb | 2 +- tests/test_notebook_setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 60fdb5ef..7029c0d8 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -67,7 +67,7 @@ "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", "RELEASE_TAG = \"v0.5.0\"\n", - "RELEASE_REF = \"4b8a32e58985d0753c2bfb7e2998893a84185533\"\n", + "RELEASE_REF = \"3b616a0a8791a0a93d5059e96c4cf311f717724d\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", "\n", diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index 96fff613..c137e76d 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -164,7 +164,7 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, target = command[5] assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) - assert namespace["RELEASE_REF"] == "4b8a32e58985d0753c2bfb7e2998893a84185533" + assert namespace["RELEASE_REF"] == "3b616a0a8791a0a93d5059e96c4cf311f717724d" assert "pinned v0.5.0 candidate" in capsys.readouterr().out From 4437067bf06339834bf08783e3c4c11e499ed744 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 13:05:48 +0000 Subject: [PATCH 03/13] Gate G1 record for the v0.5.0 release candidate; restore the PR 17 run's pin docs/conformance.md: the G1 record measured on c74d717 (Spec pin 1bed024): the whole suite, the Spec modules with no skip, P01-P35 with their tests, the golden CSVs, the notebook items, the four environment skips listed as not run, and the gaps against the conformance plan. The PR 17 run of 8 November is restored to the pin it ran with (b197ac1). Fixes #235 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/conformance.md | 168 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 155 insertions(+), 13 deletions(-) diff --git a/docs/conformance.md b/docs/conformance.md index 36dcaaba..38aba8bc 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -1,14 +1,161 @@ -# Conformance run record (Stage 1, PR 17) +# Conformance run records -The design (Stage 1, PR 17) names no place for this record, so it is kept here. Every row is a -real run; a skip is never counted as a pass. +The conformance plan of MorphoBricks-Spec (`docs/conformance-plan.md`, section 5) asks for the +test run to be recorded but names no place for the record, so it is kept here. Every row is a +real run; a skip is never counted as a pass. Owner rulings of 8 November 2026 (issue #235): +"no skips" applies to the Spec conformance modules (`tests/test_spec_*.py`); a skip elsewhere +in the suite is listed as **not run**, with its reason. -## Run of 8 November 2026 +## Gate G1 record: release candidate v0.5.0 (9 November 2026) + +| | | +|---|---| +| Release candidate | `c74d717e4f4f0a72be8b991435413adf4a2a1d37` (branch `claude/release-0.5.0`: `main` at `bf733ee` plus the v0.5.0 preparation `3b616a0` and the notebook's `RELEASE_REF` commit). The commit that adds this record changes only this file, so the code and tests are those of `c74d717` | +| Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d` (Spec `main`). Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | +| Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0, normalization 0.3.4); run as root on a case-sensitive file system | +| Commands | `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/` (the whole suite, with a per-test JUnit report), and `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules) | +| Whole suite | 5018 collected: 5014 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | +| Spec modules (`tests/test_spec_*.py`) | 1649 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | +| `tools/check.py` | Ruff and format clean, pytest 5014 passed and 4 skipped, code rules passed (exit status 0) | + +**How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1649 test cases +from `test_spec_*` modules, every one passed: none has a `skipped` (or xfail) element. (2) The +`-rsxX` summary of the run lists exactly the four skips below, none in a Spec module, and no +xfail or xpass. (3) All 21 modules of `EXPECTED_MODULES` have test cases in the report +(catalogue 376, derived design 79, discovery 11, encoding 58, gate 9, no fallback 15, pin 16, +publication 20, reader 330, reader phases 1-3 287, reader refusal paths 8, reconcile 25, +records 15, records goldens 76, refval cross-check 203, rules table 14, run aliases 15, run +description 9, RunInputs semantic 7, transitions 73, vendored missing 3), and the gate's +collection check (a deleted, emptied or skipped Spec module stops the run) did not fire. +(4) The gate itself (`tests/spec_gate.py`) turns a skip or xfail in those modules into a +failure, and its own tests (`test_spec_gate.py`, 9) passed. + +**Reference validator.** `test_spec_refval_crosscheck.py` (203 passed) compares the reader with +the vendored `refval.py` on every fixture; `test_spec_records_goldens.py` (76 passed) compares +`read_records` with the vendored `refrecords.py` on every records golden; the transitions oracle +(`test_spec_transitions.py::test_the_oracle_agrees_on_every_ordered_pair`) and the derived-CSV +oracle (`test_spec_derived_design.py`) passed. + +### Not run (4 environment skips, outside the Spec modules) + +| Test | Reason | +|---|---| +| `tests/test_cli_log.py:165` (`test_the_experiment_folder_in_another_case_is_refused`) | The file system tells names apart by case, so the case-only collision cannot be built | +| `tests/test_folder_rules.py:201` (`test_the_masks_folder_spelled_in_another_case_is_the_same_folder`) | Same: case-sensitive file system | +| `tests/test_folder_rules.py:229` (`test_output_folder_in_the_experiment_spelled_in_another_case`) | Same: case-sensitive file system | +| `tests/test_discovery.py:665` (`test_unreadable_folder_is_an_error`) | Run as root: file modes are ignored, so an unreadable folder cannot be made | + +These four behaviours are **not verified by this run**. + +### Physical scenarios P01-P35 (conformance plan section 3) + +"Passed" means every test named passed in this run. Where a test uses another fixture or a +synthetic tree than the plan names, or covers only part of the required outcome, the row says so. + +| # | Result | Tests (module: test functions) | +|---|---|---| +| P01 | Passed | `test_physical_check`: `test_p01_an_image_with_another_channel_order_is_not_homogeneous` (`v01`) | +| P02 | Passed | `test_physical_check`: `test_p02_an_image_with_one_channel_is_not_homogeneous_not_a_count_mismatch` (`v01`) | +| P03 | Passed | `test_physical_check`: `test_p03_no_case_folding_and_no_similarity` (`v11`, 2 cases) | +| P04 | Passed | `test_spec_run_description`: `test_p04_heterogeneous_layouts_are_refused_and_nothing_is_processed` (`v04`, `v05`, `v06`, `v17`), `test_p04_the_message_names_the_layouts_and_the_units_in_plain_words`; `test_heterogeneous_refusal_first` (9) | +| P05 | Passed | `test_spec_run_description`: `test_p05_binary_mask_with_values_0_to_5_is_refused_by_the_mask_step_not_the_run`, `test_p05_an_instances_mask_holding_only_0_and_1_is_accepted` (`v08`) | +| P06 | Passed, partly | `test_spec_run_aliases`: `test_p26_images_without_header_names_need_no_evidence_v12`; `test_physical_check`: `test_an_image_with_names_when_the_channel_has_none_contradicts` (`v12`); `test_physical_check`: `test_p23_pixels_that_look_like_rgb_are_never_examined` (no check reads pixels). **No test builds two unnamed units with physically swapped fluorophores**; the "preview + confirmation only" part is the notebook's channel grid (`test_notebook_channels`, 30) | +| P07 | Passed | `test_spec_reconcile`: `test_p07_a_new_technical_repeat_is_pending_and_reopens_only_its_repeat` (`v13`), `test_p07_after_the_user_assigns_the_unit_and_confirms_the_new_date_is_accepted` (`v06`) | +| P08 | Passed | `test_spec_reconcile`: `test_p08_a_renamed_folder_is_removed_plus_new_and_never_paired`, `test_p08_the_removal_is_one_grouped_question_and_publishes_only_when_answered` (`v03`) | +| P09 | Passed | `test_description_file`: `test_p09_a_symlinked_description_is_refused_by_the_reader`, `test_p09_a_dangling_symlink_is_refused_not_missing`, `test_p09_publication_refuses_a_symlink_and_neither_follows_nor_replaces_it` (`v01`) | +| P10 | Passed | `test_physical_check`: `test_p10_an_undeclared_mask_type_is_refused_and_a_declared_empty_one_is_valid` (`v07`) | +| P11 | Passed | `test_spec_run_description`: `test_p11_masks_for_none_some_then_all_images_are_valid_and_completeness_is_recorded` (`v09`) | +| P12 | Passed | `test_physical_check`: `test_p12_a_declared_unit_with_every_image_excluded_is_not_found` (`v01`) | +| P13 | Passed | `test_physical_check`: `test_p13_rgb_units_declared_rgb_match_the_header_and_are_header_verified` (`v28`), `test_a_format_without_a_colour_interpretation_is_recorded_as_not_header_verified` | +| P14 | Passed | `test_spec_run_inputs_semantic`: `test_p14_a_draft_is_never_read`, `test_p14_a_published_file_with_an_unconfirmed_repeat_is_refused`, `test_p14_a_description_changed_since_the_run_is_refused` (`v01`) | +| P15 | Passed | `test_spec_reconcile`: `test_p15_new_fovs_in_an_existing_unit_keep_the_confirmation_and_its_date` (`v13`), `test_p15_a_changed_channel_list_reopens_with_the_date_kept` (`v01`) | +| P16 | Passed | `test_spec_publication`: `test_p16_a_changed_id_is_refused_and_t02_also_drops_a_used_id`, `test_p16_t01_is_accepted_only_with_a_recorded_reconfirmation`, `test_p16_a_display_name_change_is_accepted_and_invalidates_nothing` (`v01`, `t01`, `t02`, `v16`) | +| P17 | Passed | `test_spec_publication`: `test_p17_a_second_draft_from_the_same_base_is_refused_and_not_merged` (`v01`, `v21`, `v16`; includes re-deriving B from A's file) | +| P18 | Passed | `test_spec_publication`: `test_p18_publishing_over_nothing_with_the_absent_base_is_accepted`, `test_p18_another_file_appears_before_publication`, `test_p18_a_draft_without_a_usable_base_is_refused_and_nothing_is_published` (8 cases) | +| P19 | Passed | `test_spec_publication`: `test_p19_a_retired_id_declared_again_is_refused`, `test_p19_also_when_the_returning_folder_is_the_one_the_retired_id_had` (`v25`, `t03`) | +| P20 | Passed | `test_spec_publication`: `test_p20_a_dropped_used_id_is_refused` (`v25`, `t04`), `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation` (on `v01`) | +| P21 | Passed | `test_spec_discovery`: `test_p21_a_decomposed_folder_is_refused_with_a_description`, `test_p21_a_composed_folder_is_accepted`, `test_p21_without_a_description_discovery_is_as_in_v0_4`, `test_p21_preflight_carries_the_refusal` (synthetic tree) | +| P22 | Passed | `test_spec_publication`: `test_p22_confirmed_on_stays_byte_for_byte_through_an_unrelated_publication`, `test_p22_a_new_date_on_an_unchanged_repeat_is_refused` (`t05`), `test_p22_after_a_reopening_and_a_new_confirmation_the_new_date_is_accepted` (`v01`) | +| P23 | Passed | `test_physical_check`: `test_p23_a_unit_declared_rgb_whose_images_are_two_channel_grayscale` (`v28`), `test_p23_an_rgb_tiff_in_a_unit_that_does_not_declare_it` (`v01`), `test_p23_pixels_that_look_like_rgb_are_never_examined` | +| P24 | **Not run** | The outcome is a brick's: a request through a provider manifest with or without `rgb_support`. Preprocessing has no provider manifest; it only exports the error class (`test_a_error_classes`: `A_RGB_NOT_SUPPORTED` class, code and pickling, passed). No test of (a)-(d) exists here | +| P25 | Passed | `test_spec_run_inputs_semantic`: `test_p25_each_fov_carries_the_source_label_and_the_added_repeat_group` (`v03`, `v30`), `test_p25_in_v30_grouping_by_the_source_label_would_merge_different_repeats` | +| P26 | Passed | `test_spec_run_aliases` (9 `test_p26_*`), `test_physical_check` (`test_p26_*`, 2 functions), `test_spec_discovery` (`test_p26_the_header_names_are_recorded_as_read_untrimmed`, `test_p26_the_manifest_records_the_header_name_as_read`): 19 cases (`v11`, `v31`, `v12`) | +| P27 | **Not run** | The outcome is a brick's: an object-set request with or without `binary_mask_policy`. Preprocessing only exports `A_MASK_NOT_INSTANCES` (`test_a_error_classes`, passed) and refuses `binary_mask_policy` in the file (`i112`, `E_FORBIDDEN_FIELD`, through the reader tests). (a)-(c) are not tested here | +| P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` | +| P29 | Passed | `test_spec_discovery`: `test_p29_na_and_a_leading_space_are_invalid_mask_type_names`, `test_p29_a_decomposed_mask_type_is_not_nfc`, `test_p29_the_three_cases_together`, `test_p29_valid_mask_types_are_accepted`, `test_p29_without_a_description_mask_types_are_accepted_as_in_v0_4` (synthetic tree) | +| P30 | Passed, partly | `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date`; golden `v13` (below). **Not tested:** the derived CSV of the two-date group compared with the legacy file (differing only in the earlier date) | +| P31 | **Not run** | The outcome is MorphoCoverage's (prefill, confirmation, refusal of an unconfirmed prefill). Preprocessing's part passed: editing `target` changes no digest (`test_description_write`: `test_a_display_name_target_or_treatment_key_changes_no_digest`; `v27` valid with no warning in the reader tests) | +| P32 | Passed | `test_draft`: `test_p32_forbidden_folders_get_no_proposal_and_nothing_is_adopted`, `test_p32_ordinary_folders_are_proposed_as_their_labels`; `test_spec_derived_design`: golden `v33`, `v34` and `test_the_file_names_folders_never_ids` | +| P33 | Passed | `test_spec_reconcile`: `test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id`, `test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats`, `test_p33_c_the_old_folder_removal_needs_its_confirmation_and_the_id_stays_used` (`v33`); `test_condition_folder_reopens` (2) | +| P34 | Passed, on other fixtures | `test_spec_records`: `test_a_changed_image_count_writes_the_old_and_new_counts_with_recorded_on` (no reopening, `confirmed_on` kept), `test_several_changed_members_are_written_in_condition_then_repeat_order`, `test_an_image_count_change_needs_a_date_and_writes_nothing_without_one`, `test_a_removal_and_a_count_change_in_one_publication_write_both_in_order` (RR3). They use `v06` and `v13`, not the `v01` the plan names | +| P35 | Passed, on other fixtures | `test_spec_records`: `test_a_removed_unit_writes_one_unit_record_naming_all_six_keys` (with its `reconfirmation` lines), `test_a_removed_repeat_writes_the_repeat_then_each_of_its_units`, `test_a_removed_condition_writes_the_condition_then_each_of_its_units`, `test_a_removed_technical_repeat_names_its_id_and_folder_then_its_unit`, `test_a_dropped_unit_without_a_removal_is_not_published`, `test_nothing_is_written_when_publication_is_refused`; `test_spec_publication`: `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation`. They use `v01`, `v03`, `v06`, not `v25` with `t07` (`t07` itself passes in `test_spec_transitions`) | + +### Golden CSV comparisons (conformance plan section 2, "Derived CSV") + +`tests/test_spec_derived_design.py`, 79 passed: + +- `test_the_golden_files_are_the_three_the_catalogue_lists`: `fixtures/expected/` holds exactly + `v13`, `v33`, `v34`; +- `test_the_derived_file_equals_the_golden_file_byte_for_byte`: `v13`, `v33`, `v34`, byte for byte; +- `test_the_derived_file_equals_the_oracles_on_every_valid_fixture` and + `test_the_derived_file_is_read_back_by_the_strict_reader`: every valid fixture, against + `refval.derive_csv` and through `read_confirmed_design`; +- `test_the_file_names_folders_never_ids`: `v33` carries the folder label `control`, never `ctrl`. + +Legacy parity with the `v0.4.0` tag (`tests/test_parity_v0_4_0.py`, 24 passed, the tag's `src` +run in a child Python): `plane_histograms.csv` and `experiment_design.csv` byte-identical, +identities and manifest entries equal, a v0.4.0 output folder reused with zero rows recomputed, +and the derived design equal to the bytes v0.4.0 wrote. The goldens for the other valid +fixtures are "to be added" in the Spec (conformance plan section 2); the oracle comparison covers +them meanwhile. + +### Notebook items (roadmap Gate G1; Stage 1 design section 5, items a, b, d) + +All headless (ipywidgets driven by `click()` and `.value`) and in a real kernel; all passed. + +| Item | Tests | +|---|---| +| (a) The notebook starts from raw folders | `test_notebook_check_save` (14): from a folder tree only, every answer, one click publishes `morphobricks.toml` and only then writes `experiment_design.csv` (`test_the_csv_is_written_only_at_save`, `test_save_says_saved_and_enables_run`); `test_notebook_stepper` (13); `test_notebook_description` (25); `test_notebook_channels` (30); `test_notebook_masks_screen` (9); `test_preview` (22); `test_notebook_file` (6, including `test_notebook_runs_in_a_kernel`) | +| (b) The confirmed file reloads and reconciles | `test_notebook_check_save`: `test_reopening_with_nothing_changed_folds_every_screen`, `test_a_folder_that_joins_a_replicate_asks_same_way_again_for_it_only`, `test_a_removed_folder_needs_yes_remove_before_saving` (writes the `removed` record), `test_the_draft_carries_the_pairing_across_a_reopen`; `test_notebook_row_fold` (6); `test_notebook_tag_like` (6); `test_notebook_stepper`: `test_a_reopened_experiment_opens_on_the_axes_its_only_open_step`, `test_a_changed_reopen_turns_the_affected_steps_orange` | +| (c) Legacy exports, identities and caches preserved | `test_parity_v0_4_0` (24, above); `test_notebook_check_save::test_without_describe_the_v04_cards_stay` | +| (d) Heterogeneous refusal explicit and tested | `test_heterogeneous_refusal_first` (9, including `test_the_check_and_save_list_leads_with_the_layout_refusal`); P04 above | +| Shipped notebook install | `test_notebook_file::test_remote_install_is_pinned_to_one_exact_commit`, `test_version_metadata_matches_labconstrictor_index`; `test_notebook_setup` (18, including `test_setup_installs_missing_package_from_the_exact_commit`) | + +**Not part of this record:** the browser smoke check of `docs/colab-release-check.md` (a +fresh Google Colab runtime and JupyterLab/LabConstrictor). Headless tests cannot show that a +browser renders the widgets; that check is the owner's, before the tag. + +### Gaps against the conformance plan + +The plan's exit criterion (section 5) asks that every section 2 test and every section 3 +scenario run and pass. This run does not meet it in full: + +- **P24, P27, P31 are not run**: their outcomes belong to the consuming bricks and MorphoCoverage + (conformance plan section 4 puts `A_RGB_NOT_SUPPORTED`, `A_MASK_NOT_INSTANCES` and the + MorphoCoverage prefill in Stage 2), and preprocessing has no code that could run them. +- **P06 and P30 are partly covered** (see their rows); **P34 and P35** pass on other fixtures + than the ones the plan names. +- Section 2, "Provenance": `i99` is refused (`E_FORBIDDEN_FIELD`), but no run-manifest test + records lab-tool links: preprocessing has no lab-tool provenance input yet. +- Section 2, "`confirmed_on`": no writer test with a frozen clock moved by a day; the writer + writes the model's date (`test_description_write::test_confirmed_on_is_written_as_the_model_holds_it`) + and records take the user's date, not the clock + (`test_spec_records::test_the_date_of_each_record_is_the_users_confirmation_not_the_clock`). + +So this record does not by itself declare any reader conforming; deciding Gate G1 with these +gaps is the owner's. + +## Run of 8 November 2026 (Stage 1, PR 17) + +Kept as it was run. Issue #235: this section had been relabelled to later Spec pins (`2975e0e`, +`c8f6852`, `1bed024`) without a new run. The record was written in commit `7418da1`, whose +`tests/spec_fixtures/PIN.toml` and the record's own first text both name `b197ac1`, so that is +the pin restored here. The run on the current pin is the G1 record above. | | | |---|---| | Base commit | `68b2a4b` (`origin/main`, merge of PR 211); the run included the changes of PR 17 | -| Spec pin | the full SHA in `tests/spec_fixtures/PIN.toml` (vendored `experiment-v1.md` at `1bed024`) | +| Spec pin | MorphoBricks-Spec `b197ac1` (`tests/spec_fixtures/PIN.toml` of that commit) | | Command | `python tools/check.py` (Ruff, Ruff format, then `pytest -q` over the whole `tests/` folder), then `python tools/check.py --rules` | | Whole suite | 4324 passed, 4 skipped, 0 failed (301 s) | | Spec modules (`tests/test_spec_*.py`) | 1535 passed, 0 skipped, 0 failed (the Spec gate fails a skip or xfail there) | @@ -20,11 +167,6 @@ The 4 skips are outside the Spec modules, each an environment limit of the machi the file system tells names apart by case, so the case-only collision cannot be built; - `tests/test_discovery.py:665`: the run had permissions that ignore file modes (root). -These four behaviours are therefore **not verified by this run**. - -## Not covered - -- The Spec's `fixtures/expected/` golden CSVs and the conformance items that need PRs 13 to 16 - (notebook cards) are outside this run; this is not a release-candidate sign-off for those. -- `describe --publish` publishes nothing from a draft of folders (see the pull request's open - questions), so no end-to-end publication through the command line is part of this record. +These four behaviours were therefore **not verified by that run**. Not covered by it: the Spec's +golden CSVs and the notebook items (PRs 13 to 16), and an end-to-end publication through the +command line (`describe --publish` publishes nothing from a draft of folders). From a6a0343d0da8ecb1bbdc4a3897a92a808082076d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 13:46:11 +0000 Subject: [PATCH 04/13] Close the cheap Gate G1 gaps with tests: P06, P30, P34, P35, P28, P33, confirmed_on, v27 Tests only; src and pyproject.toml are unchanged, so RELEASE_REF 3b616a0 stays valid. - P06: two unnamed v12 units with physically swapped fluorophores run unrefused, the check and the run record cannot tell them from the unswapped folder, the layout group is not batch-confirmable and the preview shows each unit's own planes (test_physical_check, test_spec_run_aliases). - P30: the v13 legacy answers, migrated and derived, give back the legacy file byte for byte; with two dates in rep1 the derived file differs only in the earlier date, and no clock is read (test_spec_derived_design). - P34 on v01 and P35 on v25 and t07: a confirmed removal from v01 publishes exactly t07 or v25; removals from the v25 base; t07 without the removal is refused (test_spec_records). - P28 on v09: each batch of masks is unchecked until the mask step reads it (test_spec_run_inputs_semantic). - P33 (a): the hand-made attachment is the fixture t12 (test_spec_reconcile). - Section 2 confirmed_on: a frozen clock moved by one day writes identical bytes and the canonical writer reads no clock (tests/frozen_clock.py, test_description_write). - Section 2 descriptive keys and digests on v27 (test_description_write). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- tests/description_tree.py | 38 ++++++ tests/frozen_clock.py | 77 ++++++++++++ tests/test_description_write.py | 51 ++++++++ tests/test_physical_check.py | 44 ++++++- tests/test_spec_derived_design.py | 80 ++++++++++++- tests/test_spec_reconcile.py | 4 +- tests/test_spec_records.py | 158 ++++++++++++++++++++++++- tests/test_spec_run_aliases.py | 36 +++++- tests/test_spec_run_inputs_semantic.py | 30 ++++- 9 files changed, 508 insertions(+), 10 deletions(-) create mode 100644 tests/frozen_clock.py diff --git a/tests/description_tree.py b/tests/description_tree.py index c355001e..dc58d9c9 100644 --- a/tests/description_tree.py +++ b/tests/description_tree.py @@ -47,3 +47,41 @@ def write_mask(masks_root, mask_type, image, values): MAPPING = prefill_mapping("CYX") + + +def two_unnamed_units(v12: bytes) -> bytes: + """Return ``v12`` with a second unit, DrugA/rep1, of the same unnamed DNA, ACTIN layout. + + Conformance P06 asks for two unnamed, identical-shape units; ``v12`` declares one. The + second is added as the file format writes it, and DrugA is listed in ``used_ids``. + """ + text = v12.decode("utf-8") + condition = '[conditions.DrugA]\nsource_folder = "DrugA"\nreference = "CTRL"\n\n' + member = ( + '[repeats.rep1.members.DrugA]\nsource_repeat = "rep1"\nimage_count = 4\n\n' + '[repeats.rep1.members.DrugA.technical_repeats.single]\nsource_folder = "single"\n' + 'channels = ["DNA", "ACTIN"]\n\n' + ) + for old, new in ( + ("[channels.DNA]", condition + "[channels.DNA]"), + ("[used_ids]", member + "[used_ids]"), + ('conditions = ["CTRL"]', 'conditions = ["CTRL", "DrugA"]'), + ): + assert text.count(old) == 1, old + text = text.replace(old, new) + return text.encode("utf-8") + + +def write_fluorophores(path, *, swapped, side=8): + """Write an unnamed ``CYX`` image: a nucleus-like disc then actin-like stripes. + + With ``swapped`` the two planes are stored the other way round, as when the fluorophores + of a unit were physically exchanged. The header names no channel either way. + """ + yy, xx = np.mgrid[0:side, 0:side] + nuclei = ((yy - side // 2) ** 2 + (xx - side // 2) ** 2 < 4).astype(np.uint8) * 200 + actin = (xx % 2 == 0).astype(np.uint8) * 60 + planes = [actin, nuclei] if swapped else [nuclei, actin] + path.parent.mkdir(parents=True, exist_ok=True) + tifffile.imwrite(path, np.stack(planes), photometric="minisblack", metadata={"axes": "CYX"}) + return path diff --git a/tests/frozen_clock.py b/tests/frozen_clock.py new file mode 100644 index 00000000..6849937e --- /dev/null +++ b/tests/frozen_clock.py @@ -0,0 +1,77 @@ +"""A frozen clock for tests: every ``date`` and ``datetime`` the package can read gives one day. + +Conformance plan section 2, ``confirmed_on``: a writer under a frozen clock moved by a day must +write identical bytes for an unchanged model. :class:`FrozenClock` replaces +:class:`datetime.date` and :class:`datetime.datetime` in the :mod:`datetime` module and in every +module of the package that imported them by name, with subclasses whose ``today``, ``now`` and +``utcnow`` return the frozen day (at noon UTC) and count each read. +""" + +from __future__ import annotations + +import datetime as dt +import importlib +import pkgutil +import sys + +import morphobricks_preprocessing + +PACKAGE = morphobricks_preprocessing.__name__ + + +def _import_package() -> list[object]: + """Import every module of the package (not ``__main__``) and return them all.""" + for info in pkgutil.walk_packages(morphobricks_preprocessing.__path__, PACKAGE + "."): + if not info.name.endswith(".__main__"): + importlib.import_module(info.name) + return [m for name, m in sorted(sys.modules.items()) if name.split(".")[0] == PACKAGE] + + +class FrozenClock: + """The day every clock read of the package returns, and how many reads there were. + + Parameters + ---------- + monkeypatch + The pytest fixture that undoes the replacement after the test. + day + The frozen day. + """ + + def __init__(self, monkeypatch, day: dt.date) -> None: + self.day = day + self.reads = 0 + clock = self + + class FrozenDate(dt.date): + @classmethod + def today(cls): + clock.reads += 1 + return cls(clock.day.year, clock.day.month, clock.day.day) + + class FrozenDateTime(dt.datetime): + @classmethod + def now(cls, tz=None): + clock.reads += 1 + noon = cls(clock.day.year, clock.day.month, clock.day.day, 12) + return noon if tz is None else noon.replace(tzinfo=dt.UTC).astimezone(tz) + + @classmethod + def utcnow(cls): + return cls.now(dt.UTC).replace(tzinfo=None) + + @classmethod + def today(cls): + return cls.now() + + self.patched: list[str] = [] + replacements = (("date", dt.date, FrozenDate), ("datetime", dt.datetime, FrozenDateTime)) + for module in [*_import_package(), dt]: + for name, real, frozen in replacements: + if vars(module).get(name) is real: + monkeypatch.setattr(module, name, frozen) + self.patched.append(f"{module.__name__}.{name}") + + def move(self, days: int) -> None: + """Move the frozen day by ``days``.""" + self.day += dt.timedelta(days=days) diff --git a/tests/test_description_write.py b/tests/test_description_write.py index 2ebd3b3f..57b398ee 100644 --- a/tests/test_description_write.py +++ b/tests/test_description_write.py @@ -7,16 +7,20 @@ from __future__ import annotations +import datetime import re import tomllib +from dataclasses import replace import pytest +from frozen_clock import FrozenClock from morphobricks_preprocessing.description import ( Description, DescriptionError, description_from_bytes, ) +from morphobricks_preprocessing.description_file import DESCRIPTION_FILENAME, RECORDS_FILENAME from morphobricks_preprocessing.description_read import read_description_bytes from morphobricks_preprocessing.description_write import ( SECTIONS, @@ -25,6 +29,8 @@ toml_key, toml_string, ) +from morphobricks_preprocessing.design_derive import derive_design +from morphobricks_preprocessing.run.description_file import ABSENT, publish_description from spec_pin import fixtures VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} @@ -102,6 +108,24 @@ def test_a_display_name_target_or_treatment_key_changes_no_digest(name): assert canonical_bytes(_model(name)) != canonical_bytes(_model("v01")) +def test_editing_only_target_or_treatment_key_of_v27_changes_no_digest_and_no_grouping(): + """Section 2 "Descriptive keys" and "Section digests" on ``v27`` (ADR 0001 GS1).""" + base = _model("v27") + edits = { + "DrugA": {"target": "YES1", "treatment_key": "dasatinib"}, + "DrugB": {"target": None, "treatment_key": None}, + "DrugC": {"target": "src", "treatment_key": "azd0530"}, # now equal to DrugB's + } + conditions = { + key: replace(condition, **edits.get(key, {})) for key, condition in base.conditions.items() + } + edited = replace(base, conditions=conditions) + assert canonical_bytes(edited) != canonical_bytes(base) + assert description_from_bytes(canonical_bytes(edited)) == edited # a valid file + assert section_digests(edited) == section_digests(base) + assert derive_design(edited) == derive_design(base) # no grouping on the descriptive keys + + def test_the_digests_cover_the_four_sections_and_a_semantic_change_moves_them(): base = _model("v01") digests = section_digests(base) @@ -181,6 +205,33 @@ def test_confirmed_on_is_written_as_the_model_holds_it(): assert 'confirmed_on = "1999-12-31"' in text and "1999-12-31" in text.split("[used_ids]")[0] +def test_a_frozen_clock_moved_by_a_day_writes_identical_bytes_for_an_unchanged_model( + monkeypatch, tmp_path +): + """Conformance plan section 2, ``confirmed_on``: the writer never reads the clock.""" + clock = FrozenClock(monkeypatch, datetime.date(2026, 11, 9)) + assert "morphobricks_preprocessing.notebook_check_save.date" in clock.patched + models = {name: _model(name) for name in VALID} + clock.reads = 0 + first = {name: canonical_bytes(model) for name, model in models.items()} + out = tmp_path / "out" + out.mkdir() + published = publish_description(out, models["v01"], base_sha256=ABSENT) + file_first = (out / DESCRIPTION_FILENAME).read_bytes() + clock.move(days=1) + assert datetime.date.today() == datetime.date(2026, 11, 10) # the clock did move + clock.reads = 0 + second = {name: canonical_bytes(model) for name, model in models.items()} + assert clock.reads == 0 # the canonical writer read no clock, on either day + assert second == first + again = publish_description(out, models["v01"], base_sha256=published.sha256) + assert (out / DESCRIPTION_FILENAME).read_bytes() == file_first == first["v01"] + assert again.sha256 == published.sha256 and again.records == () + assert not (out / RECORDS_FILENAME).exists() + assert {m.confirmed_on for m in models["v01"].repeats.values()} == {"2026-11-06"} + assert file_first.count(b'confirmed_on = "2026-11-06"') == 2 # the model's, not the clock's + + def test_a_refused_file_is_not_a_model(): with pytest.raises(DescriptionError) as error: description_from_bytes(b"schema = 1\n") diff --git a/tests/test_physical_check.py b/tests/test_physical_check.py index 6f538eaa..4678a37e 100644 --- a/tests/test_physical_check.py +++ b/tests/test_physical_check.py @@ -1,4 +1,4 @@ -"""The physical ``R_*`` checks on the Spec fixtures: conformance P01-P03, P10, P12, P13, P23, P26. +"""The physical ``R_*`` checks on Spec fixtures: conformance P01-P03, P06, P10, P12, P13, P23, P26. Experiment-v1 section 8. The folder is built from a vendored valid fixture (one small image per unit position, header names taken from the fixture's confirmed names), then one image or @@ -13,6 +13,8 @@ import pytest import tifffile +from description_tree import two_unnamed_units, write_fluorophores +from morphobricks_preprocessing.axes import AxisMapping, AxisReview from morphobricks_preprocessing.description import Description, description_from_bytes from morphobricks_preprocessing.discovery import discover_experiment from morphobricks_preprocessing.metadata import OME_ZARR, inspect_images @@ -29,6 +31,7 @@ check_physical, layout_groups, ) +from morphobricks_preprocessing.preview import channel_strip from spec_pin import fixtures VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} @@ -222,6 +225,45 @@ def test_an_image_with_names_when_the_channel_has_none_contradicts(tmp_path): assert check(root, description).codes == {HEADER_CONTRADICTION} +# P06 ------------------------------------------------------------------------------------- + + +def _p06_tree(root, *, swapped): + """CTRL/rep1 and DrugA/rep1, four unnamed images each; DrugA's fluorophores maybe swapped.""" + for condition in ("CTRL", "DrugA"): + for index in range(4): + path = root / condition / "rep1" / f"FOV{index}.tif" + write_fluorophores(path, swapped=swapped and condition == "DrugA") + return root + + +def test_p06_swapped_fluorophores_in_unnamed_units_pass_and_each_unit_needs_its_own_look( + tmp_path, +): + """P06: no refusal (a documented limit); what guards it is the preview and confirmation.""" + description = description_from_bytes(two_unnamed_units(VALID["v12"].data)) + same = check(_p06_tree(tmp_path / "same", swapped=False), description) + root = _p06_tree(tmp_path / "swapped", swapped=True) + report = check(root, description) + assert report.ok and report.issues == () and report.unchecked == () + assert report == same # the pixels are never compared: nothing tells the two apart + ctrl, drug = UnitKey("CTRL", "rep1", "single"), UnitKey("DrugA", "rep1", "single") + assert report.layouts[ctrl] == report.layouts[drug] + assert (report.layouts[ctrl].channels, report.layouts[ctrl].names) == (2, ()) + (group,) = layout_groups(report.layouts) + assert group.units == (ctrl, drug) and not group.batch_confirmable # Q3: one look per unit + mapping = AxisMapping("CYX", "CYX") + shown = {} + for condition in ("CTRL", "DrugA"): + relative = f"{condition}/rep1/FOV0.tif" + strip = channel_strip( + root / relative, relative, AxisReview(mapping, {relative: mapping}, ()) + ) + assert [p.label for p in strip.panels] == ["C0", "C1"] # no name is made up + shown[condition] = [int(p.plane.pixels.max()) for p in strip.panels] + assert shown == {"CTRL": [200, 60], "DrugA": [60, 200]} # the preview shows the swap + + # P10, P12 ------------------------------------------------------------------------------ diff --git a/tests/test_spec_derived_design.py b/tests/test_spec_derived_design.py index 14e14900..5c362765 100644 --- a/tests/test_spec_derived_design.py +++ b/tests/test_spec_derived_design.py @@ -6,15 +6,28 @@ from __future__ import annotations +import datetime import tomllib import pytest -from morphobricks_preprocessing import read_confirmed_design +from frozen_clock import FrozenClock +from morphobricks_preprocessing import discover_experiment, read_confirmed_design from morphobricks_preprocessing.description import description_from_bytes from morphobricks_preprocessing.design_derive import derive_design, write_derived_design +from morphobricks_preprocessing.draft import migrate_legacy_design +from morphobricks_preprocessing.draft_answers import ( + ChannelAnswer, + ConditionAnswer, + DraftAnswers, + RepeatAnswer, + UnitAnswer, + description_from_draft, +) +from morphobricks_preprocessing.draft_format import ABSENT, SPEC_COMMIT from spec_oracle import refval from spec_pin import fixtures, spec_root +from tiff_tree import make_tree VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} GOLDEN = sorted((spec_root() / "fixtures" / "expected").glob("*.experiment_design.csv")) @@ -55,3 +68,68 @@ def test_the_file_names_folders_never_ids(): assert [row.condition for row in rows] == [line.split(",")[0] for line in golden] assert "ctrl" not in {row.condition for row in rows} assert {row.condition for row in rows if row.control} == {"control"} + + +# P30: the legacy v13 answers, migrated, give back the legacy file ------------------------- + +V13_FOLDERS = [ + f"{c}/{r}/img{i:02d}.tif" + for c in ("CTRL", "DrugA", "DrugB") + for r in ("rep1", "rep2", "rep3") + for i in range(10) +] +TWO_DATES = { + b"CTRL,rep1,rep1,true,10,0,true,2026-10-30": b"CTRL,rep1,rep1,true,10,0,true,2026-10-01", + b"DrugA,rep1,rep1,false,10,0,true,2026-10-30": b"DrugA,rep1,rep1,false,10,0,true,2026-10-15", + b"DrugB,rep1,rep1,false,10,0,true,2026-10-30": b"DrugB,rep1,rep1,false,10,0,true,2026-10-15", +} + + +def _only_ids(draft): + """The user's answers to a migrated v13 draft: the proposed IDs and the unit channels.""" + conditions, repeats = ("CTRL", "DrugA", "DrugB"), ("rep1", "rep2", "rep3") + return DraftAnswers( + ABSENT, + SPEC_COMMIT, + {c: ConditionAnswer(id=c) for c in conditions}, + {"single": "single"}, + {r: RepeatAnswer(id=r) for r in repeats}, + {(c, r, "single"): UnitAnswer(("DNA", "ACTIN")) for c in conditions for r in repeats}, + {"DNA": ChannelAnswer("DAPI", None), "ACTIN": ChannelAnswer("F-actin", None)}, + ) + + +@pytest.mark.parametrize("two_dates", [False, True], ids=["one_date", "two_dates"]) +def test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date( + two_dates, tmp_path, monkeypatch +): + """P30: one date everywhere gives the legacy file; rep1 with two dates keeps the latest.""" + make_tree(tmp_path / "exp", V13_FOLDERS) + golden = (next(p for p in GOLDEN if p.name.startswith("v13"))).read_bytes() + legacy = golden + for old, new in TWO_DATES.items() if two_dates else (): + assert legacy.count(old) == 1 + legacy = legacy.replace(old, new) + clock = FrozenClock(monkeypatch, datetime.date(2026, 11, 9)) + draft = migrate_legacy_design(legacy, discover_experiment(tmp_path / "exp")) + description = description_from_draft(draft, _only_ids(draft)) + derived = write_derived_design(description, tmp_path / "design.csv").read_bytes() + assert clock.reads == 0 # the clock is never read + changed = [ + (old, new) + for old, new in zip(legacy.splitlines(), derived.splitlines(), strict=True) + if old != new + ] + if not two_dates: + assert derived == legacy == golden and draft.notes == () + return + assert description.repeats["rep1"].confirmed_on == "2026-10-15" + assert changed == [ # the one-date groups rep2 and rep3 are byte-identical + ( + b"CTRL,rep1,rep1,true,10,0,true,2026-10-01,1", + b"CTRL,rep1,rep1,true,10,0,true,2026-10-15,1", + ) + ] + (note,) = draft.notes + assert (note.repeat_group, note.kept) == ("rep1", "2026-10-15") + assert {day for _, _, day in note.dates} == {"2026-10-01", "2026-10-15"} diff --git a/tests/test_spec_reconcile.py b/tests/test_spec_reconcile.py index f0cadb04..e38f6722 100644 --- a/tests/test_spec_reconcile.py +++ b/tests/test_spec_reconcile.py @@ -343,8 +343,10 @@ def test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id(): "ctrl": replace(published.conditions["ctrl"], source_folder="ctl"), }, ) + t12 = next(item for item in fixtures("transitions") if item.name.startswith("t12_")) + assert description_from_bytes(t12.data) == attached # the hand-made attachment is t12 codes = {i.code for i in transition_issues(published, attached)} - assert Refusal.R_CONFIRMED_ID_CHANGED in codes # t12: a hand-made attachment is refused + assert codes == {Refusal.R_CONFIRMED_ID_CHANGED} # refused without a recorded confirmation def test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats(tmp_path): diff --git a/tests/test_spec_records.py b/tests/test_spec_records.py index ab5625ac..0143280e 100644 --- a/tests/test_spec_records.py +++ b/tests/test_spec_records.py @@ -4,7 +4,8 @@ another count, the draft and the user's answers come from the real reconciliation, and the lines are what ``publish_description`` appends to ``morphobricks.records.jsonl``, read back with ``read_records``. Dates are the user's: the confirmation date of a removal, ``recorded_on`` of a -count change. +count change. Conformance P34 and P35 also run on the fixtures the plan names: ``v01``, and +``v25`` with ``t07`` (a confirmed removal from ``v01`` publishes exactly ``t07`` or ``v25``). """ from __future__ import annotations @@ -28,6 +29,7 @@ reconfirmations, removal_questions, ) +from morphobricks_preprocessing.description_write import canonical_bytes from morphobricks_preprocessing.run import description_file from morphobricks_preprocessing.run.description_file import ( ABSENT, @@ -76,14 +78,14 @@ def on_disk(description, changes=None, without=()): return seen -def removal(published, gone, changes=None): +def removal(published, gone, changes=None, *, day=DAY): """The draft, re-confirmations and answers when the folder loses the ``gone`` units.""" rec = reconcile(published, on_disk(published, changes, without=gone)) - answers = [question.answer(DAY) for question in removal_questions(published, rec)] + answers = [question.answer(day) for question in removal_questions(published, rec)] draft = candidate(published, rec, answers) for repeat_id in rec.reopened: if repeat_id in draft.repeats: # a repeat left without members is gone, not reopened - draft = confirm_repeat(draft, repeat_id, DAY) + draft = confirm_repeat(draft, repeat_id, day) found = reconfirmations(published, draft, rec, answers) assert found.open == () return draft, found.accepted, answers @@ -385,3 +387,151 @@ def test_a_removal_and_a_count_change_in_one_publication_write_both_in_order(tmp ] assert [(r["old_image_count"], r["new_image_count"]) for r in counts] == [(9, 4), (5, 7)] assert all(r["recorded_on"] == LATER and "confirmed_on" not in r for r in counts) + + +# P34 on v01 and P35 on v25 and t07: the fixtures the conformance plan names ------------- + +TRANSITION = {item.name.split("_")[0]: item for item in fixtures("transitions")} + + +def _published_text(folder): + return (folder / DESCRIPTION_FILENAME).read_text(encoding="utf-8") + + +def test_p34_v01_one_changed_count_appends_one_line_and_keeps_every_confirmed_on(tmp_path): + published = model("v01") + base = publish_first(tmp_path, published) + before = _published_text(tmp_path) + draft = _counted(published, {key("CTRL", "rep1"): ObservedUnit(13)}) + publish_description(tmp_path, draft, base_sha256=base, recorded_on=LATER) + after = _published_text(tmp_path) + pairs = zip(before.split("\n"), after.split("\n"), strict=True) + changed = [(old, new) for old, new in pairs if old != new] + assert changed == [("image_count = 12", "image_count = 13")] # confirmed_on byte-identical + assert read_records(tmp_path) == ( + { + "type": "image_count_changed", + "condition_id": "CTRL", + "condition_source_folder": "CTRL", + "repeat_id": "rep1", + "source_repeat": "rep1", + "technical_repeats": [{"id": "single", "source_folder": "single"}], + "old_image_count": 12, + "new_image_count": 13, + "recorded_on": LATER, + }, + ) + + +def test_p34_v01_a_removal_and_count_changes_write_removed_then_one_line_per_member(tmp_path): + published = model("v01") + base = publish_first(tmp_path, published) + changes = {key("CTRL", "rep1"): ObservedUnit(13), key("DrugA", "rep1"): ObservedUnit(9)} + draft, accepted, answers = removal(published, {key("DrugA", "rep2")}, changes) + assert draft.repeats["rep1"].confirmed_on == published.repeats["rep1"].confirmed_on + result = publish_description( + tmp_path, + draft, + base_sha256=base, + confirmations=accepted, + removals=tuple(answers), + recorded_on=LATER, + ) + lines = read_records(tmp_path) + assert lines == result.records + assert [(r["type"], r.get("entity")) for r in lines] == [ + *[("reconfirmation", None)] * len(accepted), + ("removed", "unit"), + ("image_count_changed", None), + ("image_count_changed", None), + ] + counts = [r for r in lines if r["type"] == "image_count_changed"] + assert [(r["condition_id"], r["old_image_count"], r["new_image_count"]) for r in counts] == [ + ("CTRL", 12, 13), + ("DrugA", 11, 9), + ] + assert all(r["repeat_id"] == "rep1" and r["recorded_on"] == LATER for r in counts) + + +def test_p35_t07_is_what_a_confirmed_removal_from_v01_publishes(tmp_path): + """(a) The member DrugA/rep2 removed on 2026-11-07: the published file is ``t07``.""" + published = model("v01") + base = publish_first(tmp_path, published) + draft, accepted, answers = removal(published, {key("DrugA", "rep2")}, day="2026-11-07") + publish_description( + tmp_path, draft, base_sha256=base, confirmations=accepted, removals=tuple(answers) + ) + t07 = description_from_bytes(TRANSITION["t07"].data) + assert t07.repeats["rep2"].confirmed_on == "2026-11-07" + assert (tmp_path / DESCRIPTION_FILENAME).read_bytes() == canonical_bytes(t07) + assert "DrugA" in t07.used_ids["conditions"] and "rep2" in t07.used_ids["repeats"] + reconfirmed = {r["old_id"] for r in read_records(tmp_path) if r["type"] == "reconfirmation"} + assert "rep2" in reconfirmed # the reopened repeat is re-confirmed, as 4.6 requires + unit = unit_row("DrugA", "rep2", "rep2", "single", day="2026-11-07") + assert removed_lines(tmp_path) == [unit] + + +def test_p35_t07_without_the_recorded_removal_is_refused_and_writes_nothing(tmp_path): + base = publish_first(tmp_path, model("v01")) + before = snapshot(tmp_path) + with pytest.raises(PublicationRefused) as refused: + publish_description( + tmp_path, description_from_bytes(TRANSITION["t07"].data), base_sha256=base + ) + assert {i.code.value for i in refused.value.issues} == {"R_CONFIRMED_ID_CHANGED"} + assert snapshot(tmp_path) == before + assert not (tmp_path / RECORDS_FILENAME).exists() + + +def test_p35_v25_is_what_removing_the_last_members_of_drug_a_from_v01_publishes(tmp_path): + """(b), (c) condition: every DrugA member removed on 2026-11-06 gives ``v25``.""" + published = model("v01") + base = publish_first(tmp_path, published) + gone = {key("DrugA", "rep1"), key("DrugA", "rep2")} + draft, accepted, answers = removal(published, gone, day="2026-11-06") + publish_description( + tmp_path, draft, base_sha256=base, confirmations=accepted, removals=tuple(answers) + ) + assert (tmp_path / DESCRIPTION_FILENAME).read_bytes() == canonical_bytes(model("v25")) + day = "2026-11-06" + assert removed_lines(tmp_path) == [ + { + **NULLS, + "type": "removed", + "entity": "condition", + "condition_id": "DrugA", + "condition_source_folder": "DrugA", + "confirmed_on": day, + }, + unit_row("DrugA", "rep1", "rep1", "single", day=day), + unit_row("DrugA", "rep2", "rep2", "single", day=day), + ] + + +def test_p35_on_the_v25_base_a_removed_repeat_keeps_every_retired_id(tmp_path): + """(c) repeat: with ``v25`` published, CTRL/rep2 removed is the repeat, then its unit.""" + published = model("v25") + base = publish_first(tmp_path, published) + draft, accepted, answers = removal(published, {key("CTRL", "rep2")}) + assert draft.used_ids == published.used_ids # DrugA and rep2 stay listed + publish_description( + tmp_path, draft, base_sha256=base, confirmations=accepted, removals=tuple(answers) + ) + assert removed_lines(tmp_path) == [ + {**NULLS, "type": "removed", "entity": "repeat", "repeat_id": "rep2", "confirmed_on": DAY}, + unit_row("CTRL", "rep2", "rep2", "single"), + ] + assert ( + "rep2" not in description_from_bytes((tmp_path / DESCRIPTION_FILENAME).read_bytes()).repeats + ) + + +def test_p35_on_the_v25_base_an_unconfirmed_removal_writes_nothing(tmp_path): + published = model("v25") + base = publish_first(tmp_path, published) + draft, accepted, _ = removal(published, {key("CTRL", "rep2")}) + before = snapshot(tmp_path) + with pytest.raises(ValueError, match="no removal confirms"): + publish_description(tmp_path, draft, base_sha256=base, confirmations=accepted) + assert snapshot(tmp_path) == before + assert not (tmp_path / RECORDS_FILENAME).exists() diff --git a/tests/test_spec_run_aliases.py b/tests/test_spec_run_aliases.py index acf76b2e..97036949 100644 --- a/tests/test_spec_run_aliases.py +++ b/tests/test_spec_run_aliases.py @@ -1,4 +1,4 @@ -"""A run with a published description: conformance P26 (per-unit header evidence), Stage 1 PR 13. +"""A described run: conformance P26 (per-unit header evidence, Stage 1 PR 13) and P06. Experiment-v1 section 4.3 and 8. For a described run the experiment-wide v0.4 checks ``channel_names_differ`` and ``different_channel_counts`` are replaced by per-unit checks: each @@ -9,11 +9,13 @@ from __future__ import annotations +import json + import numpy as np import pytest import tifffile -from description_tree import MAPPING, unit_folder +from description_tree import MAPPING, two_unnamed_units, unit_folder, write_fluorophores from morphobricks_preprocessing import run_experiment from morphobricks_preprocessing.description import description_from_bytes from spec_pin import fixtures @@ -22,6 +24,7 @@ CONTRADICTION = "R_HEADER_CONTRADICTION" COUNT = "R_CHANNEL_COUNT_MISMATCH" HOMOGENEOUS = "R_UNIT_NOT_HOMOGENEOUS" +FIELD = "images_without_header_channel_names" GLOBAL = {"channel_names_differ", "different_channel_counts"} COMPOSED, DECOMPOSED = "Café", "Café" @@ -148,6 +151,35 @@ def test_p26_images_without_header_names_need_no_evidence_v12(tmp_path): assert result.written, [(i.code, i.message) for i in result.issues] +def _p06_run(tmp_path, *, swapped): + """Run the two-unit ``v12`` folder; DrugA's fluorophores are maybe physically swapped.""" + root, out = tmp_path / f"exp_{swapped}", tmp_path / f"out_{swapped}" + out.mkdir() + (out / "morphobricks.toml").write_bytes(two_unnamed_units(VALID["v12"].data)) + for condition in ("CTRL", "DrugA"): + for index in range(4): + path = root / condition / "rep1" / f"FOV{index}.tif" + write_fluorophores(path, swapped=swapped and condition == "DrugA") + result = _run(root, out) + manifest = json.loads((out / "run_manifest.json").read_text(encoding="utf-8")) + return result, manifest["experiment_description"] + + +def test_p06_two_unnamed_units_with_swapped_fluorophores_run_unrefused_on_the_confirmation( + tmp_path, +): + """P06: a documented limit, not a refusal; the record shows the units stand on confirmation.""" + result, entry = _p06_run(tmp_path, swapped=True) + assert result.written and result.issues == () + _, unswapped = _p06_run(tmp_path, swapped=False) + assert entry == unswapped # the described-run record cannot tell the two folders apart + listed = { + u["condition"]: [(i["header_names"], i["channel_positions"]) for i in u[FIELD]] + for u in entry["units"] + } + assert listed == {"CTRL": [("none", [0, 1])] * 4, "DrugA": [("none", [0, 1])] * 4} + + def test_p26_a_unit_with_named_and_unnamed_images_is_not_homogeneous(tmp_path): root, out = _setup(tmp_path, "v11", {"*": ["DAPI", "ACTIN"]}) _image(root / "CTRL" / "rep1" / "acquisition_1" / "img0.ome.tif", None, 2) diff --git a/tests/test_spec_run_inputs_semantic.py b/tests/test_spec_run_inputs_semantic.py index e434d648..347b68f5 100644 --- a/tests/test_spec_run_inputs_semantic.py +++ b/tests/test_spec_run_inputs_semantic.py @@ -1,4 +1,4 @@ -"""Semantic ``RunInputs``: conformance P14 and P25, and ``A_MASK_UNCHECKED`` of P28 (PR 12). +"""Semantic ``RunInputs``: conformance P14 and P25, and ``A_MASK_UNCHECKED`` of P28 (v08, v09). P14: only the published, confirmed ``morphobricks.toml`` is read; a draft or a file with an unconfirmed repeat is ``A_DRAFT_REFUSED``. P25: every position carries the key with the source @@ -114,3 +114,31 @@ def test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_che assert inputs.read_mask(fov, "nuclei").max() == 2 manifest = json.loads((out / "run_manifest.json").read_text(encoding="utf-8")) assert manifest["experiment_description"]["masks"]["TumorRegion"]["status"] == "not yet checked" + + +def test_p28_on_v09_each_batch_of_masks_is_unchecked_until_the_mask_step_reads_it(tmp_path): + """P28 on ``v09`` (partial masks), the fixture the plan names beside ``v08``.""" + root, out, found = _described_run(tmp_path, "v09") + masks = tmp_path / "masks" + _run(root, out) # no masks yet: valid + checked = [] + for batch in (found[:1], found[1:3], found[3:]): + for image in batch: + write_mask(masks, "nuclei", image, [0, 1, 2]) + _run(root, out, masks) + inputs = RunInputs.open(out) + image_of = {f"{fov.key[0]}/{fov.key[1]}/{fov.key[3]}.tif": fov for fov in inputs.fovs()} + new = [image_of[image] for image in batch] + for image in checked: # the batches read before stay checked + assert inputs.mask_check_status("Nuclei", image_of[image]) == "checked" + for fov in new: + assert inputs.mask_check_status("Nuclei", fov) == "not yet checked" + with pytest.raises(MaskUncheckedError) as raised: + inputs.read_mask(fov, "nuclei") + assert raised.value.code == "A_MASK_UNCHECKED" + assert describe_masks(prepare_mask_step(None, out)).issues == () + inputs = RunInputs.open(out) + for fov in new: + assert inputs.mask_check_status("Nuclei", fov) == "checked" + assert inputs.read_mask(fov, "nuclei").max() == 2 + checked.extend(batch) From c675f13f3de074b2b1d805f4a59f3455a9b573c3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 14:16:13 +0000 Subject: [PATCH 05/13] Gate G1 record: the gap tests, the GS1 scope ruling and the P21/P29 finding Measured on a6a0343 (tests only on top of 4437067; src and pyproject.toml equal to 3b616a0). P06, P30, P34, P35, P28 (v09) and the section 2 confirmed_on and v27 rows now pass on the fixtures the plan names. P24, P27, P31 and the lab-tool part of Provenance are deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec cdc1841). P21 and P29 are recorded as not met on v01 and v07: a run with a published description never reports R_FOLDER_NOT_NFC or R_MASK_TYPE_NAME_INVALID (source defect, left for the owner). The Colab and JupyterLab browser check is the owner's, before tagging. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/conformance.md | 107 +++++++++++++++++++++++++------------------- 1 file changed, 60 insertions(+), 47 deletions(-) diff --git a/docs/conformance.md b/docs/conformance.md index 38aba8bc..85a942e8 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -10,22 +10,22 @@ in the suite is listed as **not run**, with its reason. | | | |---|---| -| Release candidate | `c74d717e4f4f0a72be8b991435413adf4a2a1d37` (branch `claude/release-0.5.0`: `main` at `bf733ee` plus the v0.5.0 preparation `3b616a0` and the notebook's `RELEASE_REF` commit). The commit that adds this record changes only this file, so the code and tests are those of `c74d717` | -| Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d` (Spec `main`). Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | +| Release candidate | Measured on `a6a0343` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, the notebook's `RELEASE_REF` commit `c74d717`, the first record `4437067` and the G1 gap tests `a6a0343`). The commit that updates this record changes only this file, so the code and tests are those of `a6a0343`. `git diff 3b616a0 a6a0343 -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names | +| Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d`. The G1 scope ruling (ADR 0001 GS1, conformance plan sections 4 and 5) was merged later as `cdc1841`, which changes only `docs/adr/0001-experiment-description-v1.md` and `docs/conformance-plan.md`, so the pinned files are unchanged. Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | | Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0, normalization 0.3.4); run as root on a case-sensitive file system | | Commands | `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/` (the whole suite, with a per-test JUnit report), and `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules) | -| Whole suite | 5018 collected: 5014 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | -| Spec modules (`tests/test_spec_*.py`) | 1649 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | -| `tools/check.py` | Ruff and format clean, pytest 5014 passed and 4 skipped, code rules passed (exit status 0) | +| Whole suite | 5032 collected: 5028 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | +| Spec modules (`tests/test_spec_*.py`) | 1660 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | +| `tools/check.py` (on `a6a0343`) | Ruff clean, Ruff format 410 files already formatted, pytest 5028 passed and 4 skipped, code rules passed (exit status 0) | -**How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1649 test cases +**How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1660 test cases from `test_spec_*` modules, every one passed: none has a `skipped` (or xfail) element. (2) The `-rsxX` summary of the run lists exactly the four skips below, none in a Spec module, and no xfail or xpass. (3) All 21 modules of `EXPECTED_MODULES` have test cases in the report -(catalogue 376, derived design 79, discovery 11, encoding 58, gate 9, no fallback 15, pin 16, +(catalogue 376, derived design 81, discovery 11, encoding 58, gate 9, no fallback 15, pin 16, publication 20, reader 330, reader phases 1-3 287, reader refusal paths 8, reconcile 25, -records 15, records goldens 76, refval cross-check 203, rules table 14, run aliases 15, run -description 9, RunInputs semantic 7, transitions 73, vendored missing 3), and the gate's +records 22, records goldens 76, refval cross-check 203, rules table 14, run aliases 16, run +description 9, RunInputs semantic 8, transitions 73, vendored missing 3), and the gate's collection check (a deleted, emptied or skipped Spec module stops the run) did not fire. (4) The gate itself (`tests/spec_gate.py`) turns a skip or xfail in those modules into a failure, and its own tests (`test_spec_gate.py`, 9) passed. @@ -49,8 +49,9 @@ These four behaviours are **not verified by this run**. ### Physical scenarios P01-P35 (conformance plan section 3) -"Passed" means every test named passed in this run. Where a test uses another fixture or a -synthetic tree than the plan names, or covers only part of the required outcome, the row says so. +"Passed" means every test named passed in this run, on the fixtures the plan names (ADR 0001 +GS1: a scenario that names fixtures is shown on those fixtures; further fixtures are listed as +"also"). Where the required outcome is not shown on the named fixture, the row says so. | # | Result | Tests (module: test functions) | |---|---|---| @@ -59,7 +60,7 @@ synthetic tree than the plan names, or covers only part of the required outcome, | P03 | Passed | `test_physical_check`: `test_p03_no_case_folding_and_no_similarity` (`v11`, 2 cases) | | P04 | Passed | `test_spec_run_description`: `test_p04_heterogeneous_layouts_are_refused_and_nothing_is_processed` (`v04`, `v05`, `v06`, `v17`), `test_p04_the_message_names_the_layouts_and_the_units_in_plain_words`; `test_heterogeneous_refusal_first` (9) | | P05 | Passed | `test_spec_run_description`: `test_p05_binary_mask_with_values_0_to_5_is_refused_by_the_mask_step_not_the_run`, `test_p05_an_instances_mask_holding_only_0_and_1_is_accepted` (`v08`) | -| P06 | Passed, partly | `test_spec_run_aliases`: `test_p26_images_without_header_names_need_no_evidence_v12`; `test_physical_check`: `test_an_image_with_names_when_the_channel_has_none_contradicts` (`v12`); `test_physical_check`: `test_p23_pixels_that_look_like_rgb_are_never_examined` (no check reads pixels). **No test builds two unnamed units with physically swapped fluorophores**; the "preview + confirmation only" part is the notebook's channel grid (`test_notebook_channels`, 30) | +| P06 | Passed | `test_physical_check`: `test_p06_swapped_fluorophores_in_unnamed_units_pass_and_each_unit_needs_its_own_look` (`v12` with a second unnamed unit, DrugA/rep1, of the same shape; DrugA's two planes stored the other way round): no issue, the physical report equals that of the unswapped folder, the two units share one layout group that is **not** batch-confirmable (Q3), and `channel_strip` shows each unit's own planes (labels `C0`, `C1`, no name made up); `test_spec_run_aliases`: `test_p06_two_unnamed_units_with_swapped_fluorophores_run_unrefused_on_the_confirmation`: the run is written with no issue, its `experiment_description` entry equals the unswapped run's, and every image of both units is listed under `images_without_header_channel_names` (`none`, positions 0, 1). Also `test_p26_images_without_header_names_need_no_evidence_v12`, `test_an_image_with_names_when_the_channel_has_none_contradicts` (`v12`) | | P07 | Passed | `test_spec_reconcile`: `test_p07_a_new_technical_repeat_is_pending_and_reopens_only_its_repeat` (`v13`), `test_p07_after_the_user_assigns_the_unit_and_confirms_the_new_date_is_accepted` (`v06`) | | P08 | Passed | `test_spec_reconcile`: `test_p08_a_renamed_folder_is_removed_plus_new_and_never_paired`, `test_p08_the_removal_is_one_grouped_question_and_publishes_only_when_answered` (`v03`) | | P09 | Passed | `test_description_file`: `test_p09_a_symlinked_description_is_refused_by_the_reader`, `test_p09_a_dangling_symlink_is_refused_not_missing`, `test_p09_publication_refuses_a_symlink_and_neither_follows_nor_replaces_it` (`v01`) | @@ -73,26 +74,26 @@ synthetic tree than the plan names, or covers only part of the required outcome, | P17 | Passed | `test_spec_publication`: `test_p17_a_second_draft_from_the_same_base_is_refused_and_not_merged` (`v01`, `v21`, `v16`; includes re-deriving B from A's file) | | P18 | Passed | `test_spec_publication`: `test_p18_publishing_over_nothing_with_the_absent_base_is_accepted`, `test_p18_another_file_appears_before_publication`, `test_p18_a_draft_without_a_usable_base_is_refused_and_nothing_is_published` (8 cases) | | P19 | Passed | `test_spec_publication`: `test_p19_a_retired_id_declared_again_is_refused`, `test_p19_also_when_the_returning_folder_is_the_one_the_retired_id_had` (`v25`, `t03`) | -| P20 | Passed | `test_spec_publication`: `test_p20_a_dropped_used_id_is_refused` (`v25`, `t04`), `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation` (on `v01`) | -| P21 | Passed | `test_spec_discovery`: `test_p21_a_decomposed_folder_is_refused_with_a_description`, `test_p21_a_composed_folder_is_accepted`, `test_p21_without_a_description_discovery_is_as_in_v0_4`, `test_p21_preflight_carries_the_refusal` (synthetic tree) | +| P20 | Passed | `test_spec_publication`: `test_p20_a_dropped_used_id_is_refused` (`v25`, `t04`), `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation` (on `v01`); `test_spec_records`: `test_p35_v25_is_what_removing_the_last_members_of_drug_a_from_v01_publishes` (the accepted removal publishes `v25` byte for byte, DrugA kept in `used_ids`) | +| P21 | **Not met on `v01`** | `test_spec_discovery` (4 `test_p21_*`, passed) shows `R_FOLDER_NOT_NFC` only through `preflight(..., with_description=True)` on a synthetic tree. With `v01` published and a decomposed folder `CTRL/Cafe\u0301` added, `run_experiment` refuses with `R_UNIT_NOT_DECLARED` and `repeat_missing_conditions` but **not** `R_FOLDER_NOT_NFC` (my run on `a6a0343`; nothing renamed). No production caller of `preflight` passes `with_description=True` (run pipeline, mask step, notebook folder check, command line). A source defect, reported to the owner and not fixed here: a fix moves `RELEASE_REF` | | P22 | Passed | `test_spec_publication`: `test_p22_confirmed_on_stays_byte_for_byte_through_an_unrelated_publication`, `test_p22_a_new_date_on_an_unchanged_repeat_is_refused` (`t05`), `test_p22_after_a_reopening_and_a_new_confirmation_the_new_date_is_accepted` (`v01`) | | P23 | Passed | `test_physical_check`: `test_p23_a_unit_declared_rgb_whose_images_are_two_channel_grayscale` (`v28`), `test_p23_an_rgb_tiff_in_a_unit_that_does_not_declare_it` (`v01`), `test_p23_pixels_that_look_like_rgb_are_never_examined` | -| P24 | **Not run** | The outcome is a brick's: a request through a provider manifest with or without `rgb_support`. Preprocessing has no provider manifest; it only exports the error class (`test_a_error_classes`: `A_RGB_NOT_SUPPORTED` class, code and pickling, passed). No test of (a)-(d) exists here | +| P24 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (provider manifest `rgb_support`). Preprocessing's own parts passed: the `A_RGB_NOT_SUPPORTED` class (`test_a_error_classes`) and the section 2 "RGB" row (below) | | P25 | Passed | `test_spec_run_inputs_semantic`: `test_p25_each_fov_carries_the_source_label_and_the_added_repeat_group` (`v03`, `v30`), `test_p25_in_v30_grouping_by_the_source_label_would_merge_different_repeats` | | P26 | Passed | `test_spec_run_aliases` (9 `test_p26_*`), `test_physical_check` (`test_p26_*`, 2 functions), `test_spec_discovery` (`test_p26_the_header_names_are_recorded_as_read_untrimmed`, `test_p26_the_manifest_records_the_header_name_as_read`): 19 cases (`v11`, `v31`, `v12`) | -| P27 | **Not run** | The outcome is a brick's: an object-set request with or without `binary_mask_policy`. Preprocessing only exports `A_MASK_NOT_INSTANCES` (`test_a_error_classes`, passed) and refuses `binary_mask_policy` in the file (`i112`, `E_FORBIDDEN_FIELD`, through the reader tests). (a)-(c) are not tested here | -| P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` | -| P29 | Passed | `test_spec_discovery`: `test_p29_na_and_a_leading_space_are_invalid_mask_type_names`, `test_p29_a_decomposed_mask_type_is_not_nfc`, `test_p29_the_three_cases_together`, `test_p29_valid_mask_types_are_accepted`, `test_p29_without_a_description_mask_types_are_accepted_as_in_v0_4` (synthetic tree) | -| P30 | Passed, partly | `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date`; golden `v13` (below). **Not tested:** the derived CSV of the two-date group compared with the legacy file (differing only in the earlier date) | -| P31 | **Not run** | The outcome is MorphoCoverage's (prefill, confirmation, refusal of an unconfirmed prefill). Preprocessing's part passed: editing `target` changes no digest (`test_description_write`: `test_a_display_name_target_or_treatment_key_changes_no_digest`; `v27` valid with no warning in the reader tests) | +| P27 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (per-request `binary_mask_policy`). Preprocessing's own parts passed: `A_MASK_NOT_INSTANCES` (`test_a_error_classes`) and the section 2 "Mask fields" row (`i111`, `i112`, below) | +| P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`, with the 0-5 binary batch); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` (`v08`), `test_p28_on_v09_each_batch_of_masks_is_unchecked_until_the_mask_step_reads_it` (`v09`: three batches, each `not yet checked` and `A_MASK_UNCHECKED` until the mask step reads it, earlier batches stay `checked`) | +| P29 | **Not met on `v07`** | `test_spec_discovery` (5 `test_p29_*`, passed) shows `R_MASK_TYPE_NAME_INVALID` and `R_FOLDER_NOT_NFC` only through `preflight(..., with_description=True)` on a synthetic tree. With `v07` published and mask folders `NA`, ` cells` and `Cafe\u0301` beside `nuclei` and `cells`, `run_experiment` refuses with `R_MASK_TYPE_NOT_DECLARED` (and `mask_missing`) but **not** `R_MASK_TYPE_NAME_INVALID` or `R_FOLDER_NOT_NFC` (my run on `a6a0343`; nothing renamed). Same source defect as P21 | +| P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date` | +| P31 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is MorphoCoverage's (prefill, confirmation). Preprocessing's own parts passed: `v27` valid with no warning, and editing `target` or `treatment_key` of `v27` changes no digest and no derived row (section 2 rows below) | | P32 | Passed | `test_draft`: `test_p32_forbidden_folders_get_no_proposal_and_nothing_is_adopted`, `test_p32_ordinary_folders_are_proposed_as_their_labels`; `test_spec_derived_design`: golden `v33`, `v34` and `test_the_file_names_folders_never_ids` | -| P33 | Passed | `test_spec_reconcile`: `test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id`, `test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats`, `test_p33_c_the_old_folder_removal_needs_its_confirmation_and_the_id_stays_used` (`v33`); `test_condition_folder_reopens` (2) | -| P34 | Passed, on other fixtures | `test_spec_records`: `test_a_changed_image_count_writes_the_old_and_new_counts_with_recorded_on` (no reopening, `confirmed_on` kept), `test_several_changed_members_are_written_in_condition_then_repeat_order`, `test_an_image_count_change_needs_a_date_and_writes_nothing_without_one`, `test_a_removal_and_a_count_change_in_one_publication_write_both_in_order` (RR3). They use `v06` and `v13`, not the `v01` the plan names | -| P35 | Passed, on other fixtures | `test_spec_records`: `test_a_removed_unit_writes_one_unit_record_naming_all_six_keys` (with its `reconfirmation` lines), `test_a_removed_repeat_writes_the_repeat_then_each_of_its_units`, `test_a_removed_condition_writes_the_condition_then_each_of_its_units`, `test_a_removed_technical_repeat_names_its_id_and_folder_then_its_unit`, `test_a_dropped_unit_without_a_removal_is_not_published`, `test_nothing_is_written_when_publication_is_refused`; `test_spec_publication`: `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation`. They use `v01`, `v03`, `v06`, not `v25` with `t07` (`t07` itself passes in `test_spec_transitions`) | +| P33 | Passed | `test_spec_reconcile`: `test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id`, `test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats`, `test_p33_c_the_old_folder_removal_needs_its_confirmation_and_the_id_stays_used` (`v33`; (a) asserts that the hand-made attachment is `t12` and gives exactly `R_CONFIRMED_ID_CHANGED`); `t10`, `t11`, `t12` against their predecessors in `test_spec_transitions`; `test_condition_folder_reopens` (2) | +| P34 | Passed | On `v01` (`test_spec_records`): `test_p34_v01_one_changed_count_appends_one_line_and_keeps_every_confirmed_on` (one `image_count_changed` line with the section 4.6 keys, old and new counts and `recorded_on`; the published file differs from `v01` in that one `image_count` line, so no repeat reopens and every `confirmed_on` is byte-identical), `test_p34_v01_a_removal_and_count_changes_write_removed_then_one_line_per_member` (RR3: the `removed` line, then one count line per remaining member, `rep1` keeps its date). Also on `v06`, `v13`: `test_a_changed_image_count_writes_the_old_and_new_counts_with_recorded_on`, `test_several_changed_members_are_written_in_condition_then_repeat_order`, `test_an_image_count_change_needs_a_date_and_writes_nothing_without_one`, `test_a_removal_and_a_count_change_in_one_publication_write_both_in_order` | +| P35 | Passed | On `v25` and `t07` (`test_spec_records`): `test_p35_t07_is_what_a_confirmed_removal_from_v01_publishes` ((a): the confirmed removal of DrugA/rep2 on 2026-11-07 publishes `t07` byte for byte, with the `reconfirmation` lines and one `unit` line), `test_p35_t07_without_the_recorded_removal_is_refused_and_writes_nothing` (`R_CONFIRMED_ID_CHANGED`, nothing written), `test_p35_v25_is_what_removing_the_last_members_of_drug_a_from_v01_publishes` ((b), (c) condition: the `condition` line then each `unit` line, the file is `v25` byte for byte), `test_p35_on_the_v25_base_a_removed_repeat_keeps_every_retired_id` ((c) repeat on the `v25` base: the `repeat` line then its `unit` line; `used_ids` unchanged, DrugA and `rep2` kept), `test_p35_on_the_v25_base_an_unconfirmed_removal_writes_nothing`. (c) technical repeat on `v06` (`v25` has only `single`): `test_a_removed_technical_repeat_names_its_id_and_folder_then_its_unit`. Also `test_a_removed_unit_writes_one_unit_record_naming_all_six_keys`, `test_a_removed_repeat_writes_the_repeat_then_each_of_its_units`, `test_a_removed_condition_writes_the_condition_then_each_of_its_units`, `test_a_dropped_unit_without_a_removal_is_not_published`, `test_nothing_is_written_when_publication_is_refused` | ### Golden CSV comparisons (conformance plan section 2, "Derived CSV") -`tests/test_spec_derived_design.py`, 79 passed: +`tests/test_spec_derived_design.py`, 81 passed (the two P30 cases are in the P30 row): - `test_the_golden_files_are_the_three_the_catalogue_lists`: `fixtures/expected/` holds exactly `v13`, `v33`, `v34`; @@ -121,29 +122,41 @@ All headless (ipywidgets driven by `click()` and `.value`) and in a real kernel; | (d) Heterogeneous refusal explicit and tested | `test_heterogeneous_refusal_first` (9, including `test_the_check_and_save_list_leads_with_the_layout_refusal`); P04 above | | Shipped notebook install | `test_notebook_file::test_remote_install_is_pinned_to_one_exact_commit`, `test_version_metadata_matches_labconstrictor_index`; `test_notebook_setup` (18, including `test_setup_installs_missing_package_from_the_exact_commit`) | -**Not part of this record:** the browser smoke check of `docs/colab-release-check.md` (a -fresh Google Colab runtime and JupyterLab/LabConstrictor). Headless tests cannot show that a -browser renders the widgets; that check is the owner's, before the tag. - -### Gaps against the conformance plan - -The plan's exit criterion (section 5) asks that every section 2 test and every section 3 -scenario run and pass. This run does not meet it in full: - -- **P24, P27, P31 are not run**: their outcomes belong to the consuming bricks and MorphoCoverage - (conformance plan section 4 puts `A_RGB_NOT_SUPPORTED`, `A_MASK_NOT_INSTANCES` and the - MorphoCoverage prefill in Stage 2), and preprocessing has no code that could run them. -- **P06 and P30 are partly covered** (see their rows); **P34 and P35** pass on other fixtures - than the ones the plan names. -- Section 2, "Provenance": `i99` is refused (`E_FORBIDDEN_FIELD`), but no run-manifest test - records lab-tool links: preprocessing has no lab-tool provenance input yet. -- Section 2, "`confirmed_on`": no writer test with a frozen clock moved by a day; the writer - writes the model's date (`test_description_write::test_confirmed_on_is_written_as_the_model_holds_it`) - and records take the user's date, not the clock - (`test_spec_records::test_the_date_of_each_record_is_the_users_confirmation_not_the_clock`). - -So this record does not by itself declare any reader conforming; deciding Gate G1 with these -gaps is the owner's. +**Colab and JupyterLab browser check: owner, before tagging.** The browser smoke check of +`docs/colab-release-check.md` (a fresh Google Colab runtime and JupyterLab/LabConstrictor) was not +run for this record: headless tests cannot show that a browser renders the widgets. + +### Section 2 rows (conformance plan section 2) + +The `confirmed_on` row, and the rows that ADR 0001 GS1 keeps in G1 on the subjects of the +deferred items, each from this run. The other section 2 rows run in the Spec modules above (all +passed, none skipped); they were not re-audited row by row for this update. + +| Row | Result | Tests | +|---|---|---| +| `confirmed_on` | Passed | `i98` is `E_MISSING_FIELD` (`test_spec_reader`); `test_description_write`: `test_a_frozen_clock_moved_by_a_day_writes_identical_bytes_for_an_unchanged_model`: with `date` and `datetime` frozen in the `datetime` module and every package module that imports them (`tests/frozen_clock.py`), on 2026-11-09 and then 2026-11-10, the canonical bytes of every valid fixture are identical, the canonical writer reads the clock 0 times, and republishing `v01` the next day leaves `morphobricks.toml` byte-identical with no record written | +| RGB | Passed | `test_spec_reader`: `v28`, `v29` accepted; `i100`-`i106`, `i119` give exactly their codes, each with a table path and the value; `test_spec_refval_crosscheck` agrees on all of them; `test_description_write`: round trip, idempotence and section 9.1 layout of `v28`, `v29` | +| Mask fields | Passed | `test_spec_reader` and `test_spec_refval_crosscheck`: `i111` (`E_ENUM`), `i112` (`E_FORBIDDEN_FIELD`) | +| Descriptive keys, Section digests | Passed | `v27` accepted with no warning (`test_spec_reader`); `test_description_write`: `test_editing_only_target_or_treatment_key_of_v27_changes_no_digest_and_no_grouping` (new: the edited `v27` is valid, all four digests equal, the derived design rows equal), `test_a_display_name_target_or_treatment_key_changes_no_digest` (`v16`, `v21` against `v01`) | +| Provenance | `i99` passed; lab-tool links deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | `i99` is `E_FORBIDDEN_FIELD` (`test_spec_reader`, `test_spec_refval_crosscheck`). Preprocessing has no lab-tool input yet | + +### Against the conformance plan + +Gate G1 scope, ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`): P24, P27, P31 and the lab-tool-links +part of the section 2 "Provenance" row are deferred to the Stage 2 gate per ADR 0001 GS1 +(MorphoBricks-Spec `cdc1841`); nothing else is deferred. Against that scope this run shows: + +- P01-P20, P22, P23, P25, P26, P28, P30, P33-P35 passed on the fixtures the plan names. P32 + passed on trees with the `v33` and `v34` folders and on the `v33`, `v34` goldens. The section 2 + rows in the table above passed (Provenance: its `i99` part). +- **P21 and P29 are not met on their named fixtures (`v01`, `v07`).** Their codes are produced + only by `preflight(..., with_description=True)`, which no production caller passes, so a run + with a published description does not report `R_FOLDER_NOT_NFC` or `R_MASK_TYPE_NAME_INVALID` + (it refuses with `R_UNIT_NOT_DECLARED` or `R_MASK_TYPE_NOT_DECLARED` instead). This is a + source defect, left for the owner: fixing it changes `src` and so moves `RELEASE_REF`. +- The Colab and JupyterLab browser check is the owner's, before tagging. + +So this record does not by itself declare any reader conforming; deciding Gate G1 is the owner's. ## Run of 8 November 2026 (Stage 1, PR 17) From 0a2e25b4ff4dd1f638ac65c50e09b43182e71a6d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 15:18:15 +0000 Subject: [PATCH 06/13] Raise R_FOLDER_NOT_NFC and R_MASK_TYPE_NAME_INVALID on the described paths (P21, P29) The two refusals were raised only by preflight(..., with_description=True), and no production caller passed it: a run with a published morphobricks.toml refused a decomposed folder only as R_UNIT_NOT_DECLARED and the mask types NA, ' cells' and a decomposed name only as R_MASK_TYPE_NOT_DECLARED. Spec experiment-v1 section 3.3 (a folder label read from disk that is not NFC is refused by preprocessing, never normalized or renamed), section 8 and ADR 0001 AQ11 (an invalid mask type name is refused at discovery, never reaching E_ID_TEXT), section 10 (an R_* conflict blocks publication) and owner decision Q1 (only when a description is in use). - run.preflight.description_in_use: the output folder holds a published morphobricks.toml. - run_experiment, prepare_mask_step and the command line's check/run pass it as with_description; the notebook's folder check and mask re-pairing pass it, or True while the user describes the experiment; Check and save is blocked while a name is refused. - The command line's describe refuses a folder label that is not NFC. Without a published description nothing changes. Regression tests on v01 and v07 through run_experiment, the mask step, the command line and the notebook fail on c675f13. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- CHANGELOG.md | 12 ++ .../cli_describe.py | 15 +- src/morphobricks_preprocessing/cli_run.py | 2 + .../notebook_check_save.py | 9 + .../notebook_folder.py | 11 ++ .../notebook_masks.py | 10 +- .../run/mask_step.py | 3 +- .../run/pipeline.py | 3 +- .../run/preflight.py | 30 ++- tests/test_notebook_name_refusals.py | 108 +++++++++++ tests/test_spec_discovery.py | 183 +++++++++++++++++- 11 files changed, 374 insertions(+), 12 deletions(-) create mode 100644 tests/test_notebook_name_refusals.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 73be657c..e4a05e94 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,18 @@ installs `v0.5.0` (its `RELEASE_REF` is the release-preparation commit), and the of this release candidate is in `docs/conformance.md`. The normalization brick stays pinned at v0.3.4. +- **Fixed: `R_FOLDER_NOT_NFC` and `R_MASK_TYPE_NAME_INVALID` on the production paths + (conformance P21, P29).** They were raised only by `preflight(..., with_description=True)`, + which no production caller passed, so a run with a published `morphobricks.toml` refused a + decomposed folder only as `R_UNIT_NOT_DECLARED`, and the mask types `NA`, ` cells` and a + decomposed name only as `R_MASK_TYPE_NOT_DECLARED`. Now `run_experiment`, the mask step + (`prepare_mask_step`, the command line's `masks`) and the command line's `check` and `run` + pass `with_description` whenever the output folder holds a published description + (`run.preflight.description_in_use`); the notebook's folder check and its re-pairing of the + masks also pass it while the user describes the experiment, and *Check and save* is blocked + while either refusal stands; the command line's `describe` refuses a folder label that is not + NFC instead of drafting it. Nothing is renamed or normalized. A run, mask step or check + without a published description is unchanged (owner decision Q1). - **Notebook: folder pictures with a 1:1 crop, "Tag like the saved group", colour mask evidence (PR 17c).** *Images found* adds a picture of each folder's first image to the labels table, from the file header only and display only (`header_preview`, `notebook_folder_previews`): diff --git a/src/morphobricks_preprocessing/cli_describe.py b/src/morphobricks_preprocessing/cli_describe.py index 155ff9f7..c2399065 100644 --- a/src/morphobricks_preprocessing/cli_describe.py +++ b/src/morphobricks_preprocessing/cli_describe.py @@ -4,7 +4,8 @@ about the experiment description (experiment-v1) in plain language: the IDs proposed for the conditions, technical repeats and biological repeats, the repeats and their folders, the references, and every question still open. It reads the folder structure and nothing else (no -image is opened, no pixel read) and writes nothing. +image is opened, no pixel read) and writes nothing. A folder label that is not Unicode NFC is +refused (``R_FOLDER_NOT_NFC``, experiment-v1 section 3.3): nothing is drafted from it. The draft comes from the existing functions only: the legacy ``experiment_design.csv`` of the output folder, when there is one and no published description, is migrated @@ -44,6 +45,7 @@ from .discovery import DiscoveryResult, default_output_folder, discover_experiment from .draft import Draft, DraftError, draft_from_discovery, migrate_legacy_design from .draft_ids import IdKind, id_problems +from .name_issues import non_nfc_folders_issue from .report import format_issues from .run.description_run import load_description from .run.recorded import normalise_relative_path @@ -115,11 +117,14 @@ def describe_command(args: argparse.Namespace) -> int: output = Path(args.output).resolve() if args.output else default_output_folder(root) excluded = sorted({normalise_relative_path(path) for path in args.exclude}) discovery = discover_experiment(root, output) - if has_errors(discovery.issues): - print(format_issues(discovery.issues)) + # A description is drafted from the folders: a label that is not NFC is refused (Spec + # experiment-v1 section 3.3), never normalized or renamed. + issues = [*discovery.issues, *non_nfc_folders_issue(discovery.leaves)] + if has_errors(issues): + print(format_issues(issues)) return 1 - if discovery.issues: - print(format_issues(discovery.issues), end="\n\n") + if issues: + print(format_issues(issues), end="\n\n") found, problems = load_description(output) if problems: print(format_issues(problems)) diff --git a/src/morphobricks_preprocessing/cli_run.py b/src/morphobricks_preprocessing/cli_run.py index 5ee0c399..8ed7898a 100644 --- a/src/morphobricks_preprocessing/cli_run.py +++ b/src/morphobricks_preprocessing/cli_run.py @@ -55,6 +55,7 @@ from .run import MODES, RESTART, Preflight, RunMode, preflight, run_experiment from .run.lock import OutputLockedError, lock_issue, output_lock from .run.output_guard import output_masks_issues +from .run.preflight import description_in_use from .run.recorded import usable from .run.reuse import read_previous, recorded_experiment_root from .validation import ValidationIssue, has_errors @@ -107,6 +108,7 @@ def command_preflight(args: argparse.Namespace) -> Preflight: exclude=args.exclude, masks_folder=args.masks, mask_axes=args.mask_axes, + with_description=description_in_use(args.experiment, args.output), ) diff --git a/src/morphobricks_preprocessing/notebook_check_save.py b/src/morphobricks_preprocessing/notebook_check_save.py index f02bbf10..0b820306 100644 --- a/src/morphobricks_preprocessing/notebook_check_save.py +++ b/src/morphobricks_preprocessing/notebook_check_save.py @@ -40,6 +40,7 @@ ) from .draft_format import DraftFileError from .mask_answers import masks_unanswered +from .name_issues import NAME_CODES from .notebook_channel_thumbs import Thumb from .notebook_check_view import ( SAME_WAY, @@ -238,6 +239,14 @@ def _todo(self, applied: object, answers: object, problem: str) -> tuple[list, l ] if self._design_folder() is None: extra.append("an output folder that can be written (step 1)") + # Spec experiment-v1 section 3.3 and AQ11: names refused at discovery block the save. + found = () if self.preflight is None else self.preflight.issues + refused = {i.code for i in found if i.code in NAME_CODES} + extra += [ + f"the folder or mask type names that step 1 refuses ({code}): rename them on disk " + "and check the folder again" + for code in sorted(refused) + ] return questions, extra def _candidate_check(self, answers: object) -> CandidateCheck | None: diff --git a/src/morphobricks_preprocessing/notebook_folder.py b/src/morphobricks_preprocessing/notebook_folder.py index 2098ed7a..d6834a5f 100644 --- a/src/morphobricks_preprocessing/notebook_folder.py +++ b/src/morphobricks_preprocessing/notebook_folder.py @@ -23,6 +23,7 @@ labels_html, ) from .run import Preflight, preflight +from .run.preflight import description_in_use from .unit_consistency import unit_checked from .validation import ValidationIssue, plural @@ -213,6 +214,7 @@ def check_folder(self) -> Preflight | None: exclude=self._excluded(), masks_folder=masks, mask_axes=mask_axes, + with_description=self._with_description(folder), ) if self._describe: pre = unit_checked(pre) # a described run checks inside each unit @@ -235,6 +237,15 @@ def check_folder(self) -> Preflight | None: self._show_log() return pre + def _with_description(self, folder: str) -> bool: + """Return True when the folder check is that of a description (Spec experiment-v1 §3.3). + + While the user describes the experiment (a described run), and whenever the output + folder holds a published ``morphobricks.toml``, which the run uses: folder and mask + type names are then refused as the run refuses them. + """ + return self._describe or description_in_use(folder, self._output_folder()) + def _show_folder_results(self, pre: Preflight) -> None: """Show the labels, messages and pixel sizes of a new check; the design starts over.""" self.design = None diff --git a/src/morphobricks_preprocessing/notebook_masks.py b/src/morphobricks_preprocessing/notebook_masks.py index 065b297e..2cf93fab 100644 --- a/src/morphobricks_preprocessing/notebook_masks.py +++ b/src/morphobricks_preprocessing/notebook_masks.py @@ -38,7 +38,7 @@ from .notebook_options import _WIDE from .notebook_render import issue_count_chips, issues_html from .run.planned import PlannedMasks, planned_masks -from .run.preflight import remasked +from .run.preflight import description_in_use, remasked from .run.recorded import GivenInputs from .validation import ValidationIssue, has_errors, plural @@ -161,7 +161,13 @@ def _check_masks(self) -> None: plan = self._planned_masks(root) output = self._output_folder() or default_output_folder(root) with self.log.attached(): - self.preflight = remasked(self.preflight, plan.folder, plan.mask_axes or None, output) + self.preflight = remasked( + self.preflight, + plan.folder, + plan.mask_axes or None, + output, + with_description=self._describe or description_in_use(root, output), + ) self._paired_masks = _pairing(plan) self._show_issue_summary() self._refresh_run_step() diff --git a/src/morphobricks_preprocessing/run/mask_step.py b/src/morphobricks_preprocessing/run/mask_step.py index 5f6dbeb7..ea8234ee 100644 --- a/src/morphobricks_preprocessing/run/mask_step.py +++ b/src/morphobricks_preprocessing/run/mask_step.py @@ -36,7 +36,7 @@ from ..validation import ValidationIssue, has_errors from .description_run import load_description from .lock import OutputLockedError, lock_issue, output_lock -from .preflight import Preflight, preflight +from .preflight import Preflight, description_in_use, preflight from .recorded import GivenInputs, InputChange, resolve_inputs, usable from .reuse import read_previous, recorded_experiment_root @@ -201,6 +201,7 @@ def prepare_mask_step( masks_folder=resolved.masks_folder, mask_axes=resolved.mask_axes, bind_headers=True, + with_description=description_in_use(root, output), ) pre = replace(pre, masks=pre.masks or MaskInventory(folder=None)) # none found return MaskStep( diff --git a/src/morphobricks_preprocessing/run/pipeline.py b/src/morphobricks_preprocessing/run/pipeline.py index 011f2b45..24c36fd1 100644 --- a/src/morphobricks_preprocessing/run/pipeline.py +++ b/src/morphobricks_preprocessing/run/pipeline.py @@ -71,7 +71,7 @@ from .lock import OutputLockedError, lock_issue, output_lock from .modes import FULL, RESTART, RunMode, run_mode from .output_guard import output_masks_issues -from .preflight import Preflight, fit_calibration, preflight +from .preflight import Preflight, description_in_use, fit_calibration, preflight from .publish import write_report from .recorded import ( GivenInputs, @@ -256,6 +256,7 @@ def _locked_run(given: GivenInputs, output: Path, options: _Options) -> RunResul masks_folder=inputs.masks_folder, mask_axes=inputs.mask_axes or None, bind_headers=True, + with_description=description_in_use(inputs.experiment_root, output), ) design, design_problems, design_changed = design_for_run(options.design, output, previous[0]) checked = header_checks(pre, inputs, design, options.memory_limit) diff --git a/src/morphobricks_preprocessing/run/preflight.py b/src/morphobricks_preprocessing/run/preflight.py index b9d8ccdf..05d1348f 100644 --- a/src/morphobricks_preprocessing/run/preflight.py +++ b/src/morphobricks_preprocessing/run/preflight.py @@ -23,6 +23,7 @@ from ..calibration_io import load_calibration from ..conversion_record import ConversionReview, review_conversion from ..dataset import DatasetReport, check_dataset +from ..description_file import DESCRIPTION_FILENAME from ..discovery import DiscoveryResult, default_output_folder, discover_experiment from ..fingerprint import Fingerprint, current_fingerprint from ..mask_names import MASK_NAME_CODES, mask_name_issues @@ -166,6 +167,31 @@ def sample_planes( return samples +def description_in_use( + experiment_root: str | os.PathLike, output_folder: str | os.PathLike | None = None +) -> bool: + """Return True when the run's output folder holds a published ``morphobricks.toml``. + + It is what :func:`preflight`'s ``with_description`` is given on the paths that work with + the output folder's description: the run, the mask step, the command line's ``check`` and + ``run``, and the notebook (which also passes True while the user describes the experiment). + A symbolic link counts as published: the run refuses it (``R_FILE_SYMLINK``), and the + names are refused with it. Only whether the name exists is looked at; nothing is read. + + Parameters + ---------- + experiment_root + The experiment folder. + output_folder + The output folder; defaults to :func:`default_output_folder`. + """ + if output_folder is None: + output = default_output_folder(Path(experiment_root).resolve()) + else: + output = Path(output_folder) + return os.path.lexists(output / DESCRIPTION_FILENAME) + + def preflight( experiment_root: str | os.PathLike, output_folder: str | os.PathLike | None = None, @@ -204,7 +230,9 @@ def preflight( True when an experiment description is in use: folder names that are not Unicode NFC (``R_FOLDER_NOT_NFC``) and mask type names that break the Label rules (``R_MASK_TYPE_NAME_INVALID``) or are not NFC are then errors. Without one (the - default) they are accepted as in v0.4 (owner decision Q1). + default) they are accepted as in v0.4 (owner decision Q1). The run, the mask step and + the command line pass :func:`description_in_use`; the notebook also passes True while + the user describes the experiment. progress Called as ``progress(phase, done, total, path)`` while the check runs, with a phase of :data:`.phases.PHASES` (``folders``, ``headers``, ``masks``, ``acquisitions``): a phase diff --git a/tests/test_notebook_name_refusals.py b/tests/test_notebook_name_refusals.py new file mode 100644 index 00000000..9c8d2a22 --- /dev/null +++ b/tests/test_notebook_name_refusals.py @@ -0,0 +1,108 @@ +"""Conformance P21 and P29 in the notebook: the folder check refuses names as the run does. + +Spec experiment-v1 section 3.3 (``R_FOLDER_NOT_NFC``: a folder label read from disk that is not +NFC is refused, never normalized or renamed) and section 8 / AQ11 (``R_MASK_TYPE_NAME_INVALID``: +refused at discovery, never reaching ``E_ID_TEXT``). They apply when a description is in use +(owner decision Q1): while the user describes the experiment (``describe=True``), and whenever +the output folder holds a published ``morphobricks.toml``. A refused name blocks *Check and +save* (section 10: a conflicting unit blocks publication). Without a description, the folder +check is as in v0.4. +""" + +import pytest + +from channel_images import write_masks +from description_tree import build_experiment, write_image +from morphobricks_preprocessing.description import description_from_bytes +from morphobricks_preprocessing.description_file import DESCRIPTION_FILENAME +from spec_pin import fixtures + +pytest.importorskip("ipywidgets") +pytest.importorskip("matplotlib") + +from morphobricks_preprocessing.notebook_ui import PreprocessingNotebook # noqa: E402 +from test_notebook_check_save import FOLDERS, add_folder, answer_all, opened # noqa: E402 + +DECOMPOSED = "Café" +NAME_CODES = {"R_FOLDER_NOT_NFC", "R_MASK_TYPE_NAME_INVALID"} +V01 = next(item for item in fixtures("valid") if item.name.startswith("v01_")) + + +def codes(nb): + return {issue.code for issue in nb.preflight.issues} + + +def paths(nb, code): + return next(issue.paths for issue in nb.preflight.issues if issue.code == code) + + +def tree(tmp_path, extra_folder=None, mask_types=()): + """The notebook tests' experiment, with one more folder and more mask types if asked.""" + root, masks = tmp_path / "exp", tmp_path / "masks" + for i, folder in enumerate(FOLDERS): + add_folder(root, masks, folder, i + 1) + if extra_folder is not None: + add_folder(root, masks, extra_folder, 9) + for mask_type in mask_types: + write_masks(masks, {f"{folder}/a.tif": 1 for folder in FOLDERS}, {mask_type: "labels"}) + return root, masks, tmp_path / "out" + + +def checked(root, masks, out, *, describe): + """Open the notebook and check the folder (step 1) with the masks folder given.""" + nb = PreprocessingNotebook(str(root), output_folder=str(out), describe=describe) + for step in (nb.welcome, nb.folder_step, nb.axis_step, nb.masks_step, nb.run_step): + step() + nb.masks_folder.value = str(masks) + nb.check_button.click() + return nb + + +def test_p21_the_described_folder_check_refuses_a_decomposed_folder(tmp_path): + nb = checked(*tree(tmp_path, f"ctrl/{DECOMPOSED}"), describe=True) + assert paths(nb, "R_FOLDER_NOT_NFC") == (f"ctrl/{DECOMPOSED}",) + assert not nb.preflight.ok + assert (tmp_path / "exp" / "ctrl" / DECOMPOSED).is_dir() # never renamed + + +def test_p29_the_described_folder_check_refuses_the_mask_type_names(tmp_path): + nb = checked(*tree(tmp_path, mask_types=("NA", " cells", DECOMPOSED)), describe=True) + assert paths(nb, "R_MASK_TYPE_NAME_INVALID") == (" cells", "NA") + assert paths(nb, "R_FOLDER_NOT_NFC") == (DECOMPOSED,) + assert {"NA", " cells", DECOMPOSED} <= set(nb.preflight.masks.types) # as discovered + + +def test_p29_pairing_the_masks_again_keeps_the_refusal(tmp_path): + root, masks, out = tree(tmp_path) + nb = checked(root, masks, out, describe=True) + assert not NAME_CODES & codes(nb) + write_masks(masks, {f"{folder}/a.tif": 1 for folder in FOLDERS}, {"NA": "labels"}) + nb.masks_button.click() # the masks are paired again (remasked), the folder is not + assert paths(nb, "R_MASK_TYPE_NAME_INVALID") == ("NA",) + + +def test_p21_p29_a_refused_name_blocks_check_and_save(tmp_path): + root, masks, out = tree(tmp_path, mask_types=("NA",)) + nb = opened(root, masks, out) + answer_all(nb) + nb.mask_card.questions["NA"].choice.value = "instances" + assert not nb.save_experiment() + assert "R_MASK_TYPE_NAME_INVALID" in nb.todo.value + assert not (out / DESCRIPTION_FILENAME).exists() + + +def test_p21_the_v0_4_folder_check_with_a_published_description_refuses_too(tmp_path): + root, out = tmp_path / "exp", tmp_path / "out" + build_experiment(root, description_from_bytes(V01.data)) + write_image(root / "CTRL" / DECOMPOSED / "a.tif", 2) + out.mkdir() + (out / DESCRIPTION_FILENAME).write_bytes(V01.data) # the run uses it, so the check does + nb = checked(root, tmp_path / "masks", out, describe=False) + assert paths(nb, "R_FOLDER_NOT_NFC") == (f"CTRL/{DECOMPOSED}",) + + +def test_p21_p29_without_a_description_the_folder_check_is_as_in_v0_4(tmp_path): + root, masks, out = tree(tmp_path, f"ctrl/{DECOMPOSED}", ("NA", " cells", DECOMPOSED)) + nb = checked(root, masks, out, describe=False) + assert not NAME_CODES & codes(nb) + assert not (out / DESCRIPTION_FILENAME).exists() diff --git a/tests/test_spec_discovery.py b/tests/test_spec_discovery.py index 04318f57..0e7a9a7a 100644 --- a/tests/test_spec_discovery.py +++ b/tests/test_spec_discovery.py @@ -1,26 +1,37 @@ """Conformance P21, P26 (recorded header names) and P29: discovery before any description. Spec experiment-v1 §3.3 and §6, conformance plan section 4. The refusals apply only when a -description is in use (owner decision Q1): a run without one accepts what v0.4 accepted. +description is in use (owner decision Q1): a run without one accepts what v0.4 accepted. P21 and +P29 are shown on their named fixtures (``v01``, ``v07``) through the run, the mask step and the +command line (``check``, ``run``, ``masks``, ``describe``); the notebook's folder check and +*Check and save* are in ``test_notebook_name_refusals.py``. P26's comparison of header names with a description's confirmed names (``R_HEADER_CONTRADICTION``) belongs to the physical checks (Stage 1, PR 8); here the names are recorded as read, untrimmed. """ import numpy as np +import pytest import tifffile +from description_tree import MAPPING, build_experiment, write_image, write_mask from morphobricks_preprocessing import ( prefill_mapping, preflight, read_manifest, run_experiment, ) -from morphobricks_preprocessing.discovery import discover_experiment +from morphobricks_preprocessing.cli import main +from morphobricks_preprocessing.description import description_from_bytes +from morphobricks_preprocessing.discovery import default_output_folder, discover_experiment from morphobricks_preprocessing.metadata import inspect_tiff +from morphobricks_preprocessing.run.mask_step import prepare_mask_step +from morphobricks_preprocessing.run.preflight import description_in_use +from spec_pin import fixtures from tiff_tree import write_data, write_tiff DECOMPOSED = "Café" COMPOSED = "Café" +VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} def codes(issues): @@ -156,3 +167,171 @@ def test_p29_without_a_description_mask_types_are_accepted_as_in_v0_4(tmp_path): pre = preflight(root, masks_folder=masks) assert not {"R_FOLDER_NOT_NFC", "R_MASK_TYPE_NAME_INVALID"} & set(codes(pre.masks.issues)) assert not any(i.is_error for i in pre.masks.issues) + + +# P21 and P29 on their named fixtures, through the production paths -------------------- +# +# A run, the mask step and the command line refuse these names when the output folder holds +# a published description (``description_in_use``); the tests above call ``preflight`` +# directly. Without a published description everything stays as in v0.4 (owner decision Q1). + +NAME_CODES = {"R_FOLDER_NOT_NFC", "R_MASK_TYPE_NAME_INVALID"} +P29_TYPES = ("nuclei", "cells", "NA", " cells", DECOMPOSED) + + +def published(tmp_path, name, *, publish=True): + """Build the tree of fixture ``name``; publish it in the output folder unless told not to.""" + data = VALID[name].data + root, out = tmp_path / "exp", tmp_path / "out" + out.mkdir(parents=True) + if publish: + (out / "morphobricks.toml").write_bytes(data) + return root, out, build_experiment(root, description_from_bytes(data)) + + +def p21_tree(tmp_path, *, publish=True): + """``v01`` with a decomposed folder ``CTRL/Cafe\\u0301`` holding one image.""" + root, out, _ = published(tmp_path, "v01", publish=publish) + write_image(root / "CTRL" / DECOMPOSED / "a.tif", 2) + return root, out + + +def p29_tree(tmp_path, *, publish=True): + """``v07`` with the mask folders of :data:`P29_TYPES`: two declared, three refused.""" + root, out, images = published(tmp_path, "v07", publish=publish) + masks = tmp_path / "masks" + for mask_type in P29_TYPES: + for image in images: + write_mask(masks, mask_type, image, [0, 1, 2]) + return root, out, masks + + +def test_p21_a_run_with_published_v01_refuses_a_decomposed_folder(tmp_path): + root, out = p21_tree(tmp_path) + result = run_experiment(root, MAPPING, output_folder=out) + assert not result.written + found = issue(result.issues, "R_FOLDER_NOT_NFC") + assert found.is_error and found.paths == (f"CTRL/{DECOMPOSED}",) + assert "R_UNIT_NOT_DECLARED" in codes(result.issues) # the unit is new, as before + # Nothing renamed or normalized: the folder is as written, no composed twin appears. + assert (root / "CTRL" / DECOMPOSED).is_dir() + assert sorted(p.name for p in (root / "CTRL").iterdir()) == sorted([DECOMPOSED, "rep1", "rep2"]) + assert not (out / "plane_histograms.csv").exists() + + +def test_p29_a_run_with_published_v07_refuses_the_mask_type_names(tmp_path): + root, out, masks = p29_tree(tmp_path) + result = run_experiment(root, MAPPING, output_folder=out, masks_folder=masks) + assert not result.written + invalid = issue(result.issues, "R_MASK_TYPE_NAME_INVALID") + assert invalid.is_error and invalid.paths == (" cells", "NA") + not_nfc = issue(result.issues, "R_FOLDER_NOT_NFC") + assert not_nfc.is_error and not_nfc.paths == (DECOMPOSED,) + assert "R_MASK_TYPE_NOT_DECLARED" in codes(result.issues) + assert sorted(p.name for p in masks.iterdir()) == sorted(P29_TYPES) # never renamed + + +def test_p21_p29_an_undescribed_run_of_the_same_trees_is_as_in_v0_4(tmp_path): + root, out = p21_tree(tmp_path / "p21", publish=False) + with pytest.warns(UserWarning, match="No experiment description"): + result = run_experiment(root, MAPPING, output_folder=out) + assert not NAME_CODES & set(codes(result.issues)) + root, out, masks = p29_tree(tmp_path / "p29", publish=False) + with pytest.warns(UserWarning, match="No experiment description"): + result = run_experiment(root, MAPPING, output_folder=out, masks_folder=masks) + assert not NAME_CODES & set(codes(result.issues)) + + +def test_p29_the_mask_step_refuses_the_names_with_a_published_description(tmp_path): + root, out, masks = p29_tree(tmp_path) + step = prepare_mask_step(root, out, masks_folder=masks) + assert issue(step.issues, "R_MASK_TYPE_NAME_INVALID").paths == (" cells", "NA") + assert issue(step.issues, "R_FOLDER_NOT_NFC").paths == (DECOMPOSED,) + assert not step.ready + + +def test_p21_the_mask_step_refuses_a_decomposed_experiment_folder(tmp_path): + root, out = p21_tree(tmp_path) + masks = tmp_path / "masks" + write_mask(masks, "nuclei", f"CTRL/{DECOMPOSED}/a.tif", [0, 1]) + step = prepare_mask_step(root, out, masks_folder=masks) + assert issue(step.issues, "R_FOLDER_NOT_NFC").paths == (f"CTRL/{DECOMPOSED}",) + assert not step.ready + + +def test_p21_p29_the_mask_step_without_a_description_is_as_in_v0_4(tmp_path): + root, out, masks = p29_tree(tmp_path, publish=False) + step = prepare_mask_step(root, out, masks_folder=masks) + assert not NAME_CODES & set(codes(step.issues)) + + +def test_description_in_use_looks_only_at_the_published_file(tmp_path): + root, out = tmp_path / "exp", tmp_path / "out" + out.mkdir() + assert not description_in_use(root, out) + (out / "morphobricks.draft.json").write_text("{}", encoding="utf-8") + assert not description_in_use(root, out) # a draft is not a published description + (tmp_path / "elsewhere.toml").write_bytes(VALID["v01"].data) + (out / "morphobricks.toml").symlink_to(tmp_path / "elsewhere.toml") + assert description_in_use(root, out) # the run refuses the link itself (R_FILE_SYMLINK) + default = default_output_folder(root.resolve()) + default.mkdir() + (default / "morphobricks.toml").write_bytes(VALID["v01"].data) + assert description_in_use(root) + + +# The command line ------------------------------------------------------------------------ +# +# The command line prints the plain message; the report it writes holds the code. + +NFC_TEXT = "not in Unicode NFC form" +INVALID_TEXT = "cannot identify a mask type" + + +def reported(out): + """Return the codes of the issues in the output folder's ``discovery_report.json``.""" + report = (out / "discovery_report.json").read_text(encoding="utf-8") + return {code for code in NAME_CODES | {"R_MASK_TYPE_NOT_DECLARED"} if f'"{code}"' in report} + + +def test_p21_the_command_line_check_and_run_refuse_a_decomposed_folder(tmp_path, capsys): + root, out = p21_tree(tmp_path) + assert main(["check", str(root), "--output", str(out)]) == 1 + assert NFC_TEXT in capsys.readouterr().out + assert reported(out) == {"R_FOLDER_NOT_NFC"} + assert main(["run", str(root), "--output", str(out), "--axes", "CYX"]) == 1 + assert NFC_TEXT in capsys.readouterr().out + assert reported(out) == {"R_FOLDER_NOT_NFC"} + assert not (out / "plane_histograms.csv").exists() + + +def test_p29_the_command_line_check_and_masks_refuse_the_mask_type_names(tmp_path, capsys): + root, out, masks = p29_tree(tmp_path) + assert main(["check", str(root), "--output", str(out), "--masks", str(masks)]) == 1 + printed = capsys.readouterr().out + assert INVALID_TEXT in printed and NFC_TEXT in printed + assert {"R_MASK_TYPE_NAME_INVALID", "R_FOLDER_NOT_NFC"} <= reported(out) + assert main(["masks", str(root), "--output", str(out), "--masks", str(masks)]) == 1 + printed = capsys.readouterr().out + assert INVALID_TEXT in printed and NFC_TEXT in printed + assert not (out / "mask_relationships.json").exists() + + +def test_p21_the_command_line_describe_refuses_a_decomposed_folder(tmp_path, capsys): + root, out = p21_tree(tmp_path) + assert main(["describe", str(root), "--output", str(out)]) == 1 + assert NFC_TEXT in capsys.readouterr().out + # Without a published description too: describe drafts a description from the folders. + root, out = p21_tree(tmp_path / "draft", publish=False) + assert main(["describe", str(root), "--output", str(out)]) == 1 + assert NFC_TEXT in capsys.readouterr().out + assert list(out.iterdir()) == [] # describe writes nothing + + +def test_p21_p29_the_command_line_check_without_a_description_is_as_in_v0_4(tmp_path, capsys): + root, out, masks = p29_tree(tmp_path, publish=False) + write_image(root / "CTRL" / DECOMPOSED / "a.tif", 2) + assert main(["check", str(root), "--output", str(out), "--masks", str(masks)]) == 0 + printed = capsys.readouterr().out + assert NFC_TEXT not in printed and INVALID_TEXT not in printed + assert reported(out) == set() From 5cc3b07f94e4d00e652ef00603c51db28db2f0f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 15:18:28 +0000 Subject: [PATCH 07/13] Point the notebook at the P21/P29 fix commit RELEASE_REF (and the test asserting it) now names 0a2e25b, the commit that raises R_FOLDER_NOT_NFC and R_MASK_TYPE_NAME_INVALID on the described paths. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- notebooks/MorphoBricks_Preprocessing.ipynb | 2 +- tests/test_notebook_setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 7029c0d8..09acc794 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -67,7 +67,7 @@ "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", "RELEASE_TAG = \"v0.5.0\"\n", - "RELEASE_REF = \"3b616a0a8791a0a93d5059e96c4cf311f717724d\"\n", + "RELEASE_REF = \"0a2e25b4ff4dd1f638ac65c50e09b43182e71a6d\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", "\n", diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index c137e76d..c481f30e 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -164,7 +164,7 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, target = command[5] assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) - assert namespace["RELEASE_REF"] == "3b616a0a8791a0a93d5059e96c4cf311f717724d" + assert namespace["RELEASE_REF"] == "0a2e25b4ff4dd1f638ac65c50e09b43182e71a6d" assert "pinned v0.5.0 candidate" in capsys.readouterr().out From 76e09c3ee0ee7296872c60de66752446c60452a1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 9 Nov 2026 15:37:23 +0000 Subject: [PATCH 08/13] Gate G1 record: P21 and P29 met on v01 and v07, measured on 5cc3b07 Only docs/conformance.md: the measured commit, the counts (5049 collected, 5045 passed, 4 skipped; Spec modules 1671 passed, 0 skipped), the P21 and P29 rows with the new tests, and the summary against the conformance plan, with the undescribed-parity evidence. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/conformance.md | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/docs/conformance.md b/docs/conformance.md index 85a942e8..d3bd7454 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -10,19 +10,19 @@ in the suite is listed as **not run**, with its reason. | | | |---|---| -| Release candidate | Measured on `a6a0343` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, the notebook's `RELEASE_REF` commit `c74d717`, the first record `4437067` and the G1 gap tests `a6a0343`). The commit that updates this record changes only this file, so the code and tests are those of `a6a0343`. `git diff 3b616a0 a6a0343 -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names | +| Release candidate | Measured on `5cc3b07` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, `c74d717`, the first record `4437067`, the G1 gap tests `a6a0343`, the record `c675f13`, the P21/P29 fix `0a2e25b`, and `5cc3b07`, which points the notebook's `RELEASE_REF` at `0a2e25b`). The commit that updates this record changes only this file, so the code and tests are those of `5cc3b07`. `git diff 0a2e25b 5cc3b07 -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. The earlier measurement on `a6a0343` (5028 passed, 1660 in the Spec modules) found P21 and P29 not met; `0a2e25b` fixes that source defect | | Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d`. The G1 scope ruling (ADR 0001 GS1, conformance plan sections 4 and 5) was merged later as `cdc1841`, which changes only `docs/adr/0001-experiment-description-v1.md` and `docs/conformance-plan.md`, so the pinned files are unchanged. Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | | Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0, normalization 0.3.4); run as root on a case-sensitive file system | | Commands | `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/` (the whole suite, with a per-test JUnit report), and `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules) | -| Whole suite | 5032 collected: 5028 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | -| Spec modules (`tests/test_spec_*.py`) | 1660 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | -| `tools/check.py` (on `a6a0343`) | Ruff clean, Ruff format 410 files already formatted, pytest 5028 passed and 4 skipped, code rules passed (exit status 0) | +| Whole suite | 5049 collected: 5045 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | +| Spec modules (`tests/test_spec_*.py`) | 1671 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | +| `tools/check.py` (on `5cc3b07`) | Ruff clean, Ruff format 411 files already formatted, pytest 5045 passed and 4 skipped, code rules passed (exit status 0) | -**How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1660 test cases +**How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1671 test cases from `test_spec_*` modules, every one passed: none has a `skipped` (or xfail) element. (2) The `-rsxX` summary of the run lists exactly the four skips below, none in a Spec module, and no xfail or xpass. (3) All 21 modules of `EXPECTED_MODULES` have test cases in the report -(catalogue 376, derived design 81, discovery 11, encoding 58, gate 9, no fallback 15, pin 16, +(catalogue 376, derived design 81, discovery 22, encoding 58, gate 9, no fallback 15, pin 16, publication 20, reader 330, reader phases 1-3 287, reader refusal paths 8, reconcile 25, records 22, records goldens 76, refval cross-check 203, rules table 14, run aliases 16, run description 9, RunInputs semantic 8, transitions 73, vendored missing 3), and the gate's @@ -75,7 +75,7 @@ GS1: a scenario that names fixtures is shown on those fixtures; further fixtures | P18 | Passed | `test_spec_publication`: `test_p18_publishing_over_nothing_with_the_absent_base_is_accepted`, `test_p18_another_file_appears_before_publication`, `test_p18_a_draft_without_a_usable_base_is_refused_and_nothing_is_published` (8 cases) | | P19 | Passed | `test_spec_publication`: `test_p19_a_retired_id_declared_again_is_refused`, `test_p19_also_when_the_returning_folder_is_the_one_the_retired_id_had` (`v25`, `t03`) | | P20 | Passed | `test_spec_publication`: `test_p20_a_dropped_used_id_is_refused` (`v25`, `t04`), `test_p20_removing_the_last_member_of_a_condition_needs_the_recorded_confirmation` (on `v01`); `test_spec_records`: `test_p35_v25_is_what_removing_the_last_members_of_drug_a_from_v01_publishes` (the accepted removal publishes `v25` byte for byte, DrugA kept in `used_ids`) | -| P21 | **Not met on `v01`** | `test_spec_discovery` (4 `test_p21_*`, passed) shows `R_FOLDER_NOT_NFC` only through `preflight(..., with_description=True)` on a synthetic tree. With `v01` published and a decomposed folder `CTRL/Cafe\u0301` added, `run_experiment` refuses with `R_UNIT_NOT_DECLARED` and `repeat_missing_conditions` but **not** `R_FOLDER_NOT_NFC` (my run on `a6a0343`; nothing renamed). No production caller of `preflight` passes `with_description=True` (run pipeline, mask step, notebook folder check, command line). A source defect, reported to the owner and not fixed here: a fix moves `RELEASE_REF` | +| P21 | Passed | `test_spec_discovery` on `v01` published with a decomposed folder `CTRL/Cafe\u0301` added: `test_p21_a_run_with_published_v01_refuses_a_decomposed_folder` (`run_experiment` refuses with `R_FOLDER_NOT_NFC` naming `CTRL/Cafe\u0301`, beside `R_UNIT_NOT_DECLARED`; nothing written, nothing renamed, no composed twin), `test_p21_the_mask_step_refuses_a_decomposed_experiment_folder`, `test_p21_the_command_line_check_and_run_refuse_a_decomposed_folder`, `test_p21_the_command_line_describe_refuses_a_decomposed_folder`; `test_notebook_name_refusals`: `test_p21_the_described_folder_check_refuses_a_decomposed_folder`, `test_p21_the_v0_4_folder_check_with_a_published_description_refuses_too`, `test_p21_p29_a_refused_name_blocks_check_and_save`. Without a description: `test_p21_p29_an_undescribed_run_of_the_same_trees_is_as_in_v0_4`, `test_p21_p29_the_mask_step_without_a_description_is_as_in_v0_4`, `test_p21_p29_the_command_line_check_without_a_description_is_as_in_v0_4`, `test_p21_p29_without_a_description_the_folder_check_is_as_in_v0_4`. Also the 4 synthetic-tree `test_p21_*` of `preflight(..., with_description=True)`. Fixed in `0a2e25b` (on `a6a0343` the run gave only `R_UNIT_NOT_DECLARED`): each of these fails on `c675f13` except the four without a description | | P22 | Passed | `test_spec_publication`: `test_p22_confirmed_on_stays_byte_for_byte_through_an_unrelated_publication`, `test_p22_a_new_date_on_an_unchanged_repeat_is_refused` (`t05`), `test_p22_after_a_reopening_and_a_new_confirmation_the_new_date_is_accepted` (`v01`) | | P23 | Passed | `test_physical_check`: `test_p23_a_unit_declared_rgb_whose_images_are_two_channel_grayscale` (`v28`), `test_p23_an_rgb_tiff_in_a_unit_that_does_not_declare_it` (`v01`), `test_p23_pixels_that_look_like_rgb_are_never_examined` | | P24 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (provider manifest `rgb_support`). Preprocessing's own parts passed: the `A_RGB_NOT_SUPPORTED` class (`test_a_error_classes`) and the section 2 "RGB" row (below) | @@ -83,7 +83,7 @@ GS1: a scenario that names fixtures is shown on those fixtures; further fixtures | P26 | Passed | `test_spec_run_aliases` (9 `test_p26_*`), `test_physical_check` (`test_p26_*`, 2 functions), `test_spec_discovery` (`test_p26_the_header_names_are_recorded_as_read_untrimmed`, `test_p26_the_manifest_records_the_header_name_as_read`): 19 cases (`v11`, `v31`, `v12`) | | P27 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (per-request `binary_mask_policy`). Preprocessing's own parts passed: `A_MASK_NOT_INSTANCES` (`test_a_error_classes`) and the section 2 "Mask fields" row (`i111`, `i112`, below) | | P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`, with the 0-5 binary batch); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` (`v08`), `test_p28_on_v09_each_batch_of_masks_is_unchecked_until_the_mask_step_reads_it` (`v09`: three batches, each `not yet checked` and `A_MASK_UNCHECKED` until the mask step reads it, earlier batches stay `checked`) | -| P29 | **Not met on `v07`** | `test_spec_discovery` (5 `test_p29_*`, passed) shows `R_MASK_TYPE_NAME_INVALID` and `R_FOLDER_NOT_NFC` only through `preflight(..., with_description=True)` on a synthetic tree. With `v07` published and mask folders `NA`, ` cells` and `Cafe\u0301` beside `nuclei` and `cells`, `run_experiment` refuses with `R_MASK_TYPE_NOT_DECLARED` (and `mask_missing`) but **not** `R_MASK_TYPE_NAME_INVALID` or `R_FOLDER_NOT_NFC` (my run on `a6a0343`; nothing renamed). Same source defect as P21 | +| P29 | Passed | `test_spec_discovery` on `v07` published with mask folders `NA`, ` cells` and `Cafe\u0301` beside `nuclei` and `cells`: `test_p29_a_run_with_published_v07_refuses_the_mask_type_names` (`run_experiment` refuses with `R_MASK_TYPE_NAME_INVALID` naming ` cells`, `NA` and `R_FOLDER_NOT_NFC` naming `Cafe\u0301`, beside `R_MASK_TYPE_NOT_DECLARED`; nothing renamed), `test_p29_the_mask_step_refuses_the_names_with_a_published_description` (the step is not ready), `test_p29_the_command_line_check_and_masks_refuse_the_mask_type_names` (no `mask_relationships.json`); `test_notebook_name_refusals`: `test_p29_the_described_folder_check_refuses_the_mask_type_names`, `test_p29_pairing_the_masks_again_keeps_the_refusal`, `test_p21_p29_a_refused_name_blocks_check_and_save` (no `morphobricks.toml` is published, so the names never reach `E_ID_TEXT`). Without a description: the four `test_p21_p29_*_as_in_v0_4` above. Also the 5 synthetic-tree `test_p29_*`. Fixed in `0a2e25b` (on `a6a0343` the run gave only `R_MASK_TYPE_NOT_DECLARED`); the new tests fail on `c675f13` except those without a description | | P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date` | | P31 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is MorphoCoverage's (prefill, confirmation). Preprocessing's own parts passed: `v27` valid with no warning, and editing `target` or `treatment_key` of `v27` changes no digest and no derived row (section 2 rows below) | | P32 | Passed | `test_draft`: `test_p32_forbidden_folders_get_no_proposal_and_nothing_is_adopted`, `test_p32_ordinary_folders_are_proposed_as_their_labels`; `test_spec_derived_design`: golden `v33`, `v34` and `test_the_file_names_folders_never_ids` | @@ -146,14 +146,20 @@ Gate G1 scope, ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`): P24, P27, P31 and the part of the section 2 "Provenance" row are deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`); nothing else is deferred. Against that scope this run shows: -- P01-P20, P22, P23, P25, P26, P28, P30, P33-P35 passed on the fixtures the plan names. P32 +- P01-P23, P25, P26, P28-P30, P33-P35 passed on the fixtures the plan names. P32 passed on trees with the `v33` and `v34` folders and on the `v33`, `v34` goldens. The section 2 rows in the table above passed (Provenance: its `i99` part). -- **P21 and P29 are not met on their named fixtures (`v01`, `v07`).** Their codes are produced - only by `preflight(..., with_description=True)`, which no production caller passes, so a run - with a published description does not report `R_FOLDER_NOT_NFC` or `R_MASK_TYPE_NAME_INVALID` - (it refuses with `R_UNIT_NOT_DECLARED` or `R_MASK_TYPE_NOT_DECLARED` instead). This is a - source defect, left for the owner: fixing it changes `src` and so moves `RELEASE_REF`. +- P21 and P29, not met when measured on `a6a0343` (their codes came only from + `preflight(..., with_description=True)`, which no production caller passed), are met on `v01` + and `v07` since `0a2e25b`: the run, the mask step, the command line (`check`, `run`, `masks`, + `describe`) and the notebook's folder check refuse the names whenever a description is in use + (owner decision Q1), and *Check and save* does not publish while one is refused. Without a + published description the run, mask step and check are unchanged: the same trees run without + a description give byte-identical `plane_histograms.csv`, `mask_relationships.json`, + `experiment_design.csv` and reports on `c675f13` and `5cc3b07`, the manifests differing only in + `written_at` (and so the reports in the manifest's SHA-256); `test_parity_v0_4_0` (24) passed. + The command line's `describe` now refuses a decomposed folder (exit status 1) where it drafted it + before: it drafts a description, so section 3.3 applies. - The Colab and JupyterLab browser check is the owner's, before tagging. So this record does not by itself declare any reader conforming; deciding Gate G1 is the owner's. From ba415ded3ad3c5a49c008d145208430d12c78e8d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:07:03 +0000 Subject: [PATCH 09/13] Point the notebook at the merge of main into the release candidate RELEASE_REF (and the test asserting it) now names 0ea3254, the merge of origin/main (#254, #255, #257) into claude/release-0.5.0, which carries the P21/P29 fix. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- notebooks/MorphoBricks_Preprocessing.ipynb | 2 +- tests/test_notebook_setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 09acc794..42d18fc2 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -67,7 +67,7 @@ "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", "RELEASE_TAG = \"v0.5.0\"\n", - "RELEASE_REF = \"0a2e25b4ff4dd1f638ac65c50e09b43182e71a6d\"\n", + "RELEASE_REF = \"0ea3254418fb808feddb0a235fabc025bd855e9e\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", "\n", diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index c481f30e..f51861e4 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -164,7 +164,7 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, target = command[5] assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) - assert namespace["RELEASE_REF"] == "0a2e25b4ff4dd1f638ac65c50e09b43182e71a6d" + assert namespace["RELEASE_REF"] == "0ea3254418fb808feddb0a235fabc025bd855e9e" assert "pinned v0.5.0 candidate" in capsys.readouterr().out From 43d125790c01cad6d93d5294b7d5f7859a6c1102 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:25:57 +0000 Subject: [PATCH 10/13] Gate G1 record: measured on ba415de after the merge of main Only docs/conformance.md: the measured commit (ba415de, whose RELEASE_REF is the merge 0ea3254), the counts (5109 collected, 5105 passed, 4 skipped; Spec modules 1671 passed, 0 skipped), tools/check.py (415 files, 5105 passed), the P30 answers now stating the image type, and the fixes merged from main: audit batches 33/34a (#254), 34b (#255) and the pre-release follow-ups (#257). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/conformance.md | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/docs/conformance.md b/docs/conformance.md index d3bd7454..e3dfc8af 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -10,13 +10,13 @@ in the suite is listed as **not run**, with its reason. | | | |---|---| -| Release candidate | Measured on `5cc3b07` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, `c74d717`, the first record `4437067`, the G1 gap tests `a6a0343`, the record `c675f13`, the P21/P29 fix `0a2e25b`, and `5cc3b07`, which points the notebook's `RELEASE_REF` at `0a2e25b`). The commit that updates this record changes only this file, so the code and tests are those of `5cc3b07`. `git diff 0a2e25b 5cc3b07 -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. The earlier measurement on `a6a0343` (5028 passed, 1660 in the Spec modules) found P21 and P29 not met; `0a2e25b` fixes that source defect | +| Release candidate | Measured on `ba415de` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, `c74d717`, the first record `4437067`, the G1 gap tests `a6a0343`, the record `c675f13`, the P21/P29 fix `0a2e25b`, `5cc3b07`, the record `76e09c3`, then `0ea3254`, the merge of `main` at `8c573d6` (#254, #255, #257: the audit fixes listed below), and `ba415de`, which points the notebook's `RELEASE_REF` at `0ea3254`). The commit that updates this record changes only this file, so the code and tests are those of `ba415de`. `git diff 0ea3254 ba415de -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. Earlier measurements: on `a6a0343` (5028 passed, 1660 in the Spec modules) P21 and P29 were not met, and `0a2e25b` fixes that source defect; on `5cc3b07`, before the merge of `main`, 5045 passed and 1671 in the Spec modules | | Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d`. The G1 scope ruling (ADR 0001 GS1, conformance plan sections 4 and 5) was merged later as `cdc1841`, which changes only `docs/adr/0001-experiment-description-v1.md` and `docs/conformance-plan.md`, so the pinned files are unchanged. Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | | Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0, normalization 0.3.4); run as root on a case-sensitive file system | | Commands | `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/` (the whole suite, with a per-test JUnit report), and `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules) | -| Whole suite | 5049 collected: 5045 passed, 4 skipped (not run, below), 0 failed, 0 xfailed | +| Whole suite | 5109 collected: 5105 passed, 4 skipped (not run, below), 0 failed, 0 xfailed (the 60 more than on `5cc3b07` are the tests of #254, #255 and #257) | | Spec modules (`tests/test_spec_*.py`) | 1671 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | -| `tools/check.py` (on `5cc3b07`) | Ruff clean, Ruff format 411 files already formatted, pytest 5045 passed and 4 skipped, code rules passed (exit status 0) | +| `tools/check.py` (on `ba415de`) | Ruff clean, Ruff format 415 files already formatted, pytest 5105 passed and 4 skipped, code rules passed (exit status 0) | **How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1671 test cases from `test_spec_*` modules, every one passed: none has a `skipped` (or xfail) element. (2) The @@ -47,6 +47,31 @@ oracle (`test_spec_derived_design.py`) passed. These four behaviours are **not verified by this run**. +### Fixes in this candidate since the previous measurement (merged from `main` in `0ea3254`) + +Each with its own tests, all passed in this run; none changes a Spec module or the Spec pin. + +- **Audit batches 33 and 34a (#254):** what is published is what was answered, of the image + checked. A block's saved tags are exactly those shown; a unit tagging one label twice is + refused (`E_DUPLICATE_CHANNEL_IN_UNIT`); an unanswered image type is an open question + (`unit_image_type_unanswered`); `publish_draft` requires each unit's channel count + (`unit_channels_stale`, `unit_channel_count_unknown`); the notebook's folder check binds each + header to its file's fingerprint (`bind_headers=True`, beside the P21/P29 `with_description`), + and pictures, mask evidence and the save are of that revision; records that the records reader + would refuse are never written (`R_RECORDS_FILE_INVALID`). +- **Audit batch 34b (#255):** *Confirm these axes* is offered only while a preview of exactly + the offered roles is shown; the reopen reconciles with the folder check's header evidence + (channel count, raw header names, the RGB flag), so a header rewritten under the same files is + a channel change, not "nothing changed"; the axis preview is bound with the same header + fingerprints; non-finite pixels are no longer counted as crop detail. +- **Pre-release follow-ups (#257):** "prepared and imaged the same way?" is asked again only for + a replicate whose declared layout changes against the published one (experiment-v1 §10), + every other `confirmed_on` kept byte for byte (§4.4a); header contradictions are the run's + `R_HEADER_CONTRADICTION`, shown at once; *Check and save* runs publication's transition checks + (`R_CONFIRMED_ON_CHANGED`, `R_CONFIRMED_ID_CHANGED`, `R_USED_ID_*`); `describe` reads the + headers (no pixel) when it compares a published description; a unit's channel count comes from + every image of the unit. + ### Physical scenarios P01-P35 (conformance plan section 3) "Passed" means every test named passed in this run, on the fixtures the plan names (ADR 0001 @@ -84,7 +109,7 @@ GS1: a scenario that names fixtures is shown on those fixtures; further fixtures | P27 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (per-request `binary_mask_policy`). Preprocessing's own parts passed: `A_MASK_NOT_INSTANCES` (`test_a_error_classes`) and the section 2 "Mask fields" row (`i111`, `i112`, below) | | P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`, with the 0-5 binary batch); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` (`v08`), `test_p28_on_v09_each_batch_of_masks_is_unchecked_until_the_mask_step_reads_it` (`v09`: three batches, each `not yet checked` and `A_MASK_UNCHECKED` until the mask step reads it, earlier batches stay `checked`) | | P29 | Passed | `test_spec_discovery` on `v07` published with mask folders `NA`, ` cells` and `Cafe\u0301` beside `nuclei` and `cells`: `test_p29_a_run_with_published_v07_refuses_the_mask_type_names` (`run_experiment` refuses with `R_MASK_TYPE_NAME_INVALID` naming ` cells`, `NA` and `R_FOLDER_NOT_NFC` naming `Cafe\u0301`, beside `R_MASK_TYPE_NOT_DECLARED`; nothing renamed), `test_p29_the_mask_step_refuses_the_names_with_a_published_description` (the step is not ready), `test_p29_the_command_line_check_and_masks_refuse_the_mask_type_names` (no `mask_relationships.json`); `test_notebook_name_refusals`: `test_p29_the_described_folder_check_refuses_the_mask_type_names`, `test_p29_pairing_the_masks_again_keeps_the_refusal`, `test_p21_p29_a_refused_name_blocks_check_and_save` (no `morphobricks.toml` is published, so the names never reach `E_ID_TEXT`). Without a description: the four `test_p21_p29_*_as_in_v0_4` above. Also the 5 synthetic-tree `test_p29_*`. Fixed in `0a2e25b` (on `a6a0343` the run gave only `R_MASK_TYPE_NOT_DECLARED`); the new tests fail on `c675f13` except those without a description | -| P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date` | +| P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date`. The answers state each unit's image type ("channels"): since audit batch 33 an unanswered image type is an open question (`unit_image_type_unanswered`), so the test answers it, as the merge `0ea3254` records | | P31 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is MorphoCoverage's (prefill, confirmation). Preprocessing's own parts passed: `v27` valid with no warning, and editing `target` or `treatment_key` of `v27` changes no digest and no derived row (section 2 rows below) | | P32 | Passed | `test_draft`: `test_p32_forbidden_folders_get_no_proposal_and_nothing_is_adopted`, `test_p32_ordinary_folders_are_proposed_as_their_labels`; `test_spec_derived_design`: golden `v33`, `v34` and `test_the_file_names_folders_never_ids` | | P33 | Passed | `test_spec_reconcile`: `test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id`, `test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats`, `test_p33_c_the_old_folder_removal_needs_its_confirmation_and_the_id_stays_used` (`v33`; (a) asserts that the hand-made attachment is `t12` and gives exactly `R_CONFIRMED_ID_CHANGED`); `t10`, `t11`, `t12` against their predecessors in `test_spec_transitions`; `test_condition_folder_reopens` (2) | From fd7afd339bcaab7579c806a28a65bb7ecebe940c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:18:32 +0000 Subject: [PATCH 11/13] Final review of v0.5.0: two confirmations the user did not give 1. A header refusal lifted by an answer about another cell. Tagging any channel of any folder recomputed every label's header_names from every saved tag (channel_answers.synced). After images were rewritten with other header names (for example two channels swapped), tagging one channel of an unchanged folder made the R_HEADER_CONTRADICTION of the changed folders vanish: Check and save said the images agree, the save published it and the run wrote the table. Now a tag confirms the header name of the cell tagged and nothing else: synced() takes the cells of the answer, drops the names no tagged cell has any more and adds a name only for those cells. The refusal stays until the channel it names is tagged again (the resolution #257 documents). 2. A "same way?" kept for other folders or another layout in a draft. With nothing published, the answer saved in morphobricks.draft.json was kept when a folder joined or left the replicate before the next folder check (carry_over keeps repeats by label), and when a tag or an image type changed after it (reasked ran only against a published file); it was then published as the user's confirmation. Now the draft file's date is kept only when every unit the repeat has now has its whole saved channel list and type and no unit with a saved answer left it (published_answers.resumed), and an answer that changes a repeat's declared layout reopens it with or without a published file; a repeat tagged back to the layout it was answered for keeps its date. The names card says which replicate is asked again. Tests: tests/test_notebook_final_review.py (10; 8 fail before the fix), and two existing tests follow the new rule (the alias of a cell not tagged by the answer; "same way?" asked again after the image type of a folder changed). Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- CHANGELOG.md | 15 ++ docs/contract.md | 4 +- .../channel_answers.py | 29 ++- .../channel_labels.py | 7 +- .../notebook_channels_grid.py | 10 +- .../notebook_described.py | 35 ++- .../notebook_description_saved.py | 18 ++ .../published_answers.py | 63 +++++- tests/test_channel_labels.py | 4 +- tests/test_notebook_final_review.py | 206 ++++++++++++++++++ tests/test_notebook_stains_na.py | 6 + 11 files changed, 375 insertions(+), 22 deletions(-) create mode 100644 tests/test_notebook_final_review.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cbb85ce..43a06d6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,21 @@ of this release candidate is in `docs/conformance.md`. The normalization brick s v0.3.4. It also carries the audit batches 33, 34a and 34b (#254, #255) and their pre-release follow-ups (#257). +- **Fixed (final review of the release candidate): two confirmations the user did not give.** + (1) *A header refusal lifted by an answer about another cell.* Tagging any channel of any + folder recomputed every label's `header_names` from every saved tag, so after images were + rewritten with other header names (for example two channels swapped) the + `R_HEADER_CONTRADICTION` of folders the user had not tagged again vanished, *Check and save* + said the images agree and the run wrote the table. Now a tag confirms the header name of the + cell tagged, and nothing else: names no tagged cell has any more are dropped, and a name is + added only for the cells of that answer. The refusal stays until the channel it names is + tagged again. (2) *A "same way?" kept for other folders or another layout in a draft.* With + nothing published yet, the answer saved in `morphobricks.draft.json` was kept when a folder + joined or left the replicate before the next folder check, and when a tag or an image type + changed after it, and it was published as the user's confirmation. Now it is asked again in + both cases, as it already was against a published file (experiment-v1 §10); the names card + says which replicate is asked again, and a replicate tagged back to the layout it was + answered for keeps its answer. - **Fixed: `R_FOLDER_NOT_NFC` and `R_MASK_TYPE_NAME_INVALID` on the production paths (conformance P21, P29).** They were raised only by `preflight(..., with_description=True)`, which no production caller passed, so a run with a published `morphobricks.toml` refused a diff --git a/docs/contract.md b/docs/contract.md index 804bd591..c84cd460 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -680,7 +680,7 @@ One JSON object, UTF-8, two-space indent, sorted keys, LF, a final newline, no d | `repeats` | The pairing label of the draft's repeat (the folder name, or the legacy `repeat_group`) → object with only the answered keys, from `id`, `confirmed_on`, `display_name` | | `pairing` | Condition source folder → source repeat folder → the pairing label of the repeat the user paired that folder with (only folders not in the repeat of their own name; applied to the draft first, so the `repeats` keys are the paired labels; refused with `pairing_after_confirmation` once a repeat is named or confirmed, `unknown_key` for a folder the draft lacks). A reference to the condition itself or one that closes a loop is refused when saved (`E_REFERENCE_SELF`, `E_REFERENCE_CYCLE`). | | `units` | `condition folder` → `source repeat folder` → `technical-repeat folder` → `{channels, image_type?, display_name?}`; `channels` lists, for C0, C1, ... in order, a channel ID (every position needs a label the user declared, for example `unused`; an empty string is refused) or `null` (not tagged yet); `image_type` is `"rgb"` or `"channels"` (an ordinary channel image; absent means not answered), any other value is refused | -| `channels` | Channel ID → `{display_name?, header_names?}` (the label list of the notebook's *Your channels* screen; `header_names` are the raw header names of every cell the user tagged with that label, recomputed at each save: the tag is the alias confirmation) | +| `channels` | Channel ID → `{display_name?, header_names?}` (the label list of the notebook's *Your channels* screen; `header_names` are the raw header names of the cells the user tagged with that label: the tag is the alias confirmation, of the name the cell's header has when it is tagged. At each tag the names no tagged cell has any more are dropped, and only the names of the cells just tagged are added: a tag saved while the header had another name never confirms the name it has now, so `R_HEADER_CONTRADICTION` stays until that cell is tagged again) | | `masks` | Mask ID → `{source_type, representation, source_channels?, display_name?}` (the notebook's *Your masks* screen: one answer per mask type, `representation` `instances` or `binary` with no default; the ID is the mask type's folder name when it is a valid mask ID, else typed by the user; `source_channels` are declared channel IDs) | Rules. @@ -689,6 +689,6 @@ Rules. - **Publishing** (`publish_draft`, the notebook's *Check and save*) publishes under the lock and then writes the derived `experiment_design.csv` (Spec experiment-v1 §11); in a described run nothing else writes that file, so the draft (which carries the pairing) and the CSV cannot disagree. Reopening a published file with no draft takes its answers from the file; removed folders are removed only on the user's *Yes, remove* (`removals`). - **Saving** (`save_draft`) is atomic (temporary file, one replace) and under the output folder's lock, after each answer. The answers are applied to the draft and the exact bytes are read back with the strict reader first, so an answer that would be refused on resume (a `spec_commit` other than the pinned one, `spec_commit_mismatch`; an ID the reader would refuse, a folder the draft does not have, a date that is not a real day) is refused when saved and nothing is written. An incomplete draft can be saved. The file is replaced, not merged. A symbolic link at the draft's path is refused and never replaced. - **Reading** is strict and never repairs: bytes that are not UTF-8 JSON, a byte-order mark, a duplicate key, `NaN`, another `format`, an unknown `version`, an unknown key at any level, a missing key, a value of the wrong type or a date that is not a real `YYYY-MM-DD` day are `DraftFileError` (`code`: `malformed`, `unknown_version`, `unknown_key`, `R_DRAFT_BASE_MISSING`, `R_FILE_SYMLINK`). Reading accepts any 40-digit `spec_commit`; whether it is the current one is decided at resume and save, below. -- **Resume** (`load_draft`): the file's `spec_commit` must equal the pinned `SPEC_COMMIT`, else `DraftFileError` with code `spec_commit_mismatch` (owner decision, 2026-11-08): the confirmations in the file were given under other rules, so none is carried forward. The message names both commits, says the published `morphobricks.toml` is untouched and says to start a fresh draft from the folders. Library code never rewrites the stored commit and never deletes or replaces the stale draft; the caller derives a fresh draft and saves it, which replaces the file. `publish_draft` resumes first, so it refuses the same way and writes nothing. Then the file's `base_sha256` must equal the SHA-256 of the file published now (or `absent` if there is none), else `R_PUBLISHED_FILE_CHANGED` (`PublicationRefused`); the draft is never merged with the new file, it is derived again from it. An answer keyed by a folder or repeat the draft no longer has is `DraftError` (`unknown_key`), never moved to another folder. +- **Resume** (`load_draft`): the file's `spec_commit` must equal the pinned `SPEC_COMMIT`, else `DraftFileError` with code `spec_commit_mismatch` (owner decision, 2026-11-08): the confirmations in the file were given under other rules, so none is carried forward. The message names both commits, says the published `morphobricks.toml` is untouched and says to start a fresh draft from the folders. Library code never rewrites the stored commit and never deletes or replaces the stale draft; the caller derives a fresh draft and saves it, which replaces the file. `publish_draft` resumes first, so it refuses the same way and writes nothing. Then the file's `base_sha256` must equal the SHA-256 of the file published now (or `absent` if there is none), else `R_PUBLISHED_FILE_CHANGED` (`PublicationRefused`); the draft is never merged with the new file, it is derived again from it. An answer keyed by a folder or repeat the draft no longer has is `DraftError` (`unknown_key`), never moved to another folder. In the notebook's described run, a repeat's `confirmed_on` read from the draft file is kept only when every unit the repeat has now has its whole saved channel list and image type and no unit with a saved answer has left it (experiment-v1 §10: a new or removed unit or member reopens the confirmation); otherwise it is dropped and the repeat is asked again, and the card names it. The same holds for an answer given in the draft with no published file: a change of a unit's declared channel list, image type or grouping after the confirmation drops it. - **Draft to description** (`description_from_draft(draft, answers, *, channel_counts=None)`, pure): refuses with `DraftIncomplete`, listing **every** missing confirmation once (`id_pending`, `repeat_not_confirmed`, `confirmed_on_missing`, `control_unresolved`, `legacy_folder_missing`, `unit_channels_missing` for a unit with no channel list, `unit_image_type_unanswered` for a unit whose image type is not answered `channels` or `rgb` (an old draft without one reopens with the question asked, never as `channels`), `unit_channel_untagged` for a position still `null`, `channel_undeclared` for a channel a unit names that is not declared; with `channel_counts`, the channel count of each unit's image now read through the confirmed axis roles, also `unit_channels_stale` for tags saved for another count and `unit_channel_count_unknown` for a unit without a count), otherwise returns the `Description`. A unit tagging one label at two positions is refused by `apply_answers` before anything is saved (`E_DUPLICATE_CHANNEL_IN_UNIT`). Nothing is confirmed for the user. `used_ids` is the draft's retired IDs plus every ID confirmed now. The description is then validated by the strict reader at publication; the function does not repeat the reader's rules. - **Publication** (`publish_draft(..., channel_counts=...)`, the counts required): under one hold of the lock the draft is resumed, the description built with the counts, published with all the checks of `publish_description` (symbolic link, base, transition, strict reader on the exact bytes), and then the draft file is deleted. Any refusal writes nothing: the draft and the published file stay byte for byte as they were. If the draft cannot be deleted after a successful publication, the error is raised and the publication stands; the leftover draft records a base that no longer matches and is refused on resume. diff --git a/src/morphobricks_preprocessing/channel_answers.py b/src/morphobricks_preprocessing/channel_answers.py index 6a0a2bcf..6c54993d 100644 --- a/src/morphobricks_preprocessing/channel_answers.py +++ b/src/morphobricks_preprocessing/channel_answers.py @@ -130,19 +130,30 @@ def untagged_by(answers: DraftAnswers, labels: Collection[str]) -> int: ) -def synced(answers: DraftAnswers, layouts: Sequence[UnitLayout]) -> DraftAnswers: +def synced( + answers: DraftAnswers, + layouts: Sequence[UnitLayout], + tagged: Collection[tuple[UnitKey, int]] = (), +) -> DraftAnswers: """Return the answers with the derived parts recomputed from the saved tags. - Each label's ``header_names`` become the raw header names of every cell tagged with it (None - when none has a name), and ``R``, ``G``, ``B`` are declared exactly when some unit is a - colour unit. + ``tagged`` are the cells ``(unit key, C index)`` the user tagged with this answer. A label's + ``header_names`` are the raw header names of the cells tagged with it that the user + confirmed: the names of the cells of ``tagged``, and of the names the label already had, + those some tagged cell still has (None when none is left). A tag saved while the cell's + header had another name confirms nothing about the name it has now: that name is never + added for the user, so the run's ``R_HEADER_CONTRADICTION`` stays until that cell is + tagged again. ``R``, ``G``, ``B`` are declared exactly when some unit is a colour unit. """ found = aliases(layouts, answers.units) - channels = { - k: replace(v, header_names=found.get(k)) - for k, v in answers.channels.items() - if k not in RGB_IDS - } + fresh = aliases(layouts, answers.units, set(tagged)) + channels = {} + for key, channel in answers.channels.items(): + if key in RGB_IDS: + continue + before = set(channel.header_names or ()) + names = {name for name in found.get(key, ()) if name in before} | set(fresh.get(key, ())) + channels[key] = replace(channel, header_names=tuple(sorted(names)) or None) if any(u.image_type == "rgb" for u in answers.units.values()): channels.update({token: ChannelAnswer() for token in RGB_IDS}) return replace(answers, channels=channels) diff --git a/src/morphobricks_preprocessing/channel_labels.py b/src/morphobricks_preprocessing/channel_labels.py index ceff0f25..0c9e0f0d 100644 --- a/src/morphobricks_preprocessing/channel_labels.py +++ b/src/morphobricks_preprocessing/channel_labels.py @@ -294,13 +294,16 @@ def positions( def aliases( - layouts: Sequence[UnitLayout], units: Mapping[UnitKey, UnitAnswer] + layouts: Sequence[UnitLayout], + units: Mapping[UnitKey, UnitAnswer], + only: Collection[tuple[UnitKey, int]] | None = None, ) -> dict[str, tuple[str, ...]]: """Return the raw header names of every cell tagged with a label, by label, sorted. The tag is the user's confirmation that the header name is evidence for the label (experiment-v1 section 4.3); names are kept exactly as read. A colour unit has none (``R``, ``G`` and ``B`` take no aliases), and a label whose cells have no header name is omitted. + With ``only``, just the cells ``(unit key, C index)`` it lists are read. """ found: dict[str, set[str]] = {} for layout in layouts: @@ -309,6 +312,8 @@ def aliases( continue for index, tag in enumerate(tags_of(layout, answer)): name = layout.name(index) + if only is not None and (layout.key, index) not in only: + continue if tag and name: found.setdefault(tag, set()).add(name) return {label: tuple(sorted(names)) for label, names in found.items()} diff --git a/src/morphobricks_preprocessing/notebook_channels_grid.py b/src/morphobricks_preprocessing/notebook_channels_grid.py index a7aac0e4..c6e66ca2 100644 --- a/src/morphobricks_preprocessing/notebook_channels_grid.py +++ b/src/morphobricks_preprocessing/notebook_channels_grid.py @@ -221,7 +221,8 @@ def _tagged(self, units: Sequence[UnitLayout], index: int, value: str | None) -> tags[index] = value for layout in units: # every folder of the block holds these tags answers = with_tags(answers, layout, tags) - if self.store.update(synced(answers, self.layouts)): + cells = [(layout.key, index) for layout in units] # the cells this answer tags + if self.store.update(synced(answers, self.layouts, cells)): self.message = "" else: self.message = ui.status("error", "Not saved.", [self.store.refusal()]) @@ -249,11 +250,13 @@ def _confirm_layout(self, units: Sequence[UnitLayout]) -> None: """ answers = self.store.answers() rgb = [u.key for u in units if self.place_types and self.types.proposed(u)] + cells = [] for layout in units: shown, proposed = self.effective(layout) if proposed: answers = with_tags(answers, layout, shown) - ok = self.store.update(synced(answers, self.layouts)) + cells += [(layout.key, index) for index in proposed] + ok = self.store.update(synced(answers, self.layouts, cells)) self.message = "" if ok else ui.status("error", "Not saved.", [self.store.refusal()]) if ok and rgb: self.types.confirm_header(rgb) @@ -303,7 +306,8 @@ def tag_like(self, units: Sequence[UnitLayout], tags: tuple[str, ...]) -> None: answers = self.store.answers() for layout in units: answers = with_tags(answers, layout, tags) - ok = self.store.update(synced(answers, self.layouts)) + cells = [(layout.key, index) for layout in units for index in range(layout.channels)] + ok = self.store.update(synced(answers, self.layouts, cells)) self.message = "" if ok else ui.status("error", "Not saved.", [self.store.refusal()]) self.refresh() self._on_change() diff --git a/src/morphobricks_preprocessing/notebook_described.py b/src/morphobricks_preprocessing/notebook_described.py index 2ecd4f0a..22561d36 100644 --- a/src/morphobricks_preprocessing/notebook_described.py +++ b/src/morphobricks_preprocessing/notebook_described.py @@ -40,7 +40,14 @@ from .draft_answers import DraftAnswers, UnitKey, apply_answers, carry_over from .fingerprint import current_fingerprint from .notebook_row_fold import reopen_units -from .published_answers import published_answers, reasked, reopened, retired_ids +from .published_answers import ( + confirmed_layouts, + published_answers, + reasked, + reopened, + resumed, + retired_ids, +) from .run import Preflight @@ -215,15 +222,35 @@ def _answer(self, answers: DraftAnswers, *, rebuild: bool = True) -> bool: dropped (:func:`.published_answers.reasked`). A header name added to a channel changes no layout and keeps the date (section 4.4a). Nothing is ticked for the user. """ - state = self._published() if self._describe else None before = self.draft_answers - if state is not None and before is not None and self._draft_base is not None: + if self._describe and before is not None and self._draft_base is not None: + state = self._published() + published = None if state is None else state.description try: - answers, _ = reasked(self._draft_base, before, answers, state.description) + # The layouts "same way?" was answered for: kept while a tag passes through + # no tag, forgotten at each new check of the folders (the base draft changes). + if getattr(self, "_same_way_base", None) is not self._draft_base: + self._same_way_base, self._same_way_for = self._draft_base, {} + self._same_way_for.update(confirmed_layouts(self._draft_base, before)) + answers, _ = reasked( + self._draft_base, before, answers, published, self._same_way_for + ) except DraftError: pass # the answer itself is refused below, with its reason return super()._answer(answers, rebuild=rebuild) + def _still_confirmed( + self, saved: DraftAnswers, kept: DraftAnswers + ) -> tuple[DraftAnswers, tuple[str, ...]]: + """Take back the saved "same way?" of a replicate whose folders are not those answered. + + The draft file is read after a new check of the folders: a folder that joined or left + a replicate since the answer reopens it (:func:`.published_answers.resumed`). + """ + if not self._describe: + return super()._still_confirmed(saved, kept) + return resumed(self._draft_base, saved, kept) + def _sync_description(self) -> None: """Take the design card's pairing into the answers; changed replicates are asked again.""" if not self._describe: diff --git a/src/morphobricks_preprocessing/notebook_description_saved.py b/src/morphobricks_preprocessing/notebook_description_saved.py index 3b49300b..29996e43 100644 --- a/src/morphobricks_preprocessing/notebook_description_saved.py +++ b/src/morphobricks_preprocessing/notebook_description_saved.py @@ -43,17 +43,35 @@ def _read_saved_draft(self, output: Path) -> tuple[DraftAnswers | None, str]: pairing = self._resume_pairing(saved) answers, dropped = carry_over(self._draft_base, saved, pairing) try: + answers, asked = self._still_confirmed(saved, answers) apply_answers(self._draft_base, answers) except DraftError as exc: return None, self._set_aside(str(exc)) lines = [f"From {DRAFT_FILENAME} in the output folder."] if dict(saved.pairing) != pairing: lines.append("The pairing of the experiment design above is used.") + if asked: + lines.append( + "Asked again on Check and save (a folder joined or left it since your answer " + '"the same way", or its channels were not all tagged then): ' + + ", ".join(asked) + + "." + ) if dropped: lines.append("No longer in the experiment, so not kept: " + "; ".join(dropped) + ".") + if dropped or asked: return answers, ui.status("warn", "Your saved answers were read.", lines) return answers, ui.status("info", "Your saved answers were read.", lines) + def _still_confirmed( + self, saved: DraftAnswers, kept: DraftAnswers + ) -> tuple[DraftAnswers, tuple[str, ...]]: + """Return the kept answers and the repeats asked again (none here: the design card asks). + + A described run overrides this: its "same way?" answers live in the draft. + """ + return kept, () + def _resume_pairing(self, saved: DraftAnswers) -> dict[tuple[str, str], str]: """Return the pairing that holds on resume: the experiment design card's.""" return self._design_pairing() diff --git a/src/morphobricks_preprocessing/published_answers.py b/src/morphobricks_preprocessing/published_answers.py index 57875fcb..4f8fc004 100644 --- a/src/morphobricks_preprocessing/published_answers.py +++ b/src/morphobricks_preprocessing/published_answers.py @@ -17,6 +17,7 @@ from __future__ import annotations +from collections.abc import Mapping from dataclasses import replace from .description import USED_KINDS, Description @@ -213,8 +214,26 @@ def _declared(layout: Layout | None) -> bool: ) +def confirmed_layouts(draft: Draft, answers: DraftAnswers) -> dict[str, Layout]: + """Return the layout of every repeat that has its date and declares every unit's channels. + + The layout a "same way?" answer stands for, as long as the answers hold both. + """ + return { + label: layout + for label, layout in layouts(draft, answers).items() + if _declared(layout) + and label in answers.repeats + and answers.repeats[label].confirmed_on is not None + } + + def reasked( - draft: Draft, before: DraftAnswers, after: DraftAnswers, published: Description + draft: Draft, + before: DraftAnswers, + after: DraftAnswers, + published: Description | None, + given: Mapping[str, Layout] | None = None, ) -> tuple[DraftAnswers, tuple[str, ...]]: """Return ``after`` with the date of every repeat an answer just reopened taken back. @@ -225,9 +244,17 @@ def reasked( asked again; nothing is ever ticked here. A layout with an untagged channel declares no list yet (a tag being changed passes through it), so it reopens nothing by itself: the candidate cannot be saved while a tag is missing. + + ``given`` are the layouts the user answered "same way?" for in the draft + (:func:`confirmed_layouts` of the answers that last held a whole layout with its date): a + repeat that comes back to that layout keeps its date, as one that comes back to the + published layout does. Without a published file (``published`` None) no layout is the + published one: a "same way?" answered in the draft stands for the layout it was given for, + and for no other. """ old, new = layouts(draft, before), layouts(draft, after) - saved = published_layouts(published) + saved = {} if published is None else published_layouts(published) + answered = given or {} changed = tuple( sorted( label @@ -236,6 +263,7 @@ def reasked( and _declared(new.get(label)) and new.get(label) != old.get(label) and new.get(label) != saved.get(label) + and new.get(label) != answered.get(label) ) ) repeats = dict(after.repeats) @@ -244,6 +272,35 @@ def reasked( return replace(after, repeats=repeats), changed +def resumed( + draft: Draft, saved: DraftAnswers, kept: DraftAnswers +) -> tuple[DraftAnswers, tuple[str, ...]]: + """Return ``kept`` with the "same way?" taken back where the folders are not those answered. + + ``saved`` are the answers as the draft file holds them and ``kept`` those that still apply + to ``draft``, the draft of the folders now (:func:`.draft_answers.carry_over`). The file + does not hold the folders a "same way?" was answered for, only the answers of each unit. + A repeat's date is kept only when every unit it has now has its whole saved channel list + and type, and no unit with a saved answer has left it (section 10: a new or removed unit or + member reopens the confirmation). A folder that joined the repeat has no saved answer, so + the repeat is asked again; so is a repeat confirmed before its channels were all tagged + (tagging them reopens it anyway, :func:`reasked`). Nothing is ticked here. + """ + now = layouts(draft, kept) + left = {saved.pairing.get(key[:2], key[1]) for key in saved.units if key not in kept.units} + changed = tuple( + sorted( + label + for label, answer in kept.repeats.items() + if answer.confirmed_on is not None and (label in left or not _declared(now.get(label))) + ) + ) + repeats = dict(kept.repeats) + for label in changed: + repeats[label] = replace(repeats[label], confirmed_on=None) + return replace(kept, repeats=repeats), changed + + def kept_date( draft: Draft, answers: DraftAnswers, published: Description, label: str ) -> str | None: @@ -264,11 +321,13 @@ def kept_date( __all__ = [ "CHANNELS_TYPE", + "confirmed_layouts", "kept_date", "layouts", "published_answers", "published_layouts", "reasked", "reopened", + "resumed", "retired_ids", ] diff --git a/tests/test_channel_labels.py b/tests/test_channel_labels.py index c42deb37..d007cd5d 100644 --- a/tests/test_channel_labels.py +++ b/tests/test_channel_labels.py @@ -160,7 +160,9 @@ def test_tags_labels_and_types_change_one_slice_and_keep_the_rest(): tagged = with_tags(with_labels(base, {"DNA": "DAPI", "ACTIN": None}), lay, ["DNA", None]) assert tagged.units[lay.key] == UnitAnswer(("DNA", None)) assert tagged.channels["DNA"].display_name == "DAPI" - assert synced(tagged, [lay]).channels["DNA"].header_names == ("DAPI",) + # the tag confirms the header name of the cell tagged; a tag saved earlier confirms none + assert synced(tagged, [lay], [(lay.key, 0)]).channels["DNA"].header_names == ("DAPI",) + assert synced(tagged, [lay]).channels["DNA"].header_names is None # a label that is gone takes its tags back, never moves them to another label gone = with_labels(tagged, {"ACTIN": None}) assert gone.units[lay.key].channels == (None, None) diff --git a/tests/test_notebook_final_review.py b/tests/test_notebook_final_review.py new file mode 100644 index 00000000..a6728505 --- /dev/null +++ b/tests/test_notebook_final_review.py @@ -0,0 +1,206 @@ +"""Final review of v0.5.0: what is confirmed is what the user answered, for the folders shown. + +- A header refusal is never lifted by an answer about another cell. A tag confirms the header + name of the cell the user tagged; a tag saved while the cell's header had another name + confirms nothing about the name it has now. Before the fix, tagging any channel of any folder + recomputed every label's header names from every saved tag, so ``R_HEADER_CONTRADICTION`` of + folders the user never looked at again vanished, the save said "the images agree" and the run + wrote the table. +- A "same way?" answered in a draft (nothing published yet) is the answer for the folders and + the layout it was given for. Before the fix it was kept when a folder joined or left the + replicate before the next check, and when a tag or an image type changed after it, and it + was published as the user's confirmation. +""" + +import re +import shutil + +import pytest + +from channel_images import write_channels +from morphobricks_preprocessing.description_file import DESCRIPTION_FILENAME +from test_notebook_check_save import ( # noqa: F401 + FOLDERS, + KINDS, + add_folder, + answer_all, + exp, + opened, + saved_nb, + tag, +) + +pytest.importorskip("ipywidgets") +pytest.importorskip("matplotlib") + +CTRL, DRUG = ("ctrl", "rep1", "single"), ("drug", "rep1", "single") + + +def _text(html): + return re.sub(r"<[^>]+>", " ", html) + + +def _dates(nb): + return {label: r.confirmed_on for label, r in nb.draft_answers.repeats.items()} + + +def _retag(nb, key, index, label): + """Tag one channel again by hand (through no tag, as a re-tag passes).""" + nb.channel_grid.cells[(key, index)].value = None + nb.channel_grid.cells[(key, index)].value = label + + +# --- a header refusal is lifted only by tagging the cell it names ---------------------------- + + +def _saved_then_drug_renamed(folders): + """Save the experiment, then rewrite the drug folders with channel 1 named Hoechst.""" + saved_nb(folders) + for seed, folder in enumerate(FOLDERS[2:]): + write_channels(folders[0] / folder / "a.tif", ["Hoechst", "Actin", "Ch3"], KINDS, seed=seed) + nb = opened(*folders) + nb._refresh_check_save() + assert "R_HEADER_CONTRADICTION" in _text(nb.checks.value) + assert nb.draft_answers.channels["DNA"].header_names == ("DAPI",) + return nb + + +def test_a_header_refusal_is_not_lifted_by_tagging_another_folder(exp): # noqa: F811 + nb = _saved_then_drug_renamed(exp) + _retag(nb, CTRL, 2, "PHASE") # a folder whose images did not change + nb._refresh_check_save() + assert nb.draft_answers.channels["DNA"].header_names == ("DAPI",) # Hoechst: not confirmed + shown = _text(nb.checks.value) + assert "The images contradict it: the run would stop" in shown + assert "R_HEADER_CONTRADICTION" in shown and "'Hoechst'" in shown + assert not nb._check.can_run and not nb._run_allowed() and nb.run() is None + + +def test_a_header_refusal_is_not_lifted_by_tagging_another_channel_of_the_folder(exp): # noqa: F811 + nb = _saved_then_drug_renamed(exp) + _retag(nb, DRUG, 2, "PHASE") # the changed folders, but not the channel renamed + nb._refresh_check_save() + assert nb.draft_answers.channels["DNA"].header_names == ("DAPI",) + assert "R_HEADER_CONTRADICTION" in _text(nb.checks.value) and not nb._check.can_run + + +def test_tagging_the_renamed_channel_again_confirms_its_name_and_lifts_the_refusal(exp): # noqa: F811 + nb = _saved_then_drug_renamed(exp) + _retag(nb, DRUG, 0, "DNA") # the user looks at channel 1 of the drug folders: still DNA + nb._refresh_check_save() + assert set(nb.draft_answers.channels["DNA"].header_names) == {"DAPI", "Hoechst"} + assert nb._check.can_run and not nb._check.refusals + assert nb.save_experiment(), _text(nb.save_status.value) + assert nb.run().written + + +def test_swapped_channels_stay_refused_until_their_own_cells_are_tagged(exp): # noqa: F811 + # A draft (nothing published): the drug images are rewritten with the two channels swapped. + root, masks, out = exp + first = opened(*exp) + answer_all(first) + for seed, folder in enumerate(FOLDERS[2:]): + swapped = ["filaments", "blobs", "cells"] + write_channels(root / folder / "a.tif", ["Actin", "DAPI", "Ch3"], swapped, seed=seed) + nb = opened(*exp) + _retag(nb, CTRL, 2, "PHASE") + nb._refresh_check_save() + assert nb.draft_answers.channels["DNA"].header_names == ("DAPI",) + assert nb.draft_answers.channels["ACTIN"].header_names == ("Actin",) + assert "R_HEADER_CONTRADICTION" in _text(nb.checks.value) and not nb._check.can_run + for index in (0, 1): + nb.channel_grid.cells[(DRUG, index)].value = None + nb.channel_grid.cells[(DRUG, 0)].value = "ACTIN" + nb.channel_grid.cells[(DRUG, 1)].value = "DNA" + nb._refresh_check_save() + assert nb.draft_answers.channels["DNA"].header_names == ("DAPI",) + assert "R_HEADER_CONTRADICTION" not in _text(nb.checks.value) + + +# --- "same way?" in a draft is the answer for the folders and layout it was given for -------- + + +def _joined(nb, out): + """Answer what the new folder needs; the replicate it joined must be asked again.""" + assert _dates(nb) == {"rep1": None, "rep2": _dates(nb)["rep2"]} and _dates(nb)["rep2"] + assert "Asked again on Check and save" in nb.names_status.value + tag(nb) + nb.reference_boxes["drugB"].value = "ctrl" + nb._refresh_check_save() + assert not nb.same_way["rep1"].value and nb.same_way["rep2"].value + assert "replicate rep1: answer" in _text(nb.todo.value) + assert nb.save_button.disabled and not nb.save_experiment() + assert not (out / DESCRIPTION_FILENAME).exists() + nb.same_way["rep1"].button.click() # the user's answer, for the three folders + assert nb.save_experiment(), _text(nb.save_status.value) + assert "[repeats.rep1.members.drugB]" in (out / DESCRIPTION_FILENAME).read_text("utf-8") + + +def test_a_folder_that_joined_a_replicate_reopens_its_same_way_in_a_resumed_draft(exp): # noqa: F811 + root, masks, out = exp + answer_all(opened(*exp)) # every answer in the draft, nothing published + add_folder(root, masks, "drugB/rep1", 9) + _joined(opened(*exp), out) # a new notebook + + +def test_a_folder_that_joined_a_replicate_reopens_its_same_way_on_a_new_check(exp): # noqa: F811 + root, masks, out = exp + nb = opened(*exp) + answer_all(nb) + add_folder(root, masks, "drugB/rep1", 9) + nb.check_button.click() # the same notebook + nb.preview_checked.set(True) + nb.show_preview() + nb.confirm_button.click() + _joined(nb, out) + + +def test_a_folder_that_left_a_replicate_reopens_its_same_way_in_a_resumed_draft(exp): # noqa: F811 + root, masks, out = exp + answer_all(opened(*exp)) + shutil.rmtree(root / "drug" / "rep1") + nb = opened(*exp) + assert _dates(nb)["rep1"] is None and _dates(nb)["rep2"] + nb._refresh_check_save() + assert nb.save_button.disabled and not nb.same_way["rep1"].value + + +def test_a_draft_resumed_on_unchanged_folders_keeps_every_same_way(exp): # noqa: F811 + first = opened(*exp) + answer_all(first) + nb = opened(*exp) + assert _dates(nb) == _dates(first) and all(_dates(nb).values()) + assert "Asked again" not in nb.names_status.value + nb._refresh_check_save() + assert not nb.save_button.disabled and nb.save_experiment() + + +def test_a_tag_changed_after_same_way_asks_it_again_in_a_draft(exp): # noqa: F811 + root, masks, out = exp + nb = opened(*exp) + answer_all(nb) + assert all(_dates(nb).values()) + for index in (0, 1): + nb.channel_grid.cells[(CTRL, index)].value = None + assert all(_dates(nb).values()) # an untagged channel on the way declares no list + nb.channel_grid.cells[(CTRL, 0)].value = "DNA" + nb.channel_grid.cells[(CTRL, 1)].value = "ACTIN" + assert all(_dates(nb).values()) # tagged as it was answered for: nothing is asked again + for index in (0, 1): + nb.channel_grid.cells[(CTRL, index)].value = None + nb.channel_grid.cells[(CTRL, 0)].value = "ACTIN" + nb.channel_grid.cells[(CTRL, 1)].value = "DNA" + assert _dates(nb) == {"rep1": None, "rep2": None} # the four folders are one block + nb._refresh_check_save() + assert not any(toggle.value for toggle in nb.same_way.values()) + assert nb.save_button.disabled and not nb.save_experiment() + assert not (out / DESCRIPTION_FILENAME).exists() + + +def test_an_image_type_changed_after_same_way_asks_it_again_in_a_draft(exp): # noqa: F811 + nb = opened(*exp) + answer_all(nb) + nb.channel_types.radios[("drug", "rep2", "single")].value = "rgb" + assert _dates(nb)["rep2"] is None and _dates(nb)["rep1"] + nb._refresh_check_save() + assert nb.save_button.disabled and not nb.same_way["rep2"].value diff --git a/tests/test_notebook_stains_na.py b/tests/test_notebook_stains_na.py index c29632f8..17a5d68c 100644 --- a/tests/test_notebook_stains_na.py +++ b/tests/test_notebook_stains_na.py @@ -146,6 +146,12 @@ def test_labels_declared_before_every_folder_became_colour_stay_saved_and_unused nb.channel_types.radios[key].value = "rgb" # every folder is colour again assert nb._stains_not_applicable and card.labels == ["ACTIN", "DNA", "PHASE"] assert "stay saved, unused" in card.na_note.value + # The folder's image type changed after its replicate's "same way?" was answered: that + # answer was for another layout, so it is asked again, never kept for the user. + assert nb.save_button.disabled and not nb.save_experiment() + asked = [label for label, toggle in nb.same_way.items() if not toggle.value] + assert asked == ["rep1"] + nb.same_way["rep1"].button.click() assert nb.save_experiment() text = (tmp_path / "out" / DESCRIPTION_FILENAME).read_text(encoding="utf-8") result = refval().validate(text.encode("utf-8"))[0] From 1e5390c9db4e1c9dcf0212864a26e3fc5c5a72a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:18:32 +0000 Subject: [PATCH 12/13] Point the notebook at the final-review fix RELEASE_REF (and the test asserting it) now names fd7afd339bcaab7579c806a28a65bb7ecebe940c, the commit that carries the two final-review fixes on top of the merge of main (0ea3254). Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- notebooks/MorphoBricks_Preprocessing.ipynb | 2 +- tests/test_notebook_setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 42d18fc2..3644a17c 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -67,7 +67,7 @@ "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", "RELEASE_TAG = \"v0.5.0\"\n", - "RELEASE_REF = \"0ea3254418fb808feddb0a235fabc025bd855e9e\"\n", + "RELEASE_REF = \"fd7afd339bcaab7579c806a28a65bb7ecebe940c\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", "\n", diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index f51861e4..126c9068 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -164,7 +164,7 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, target = command[5] assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) - assert namespace["RELEASE_REF"] == "0ea3254418fb808feddb0a235fabc025bd855e9e" + assert namespace["RELEASE_REF"] == "fd7afd339bcaab7579c806a28a65bb7ecebe940c" assert "pinned v0.5.0 candidate" in capsys.readouterr().out From 18673aa75b9ab34665d07e881cbdcdfc45415e1b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:38:26 +0000 Subject: [PATCH 13/13] Gate G1 record: measured on 1e5390c after the final-review fix The record of the release candidate v0.5.0 is measured again on 1e5390c (the final-review fix fd7afd3 and the RELEASE_REF commit): 5115 passed, 4 skipped (environment, outside the Spec modules), Spec modules 1671 passed with 0 skipped, tools/check.py exit 0. It lists the two final-review fixes. Only this file changes. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/conformance.md | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/conformance.md b/docs/conformance.md index e3dfc8af..1e01ee8f 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -6,17 +6,17 @@ real run; a skip is never counted as a pass. Owner rulings of 8 November 2026 (i "no skips" applies to the Spec conformance modules (`tests/test_spec_*.py`); a skip elsewhere in the suite is listed as **not run**, with its reason. -## Gate G1 record: release candidate v0.5.0 (9 November 2026) +## Gate G1 record: release candidate v0.5.0 (10 November 2026) | | | |---|---| -| Release candidate | Measured on `ba415de` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, `c74d717`, the first record `4437067`, the G1 gap tests `a6a0343`, the record `c675f13`, the P21/P29 fix `0a2e25b`, `5cc3b07`, the record `76e09c3`, then `0ea3254`, the merge of `main` at `8c573d6` (#254, #255, #257: the audit fixes listed below), and `ba415de`, which points the notebook's `RELEASE_REF` at `0ea3254`). The commit that updates this record changes only this file, so the code and tests are those of `ba415de`. `git diff 0ea3254 ba415de -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. Earlier measurements: on `a6a0343` (5028 passed, 1660 in the Spec modules) P21 and P29 were not met, and `0a2e25b` fixes that source defect; on `5cc3b07`, before the merge of `main`, 5045 passed and 1671 in the Spec modules | +| Release candidate | Measured on `1e5390c` (branch `claude/release-0.5.0`: `main` at `bf733ee`, the v0.5.0 preparation `3b616a0`, `c74d717`, the first record `4437067`, the G1 gap tests `a6a0343`, the record `c675f13`, the P21/P29 fix `0a2e25b`, `5cc3b07`, the record `76e09c3`, then `0ea3254`, the merge of `main` at `8c573d6` (#254, #255, #257: the audit fixes listed below), `ba415de`, the record `43d1257`, then the final-review fix `fd7afd3` (below), and `1e5390c`, which points the notebook's `RELEASE_REF` at `fd7afd3`). The commit that updates this record changes only this file, so the code and tests are those of `1e5390c`. `git diff fd7afd3 1e5390c -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. Earlier measurements: on `ba415de`, before the final review, 5105 passed and 1671 in the Spec modules; on `a6a0343` (5028 passed, 1660 in the Spec modules) P21 and P29 were not met, and `0a2e25b` fixes that source defect; on `5cc3b07`, before the merge of `main`, 5045 passed and 1671 in the Spec modules | | Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `1bed024e278d104ba4b8f63ae76c020fb2e3470d`. The G1 scope ruling (ADR 0001 GS1, conformance plan sections 4 and 5) was merged later as `cdc1841`, which changes only `docs/adr/0001-experiment-description-v1.md` and `docs/conformance-plan.md`, so the pinned files are unchanged. Fixtures tree `f61b1c2`, `experiment-v1.md` blob `b12678c`, `refval.py` blob `0ea8309`, `refrecords.py` blob `ef06a1e`, each also compared by hand with `git rev-parse 1bed024:` in a fresh clone of the Spec: all four equal | | Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0, normalization 0.3.4); run as root on a case-sensitive file system | | Commands | `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/` (the whole suite, with a per-test JUnit report), and `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules) | -| Whole suite | 5109 collected: 5105 passed, 4 skipped (not run, below), 0 failed, 0 xfailed (the 60 more than on `5cc3b07` are the tests of #254, #255 and #257) | +| Whole suite | 5119 collected: 5115 passed, 4 skipped (not run, below), 0 failed, 0 xfailed (the 10 more than on `ba415de` are `tests/test_notebook_final_review.py`; the 60 more on `ba415de` than on `5cc3b07` are the tests of #254, #255 and #257) | | Spec modules (`tests/test_spec_*.py`) | 1671 passed, **0 skipped, 0 xfailed, 0 failed**, from all 21 modules of `tests/spec_gate.py` `EXPECTED_MODULES` | -| `tools/check.py` (on `ba415de`) | Ruff clean, Ruff format 415 files already formatted, pytest 5105 passed and 4 skipped, code rules passed (exit status 0) | +| `tools/check.py` (on `1e5390c`) | Ruff clean, Ruff format 416 files already formatted, pytest 5115 passed and 4 skipped, code rules passed (exit status 0) | **How "no skips" was checked for the Spec modules.** (1) The JUnit report has 1671 test cases from `test_spec_*` modules, every one passed: none has a `skipped` (or xfail) element. (2) The @@ -47,7 +47,26 @@ oracle (`test_spec_derived_design.py`) passed. These four behaviours are **not verified by this run**. -### Fixes in this candidate since the previous measurement (merged from `main` in `0ea3254`) +### Final-review fixes (`fd7afd3`), since the measurement on `ba415de` + +Found by the final adversarial review of the candidate, by driving the notebook headless; each +has regression tests in `tests/test_notebook_final_review.py` (10 tests, 8 of which fail on +`ba415de`), all passed in this run. No Spec module and no Spec pin changes. + +- **A header refusal lifted by an answer about another cell.** Tagging any channel of any folder + recomputed every label's `header_names` from every saved tag, so after images were rewritten + with other header names the `R_HEADER_CONTRADICTION` of folders the user had not tagged again + vanished, the save published it and the run wrote the table. Now a tag confirms the header + name of the cell tagged and nothing else (`channel_answers.synced`); the refusal stays until + the channel it names is tagged again. +- **A "same way?" kept for other folders or another layout in a draft.** With nothing published, + the date saved in `morphobricks.draft.json` was kept when a folder joined or left the + replicate, and when a tag or an image type changed after it, and was then published as the + user's confirmation. Now it is asked again (`published_answers.resumed`, and `reasked` with or + without a published file); a replicate brought back to the layout it was answered for keeps + its answer. + +### Fixes merged from `main` in `0ea3254` (measured first on `ba415de`) Each with its own tests, all passed in this run; none changes a Spec module or the Spec pin.