From 53f1c72c3c650bb95456294d3457539daf0af4c8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 18:19:16 +0000 Subject: [PATCH 1/2] Drop the legacy experiment_design.csv migration (PP-03) Spec ADR 0002 AR10 (9) and experiment-v1 section 7 (MorphoBricks-Spec PR 25): an experiment_design.csv in an output folder is never read to propose or prefill a draft. - Re-vendor the Spec at cc9820f with tools/vendor.py (--check identical); SPEC_COMMIT follows. - Remove migrate_legacy_design, LegacyMigrationRefused, MigrationNote, the legacy rule check, Draft.origin/legacy_missing/notes/unresolved_controls, the legacy_folder_missing and control_unresolved questions, and their sites in describe and the notebook text table. - Keep parse_design_bytes and design_issues (the written file's tests and the run's check use them). - P30 follows the amended Spec; migration tests deleted; add tests that an orphan experiment_design.csv is ignored by draft creation, describe and the notebook. - Docs and CHANGELOG (Breaking). Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- AGENTS.md | 9 +- CHANGELOG.md | 28 +++ docs/conformance.md | 2 +- docs/contract.md | 8 +- docs/user_guide.md | 22 +- .../cli_describe.py | 56 ++--- src/morphobricks_preprocessing/design.py | 8 +- .../design_derive.py | 2 +- src/morphobricks_preprocessing/draft.py | 220 +----------------- .../draft_answers.py | 6 +- .../draft_format.py | 2 +- .../notebook_check_view.py | 2 - .../run/draft_file.py | 4 +- tests/spec_fixtures/PIN.toml | 6 +- tests/spec_fixtures/README.md | 46 ++-- tests/spec_fixtures/docs/experiment-v1.md | 49 ++-- tests/spec_fixtures/fixtures/README.md | 20 +- tests/test_cli_describe.py | 70 +++--- tests/test_design.py | 6 +- tests/test_design_file.py | 2 +- tests/test_draft.py | 210 +---------------- tests/test_draft_answers.py | 29 +-- tests/test_orphan_design_csv.py | 84 +++++++ tests/test_spec_derived_design.py | 95 ++------ 24 files changed, 255 insertions(+), 731 deletions(-) create mode 100644 tests/test_orphan_design_csv.py diff --git a/AGENTS.md b/AGENTS.md index 3e0a4d17..bf611838 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,7 +90,7 @@ src/morphobricks_preprocessing/ design.py the rows of experiment_design.csv (an output only since v0.5.1): DesignTable/ DesignRow (read-only), the run's reconcile_design and design_issues against the folders, write_experiment_design, parse_design_bytes (the strict parser - of the draft's legacy migration); no reader for other bricks (SWEEP-PP) + of a written file; nothing reads the file to prefill a draft); no reader for other bricks (SWEEP-PP) design_derive.py derive_design/write_derived_design: experiment_design.csv from a published description (Spec experiment-v1 11), written by design.write_experiment_design design_values.py the design file's header check (no column twice or unnamed, #83), booleans, @@ -129,7 +129,7 @@ src/morphobricks_preprocessing/ draft_ids.py proposed IDs: the folder label or raw header name only when the reader would accept it as written, else none (never repaired); draft.py the draft (IdSlot proposed/confirmed, open questions, unit_rows with condition_id, - repeat_group, technical_repeat_id) and migrate_legacy_design + repeat_group, technical_repeat_id) image_type.py image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the brick's check_rgb), image_type_issues (rgb_not_possible) @@ -663,8 +663,9 @@ functions the tests use. `load_design`, the strict `read_confirmed_design`/`ConfirmedDesign`/`DesignCheckError` and `RunInputs.design()` are removed; bricks take `repeat_group` from the description (`Fov.repeat_group`). `parse_design_bytes` (strict, `DesignFormatError` naming line and - column, also for a header naming a column twice or not at all, #83) remains for the draft's - experiment-v1 §7 migration of a legacy file. + column, also for a header naming a column twice or not at all, #83) remains to check a + written file; the draft's legacy migration was removed (Spec ADR 0002 AR10 (9)), and no code + path reads an `experiment_design.csv` to propose or prefill a draft. - **The run's check:** `reconcile_design` keeps the derived rows of known folders, reopens only groups that gain a folder (`design_not_confirmed`), drops removed folders. - **v0.5.1: an output only.** Every run is described (`A_DESCRIPTION_REQUIRED` without a diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d0bc070..c2cef5d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,34 @@ v0.5.1: no run before v0.5.1 is read.) ## Unreleased (v0.5.1) +### The legacy `experiment_design.csv` migration is removed (inventory PP-03) + +Spec ADR 0002 AR10 (9) and experiment-v1 §7 (MorphoBricks-Spec PR 25): no backward +compatibility before the first pipeline release. The Spec is re-vendored at `cc9820f` +(`tools/vendor.py`; `--check`: identical). +`draft_format.SPEC_COMMIT` follows the pin, so an answers draft file (`morphobricks.draft.json`) +saved under `1bed024` is refused as before (`spec_commit_mismatch`) and not carried over. + +- **Breaking: an old `experiment_design.csv` is no longer migrated.** Preprocessing never reads + an `experiment_design.csv` from an output folder to propose or prefill a draft or a + description (Spec §7); `describe` and the notebook build the draft from the folders alone. + Describe the experiment in the notebook. The derived `experiment_design.csv` stays an output. +- Removed: `draft.migrate_legacy_design`, `LegacyMigrationRefused`, `MigrationNote`, + `Draft.origin`, `Draft.legacy_missing`, `Draft.notes`, `Draft.unresolved_controls` and the + open questions `legacy_folder_missing` and `control_unresolved` (nothing else produced them), + with their texts in `describe` and the notebook. `describe` no longer says "cannot be drafted + from experiment_design.csv". +- Kept: `design.parse_design_bytes` and `design_issues` (the run's reconcile check and the tests + of the written file still use them). +- Tests: the Spec scenario P30 follows the amended Spec (`v13` derived twice with a frozen clock + moved by a day; both files equal the `v13` golden; the clock is never read). The migration + tests are deleted; `tests/test_orphan_design_csv.py` and `test_cli_describe` check that an + orphan `experiment_design.csv` in the output folder is ignored by draft creation, `describe` + and the notebook (open and reopen). +- The valid Spec fixtures give byte-identical `plane_histograms.csv`, `experiment_design.csv` + and manifest (apart from times) before and after; the refusal codes of the 135 invalid + fixtures are unchanged. + ### Normalization pinned at v0.4.0 (#262) - **`morphobricks-normalization` moves from v0.3.4 to v0.4.0** (tag `c952263`). It is diff --git a/docs/conformance.md b/docs/conformance.md index b4cfccec..0efeef76 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -128,7 +128,7 @@ GS1: a scenario that names fixtures is shown on those fixtures; further fixtures | P27 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is a brick's (per-request `binary_mask_policy`). Preprocessing's own parts passed: `A_MASK_NOT_INSTANCES` (`test_a_error_classes`) and the section 2 "Mask fields" row (`i111`, `i112`, below) | | P28 | Passed | `test_spec_run_description`: `test_p28_each_new_batch_is_not_yet_checked_until_the_mask_step_has_read_it` (`v08`, with the 0-5 binary batch); `test_spec_run_inputs_semantic`: `test_p28_a_mask_file_not_yet_checked_is_refused_when_read_until_the_step_checked_it` (`v08`), `test_p28_on_v09_each_batch_of_masks_is_unchecked_until_the_mask_step_reads_it` (`v09`: three batches, each `not yet checked` and `A_MASK_UNCHECKED` until the mask step reads it, earlier batches stay `checked`) | | P29 | Passed | `test_spec_discovery` on `v07` published with mask folders `NA`, ` cells` and `Cafe\u0301` beside `nuclei` and `cells`: `test_p29_a_run_with_published_v07_refuses_the_mask_type_names` (`run_experiment` refuses with `R_MASK_TYPE_NAME_INVALID` naming ` cells`, `NA` and `R_FOLDER_NOT_NFC` naming `Cafe\u0301`, beside `R_MASK_TYPE_NOT_DECLARED`; nothing renamed), `test_p29_the_mask_step_refuses_the_names_with_a_published_description` (the step is not ready), `test_p29_the_command_line_check_and_masks_refuse_the_mask_type_names` (no `mask_relationships.json`); `test_notebook_name_refusals`: `test_p29_the_described_folder_check_refuses_the_mask_type_names`, `test_p29_pairing_the_masks_again_keeps_the_refusal`, `test_p21_p29_a_refused_name_blocks_check_and_save` (no `morphobricks.toml` is published, so the names never reach `E_ID_TEXT`). Without a description: the four `test_p21_p29_*_as_in_v0_4` above. Also the 5 synthetic-tree `test_p29_*`. Fixed in `0a2e25b` (on `a6a0343` the run gave only `R_MASK_TYPE_NOT_DECLARED`); the new tests fail on `c675f13` except those without a description | -| P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date`. The answers state each unit's image type ("channels"): since audit batch 33 an unanswered image type is an open question (`unit_image_type_unanswered`), so the test answers it, as the merge `0ea3254` records | +| P30 | Passed | `test_spec_derived_design`: `test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date` (`v13` tree, 2 cases): the `v13` legacy file migrated, answered with the proposed IDs and derived again is the legacy file byte for byte (and the golden); with `rep1` rows dated 2026-10-01 and 2026-10-15, the repeat takes 2026-10-15, the note lists both dates, and the derived file differs from the legacy file in one line only, the CTRL/rep1 date (the one-date groups `rep2`, `rep3` are byte-identical); under a frozen clock no clock is read. Also `test_draft`: `test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both`, `test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date`. The answers state each unit's image type ("channels"): since audit batch 33 an unanswered image type is an open question (`unit_image_type_unanswered`), so the test answers it, as the merge `0ea3254` records **Amended after this run** (10 November 2026, re-pin to Spec `cc9820f`, ADR 0002 AR10 (9)): the legacy-CSV case was removed from P30; the test is now `test_p30_v13_derived_twice_with_a_clock_moved_by_a_day_equals_the_golden` and the migration tests named here are deleted. This row records the run on `1bed024`. | | P31 | Deferred to the Stage 2 gate per ADR 0001 GS1 (MorphoBricks-Spec `cdc1841`) | The outcome is MorphoCoverage's (prefill, confirmation). Preprocessing's own parts passed: `v27` valid with no warning, and editing `target` or `treatment_key` of `v27` changes no digest and no derived row (section 2 rows below) | | P32 | Passed | `test_draft`: `test_p32_forbidden_folders_get_no_proposal_and_nothing_is_adopted`, `test_p32_ordinary_folders_are_proposed_as_their_labels`; `test_spec_derived_design`: golden `v33`, `v34` and `test_the_file_names_folders_never_ids` | | P33 | Passed | `test_spec_reconcile`: `test_p33_a_a_renamed_condition_folder_is_not_attached_to_the_old_id`, `test_p33_b_the_statement_keeps_the_id_moves_the_folder_and_reopens_the_repeats`, `test_p33_c_the_old_folder_removal_needs_its_confirmation_and_the_id_stays_used` (`v33`; (a) asserts that the hand-made attachment is `t12` and gives exactly `R_CONFIRMED_ID_CHANGED`); `t10`, `t11`, `t12` against their predecessors in `test_spec_transitions`; `test_condition_folder_reopens` (2) | diff --git a/docs/contract.md b/docs/contract.md index 86a75889..8f421343 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -525,8 +525,8 @@ by preprocessing or any other brick (owner answer 3, Spec ADR 0002 AR11 (3)). Th `default_design`, `apply_design_edit`/`DesignEdit`, owner answer 4). What stays: `write_experiment_design`, `DESIGN_COLUMNS`, `design_derive`, the manifest's `experiment_design_sha256`, the rows as `DesignTable` (read-only) with the run's check -`reconcile_design`/`design_issues`, and the strict `parse_design_bytes` of the draft's -experiment-v1 §7 migration (`draft.migrate_legacy_design`). +`reconcile_design`/`design_issues`, and the strict `parse_design_bytes` (for tests and for checking a written file; the draft's +legacy migration, `draft.migrate_legacy_design`, was removed under Spec ADR 0002 AR10 (9)). | Column | Meaning | |---|---| @@ -545,12 +545,12 @@ experiment-v1 §7 migration (`draft.migrate_legacy_design`). | Confirmation | Per repeat group: "Were these images prepared and acquired under the same circumstances, so they are comparable with each other?". A group is confirmed when all its rows are. Without biological-repeat folders (every condition `single`), the whole experiment is one group and the question is asked once (N31). Missing technical-repeat folders need no question. | | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | | Run's check | The run brings the derived rows up to date with the folders found (`reconcile_design`): folders in them keep their group, control mark and confirmation (added or removed images within a folder keep it too); a folder found but not described joins the group named after it, inherits its condition's control mark, and that group is refused as not confirmed (`design_not_confirmed`); folders no longer found are dropped. Nothing is read from the output folder. | -| Reading | *(removed in v0.5.1, SWEEP-PP)* No reader of the file for its answers remains: the lenient `read_experiment_design`, `load_design` and the strict `read_confirmed_design` are gone. `parse_design_bytes` parses the file exactly as `write_experiment_design` writes it (UTF-8 without a byte-order mark, the columns in order, `true`/`false`, whole counts, `YYYY-MM-DD`, identities as §16, a header naming a column twice or not at all refused naming it, issue #83), only for the draft's experiment-v1 §7 migration of a legacy file; it raises `DesignFormatError` naming the line and column. `design_issues` gives `invalid_experiment_design` for an unusable repeat-group name or date in rows built in code, and `write_experiment_design` refuses to write them. `DesignTable.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping. | +| Reading | *(removed in v0.5.1, SWEEP-PP)* No reader of the file for its answers remains: the lenient `read_experiment_design`, `load_design` and the strict `read_confirmed_design` are gone. `parse_design_bytes` parses the file exactly as `write_experiment_design` writes it (UTF-8 without a byte-order mark, the columns in order, `true`/`false`, whole counts, `YYYY-MM-DD`, identities as §16, a header naming a column twice or not at all refused naming it, issue #83), only to check a file written by this brick (no code path reads an `experiment_design.csv` to propose or prefill a draft: experiment-v1 §7, Spec ADR 0002 AR10 (9)); it raises `DesignFormatError` naming the line and column. `design_issues` gives `invalid_experiment_design` for an unusable repeat-group name or date in rows built in code, and `write_experiment_design` refuses to write them. `DesignTable.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping. | | Notebook | The pairing is answered on the design card and kept in the draft (`morphobricks.draft.json`), confirmed with the rest on *Check and save*; the notebook never writes or reads `experiment_design.csv` (v0.5.1). The card is built on the draft (SWEEP-PP, owner answer 4): one row per folder of the draft's base with the repeat it is paired with (`PreprocessingNotebook.pairing`); a change is a draft answer at once, and a replicate whose folders it changes is asked "same way?" again. There are no comparability questions or control boxes on the card (they are on *Check and save* and the references). | | Command line | *(removed in v0.5.1)* `run --design FILE` and the template `run` wrote: `run` refuses an output folder without a published description (§0). | | Template from `check` | *(removed in v0.5.1)* `check` writes only its report. | | `design` command | *(removed in v0.5.1)*: the pairing, confirmations and references are answered in the notebook's describe steps. `apply_design_edit` and `DesignEdit` were removed in SWEEP-PP. | -| `describe` command (Stage 1, PR 17) | `describe EXPERIMENT [--output FOLDER] [--exclude PATH] [--publish]` prints the draft experiment description (experiment-v1 §12) in plain language: the proposed ID of each condition, technical-repeat folder and biological repeat (no proposal, with the rule's code, for a folder that cannot be an ID: never repaired), each repeat with its folders and image counts and whether it is confirmed (and the date), the references, the notes of a migration with several dates, and every open question with what the user has to do. The draft is the legacy `experiment_design.csv` of the output folder migrated (`draft.migrate_legacy_design`; one that cannot be read or breaks a rule is exit status 1 and left as it is) when there is one and no published `morphobricks.toml`, else the draft of the folders (`draft.draft_from_discovery`); when `morphobricks.toml` is published (`run.description_run.load_description`), the folders are compared with it (`description_reconcile.reconcile`: unchanged, new, gone, image counts, repeats to confirm again), reading the folder structure only, so the channels are not compared. It reads no image and writes nothing: no output folder, draft, description, design or records file. **It confirms nothing**: an ID, the comparability of a repeat (N35), a reference, a removal and every date are the user's answers (notebook), never given by the command or read from the clock. `--publish` publishes a draft only when no question is open and every decision is confirmed; otherwise it prints `Not published. What is missing:` with every item and exits with status 1. This version holds no answers outside the notebook (the draft file is not in the package), so `--publish` refuses every draft built from folders and publishes nothing; for a published description that matches the folders it says `Nothing to publish` (exit 0); a new, gone or changed folder is listed and the file is left as it is. An error (experiment folder not found, discovery errors, unreadable description or records) is exit status 1; the command takes no lock. | +| `describe` command (Stage 1, PR 17) | `describe EXPERIMENT [--output FOLDER] [--exclude PATH] [--publish]` prints the draft experiment description (experiment-v1 §12) in plain language: the proposed ID of each condition, technical-repeat folder and biological repeat (no proposal, with the rule's code, for a folder that cannot be an ID: never repaired), each repeat with its folders and image counts and whether it is confirmed (and the date), the references, and every open question with what the user has to do. The draft is always the draft of the folders (`draft.draft_from_discovery`); an `experiment_design.csv` in the output folder is never read to propose or prefill it (experiment-v1 §7; the legacy migration was removed under Spec ADR 0002 AR10 (9)); when `morphobricks.toml` is published (`run.description_run.load_description`), the folders are compared with it (`description_reconcile.reconcile`: unchanged, new, gone, image counts, repeats to confirm again), reading the folder structure only, so the channels are not compared. It reads no image and writes nothing: no output folder, draft, description, design or records file. **It confirms nothing**: an ID, the comparability of a repeat (N35), a reference, a removal and every date are the user's answers (notebook), never given by the command or read from the clock. `--publish` publishes a draft only when no question is open and every decision is confirmed; otherwise it prints `Not published. What is missing:` with every item and exits with status 1. This version holds no answers outside the notebook (the draft file is not in the package), so `--publish` refuses every draft built from folders and publishes nothing; for a published description that matches the folders it says `Nothing to publish` (exit 0); a new, gone or changed folder is listed and the file is left as it is. An error (experiment folder not found, discovery errors, unreadable description or records) is exit status 1; the command takes no lock. | | Run | The run derives the design from the published description and writes it with the table, published before the manifest, which records its SHA-256 (§9). `run_experiment` takes no design. `plane_histograms.csv` does not depend on the design. | ## 13. Image type diff --git a/docs/user_guide.md b/docs/user_guide.md index 8af9768f..712f34d3 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -293,10 +293,8 @@ results or comparing another grouping. - Click **Check folder**. Only folder names and image headers are read. Like the command line's `check`, it writes its report, **discovery_report.txt** and **discovery_report.json** (§6), to the output folder; **Show the report** under the - status box shows it. When the output folder holds no **experiment_design.csv**, it also - writes one prefilled from the folders, with nothing confirmed and no control, for the - **Experiment design** questions below. (When another brick embeds this step without - the output folder, the check writes nothing.) + status box shows it. (When another brick embeds this step without the output folder, + the check writes nothing.) - If another run holds the output folder's lock, **The output folder is locked** names it (computer, process ID, start time). Wait for it to finish. If it was stopped (on Colab, the runtime disconnected) and no other session uses the folder, press **Remove the lock @@ -339,11 +337,9 @@ results or comparing another grouping. is confirmed for you. Next time, if the folders are the same, they are loaded and the questions are folded away; **Change the experiment design** shows them again. A new biological-repeat folder asks again only for the group it joins. - - If the saved **experiment_design.csv** cannot be read (for example a spreadsheet - program rewrote a date such as 2026-10-26 as 26/10/2026), a red box shows why, - naming the line and column, and the file is left as it is. Correct the file and - undo and confirm the labels again to read it, or answer the questions again: the - first answer replaces the file. + - An **experiment_design.csv** that an earlier version left in the output folder is not + read and does not prefill these questions: describe the experiment here, in the + notebook. The file is only an output, derived from the published description. - The design belongs to its output folder. If you change the output folder, the design is read from the new folder (its own answers and controls, a blue box says so), or, when it holds none, the questions are asked again: answers given for the previous @@ -769,11 +765,9 @@ what each of two controls is compared against. no pixel is read) and **writes nothing**: no output folder, no draft, no description. `--output FOLDER` names the output folder to read (default: next to the experiment), `--exclude PATH` leaves an image out, as in `check`. -- If the output folder holds an `experiment_design.csv` from an earlier version and no - published `morphobricks.toml`, the draft is that design migrated: its confirmed repeats - and their dates are shown as you gave them, and the IDs are still only proposals. A design - that cannot be read, or breaks the rules of the description, is refused with the reason and - left as it is. +- The draft is always built from the folders. An `experiment_design.csv` in the output folder + (from an earlier version, say) is not read and proposes or prefills nothing: describe the + experiment in the notebook. - If a `morphobricks.toml` is published, `describe` compares it with the folders instead: how many acquisitions are as published, which are new or gone, which image counts changed and which units' image headers no longer fit their published channels and which repeats must diff --git a/src/morphobricks_preprocessing/cli_describe.py b/src/morphobricks_preprocessing/cli_describe.py index 1e36e52e..b2ba14ac 100644 --- a/src/morphobricks_preprocessing/cli_describe.py +++ b/src/morphobricks_preprocessing/cli_describe.py @@ -8,13 +8,14 @@ label that is not Unicode NFC is refused (``R_FOLDER_NOT_NFC``, experiment-v1 section 3.3): nothing is drafted from it. -The draft comes from the existing functions only: the legacy ``experiment_design.csv`` of the -output folder, when there is one and no published description, is migrated -(:func:`.draft.migrate_legacy_design`); otherwise the draft is built from the folders -(:func:`.draft.draft_from_discovery`); when ``morphobricks.toml`` is published, the folders and -the headers of their images are compared with it (:func:`.reconcile_observe.observe_folders` -with the headers, then :func:`.description_reconcile.reconcile`), exactly as the notebook -compares them on reopening, so both say the same. This module adds no rule. +The draft comes from the existing functions only: it is built from the folders +(:func:`.draft.draft_from_discovery`); no file of the output folder, ``experiment_design.csv`` +included, is read to propose or prefill it (experiment-v1 section 7); when ``morphobricks.toml`` +is published, the folders and the headers of their images are compared with it +(:func:`.reconcile_observe.observe_folders` with the headers, then +:func:`.description_reconcile.reconcile`), +exactly as the notebook compares them on reopening, so both say the same. This module +adds no rule. **Nothing is confirmed here.** An ID, a repeat's comparability (N35), a reference and a removal are the user's answers, given in the notebook; the command never answers for the user and never @@ -26,7 +27,7 @@ publish. Exit status: 1 for an error (a folder that cannot be described, a description that cannot be -read, a legacy design that breaks the rules, a publication refused), else 0, as every command. +read, a publication refused), else 0, as every command. """ from __future__ import annotations @@ -45,9 +46,8 @@ UnitFolders, reconcile, ) -from .design import DESIGN_FILENAME, DesignFormatError from .discovery import DiscoveryResult, default_output_folder, discover_experiment -from .draft import Draft, DraftError, draft_from_discovery, migrate_legacy_design +from .draft import Draft, draft_from_discovery from .draft_ids import IdKind, id_problems from .metadata import inspect_images from .name_issues import non_nfc_folders_issue @@ -64,14 +64,6 @@ "circumstances, so they are comparable" ), "confirmed_on_missing": "{subject} is confirmed but has no date: confirm it again", - "control_unresolved": ( - "condition {subject} is marked as a control, as is another one: say what each " - "condition is compared against" - ), - "legacy_folder_missing": ( - "the legacy design lists {subject}, which is no longer in the folder: restore it or " - "confirm that it is removed" - ), } """What the user must do for each open question of a draft, in plain words.""" @@ -99,7 +91,7 @@ def add_describe_parser(commands: argparse._SubParsersAction) -> argparse.Argume describe.add_argument( "--output", metavar="FOLDER", - help="output folder holding experiment_design.csv or morphobricks.toml " + help="output folder holding morphobricks.toml " "(default: next to the experiment, _MorphoBricks_Preprocessing)", ) describe.add_argument( @@ -140,12 +132,7 @@ def describe_command(args: argparse.Namespace) -> int: if found is not None: lines, blockers = _published(found.description, discovery, excluded, output) else: - try: - draft = _draft(discovery, excluded, output) - except (DesignFormatError, DraftError) as exc: - print(f"The experiment description cannot be drafted from {output / DESIGN_FILENAME}") - print(f" {exc}") - return 1 + draft = draft_from_discovery(discovery, excluded) lines, blockers = draft_lines(draft, root), draft_blockers(draft) print("\n".join(lines)) if not args.publish: @@ -159,14 +146,6 @@ def describe_command(args: argparse.Namespace) -> int: return 1 -def _draft(discovery: DiscoveryResult, excluded: list[str], output: Path) -> Draft: - """Build the draft: the legacy design migrated when the output folder holds one.""" - legacy = output / DESIGN_FILENAME - if legacy.is_file(): - return migrate_legacy_design(legacy.read_bytes(), discovery, excluded, name=legacy.name) - return draft_from_discovery(discovery, excluded) - - def _proposal(kind: IdKind, folder: str, slot_proposed: str | None) -> str: """Say what is proposed for a folder, or why nothing is.""" if slot_proposed is not None: @@ -177,14 +156,9 @@ def _proposal(kind: IdKind, folder: str, slot_proposed: str | None) -> str: def draft_lines(draft: Draft, root: Path) -> list[str]: """Return the plain-language report of a draft.""" - source = ( - "the legacy experiment_design.csv of the output folder" - if draft.origin == "legacy" - else "the folders alone" - ) out = [ f"Draft experiment description of {root}", - f"Drawn from {source}. Nothing here is confirmed: IDs below are proposals.", + "Drawn from the folders alone. Nothing here is confirmed: IDs below are proposals.", "", "Conditions (folder: proposal)", ] @@ -207,10 +181,6 @@ def draft_lines(draft: Draft, root: Path) -> list[str]: if draft.references: out += ["", "References (condition is compared against)"] out += [f" {c} -> {r}" for c, r in sorted(draft.references.items())] - for note in draft.notes: - listed = "; ".join(f"{c}/{r}: {d or 'no date'}" for c, r, d in note.dates) - out += ["", f"Several dates in the legacy design for {note.repeat_group} ({listed})."] - out.append(f" The draft keeps the latest, {note.kept or 'none'}; you confirm it again.") out += _question_lines(draft) return out diff --git a/src/morphobricks_preprocessing/design.py b/src/morphobricks_preprocessing/design.py index dc50bed4..45e5674d 100644 --- a/src/morphobricks_preprocessing/design.py +++ b/src/morphobricks_preprocessing/design.py @@ -10,8 +10,9 @@ holds rows and answers questions about them; it has no editing methods. :func:`reconcile_design` and :func:`design_issues` are the run's check of the derived rows -against the folders found; :func:`parse_design_bytes` is the strict parser the draft's -experiment-v1 section 7 migration of a legacy file uses (:func:`.draft.migrate_legacy_design`). +against the folders found; :func:`parse_design_bytes` is the strict parser of a file +:func:`write_experiment_design` wrote. No brick reads the file to propose or prefill a draft +(experiment-v1 section 7); the parser is for tests and for checking a written file. The lenient reader, the default pairing, the editor model and the strict reader for other bricks (``read_confirmed_design``) were removed in v0.5.1 (SWEEP-PP). Nothing here reads images. """ @@ -337,7 +338,8 @@ def parse_design_bytes(data: bytes, name: str = DESIGN_FILENAME) -> DesignTable: Strict (experiment-v1 section 7): UTF-8 without a byte-order mark, exactly :data:`DESIGN_COLUMNS` in that order, one value per column, no outer whitespace, ``true`` or ``false``, whole counts, ISO dates, identities as :func:`.discovery.check_identity_text` - takes them and no case collision. Only the draft's legacy migration uses it. + takes them and no case collision. + Nothing in this package reads an ``experiment_design.csv`` back as an input. Raises ------ diff --git a/src/morphobricks_preprocessing/design_derive.py b/src/morphobricks_preprocessing/design_derive.py index 280502b3..2bdb03ff 100644 --- a/src/morphobricks_preprocessing/design_derive.py +++ b/src/morphobricks_preprocessing/design_derive.py @@ -33,7 +33,7 @@ def derive_design(description: Description) -> DesignTable: ------ DesignFormatError From :func:`.design.write_experiment_design` when written, if a repeat ID is not a name - the legacy readers accept (it is never repaired). + the strict parser accepts (it is never repaired). """ referenced = {c.reference for c in description.conditions.values() if c.reference is not None} rows = [] diff --git a/src/morphobricks_preprocessing/draft.py b/src/morphobricks_preprocessing/draft.py index 1d6989d0..8b267cc1 100644 --- a/src/morphobricks_preprocessing/draft.py +++ b/src/morphobricks_preprocessing/draft.py @@ -1,9 +1,9 @@ -"""The draft of an experiment description, and the migration of a legacy ``experiment_design.csv``. +"""The draft of an experiment description. A draft is what discovery can say before the user has answered (experiment-v1 section 12): the folders found, the IDs proposed for them, the pairing of biological repeats, and what is still open. It is **not** a description and is never read as one: an ID is *proposed* until the user -confirms it, ``confirmed_on`` is a date the user (or a legacy file) gave, never a clock read, +confirms it, ``confirmed_on`` is a date the user gave, never a clock read, and a draft with an open question cannot be published. Nothing here writes a file, reads an image or reads the clock. @@ -22,16 +22,9 @@ from morphobricks_normalization.records import SINGLE from .description_identity import nfc -from .description_read import PHASES -from .description_report import DescriptionIssue -from .description_rules import SCHEMA_ID -from .design import DesignTable, design_issues, parse_design_bytes from .discovery import DiscoveryResult from .draft_ids import IdKind, id_problems, propose_id -PLACEHOLDER_DATE = "2000-01-01" -"""Stands in for a missing date in the rule-check document only; it never leaves that function.""" - class DraftError(ValueError): """Raised when a draft step is refused; ``codes`` are the reasons (Spec codes if any).""" @@ -41,21 +34,6 @@ def __init__(self, message: str, codes: tuple[str, ...] = ()) -> None: super().__init__(message) -class LegacyMigrationRefused(DraftError): - """Raised when a legacy design violates experiment-v1: refused, never repaired (section 7). - - ``issues`` are the reader's issues (code, table path, value) of the first failing phase; - the paths use stand-in condition IDs where the folder is not an allowed ID. - """ - - def __init__(self, message: str, issues: tuple[DescriptionIssue, ...] = ()) -> None: - self.issues = issues - codes = tuple(dict.fromkeys(issue.code.value for issue in issues)) or ( - "repeat_group_conflict", - ) - super().__init__(message, codes) - - @dataclass(frozen=True) class IdSlot: """An ID that may be proposed and is adopted only when the user confirms one. @@ -109,17 +87,6 @@ class DraftRepeat: confirmed_on: str = "" -@dataclass(frozen=True) -class MigrationNote: - """The original per-folder dates of a legacy group that had several (AQ12).""" - - repeat_group: str - dates: tuple[tuple[str, str, str], ...] - """``(condition folder, source repeat, date)``, ``""`` for a folder with no date.""" - kept: str - """The date the draft carries: the latest of them, never the clock.""" - - @dataclass(frozen=True) class OpenQuestion: """Something the user must answer before the draft can be published.""" @@ -151,23 +118,18 @@ class Draft: """A draft description: IDs beside the source folders, and the open questions. ``conditions`` and ``technical_repeats`` are keyed by source folder (one folder, one ID); - ``repeats`` by the pairing label (the folder name for the default pairing, the legacy - ``repeat_group`` for a migration). ``references`` maps a condition folder to the folder of - the condition it is compared against; ``unresolved_controls`` lists the folders a legacy - design marked as controls when there were two or more (no name-stem guessing), and - ``legacy_missing`` the legacy folders no longer found. ``origin`` is ``"discovery"`` or - ``"legacy"``. + ``repeats`` by the pairing label (the folder name for the default pairing). + ``references`` maps a condition folder to the folder of the condition it is compared + against. The draft is always derived from the folders; no file of the output folder + (``experiment_design.csv`` included) is read to propose or prefill it (experiment-v1 + section 7). """ conditions: Mapping[str, IdSlot] technical_repeats: Mapping[str, IdSlot] repeats: Mapping[str, DraftRepeat] references: Mapping[str, str] = field(default_factory=dict) - unresolved_controls: tuple[str, ...] = () - legacy_missing: tuple[tuple[str, str], ...] = () - notes: tuple[MigrationNote, ...] = () used_ids: Mapping[str, tuple[str, ...]] = field(default_factory=dict) - origin: str = "discovery" def open_questions(self) -> tuple[OpenQuestion, ...]: """Return every question still open; a draft with any cannot be published.""" @@ -188,8 +150,6 @@ def open_questions(self) -> tuple[OpenQuestion, ...]: out.append(OpenQuestion("repeat_not_confirmed", f"repeat {group}")) elif not repeat.confirmed_on: out.append(OpenQuestion("confirmed_on_missing", f"repeat {group}")) - out += [OpenQuestion("control_unresolved", folder) for folder in self.unresolved_controls] - out += [OpenQuestion("legacy_folder_missing", f"{c} / {r}") for c, r in self.legacy_missing] return tuple(out) @property @@ -380,169 +340,6 @@ def draft_from_discovery( return _draft(_members(discovery, set(excluded)), {}, used) -def _legacy_groups(legacy: DesignTable) -> dict[tuple[str, str], str]: - issues = [issue for issue in design_issues(legacy) if issue.code == "repeat_group_conflict"] - if issues: - raise LegacyMigrationRefused(issues[0].message) - return legacy.repeat_groups() - - -def _probe(draft: Draft) -> dict[str, object]: - """Return the draft as a document for the reader's rules only; stand-ins fill missing IDs. - - Condition, technical-repeat and repeat folders that are not allowed IDs get a stand-in ID - (their question stays open, section 4.6a), and one channel stands in for the layouts the - draft does not know. Nothing of it is stored. - """ - stand_in = {} - repeat_slots = {group: repeat.id for group, repeat in draft.repeats.items()} - for kind, slots in ( - ("conditions", draft.conditions), - ("technical", draft.technical_repeats), - ("repeats", repeat_slots), - ): - for folder, slot in slots.items(): - name = folder if folder == SINGLE and kind != "repeats" else slot.proposed - number = len(stand_in) + 1 - while name is None or (kind, name) in stand_in.values(): - name = f"{kind} {number}" - number += 1 - stand_in[(kind, folder)] = (kind, name) - cid = {folder: name for (kind, folder), (_, name) in stand_in.items() if kind == "conditions"} - tid = {folder: name for (kind, folder), (_, name) in stand_in.items() if kind == "technical"} - rid = {folder: name for (kind, folder), (_, name) in stand_in.items() if kind == "repeats"} - return { - "schema": SCHEMA_ID, - "schema_version": 1, - "conditions": {cid[f]: {"source_folder": f} for f in draft.conditions}, - "channels": {"Probe": {}}, - "repeats": { - rid[group]: { - "confirmed": True, - "confirmed_on": repeat.confirmed_on or PLACEHOLDER_DATE, - "members": { - cid[m.condition]: { - "source_repeat": m.source_repeat, - "image_count": m.image_count, - "technical_repeats": { - tid[u.source_folder]: { - "source_folder": u.source_folder, - "channels": ["Probe"], - } - for u in m.units - }, - } - for m in repeat.members - }, - } - for group, repeat in draft.repeats.items() - }, - "used_ids": { - "conditions": sorted(cid.values()), - "channels": ["Probe"], - "repeats": sorted(rid.values()), - "technical_repeats": sorted(tid.values()), - }, - } - - -def _check_legacy_rules(draft: Draft) -> None: - """Refuse a migrated draft that experiment-v1 would refuse, with the reader's codes.""" - doc = _probe(draft) - for _, check in PHASES: - issues = check(doc) - if issues: - codes = ", ".join(sorted({i.code.value for i in issues.items})) - raise LegacyMigrationRefused( - f"the legacy design breaks the rules of the experiment description ({codes}); " - "it is not repaired: change the repeat groups, or start a new design", - tuple(issues.items), - ) - - -def _migrated_repeat( - group: str, repeat: DraftRepeat, legacy: DesignTable -) -> tuple[DraftRepeat, MigrationNote | None]: - """Carry a legacy group's confirmation: only if every folder is confirmed, latest date.""" - rows = [legacy.row(member.key) for member in repeat.members] - known = [row for row in rows if row is not None] - dates = sorted({row.confirmed_on for row in known if row.confirmed_on}) - confirmed = len(known) == len(rows) and all(row.confirmed for row in known) - kept = dates[-1] if confirmed and dates else "" - note = None - if len(dates) > 1: - listed = tuple( - (m.condition, m.source_repeat, row.confirmed_on if row else "") - for m, row in zip(repeat.members, rows, strict=True) - ) - note = MigrationNote(group, listed, kept) - return replace(repeat, confirmed=confirmed, confirmed_on=kept), note - - -def migrate_legacy_design( - data: bytes, - discovery: DiscoveryResult, - excluded: Iterable[str] = (), - *, - name: str = "experiment_design.csv", - used_ids: Mapping[str, Iterable[str]] | None = None, -) -> Draft: - """Migrate the bytes of a legacy ``experiment_design.csv`` to a draft (experiment-v1 section 7). - - The file is read strictly (:func:`.design.parse_design_bytes`: ``yes``, ``no`` and empty - booleans are refused, not coerced). Its ``repeat_group`` becomes the proposed repeat ID, - ``biological_repeat`` the source repeat, and the ``condition`` label the condition's source - folder, whose ID is proposed as that label when allowed. Technical repeats and counts come - from ``discovery``; channels are not part of a legacy file. A group is confirmed only if - every one of its folders is; with several dates it takes the latest and a - :class:`MigrationNote` lists each folder's date; the clock is never read, and a confirmed - group without any date stays open (``confirmed_on_missing``). A ``repeat_group`` that is not - an allowed repeat ID (for example ``z``) is not refused: it migrates with no proposed ID and - an open question, so the user names the repeat (section 4.6a). Controls: none marked, no - reference; one marked, every other condition references it; two or more, an unresolved - question and no reference. A folder found now but not in the file is a new pending member - of the group named after it, which therefore is not confirmed. A legacy folder no longer - found is listed in ``legacy_missing`` and blocks publication. IDs stay proposals: the user - confirms them. - - Raises - ------ - DesignFormatError - If the bytes are not a design this brick writes (:func:`.design.parse_design_bytes`). - LegacyMigrationRefused - If a repeat group holds two folders of one condition, or the migrated design breaks a - rule of the experiment description (for example ``E_REPEAT_ID_SHADOWS_SOURCE``): it is - refused with the code, never repaired. - """ - legacy = parse_design_bytes(data, name) - used = {kind: tuple(values) for kind, values in (used_ids or {}).items()} - members = _members(discovery, set(excluded)) - draft = _draft(members, _legacy_groups(legacy), used) - notes, repeats = [], {} - for group, repeat in draft.repeats.items(): - migrated, note = _migrated_repeat(group, repeat, legacy) - proposal = group if not id_problems("repeats", group, used.get("repeats", ())) else None - repeats[group] = replace(migrated, id=IdSlot(proposal)) - notes += [note] if note else [] - present = {m.condition for m in members} - marked = sorted(c for c in legacy.controls if c in present) - references = ( - {c: marked[0] for c in sorted(present) if c != marked[0]} if len(marked) == 1 else {} - ) - found = {m.key for m in members} - draft = replace( - draft, - repeats=MappingProxyType(repeats), - references=MappingProxyType(references), - unresolved_controls=tuple(marked) if len(marked) > 1 else (), - legacy_missing=tuple(sorted(row.key for row in legacy.rows if row.key not in found)), - notes=tuple(notes), - origin="legacy", - ) - _check_legacy_rules(draft) - return draft - - __all__ = [ "Draft", "DraftError", @@ -550,10 +347,7 @@ def migrate_legacy_design( "DraftRepeat", "DraftUnit", "IdSlot", - "LegacyMigrationRefused", - "MigrationNote", "OpenQuestion", "UnitRow", "draft_from_discovery", - "migrate_legacy_design", ] diff --git a/src/morphobricks_preprocessing/draft_answers.py b/src/morphobricks_preprocessing/draft_answers.py index 5fb6009d..a2987746 100644 --- a/src/morphobricks_preprocessing/draft_answers.py +++ b/src/morphobricks_preprocessing/draft_answers.py @@ -3,7 +3,7 @@ A :class:`.draft.Draft` is what discovery can say; the answers are what the user said: the IDs they confirmed, the date of each repeat's confirmation, the channel of every C index of every imaging unit, the declared masks, optional references and display names. They are kept apart -from the draft because the draft is derived again from the folder (or from a legacy design) on +from the draft because the draft is derived again from the folder on every resume, while the answers are the only thing the user typed or clicked. The answers are keyed by the draft's own stable keys (source folder labels, the pairing label of a repeat), so an answer for a folder the draft no longer has is refused (``unknown_key``), never carried over @@ -352,7 +352,7 @@ def missing_confirmations( ) -> tuple[OpenQuestion, ...]: """Return every confirmation still missing for ``draft`` with ``answers`` applied. - The draft's own open questions (IDs, repeats, controls, legacy folders), plus the channels + The draft's own open questions (IDs, repeats), plus the channels of every imaging unit that has none (``unit_channels_missing``), every unit whose image type is not answered ``"channels"`` or ``"rgb"`` (``unit_image_type_unanswered``: an unanswered type is never published as an ordinary channel image), every position not @@ -426,7 +426,7 @@ def description_from_draft( """Build the description of a draft whose every science decision the user confirmed. Pure: no file is read or written and no clock is read. Each ID and ``confirmed_on`` is the - user's, from ``answers`` (or carried by a migrated ``draft``); ``used_ids`` is the draft's + user's, from ``answers``; ``used_ids`` is the draft's retired IDs plus every ID confirmed now. ``channel_counts`` are as :func:`missing_confirmations` takes them. diff --git a/src/morphobricks_preprocessing/draft_format.py b/src/morphobricks_preprocessing/draft_format.py index ee11dc5e..b32934e7 100644 --- a/src/morphobricks_preprocessing/draft_format.py +++ b/src/morphobricks_preprocessing/draft_format.py @@ -35,7 +35,7 @@ DRAFT_VERSION = 3 IMAGE_TYPES = ("rgb", "channels") """The image types a draft may hold: ``channels`` is the answer "ordinary channel image".""" -SPEC_COMMIT = "1bed024e278d104ba4b8f63ae76c020fb2e3470d" +SPEC_COMMIT = "cc9820f5b84f01bbbb5c8e3f54e70ee13cb5e9eb" """The MorphoBricks-Spec commit the rules of this brick are pinned to (``tests/spec_fixtures``).""" BASE_MISSING = "R_DRAFT_BASE_MISSING" SHA256 = re.compile(r"[0-9a-f]{64}") diff --git a/src/morphobricks_preprocessing/notebook_check_view.py b/src/morphobricks_preprocessing/notebook_check_view.py index b397e5b1..1f70b665 100644 --- a/src/morphobricks_preprocessing/notebook_check_view.py +++ b/src/morphobricks_preprocessing/notebook_check_view.py @@ -41,8 +41,6 @@ "(Axes) and check that every image of it can be read and has the same number of " "channels" ), - "control_unresolved": "say what {subject} is compared against", - "legacy_folder_missing": "{subject} is gone: confirm the removal", } diff --git a/src/morphobricks_preprocessing/run/draft_file.py b/src/morphobricks_preprocessing/run/draft_file.py index 0a61fb89..083f3335 100644 --- a/src/morphobricks_preprocessing/run/draft_file.py +++ b/src/morphobricks_preprocessing/run/draft_file.py @@ -110,7 +110,7 @@ def save_draft(output_folder: str | os.PathLike, draft: Draft, answers: DraftAns output_folder The output folder; created if needed. draft - The draft derived from the folder (discovery or a legacy design), without the answers. + The draft derived from the folder, without the answers. answers All the user's answers so far (the file is replaced, not merged). @@ -181,7 +181,7 @@ def load_draft(output_folder: str | os.PathLike, draft: Draft) -> ResumedDraft | output_folder The output folder. draft - The draft derived again from the folder (or the legacy design) now. + The draft derived again from the folder now. Raises ------ diff --git a/tests/spec_fixtures/PIN.toml b/tests/spec_fixtures/PIN.toml index 4deebf9e..dcc4a915 100644 --- a/tests/spec_fixtures/PIN.toml +++ b/tests/spec_fixtures/PIN.toml @@ -1,14 +1,14 @@ # The MorphoBricks-Spec commit these files are copied from (conformance plan section 1). # Updating the pin is a reviewed change: see README.md in this folder. repository = "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/CellMigrationLab/MorphoBricks-Spec" -commit = "1bed024e278d104ba4b8f63ae76c020fb2e3470d" +commit = "cc9820f5b84f01bbbb5c8e3f54e70ee13cb5e9eb" # `git rev-parse :fixtures`, the git tree-object ID of the fixtures/ folder. The # vendored folder must hash to it, so one changed, added or missing byte fails a test. -fixtures_tree = "f61b1c2ba96a36eea9b2c101b13f6d7a3ff95b8b" +fixtures_tree = "40fe48c6d735266ca7608fab0c3a564affc68171" # `git rev-parse :`, the git blob ID of each other vendored file. [files] -"docs/experiment-v1.md" = "b12678cbac2c83e7a4d64cdc2883d4a8e327bb1f" +"docs/experiment-v1.md" = "d21388ad9e48608ce42adb0f7b0a4c600cb2c9a7" "tools/audit/refval.py" = "0ea83094cfa8c1bcb6ea596e17b3f0c948ad5dfb" "tools/audit/refrecords.py" = "ef06a1ea59a93c695dbba21d2ee792667ad8e8eb" diff --git a/tests/spec_fixtures/README.md b/tests/spec_fixtures/README.md index 51e348d1..37d1118a 100644 --- a/tests/spec_fixtures/README.md +++ b/tests/spec_fixtures/README.md @@ -1,37 +1,19 @@ # Vendored MorphoBricks-Spec files -A partial copy of [MorphoBricks-Spec](https://github.com/CellMigrationLab/MorphoBricks-Spec) -at the commit named in `PIN.toml`, laid out as in the Spec repository: +**Do not edit anything in this folder.** It is a byte-exact, pinned copy of +[MorphoBricks-Spec](https://github.com/CellMigrationLab/MorphoBricks-Spec) made by the Spec's shared sync tool. Regenerate it with +`vendor.py`, never by hand: -| Here | In the Spec | Used for | -|---|---|---| -| `fixtures/` | `fixtures/` | The conformance fixtures: every test of the experiment-description reader runs on them | -| `docs/experiment-v1.md` | `docs/experiment-v1.md` | The normative text; tests compare the reader's tables of codes, names and fields with it | -| `tools/audit/refval.py` | `tools/audit/refval.py` | The reference validator, used **only as a test oracle**; nothing in `src/` imports it | -| `tools/audit/refrecords.py` | `tools/audit/refrecords.py` | The reference validator of the records file (§4.6), imported by `refval.py`; also a test oracle only | + python /tools/vendor.py --into --commit -Rules (conformance plan, sections 1 and 2): +`PIN.toml` names the Spec commit and the git hash of each pinned path. The pin test hashes +this folder against it offline, and a changed, added or missing byte fails it. The copy is a +test fixture, not a runtime dependency. A fixture that a test fails is a finding for the Spec +repository, not something to change here. How to re-pin: `docs/vendoring.md` in the Spec. -- **Never edit these files.** `tests/test_spec_pin.py` hashes `fixtures/` with git's tree - algorithm and each other file with git's blob algorithm, and fails when one byte differs - or a file is added or missing. A fixture that a reader fails is a finding for the Spec - repository, not something to change here. -- **The pin is a full commit SHA**, never a branch. -- **A skipped Spec test is a failure.** Every test module named `tests/test_spec_*.py` is - held to it by `tests/spec_gate.py`, so a missing package, file or weight cannot turn a - conformance test into a silent pass. -- The reader has no copy of the rules besides its one table module - (`tests/test_spec_no_fallback.py`). - -## Updating the pin - -1. In a checkout of the Spec, check out the new commit and read - `git rev-parse HEAD`, `git rev-parse HEAD:fixtures`, - `git rev-parse HEAD:docs/experiment-v1.md`, `git rev-parse HEAD:tools/audit/refval.py` and - `git rev-parse HEAD:tools/audit/refrecords.py`. -2. Replace the vendored files from that commit, for example - `git archive HEAD fixtures | tar -x -C tests/spec_fixtures`, and `git show HEAD:` for - the three single files. -3. Put the five values in `PIN.toml`. -4. Run the whole suite. The reader, its tables and its tests change in the same pull - request when the Spec changed text or fixtures (the Spec changes first). +| Here | In the Spec | +|---|---| +| `fixtures/` | `fixtures/` | +| `docs/experiment-v1.md` | `docs/experiment-v1.md` | +| `tools/audit/refval.py` | `tools/audit/refval.py` | +| `tools/audit/refrecords.py` | `tools/audit/refrecords.py` | diff --git a/tests/spec_fixtures/docs/experiment-v1.md b/tests/spec_fixtures/docs/experiment-v1.md index b12678cb..d21388ad 100644 --- a/tests/spec_fixtures/docs/experiment-v1.md +++ b/tests/spec_fixtures/docs/experiment-v1.md @@ -147,7 +147,7 @@ Time-lapses (T > 1) are allowed: the file has no T field, and the homogeneity of spelling; a consumer MUST NOT replace the source label by the canonical repeat ID in any of them. - The canonical repeat ID is **added** next to the key, as the field `repeat_group` (the column - name of the legacy design file, §11). Grouping and pairing MUST use `repeat_group`: pooling of + name of the derived `experiment_design.csv`, §11). Grouping and pairing MUST use `repeat_group`: pooling of normalization and scaling, pairing of conditions and families in MorphoCoverage, and any statement about biological N. A consumer MUST NOT group or pair by `biological_repeat` when a description exists, and MUST NOT fall back to folder-name grouping. @@ -296,10 +296,10 @@ accepted, **declared** image type, per imaging unit: brick without a provider manifest states the same declaration in its documented capabilities. The key declares support only; it gives RGB no meaning (the bullet above), and it does not change a description: `image_type = "rgb"` is still user-declared and never inferred. -- The refusal applies to a **described** run (one opened from a published description). A run - without a description has no declared image type to refuse on; what a brick does with colour - images there is the brick's own behaviour (owner decision RC1: such runs keep working as today, - with a warning), not a Spec rule. +- The refusal applies to a **described** run (one opened from a published description). There is + no other run: a run without a published description is refused with `A_DESCRIPTION_REQUIRED` + before any capability is checked (ADR 0002 AR10, which supersedes owner decision RC1's clause + that such runs keep working with a warning). - A file MAY hold RGB and non-RGB units; their lists differ, so that layout is heterogeneous (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED` in Stage 1-2). A unit's `image_type` is part of the `repeats` section digest (§9.3). @@ -699,26 +699,11 @@ validation: a reader MUST NOT try to interpret fields of a version it does not s new folder to the existing ID changes that ID's `source_folder`, which is a reassignment and follows the removed/reassigned rules of §4.6 (recorded user confirmation, else `R_CONFIRMED_ID_CHANGED`). The ID never changes. -- **Legacy `experiment_design.csv` → v1:** read with the strict reader (no `yes`/`no`/empty - coercion). `repeat_group` → canonical repeat ID; `biological_repeat` → `source_repeat`; - `image_count`, `confirmed`, `confirmed_on` copied (see the next point for a group whose rows - have different dates); the legacy `condition` label becomes the condition's `source_folder`, - and the condition ID is proposed as that label when allowed (§4.6a), else another proposed ID - the user confirms; technical repeats (folder, `source_folder`, ID) and channel lists come from - discovery and user confirmation. Controls: 0 marked → no `reference`; 1 marked → every other - condition references it; ≥ 2 marked → a draft with unresolved references that cannot be - cannot be published until the user answers (no name-stem guessing). A legacy design that violates v1 - rules (e.g. `E_REPEAT_ID_SHADOWS_SOURCE`) MUST be refused with the code, not repaired. A legacy - condition label on the reserved or forbidden lists is no longer a violation: it is the - `source_folder` and the user confirms another ID. -- **Legacy groups with different `confirmed_on` (AQ12).** v1 has one date per repeat. A legacy - `repeat_group` whose rows carry different dates takes the **latest** date (what preprocessing's - `ExperimentDesign.confirmed_on` returns today, `design.py:216-218`) and the migration records a - note listing each folder's original date. It never takes the clock. Consequently the - byte-identity of the derived `experiment_design.csv` (§11) is guaranteed **only** for repeat - groups with a single `confirmed_on`; for a group with several dates the derived rows carry the - latest date and only the note keeps the original per-row dates. A group with no dated row - cannot be published until the user supplies the date of their confirmation. +- **No migration from a legacy `experiment_design.csv`** (ADR 0002 AR10 (9), owner decision of + 10 November 2026). Preprocessing MUST NOT read an `experiment_design.csv` found in an output + folder to propose or prefill a description or a draft: a description is created only from + discovery of the experiment folder and the user's answers (§10). The derived + `experiment_design.csv` of §11 is an output only and is never read back. ## 8. Validation boundaries @@ -862,7 +847,7 @@ silently; edits go to a separate draft and are published atomically after valida ## 11. Derived `experiment_design.csv` (compatibility output) -Preprocessing MUST derive the legacy file (schema version 1, preprocessing contract §12) +Preprocessing MUST derive the file (schema version 1, preprocessing contract §12) deterministically from a published file, with the contract's byte rules: | CSV column | Value | @@ -886,22 +871,14 @@ or a byte-order mark, so these four are not yet normative; the golden files use quoting (no golden value needs it), LF after every row including the last, and no byte-order mark, as today's writer does. The CSV carries the **source folder labels exactly as today** and no technical repeat or condition ID, so the choice of a condition or technical repeat ID (for example `ctrl` for -the folder `control`) never changes it. For a legacy dataset whose `repeat_group` values migrate -to the canonical repeat IDs and whose controls migrate deterministically, the derived file MUST be -byte-identical to the one preprocessing writes today. -Known loss: a legacy design marking a control that no other condition references (e.g. one -condition only) exports `control = false`. For a migrated group whose legacy rows carried -different `confirmed_on` dates (§7, AQ12) the derived file carries the group's one date, the -latest, on every row. **Byte-identity is guaranteed only for repeat groups with a single -`confirmed_on`**; for a group with several dates the earlier per-row dates are the only difference -and are recorded in the migration note, which is their only record. +the folder `control`) never changes it. Expected `experiment_design.csv` outputs (golden files) for `v13`, `v33` and `v34` are in `fixtures/expected/`; a derived file MUST equal the golden file byte for byte. ## 12. Drafts -A draft (reconciliation proposal, unconfirmed repeats, unresolved legacy references) is internal +A draft (reconciliation proposal, unconfirmed repeats) is internal to preprocessing and records its base SHA-256 (§10). It is never named `morphobricks.toml`. Downstream consumers (`RunInputs`) read only the confirmed, published `morphobricks.toml` and MUST refuse a draft or any file with an unconfirmed repeat (`A_DRAFT_REFUSED`; a published-path file with `confirmed = false` is also diff --git a/tests/spec_fixtures/fixtures/README.md b/tests/spec_fixtures/fixtures/README.md index 050430ba..58f37f81 100644 --- a/tests/spec_fixtures/fixtures/README.md +++ b/tests/spec_fixtures/fixtures/README.md @@ -36,8 +36,8 @@ their section. predecessor file and are not part of the single-file valid/invalid rules. Publication-base and folder-name rules need no TOML of their own: they run on named fixtures in scenarios P17-P22. - Rules that need images, a run or a consumer (header matching, image type against the header, - mask checks and conversions, mask type names, the FOV key, the `target` prefill, legacy - migration dates) have no single-file TOML form: each is a scenario P23-P33 of the conformance + mask checks and conversions, mask type names, the FOV key, the `target` prefill, derived-CSV + dates) have no single-file TOML form: each is a scenario P23-P33 of the conformance plan on a named fixture, listed in the coverage tables below. - A copy of this tree vendored by a reader MUST hash to the pinned SHA's `fixtures` tree (conformance plan §1). @@ -124,12 +124,12 @@ This is a check of the fixtures against the text, not a conformance claim for an | 1 | Two technical repeats of different counts/orders stay ONE biological replicate | `v06` | Stage 2-3: N per repeat in normalization, embeddings, MorphoCoverage | | 2 | One deviating FOV in a technical repeat refused with image/C-index evidence | `i20`; P01, P02 | Stage 1: `R_UNIT_NOT_HOMOGENEOUS` | | 3 | Swapped C order never mixes ACTIN and DNA in normalization or CellPatcher | `v04`, `v06` | Stage 1 refusal; Stage 3 pooling by canonical ID | -| 4 | Homogeneous legacy datasets keep numerical parity and existing refusals | `v13` | Stage 1: byte-identical derived CSV; Stage 3 parity | +| 4 | A homogeneous described dataset (folder-name IDs confirmed by the user) keeps the existing refusals; no parity against v0.4.0 (ADR 0002 AR10) | `v13` | Stage 1: byte-identical derived CSV (as reviewed for v0.5.0); no legacy numerical parity gate | | 5 | `DNA` with confirmed header `DAPI` passes; unconfirmed contradiction refuses | `v11`; P03 | Stage 1: `R_HEADER_CONTRADICTION` | | 6 | A binary region mask never becomes an instance object set | `v08` | Stage 2: `A_MASK_NOT_INSTANCES` | -| 7 | N35 confirmation survives migration and reopens on discovery changes | `v13`, `i37`; P07 | Stage 1 migration and reconciliation | +| 7 | N35 confirmation survives reconciliation and reopens on discovery changes (the legacy-CSV migration is removed, ADR 0002 AR10 (9)) | `v13`, `i37`; P07 | Stage 1 reconciliation | | 8 | Distinct source repeat names paired by the user give one canonical ID in MorphoCoverage | `v03` | Stage 2 (S2.5) | -| 9 | Missing channels in requested scope: refusal or recorded user subset, never silent drop; a channel no image has is the same refusal (SI1) | `v05`, `v17` | Stage 2: `A_SCOPE_INCOMPLETE` | +| 9 | Missing channels in requested scope: refusal or recorded user subset, never silent drop; a channel no image has is the same refusal (SI1) | `v05`, `v17` | `A_SCOPE_INCOMPLETE`, shown at Stage 3 / gate G3 for both: their layouts are refused at Stages 1-2 (P04); owner decision GS2 for both | | 10 | Metadata-only changes keep valid pixel-derived caches | `v01` vs `v21` | Stage 1: section digests | **Further required cases:** @@ -150,7 +150,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | Implicit-conversion traps | `i07`, `i08`, `i12`-`i14`, `i19`, `i27`-`i30`, `i33`, `i34`, `i36`, `i38`, `i39`; valid side `v15` | | Reference DAG | `i64`-`i68`, `i41`; `v10`, `v19` | | Swapped channel order | `v04`, `v06` | -| Extra / missing channel | `v05`, `v17`, `v22`, `i69` | +| Extra / missing channel | `v05`, `v17` (both Stage 3 / G3, GS2: P04 refuses their layouts at Stages 1-2), `v22`, `i69` | | Within-unit divergence | `i20`; P01, P02 | | Instance vs binary masks | `v07`, `v08`, `i42`, `i43`, `i111`, `i112`; P05, P27, P28 | | Partial masks | `v09` | @@ -167,7 +167,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | FOV key keeps the source label, `repeat_group` added | `v03`, `v30`; P25 | | Mask checks: when, unchecked masks, type names | `v09`; P05, P28, P29 | | Descriptive keys prefill only | `v27`; P31 | -| Legacy migration dates (byte-identity only for a single `confirmed_on`) | `v13`; P30 | +| Derived CSV dates (every row carries its repeat's one `confirmed_on`; no clock read) | `v13`; P30 | | Derived CSV keeps the source folder labels | `v13`, `v33`; P32 | | Aliases (header evidence) | `v11`, `v12`, `v31`, `i46`, `i81`, `i110`; P03, P26 | | Versioning (offline constant, no default) | `i09`-`i15` | @@ -212,10 +212,10 @@ This is a check of the fixtures against the text, not a conformance claim for an | AQ9 | Binary-as-instances only by an explicit, per-request, recorded conversion | `v08`, `i112`; P27 | | AQ10 | Mask representation checked in the mask step; publication records "not yet checked"; unchecked masks refuse | `v09`; P05, P28 | | AQ11 | A discovered mask type breaking the Label rules: `R_MASK_TYPE_NAME_INVALID` | P29 | -| AQ12 | Legacy groups with different `confirmed_on` take the latest date, with a note | `v13`; P30 | +| AQ12 | Legacy groups with different `confirmed_on` take the latest date, with a note. **Superseded 10 November 2026** (ADR 0002 AR10 (9): no legacy-CSV migration) | none (P30 now tests the derived output only) | | AQ13 | Forbidden names apply to ID keys; NFC of channel references; one code per broken rule | `i113`-`i118` | | FR1 | Folder labels are never restricted by the reserved or forbidden names: conditions and technical repeats have a canonical ID separate from the required `source_folder`; one folder, one ID; a folder change for an existing ID is a reassignment; the derived CSV keeps the folder labels | `v33`, `v34`, `i113`, `i115`, `i120`-`i127`, `t10`-`t12`; P32, P33 | -| FR2 | Derived CSV byte-identity is guaranteed only for repeat groups with a single `confirmed_on`; a legacy group with several dates gets the latest, the migration note keeps the per-row dates | `v13`; P30 | +| FR2 | Derived CSV byte-identity is guaranteed only for repeat groups with a single `confirmed_on`; a legacy group with several dates gets the latest, the migration note keeps the per-row dates. **Superseded 10 November 2026** (ADR 0002 AR10 (9): no legacy-CSV migration; every published repeat has one `confirmed_on`) | none (P30 now tests the derived output only) | | RR1 | Record type `image_count_changed` (a confirmed member's image count changes; nothing reopens) | P34 | | RR3 | `image_count_changed` is written for every confirmed member whose count changed, also when the publication changes other things | P34 | | RR2 | Record type `removed` for every confirmed removal of a unit, technical repeat, repeat or condition | P35 | @@ -242,7 +242,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | [`v10_reference_chain`](valid/v10_reference_chain.toml) | RFC case 11 | pass | yes | Reference chains VEC_SiCol6a1 -> VEC_SiNC -> VEC and THBS5_SiCol6a1 -> THBS5_SiNC -> THBS5; shared treatment_key across backgrounds. | | [`v11_header_aliases`](valid/v11_header_aliases.toml) | Adversarial 5 | pass | yes | Canonical DNA confirmed with header evidence DAPI and Hoechst 33342; canonical ACTIN with an exact-match header name that still had to be confirmed. A header name outside these lists is R_HEADER_CONTRADICTION. | | [`v12_unnamed_headers`](valid/v12_unnamed_headers.toml) | Extra: aliases | pass | yes | No header evidence at all (ImageJ TIFFs carry no channel names); the user's confirmation is the only guard. Undetectable physical swaps are a documented limit. | -| [`v13_homogeneous_legacy`](valid/v13_homogeneous_legacy.toml) | Adversarial 4 | pass | yes | A legacy homogeneous dataset: folder-name repeat IDs, one control, same layout everywhere (IDs proposed equal to the folder labels). The derived experiment_design.csv, which carries the `source_folder` labels, must be byte-identical to today's; numerical outputs must keep parity. | +| [`v13_homogeneous_legacy`](valid/v13_homogeneous_legacy.toml) | Adversarial 4 | pass | yes | A legacy homogeneous dataset: folder-name repeat IDs, one control, same layout everywhere (IDs proposed equal to the folder labels). The derived experiment_design.csv, which carries the `source_folder` labels, must be byte-identical to its golden file (a Stage 1 check); there is no numerical parity against v0.4.0 (ADR 0002 AR10). | | [`v14_no_repeat_folders`](valid/v14_no_repeat_folders.toml) | Extra: single convention | pass | yes | No biological-repeat or technical-repeat folders: the source label is `single` (N31, one group for the whole experiment). | | [`v15_toml_keys_are_text`](valid/v15_toml_keys_are_text.toml) | Extra: implicit-conversion traps (valid side) | pass | yes | Folder labels that YAML would retype (2026-01-01, 012, NO, 1.0, off) stay exact text in TOML keys and strings. The quoted key "1.0" is needed: a bare 1.0 would be a dotted key. | | [`v16_display_name_change`](valid/v16_display_name_change.toml) | Owner refinement: immutable IDs | pass | yes | v01 with every `display_name` changed or added (conditions, channels, repeats). Identity is identical: all four section digests equal v01's, nothing may be invalidated. | diff --git a/tests/test_cli_describe.py b/tests/test_cli_describe.py index e1ba20f5..6837cd80 100644 --- a/tests/test_cli_describe.py +++ b/tests/test_cli_describe.py @@ -81,24 +81,38 @@ def test_a_folder_that_cannot_be_an_id_has_no_proposal_and_is_not_repaired(tmp_p assert "DrugA: proposed ID 'DrugA'" in out -def test_describe_reads_the_legacy_design_without_confirming_anything(experiment, tmp_path, capsys): +def orphan_designs(): + """An ``experiment_design.csv`` as an older version wrote it, and one that is no design.""" + rows = [row(c, r, control=c == "CTRL") for c in ("CTRL", "DrugA") for r in ("rep1", "rep2")] + return { + "a confirmed design": rows, + "no design at all": None, + } + + +@pytest.mark.parametrize("kind", ["a confirmed design", "no design at all"]) +@pytest.mark.parametrize("publish", [False, True]) +def test_an_experiment_design_csv_in_the_output_folder_is_ignored_by_describe( + experiment, tmp_path, capsys, kind, publish +): + """Experiment-v1 section 7: nothing proposes or prefills a draft from the derived CSV.""" + arguments = ["describe", str(experiment), "--output", str(tmp_path / "out")] + arguments += ["--publish"] if publish else [] + status = main(arguments) + expected = (status, capsys.readouterr().out) output = tmp_path / "out" - rows = [ - row("CTRL", "rep1", control=True), - row("CTRL", "rep2", control=True), - row("DrugA", "rep1", control=True, confirmed=False, on=""), - row("DrugA", "rep2", control=True, confirmed=False, on=""), - ] output.mkdir() - write_experiment_design(DesignTable(tuple(rows)), output / DESIGN) + rows = orphan_designs()[kind] + if rows is None: + (output / DESIGN).write_text("not,a,design\n1,2,3\n", encoding="utf-8") + else: + write_experiment_design(DesignTable(tuple(rows)), output / DESIGN) before = listing(tmp_path) - assert main(["describe", str(experiment), "--output", str(output)]) == 0 - out = capsys.readouterr().out - assert "legacy experiment_design.csv" in out - assert "DrugA -> CTRL" not in out # two controls: nothing is guessed - assert "marked as a control, as is another one" in out - assert "rep1: proposed ID 'rep1'; not confirmed" in out - assert listing(tmp_path) == before + status = main(arguments) + assert (status, capsys.readouterr().out) == expected + assert "confirmed on" not in expected[1] and "legacy" not in expected[1] + assert " CTRL -> " not in expected[1] # no reference taken from the control column + assert listing(tmp_path) == before # the orphan is neither rewritten nor removed def test_publish_refuses_a_draft_with_open_questions_and_lists_them(experiment, tmp_path, capsys): @@ -114,20 +128,6 @@ def test_publish_refuses_a_draft_with_open_questions_and_lists_them(experiment, assert listing(tmp_path) == before -def test_publish_never_confirms_for_the_user(experiment, tmp_path, capsys): - """A fully confirmed legacy design still leaves the IDs to the user: nothing is adopted.""" - output = tmp_path / "out" - output.mkdir() - rows = [row(c, r) for c in ("CTRL", "DrugA") for r in ("rep1", "rep2")] - write_experiment_design(DesignTable(tuple(rows)), output / DESIGN) - before = listing(tmp_path) - assert main(["describe", str(experiment), "--output", str(output), "--publish"]) == 1 - out = capsys.readouterr().out - assert "choose the ID of condition CTRL" in out - assert "confirmed on 2026-10-30" in out # the user's own earlier confirmation, as given - assert listing(tmp_path) == before # no design rewritten, no description, no draft - - def test_a_complete_draft_is_still_not_publishable_from_the_command_line(tmp_path): """Every ID and repeat confirmed leaves only what the draft does not hold.""" from morphobricks_preprocessing.cli_describe import NOT_IN_DRAFT, draft_blockers @@ -151,18 +151,6 @@ def test_a_complete_draft_is_still_not_publishable_from_the_command_line(tmp_pat assert draft_blockers(done) == [NOT_IN_DRAFT] -def test_a_legacy_design_that_cannot_be_read_is_refused_and_left_alone( - experiment, tmp_path, capsys -): - output = tmp_path / "out" - output.mkdir() - (output / DESIGN).write_text("not,a,design\n1,2,3\n", encoding="utf-8") - before = listing(tmp_path) - assert main(["describe", str(experiment), "--output", str(output)]) == 1 - assert "cannot be drafted" in capsys.readouterr().out - assert listing(tmp_path) == before - - def test_a_missing_experiment_folder_is_an_error(tmp_path, capsys): assert main(["describe", str(tmp_path / "nowhere")]) == 1 assert "does not exist or is not a folder" in capsys.readouterr().out diff --git a/tests/test_design.py b/tests/test_design.py index 0bf56abe..707b5589 100644 --- a/tests/test_design.py +++ b/tests/test_design.py @@ -2,9 +2,9 @@ The file is an output only (v0.5.1, Spec ADR 0002 AR11 (3)): the run derives it from the published description and checks it against the folders found (:func:`reconcile_design`, -:func:`design_issues`); :func:`parse_design_bytes` is the strict parser of the draft's legacy -migration. The editor model, the default pairing, the lenient reader, ``load_design`` and the -strict reader for other bricks were removed (SWEEP-PP). +:func:`design_issues`); :func:`parse_design_bytes` is the strict parser of a file the writer wrote +(no brick reads the file as an input). The editor model, the default pairing, the lenient +reader, ``load_design`` and the strict reader for other bricks were removed (SWEEP-PP). """ from dataclasses import replace diff --git a/tests/test_design_file.py b/tests/test_design_file.py index 9a11d9cb..3475e268 100644 --- a/tests/test_design_file.py +++ b/tests/test_design_file.py @@ -3,7 +3,7 @@ The file is an output only (v0.5.1, Spec ADR 0002 AR11 (3)): no brick reads it back for its repeat groups (they come from the description through ``RunInputs``). Its golden here is parsed in the test with the standard ``csv`` module; :func:`parse_design_bytes` is the strict parser of -the draft's experiment-v1 section 7 migration, which refuses anything doubtful. The strict +a file the writer wrote, which refuses anything doubtful. The strict reader for other bricks (``read_confirmed_design``, ``ConfirmedDesign``, ``DesignCheckError``) was removed in SWEEP-PP. """ diff --git a/tests/test_draft.py b/tests/test_draft.py index bc6f07c5..2e2e21f1 100644 --- a/tests/test_draft.py +++ b/tests/test_draft.py @@ -1,6 +1,6 @@ -"""The draft of an experiment description: proposed IDs and the legacy design migration. +"""The draft of an experiment description: proposed IDs. -Experiment-v1 sections 4.6a and 7, conformance scenarios P30 and P32 (the proposal part). The +Experiment-v1 section 4.6a, conformance scenario P32 (the proposal part). The owner's answers: Q4 a header name is proposed as a channel ID only when it is valid as it stands, Q6 the proposal rule covers repeat IDs and the condition ``single`` rule, Q8 the added fields are ``condition_id``, ``repeat_group`` and ``technical_repeat_id``. @@ -8,14 +8,8 @@ import pytest -from morphobricks_preprocessing import DesignFormatError, discover_experiment -from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design -from morphobricks_preprocessing.draft import ( - DraftError, - LegacyMigrationRefused, - draft_from_discovery, - migrate_legacy_design, -) +from morphobricks_preprocessing import discover_experiment +from morphobricks_preprocessing.draft import DraftError, draft_from_discovery from morphobricks_preprocessing.draft_ids import ( id_problems, propose_channel_ids, @@ -30,25 +24,6 @@ def folders(root, *paths): return discover_experiment(root) -def legacy(tmp_path, rows): - """Return the bytes of a legacy design with ``rows`` written by the brick's own writer.""" - path = tmp_path / "legacy.csv" - write_experiment_design(DesignTable(tuple(rows)), path) - return path.read_bytes() - - -def row(condition, repeat, group=None, *, control=False, on="2026-10-30", confirmed=True): - return DesignRow( - condition, - repeat, - group or repeat, - control=control, - image_count=2, - confirmed=confirmed, - confirmed_on=on if confirmed else "", - ) - - # --- proposed IDs (P32, Q4, Q6) --------------------------------------------------------------- @@ -201,180 +176,3 @@ def test_a_proposal_keeps_the_raw_header_name_and_its_index(): (first, second) = propose_channel_ids(["DAPI", " raw "]) assert (first.index, first.header_name, first.proposed_id) == (0, "DAPI", "DAPI") assert (second.index, second.header_name, second.proposed_id) == (1, " raw ", None) - - -# --- legacy migration (P30) -------------------------------------------------------------------- - - -def test_a_legacy_design_migrates_to_proposals_and_open_confirmations(tmp_path): - discovery = folders(tmp_path / "exp", "CTRL/rep1", "CTRL/rep2", "DrugA/rep1", "DrugA/rep2") - data = legacy( - tmp_path, - [ - row("CTRL", "rep1", control=True), - row("CTRL", "rep2", control=True), - row("DrugA", "rep1"), - row("DrugA", "rep2"), - ], - ) - draft = migrate_legacy_design(data, discovery) - assert draft.origin == "legacy" - assert {g: (r.id.proposed, r.id.confirmed) for g, r in draft.repeats.items()} == { - "rep1": ("rep1", None), - "rep2": ("rep2", None), - } - assert {r.confirmed_on for r in draft.repeats.values()} == {"2026-10-30"} - assert all(r.confirmed for r in draft.repeats.values()) - assert dict(draft.references) == {"DrugA": "CTRL"} - assert draft.notes == () - assert {q.code for q in draft.open_questions()} == {"id_pending"} # IDs not adopted - assert not draft.publishable - for kind, key in [ - ("conditions", "CTRL"), - ("conditions", "DrugA"), - ("technical_repeats", "single"), - ("repeats", "rep1"), - ("repeats", "rep2"), - ]: - draft = draft.confirm_id(kind, key, key) - assert draft.open_questions() == () - assert draft.publishable - - -def test_p30_a_group_with_two_dates_takes_the_latest_and_a_note_lists_both(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1", "A/rep2", "B/rep2") - data = legacy( - tmp_path, - [ - row("A", "rep1", "G1", on="2026-10-01"), - row("B", "rep1", "G1", on="2026-10-15"), - row("A", "rep2", "G2", on="2026-10-20"), - row("B", "rep2", "G2", on="2026-10-20"), - ], - ) - draft = migrate_legacy_design(data, discovery) - assert draft.repeats["G1"].confirmed_on == "2026-10-15" - assert draft.repeats["G2"].confirmed_on == "2026-10-20" - (note,) = draft.notes - assert note.repeat_group == "G1" - assert note.kept == "2026-10-15" - assert set(note.dates) == {("A", "rep1", "2026-10-01"), ("B", "rep1", "2026-10-15")} - - -def test_a_confirmed_group_without_any_date_stays_open_and_takes_no_clock_date(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1") - data = legacy(tmp_path, [row("A", "rep1", on=""), row("B", "rep1", on="")]) - draft = migrate_legacy_design(data, discovery) - assert draft.repeats["rep1"].confirmed is True - assert draft.repeats["rep1"].confirmed_on == "" - assert ("confirmed_on_missing", "repeat rep1") in { - (q.code, q.subject) for q in draft.open_questions() - } - - -def test_an_unconfirmed_legacy_group_stays_unconfirmed(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1") - data = legacy(tmp_path, [row("A", "rep1", confirmed=False), row("B", "rep1")]) - repeat = migrate_legacy_design(data, discovery).repeats["rep1"] - assert (repeat.confirmed, repeat.confirmed_on) == (False, "") - - -@pytest.mark.parametrize( - ("marked", "references", "unresolved"), - [ - ((), {}, ()), - (("A",), {"B": "A", "C": "A"}, ()), - (("A", "B"), {}, ("A", "B")), - ], -) -def test_legacy_controls_give_no_reference_one_reference_or_an_open_question( - tmp_path, marked, references, unresolved -): - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1", "C/rep1") - data = legacy(tmp_path, [row(c, "rep1", control=c in marked) for c in "ABC"]) - draft = migrate_legacy_design(data, discovery) - assert dict(draft.references) == references - assert draft.unresolved_controls == unresolved - assert ( - ("control_unresolved", "A") in {(q.code, q.subject) for q in draft.open_questions()} - ) == (len(unresolved) > 1) - - -def test_a_legacy_condition_on_the_forbidden_list_migrates_with_no_proposed_id(tmp_path): - discovery = folders(tmp_path / "exp", "control/rep1", "DrugA/rep1") - data = legacy(tmp_path, [row("control", "rep1", control=True), row("DrugA", "rep1")]) - draft = migrate_legacy_design(data, discovery) # not refused - assert draft.conditions["control"].proposed is None - assert dict(draft.references) == {"DrugA": "control"} # by folder; the ID is confirmed later - assert ("id_pending", "condition control") in { - (q.code, q.subject) for q in draft.open_questions() - } - assert ( - draft.confirm_id("conditions", "control", "ctrl").conditions["control"].confirmed == "ctrl" - ) - - -def test_lenient_legacy_values_are_refused_not_coerced(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1") - data = legacy(tmp_path, [row("A", "rep1")]).replace(b",true,", b",yes,") - with pytest.raises(DesignFormatError): - migrate_legacy_design(data, discovery) - - -def test_a_legacy_design_that_breaks_a_rule_is_refused_with_the_code(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1", "A/rep2", "B/rep1", "B/rep2") - swapped = legacy( - tmp_path, - [ - row("A", "rep1", "rep2"), - row("A", "rep2", "rep1"), - row("B", "rep1", "rep2"), - row("B", "rep2", "rep1"), - ], - ) - with pytest.raises(LegacyMigrationRefused) as refused: - migrate_legacy_design(swapped, discovery) - assert refused.value.codes == ("E_REPEAT_ID_SHADOWS_SOURCE",) - twice = legacy(tmp_path, [row("A", "rep1", "G"), row("A", "rep2", "G")]) - with pytest.raises(LegacyMigrationRefused) as refused: - migrate_legacy_design(twice, discovery) - assert refused.value.codes == ("repeat_group_conflict",) - - -@pytest.mark.parametrize("group", ["z", "files", "overrides"]) -def test_a_legacy_repeat_group_that_is_not_an_allowed_id_opens_a_question(tmp_path, group): - """Owner decision 2026-11-07, Spec 4.6a: not refused, no proposal, never repaired.""" - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1") - data = legacy(tmp_path, [row("A", "rep1", group), row("B", "rep1", group)]) - draft = migrate_legacy_design(data, discovery) # not refused - repeat = draft.repeats[group] - assert repeat.id.proposed is None and repeat.id.confirmed is None - assert repeat.confirmed and repeat.confirmed_on == "2026-10-30" # the legacy answer is kept - assert [m.source_repeat for m in repeat.members] == ["rep1", "rep1"] - assert ("id_pending", f"repeat {group}") in { - (q.code, q.subject) for q in draft.open_questions() - } - assert not draft.publishable - with pytest.raises(DraftError) as refused: # the user cannot adopt it unchanged - draft.confirm_id("repeats", group, group) - assert "E_FORBIDDEN_FIELD" in refused.value.codes - named = draft.confirm_id("repeats", group, "R1") - assert named.repeats[group].id.confirmed == "R1" - - -def test_new_and_missing_folders_are_never_merged_silently(tmp_path): - discovery = folders(tmp_path / "exp", "A/rep1", "B/rep1", "B/rep2") - data = legacy(tmp_path, [row("A", "rep1"), row("B", "rep1"), row("B", "gone")]) - draft = migrate_legacy_design(data, discovery) - assert draft.legacy_missing == (("B", "gone"),) - assert draft.repeats["rep1"].confirmed is True - assert draft.repeats["rep2"].confirmed is False # a new folder is pending, not confirmed - codes = {(q.code, q.subject) for q in draft.open_questions()} - assert ("legacy_folder_missing", "B / gone") in codes - assert ("repeat_not_confirmed", "repeat rep2") in codes - joined = legacy(tmp_path, [row("A", "rep1"), row("B", "rep1")]) - rejoined = migrate_legacy_design( - joined, folders(tmp_path / "exp2", "A/rep1", "B/rep1", "C/rep1") - ) - assert rejoined.repeats["rep1"].confirmed is False # the new folder joined; asked again - assert rejoined.repeats["rep1"].confirmed_on == "" diff --git a/tests/test_draft_answers.py b/tests/test_draft_answers.py index 6ccfc1ff..439dcfd3 100644 --- a/tests/test_draft_answers.py +++ b/tests/test_draft_answers.py @@ -10,15 +10,13 @@ ) from morphobricks_preprocessing import discover_experiment from morphobricks_preprocessing.description import Channel, Condition -from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design -from morphobricks_preprocessing.draft import DraftError, draft_from_discovery, migrate_legacy_design +from morphobricks_preprocessing.draft import DraftError, draft_from_discovery from morphobricks_preprocessing.draft_answers import ( ConditionAnswer, DraftAnswers, DraftIncomplete, RepeatAnswer, UnitAnswer, - apply_answers, confirm_repeat, description_from_draft, ) @@ -152,28 +150,3 @@ def test_retired_ids_stay_in_used_ids(exp): description = description_from_draft(base, complete()) assert description.used_ids["repeats"] == ("old", "rep1", "rep2") assert description.used_ids["channels"] == ("ACTIN", "DNA", "GFP") - - -def test_a_legacy_confirmation_is_carried_not_asked_again(tmp_path): - make_tree(tmp_path / "exp", PATHS) - rows = [ - DesignRow(c, r, r, control=False, image_count=2, confirmed=True, confirmed_on="2026-10-30") - for c in ("CTRL", "DrugA") - for r in ("rep1", "rep2") - ] - csv = tmp_path / "legacy.csv" - write_experiment_design(DesignTable(tuple(rows)), csv) - draft = migrate_legacy_design(csv.read_bytes(), discover_experiment(tmp_path / "exp")) - answers = complete() - only_ids = DraftAnswers( - ABSENT, - SPEC_COMMIT, - {k: ConditionAnswer(id=k) for k in ("CTRL", "DrugA")}, - answers.technical_repeats, - {k: RepeatAnswer(id=k) for k in ("rep1", "rep2")}, - answers.units, - answers.channels, - ) - description = description_from_draft(draft, only_ids) - assert {r.confirmed_on for r in description.repeats.values()} == {"2026-10-30"} - assert apply_answers(draft, only_ids).repeats["rep1"].confirmed_on == "2026-10-30" diff --git a/tests/test_orphan_design_csv.py b/tests/test_orphan_design_csv.py new file mode 100644 index 00000000..2488fc2d --- /dev/null +++ b/tests/test_orphan_design_csv.py @@ -0,0 +1,84 @@ +"""An ``experiment_design.csv`` in the output folder never proposes or prefills a draft. + +Experiment-v1 section 7 (Spec ADR 0002 AR10 (9)): the derived CSV is an output and is never read +back. The orphan below is exactly what the removed migration used to take: confirmed repeats +with dates, a control, and groups that differ from the default pairing. Draft creation, +``describe`` and the notebook (first open and reopen) must give what they give without it. +""" + +import pytest + +from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design +from morphobricks_preprocessing.discovery import default_output_folder, discover_experiment +from morphobricks_preprocessing.draft import draft_from_discovery +from tiff_tree import design_tree + +FOLDERS = ["ctrl/rep1", "ctrl/rep2", "drug/rep1", "drug/rep2"] + + +def orphan(output): + """Write a fully confirmed design with a control and odd groups; return its bytes.""" + rows = [ + DesignRow( + c, + r, + f"G-{r}", + control=c == "ctrl", + image_count=2, + confirmed=True, + confirmed_on="2026-10-30", + ) + for c in ("ctrl", "drug") + for r in ("rep1", "rep2") + ] + path = output / "experiment_design.csv" + write_experiment_design(DesignTable(tuple(rows)), path) + return path.read_bytes() + + +@pytest.fixture +def root(tmp_path): + return design_tree(tmp_path / "exp", FOLDERS) + + +def test_draft_creation_ignores_an_orphan_design_csv(root): + output = default_output_folder(root) + expected = draft_from_discovery(discover_experiment(root, output)) + output.mkdir(parents=True) + data = orphan(output) + draft = draft_from_discovery(discover_experiment(root, output)) + assert draft == expected + assert set(draft.repeats) == {"rep1", "rep2"} # not the file's G-rep1, G-rep2 + assert not draft.references + assert not any(r.confirmed or r.confirmed_on for r in draft.repeats.values()) + assert (output / "experiment_design.csv").read_bytes() == data + + +def test_the_notebook_ignores_an_orphan_design_csv_on_open_and_on_reopen(root, tmp_path): + pytest.importorskip("ipywidgets") + pytest.importorskip("matplotlib") + from morphobricks_preprocessing.notebook_ui import PreprocessingNotebook + + def opened(output): + nb = PreprocessingNotebook(str(root), output_folder=str(output)) + for step in (nb.welcome, nb.folder_step, nb.axis_step, nb.run_step, nb.results_step): + step() + nb.check_button.click() + return nb + + def seen(nb): + draft = nb.draft + return ( + sorted(draft.repeats), + dict(draft.references), + [(r.confirmed, r.confirmed_on) for r in draft.repeats.values()], + [(q.code, q.subject) for q in draft.open_questions()], + ) + + baseline = seen(opened(tmp_path / "plain")) + output = tmp_path / "with-orphan" + output.mkdir() + data = orphan(output) + assert seen(opened(output)) == baseline + assert seen(opened(output)) == baseline # reopened on the same output folder + assert (output / "experiment_design.csv").read_bytes() == data diff --git a/tests/test_spec_derived_design.py b/tests/test_spec_derived_design.py index 546a5932..4f543aac 100644 --- a/tests/test_spec_derived_design.py +++ b/tests/test_spec_derived_design.py @@ -12,22 +12,11 @@ import pytest from frozen_clock import FrozenClock -from morphobricks_preprocessing import discover_experiment, parse_design_bytes +from morphobricks_preprocessing import parse_design_bytes from morphobricks_preprocessing.description import description_from_bytes from morphobricks_preprocessing.design_derive import derive_design, write_derived_design -from morphobricks_preprocessing.draft import migrate_legacy_design -from morphobricks_preprocessing.draft_answers import ( - ChannelAnswer, - ConditionAnswer, - DraftAnswers, - RepeatAnswer, - UnitAnswer, - description_from_draft, -) -from morphobricks_preprocessing.draft_format import ABSENT, SPEC_COMMIT from spec_oracle import refval from spec_pin import fixtures, spec_root -from tiff_tree import make_tree VALID = {item.name.split("_")[0]: item for item in fixtures("valid")} GOLDEN = sorted((spec_root() / "fixtures" / "expected").glob("*.experiment_design.csv")) @@ -70,74 +59,20 @@ def test_the_file_names_folders_never_ids(): assert {row.condition for row in rows if row.control} == {"control"} -# P30: the legacy v13 answers, migrated, give back the legacy file ------------------------- - -V13_FOLDERS = [ - f"{c}/{r}/img{i:02d}.tif" - for c in ("CTRL", "DrugA", "DrugB") - for r in ("rep1", "rep2", "rep3") - for i in range(10) -] -TWO_DATES = { - b"CTRL,rep1,rep1,true,10,0,true,2026-10-30": b"CTRL,rep1,rep1,true,10,0,true,2026-10-01", - b"DrugA,rep1,rep1,false,10,0,true,2026-10-30": b"DrugA,rep1,rep1,false,10,0,true,2026-10-15", - b"DrugB,rep1,rep1,false,10,0,true,2026-10-30": b"DrugB,rep1,rep1,false,10,0,true,2026-10-15", -} - - -def _only_ids(draft): - """The user's answers to a migrated v13 draft: the proposed IDs, the unit channels and type. - - The image type is answered as an ordinary channel image ("channels"): an unanswered type is - an open question (``unit_image_type_unanswered``, audit batch 33). - """ - conditions, repeats = ("CTRL", "DrugA", "DrugB"), ("rep1", "rep2", "rep3") - return DraftAnswers( - ABSENT, - SPEC_COMMIT, - {c: ConditionAnswer(id=c) for c in conditions}, - {"single": "single"}, - {r: RepeatAnswer(id=r) for r in repeats}, - { - (c, r, "single"): UnitAnswer(("DNA", "ACTIN"), "channels") - for c in conditions - for r in repeats - }, - {"DNA": ChannelAnswer("DAPI", None), "ACTIN": ChannelAnswer("F-actin", None)}, - ) - - -@pytest.mark.parametrize("two_dates", [False, True], ids=["one_date", "two_dates"]) -def test_p30_the_derived_file_of_migrated_legacy_answers_differs_only_in_the_earlier_date( - two_dates, tmp_path, monkeypatch -): - """P30: one date everywhere gives the legacy file; rep1 with two dates keeps the latest.""" - make_tree(tmp_path / "exp", V13_FOLDERS) - golden = (next(p for p in GOLDEN if p.name.startswith("v13"))).read_bytes() - legacy = golden - for old, new in TWO_DATES.items() if two_dates else (): - assert legacy.count(old) == 1 - legacy = legacy.replace(old, new) +# P30: v13 derived twice under a frozen clock moved by a day ------------------------------ + + +def test_p30_v13_derived_twice_with_a_clock_moved_by_a_day_equals_the_golden(tmp_path, monkeypatch): + """P30 (amended 10 Nov 2026, ADR 0002 AR10 (9)): no legacy CSV is read; the clock never is.""" + golden = next(p for p in GOLDEN if p.name.startswith("v13")).read_bytes() + description = description_from_bytes(VALID["v13"].data) clock = FrozenClock(monkeypatch, datetime.date(2026, 11, 9)) - draft = migrate_legacy_design(legacy, discover_experiment(tmp_path / "exp")) - description = description_from_draft(draft, _only_ids(draft)) - derived = write_derived_design(description, tmp_path / "design.csv").read_bytes() + first = write_derived_design(description, tmp_path / "first.csv") + clock.move(1) + second = write_derived_design(description, tmp_path / "second.csv") assert clock.reads == 0 # the clock is never read - changed = [ - (old, new) - for old, new in zip(legacy.splitlines(), derived.splitlines(), strict=True) - if old != new - ] - if not two_dates: - assert derived == legacy == golden and draft.notes == () - return - assert description.repeats["rep1"].confirmed_on == "2026-10-15" - assert changed == [ # the one-date groups rep2 and rep3 are byte-identical - ( - b"CTRL,rep1,rep1,true,10,0,true,2026-10-01,1", - b"CTRL,rep1,rep1,true,10,0,true,2026-10-15,1", + assert first.read_bytes() == second.read_bytes() == golden + for row in derive_design(description).rows: # every row carries its repeat's date + assert ( + row.confirmed_on == description.repeats[row.repeat_group].confirmed_on == "2026-10-30" ) - ] - (note,) = draft.notes - assert (note.repeat_group, note.kept) == ("rep1", "2026-10-15") - assert {day for _, _, day in note.dates} == {"2026-10-01", "2026-10-15"} From f4bc30026fb8d62b38e0b6dea7d701427d180376 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 18:34:28 +0000 Subject: [PATCH 2/2] Remove stale legacy-migration references from the contract Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- docs/contract.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/contract.md b/docs/contract.md index 8f421343..c1fd01e8 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -741,7 +741,7 @@ instead of keeping copies. | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. A size that cannot be converted to a positive finite length in micrometres (`1e308 m`) is unknown, never infinite, and an infinite or NaN value is never the same as any other (issue #93; before, `inf` matched every size). The folder check's `different_pixel_sizes` uses both (§11). | | Experiment design | *(removed in v0.5.1, SWEEP-PP)* `read_confirmed_design`, `ConfirmedDesign`, `DesignCheckError`, `load_design` | `experiment_design.csv` is an output only (Spec ADR 0002 AR11 (3)); no brick reads it for its repeat groups. A brick takes `repeat_group` from the published description through `RunInputs` (`Fov.repeat_group`, `resolve_request`). The codes `missing_experiment_design`, `unreadable_experiment_design` and `changed_experiment_design` are no longer raised by this package (`RunInputs.open` still refuses a design file whose SHA-256 is not the manifest's). | | Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are found by `resolve_request` (§9.1) against the consumer's declared capability, and a brick raises them through `ResolvedRequest.require()` (or reports its `refusals`). A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. `Fov.image_type` (§9.1) is each position's declared `rgb` or `channels`; there is no run-wide image type. `RunInputs.resolve_request()` (§9.1) is the one interpreter of a described run for every route: a brick builds a `Request` and reads through the `FovSelection`s it returns, so channel-by-ID, scope, mask-representation and RGB rules have one implementation; `VolumeNotSupportedError` (`A_VOLUME_NOT_SUPPORTED`) and `DescriptionRequiredError` (`A_DESCRIPTION_REQUIRED`, raised by `RunInputs.open`) are the resolver contract's own codes (MorphoBricks-Spec ADR 0002), not experiment-v1 §8's. | -| Experiment description (Stage 1) | `read_description(path)`, `description_consume.read_published_description(path)` (the published `morphobricks.toml` only), `run.description_file.publish_description` (the only writer), `draft.draft_from_discovery`, `draft.migrate_legacy_design`, `description_reconcile.reconcile` | A brick reads the **published** `morphobricks.toml` through `RunInputs` (§9.1) and never a draft: a draft is internal to preprocessing, is never named `morphobricks.toml` and is refused with `A_DRAFT_REFUSED` (experiment-v1 §12). The command line shows the draft (`describe`, §12) and confirms nothing: the user's answers are the only way an ID, a repeat or a removal becomes confirmed. The owner's answers Q1 to Q8 of the Stage 1 design are in plan §12, question 78. | +| Experiment description (Stage 1) | `read_description(path)`, `description_consume.read_published_description(path)` (the published `morphobricks.toml` only), `run.description_file.publish_description` (the only writer), `draft.draft_from_discovery`, `description_reconcile.reconcile` | A brick reads the **published** `morphobricks.toml` through `RunInputs` (§9.1) and never a draft: a draft is internal to preprocessing, is never named `morphobricks.toml` and is refused with `A_DRAFT_REFUSED` (experiment-v1 §12). The command line shows the draft (`describe`, §12) and confirms nothing: the user's answers are the only way an ID, a repeat or a removal becomes confirmed. The owner's answers Q1 to Q8 of the Stage 1 design are in plan §12, question 78. | ## 17. Named-axis copies (`convert`, v0.4.0, design v0.4 §5) @@ -767,7 +767,7 @@ exclude=, missing_as_singleton=, memory_limit=, progress=)` (`conversion`), the ## 18. The description draft (`morphobricks.draft.json`, Stage 1) -The answers a user has given while describing an experiment, before any `morphobricks.toml` is published (experiment-v1 §10, §12). It lives in the output folder, is never named `morphobricks.toml`, has no `schema` key, and no consumer reads it as a description (`RunInputs` refuses a draft: `A_DRAFT_REFUSED`). It holds **answers only**: the folders, counts and proposals are derived again from the folder (`draft_from_discovery`) or the legacy design (`migrate_legacy_design`) on every resume, so a changed folder is never hidden by an old file. Code: `draft_answers.py` (the answers, `apply_answers`, `description_from_draft`), `draft_format.py` (strict bytes), `run/draft_file.py` (`new_answers`, `save_draft`, `load_draft`, `publish_draft`). +The answers a user has given while describing an experiment, before any `morphobricks.toml` is published (experiment-v1 §10, §12). It lives in the output folder, is never named `morphobricks.toml`, has no `schema` key, and no consumer reads it as a description (`RunInputs` refuses a draft: `A_DRAFT_REFUSED`). It holds **answers only**: the folders, counts and proposals are derived again from the folder (`draft_from_discovery`) on every resume, so a changed folder is never hidden by an old file. Code: `draft_answers.py` (the answers, `apply_answers`, `description_from_draft`), `draft_format.py` (strict bytes), `run/draft_file.py` (`new_answers`, `save_draft`, `load_draft`, `publish_draft`). One JSON object, UTF-8, two-space indent, sorted keys, LF, a final newline, no duplicate key and no `NaN`. Every key below is required; a key of this list is the only key allowed at its level. @@ -779,7 +779,7 @@ One JSON object, UTF-8, two-space indent, sorted keys, LF, a final newline, no d | `spec_commit` | The 40-digit MorphoBricks-Spec commit the rules are pinned to (`SPEC_COMMIT`, equal to `tests/spec_fixtures/PIN.toml`) | | `conditions` | Condition **source folder** → object with only the keys that were answered, from `id`, `display_name`, `reference` (the source folder of the condition it is compared with, or `""` when the user answered that it is compared with nothing; absent means not answered), `target`, `treatment_key` | | `technical_repeats` | Technical-repeat source folder (`single` for none) → its confirmed ID | -| `repeats` | The pairing label of the draft's repeat (the folder name, or the legacy `repeat_group`) → object with only the answered keys, from `id`, `confirmed_on`, `display_name` | +| `repeats` | The pairing label of the draft's repeat (the folder name) → object with only the answered keys, from `id`, `confirmed_on`, `display_name` | | `pairing` | Condition source folder → source repeat folder → the pairing label of the repeat the user paired that folder with (only folders not in the repeat of their own name; applied to the draft first, so the `repeats` keys are the paired labels; refused with `pairing_after_confirmation` once a repeat is named or confirmed, `unknown_key` for a folder the draft lacks). A reference to the condition itself or one that closes a loop is refused when saved (`E_REFERENCE_SELF`, `E_REFERENCE_CYCLE`). | | `units` | `condition folder` → `source repeat folder` → `technical-repeat folder` → `{channels, image_type?, display_name?}`; `channels` lists, for C0, C1, ... in order, a channel ID (every position needs a label the user declared, for example `unused`; an empty string is refused) or `null` (not tagged yet); `image_type` is `"rgb"` or `"channels"` (an ordinary channel image; absent means not answered), any other value is refused | | `channels` | Channel ID → `{display_name?, header_names?}` (the label list of the notebook's *Your channels* screen; `header_names` are the raw header names of the cells the user tagged with that label: the tag is the alias confirmation, of the name the cell's header has when it is tagged. At each tag the names no tagged cell has any more are dropped, and only the names of the cells just tagged are added: a tag saved while the header had another name never confirms the name it has now, so `R_HEADER_CONTRADICTION` stays until that cell is tagged again) | @@ -792,5 +792,5 @@ Rules. - **Saving** (`save_draft`) is atomic (temporary file, one replace) and under the output folder's lock, after each answer. The answers are applied to the draft and the exact bytes are read back with the strict reader first, so an answer that would be refused on resume (a `spec_commit` other than the pinned one, `spec_commit_mismatch`; an ID the reader would refuse, a folder the draft does not have, a date that is not a real day) is refused when saved and nothing is written. An incomplete draft can be saved. The file is replaced, not merged. A symbolic link at the draft's path is refused and never replaced. - **Reading** is strict and never repairs: bytes that are not UTF-8 JSON, a byte-order mark, a duplicate key, `NaN`, another `format`, an unknown `version`, an unknown key at any level, a missing key, a value of the wrong type or a date that is not a real `YYYY-MM-DD` day are `DraftFileError` (`code`: `malformed`, `unknown_version`, `unknown_key`, `R_DRAFT_BASE_MISSING`, `R_FILE_SYMLINK`). Reading accepts any 40-digit `spec_commit`; whether it is the current one is decided at resume and save, below. - **Resume** (`load_draft`): the file's `spec_commit` must equal the pinned `SPEC_COMMIT`, else `DraftFileError` with code `spec_commit_mismatch` (owner decision, 2026-11-08): the confirmations in the file were given under other rules, so none is carried forward. The message names both commits, says the published `morphobricks.toml` is untouched and says to start a fresh draft from the folders. Library code never rewrites the stored commit and never deletes or replaces the stale draft; the caller derives a fresh draft and saves it, which replaces the file. `publish_draft` resumes first, so it refuses the same way and writes nothing. Then the file's `base_sha256` must equal the SHA-256 of the file published now (or `absent` if there is none), else `R_PUBLISHED_FILE_CHANGED` (`PublicationRefused`); the draft is never merged with the new file, it is derived again from it. An answer keyed by a folder or repeat the draft no longer has is `DraftError` (`unknown_key`), never moved to another folder. In the notebook's described run, a repeat's `confirmed_on` read from the draft file is kept only when every unit the repeat has now has its whole saved channel list and image type and no unit with a saved answer has left it (experiment-v1 §10: a new or removed unit or member reopens the confirmation); otherwise it is dropped and the repeat is asked again, and the card names it. The same holds for an answer given in the draft with no published file: a change of a unit's declared channel list, image type or grouping after the confirmation drops it. -- **Draft to description** (`description_from_draft(draft, answers, *, channel_counts=None)`, pure): refuses with `DraftIncomplete`, listing **every** missing confirmation once (`id_pending`, `repeat_not_confirmed`, `confirmed_on_missing`, `control_unresolved`, `legacy_folder_missing`, `unit_channels_missing` for a unit with no channel list, `unit_image_type_unanswered` for a unit whose image type is not answered `channels` or `rgb` (an old draft without one reopens with the question asked, never as `channels`), `unit_channel_untagged` for a position still `null`, `channel_undeclared` for a channel a unit names that is not declared; with `channel_counts`, the channel count of each unit's image now read through the confirmed axis roles, also `unit_channels_stale` for tags saved for another count and `unit_channel_count_unknown` for a unit without a count), otherwise returns the `Description`. A unit tagging one label at two positions is refused by `apply_answers` before anything is saved (`E_DUPLICATE_CHANNEL_IN_UNIT`). Nothing is confirmed for the user. `used_ids` is the draft's retired IDs plus every ID confirmed now. The description is then validated by the strict reader at publication; the function does not repeat the reader's rules. +- **Draft to description** (`description_from_draft(draft, answers, *, channel_counts=None)`, pure): refuses with `DraftIncomplete`, listing **every** missing confirmation once (`id_pending`, `repeat_not_confirmed`, `confirmed_on_missing`, `unit_channels_missing` for a unit with no channel list, `unit_image_type_unanswered` for a unit whose image type is not answered `channels` or `rgb` (an old draft without one reopens with the question asked, never as `channels`), `unit_channel_untagged` for a position still `null`, `channel_undeclared` for a channel a unit names that is not declared; with `channel_counts`, the channel count of each unit's image now read through the confirmed axis roles, also `unit_channels_stale` for tags saved for another count and `unit_channel_count_unknown` for a unit without a count), otherwise returns the `Description`. A unit tagging one label at two positions is refused by `apply_answers` before anything is saved (`E_DUPLICATE_CHANNEL_IN_UNIT`). Nothing is confirmed for the user. `used_ids` is the draft's retired IDs plus every ID confirmed now. The description is then validated by the strict reader at publication; the function does not repeat the reader's rules. - **Publication** (`publish_draft(..., channel_counts=...)`, the counts required): under one hold of the lock the draft is resumed, the description built with the counts, published with all the checks of `publish_description` (symbolic link, base, transition, strict reader on the exact bytes), and then the draft file is deleted. Any refusal writes nothing: the draft and the published file stay byte for byte as they were. If the draft cannot be deleted after a successful publication, the error is raised and the publication stands; the leftover draft records a base that no longer matches and is refused on resume.