diff --git a/AGENTS.md b/AGENTS.md index cfb92f47..c7cd1bc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -87,9 +87,10 @@ src/morphobricks_preprocessing/ is wrong with a malformed one (PhysicalSizes.invalid, issue #94) dataset.py dataset consistency from the folders: biological repeats per condition, balance table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2) - design.py experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with - the folders on reruns, design_issues, read/write experiment_design.csv, load_design; - read_confirmed_design/ConfirmedDesign (the strict reader other bricks use, §16) + design.py the rows of experiment_design.csv (an output only since v0.5.1): DesignTable/ + DesignRow (read-only), the run's reconcile_design and design_issues against + the folders, write_experiment_design, parse_design_bytes (the strict parser + of the draft's legacy migration); no reader for other bricks (SWEEP-PP) design_derive.py derive_design/write_derived_design: experiment_design.csv from a published description (Spec experiment-v1 11), written by design.write_experiment_design design_values.py the design file's header check (no column twice or unnamed, #83), booleans, @@ -657,15 +658,15 @@ functions the tests use. - **Confirmation** per group (`design_not_confirmed` until then); all-`single` experiments get one whole-experiment question. Controls are recorded only. - **`experiment_design.csv`** (schema 1): one row per condition x biological-repeat folder, - sorted; counts at confirmation; `confirmed_on` ISO date. Lenient reader (any-case - booleans, BOM); `DesignFormatError` otherwise, also for a header naming a column twice - or not at all (#83, both readers), a repeat group that is not a - D-e identity or a `confirmed_on` that is not `YYYY-MM-DD` (naming line and column). - Other bricks read it with the strict `read_confirmed_design` (contract §16, - question 40): nothing filled in, every group confirmed, SHA-256 of the bytes parsed; - `RunInputs.design()` uses it too (#83). -- **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that - gain a folder, drops removed folders. + sorted; counts at confirmation; `confirmed_on` ISO date. Written by the run only. No brick + reads it back (v0.5.1, SWEEP-PP; Spec ADR 0002 AR11 (3)): the lenient reader, + `load_design`, the strict `read_confirmed_design`/`ConfirmedDesign`/`DesignCheckError` and + `RunInputs.design()` are removed; bricks take `repeat_group` from the description + (`Fov.repeat_group`). `parse_design_bytes` (strict, `DesignFormatError` naming line and + column, also for a header naming a column twice or not at all, #83) remains for the draft's + experiment-v1 §7 migration of a legacy file. +- **The run's check:** `reconcile_design` keeps the derived rows of known folders, reopens only + groups that gain a folder (`design_not_confirmed`), drops removed folders. - **v0.5.1: an output only.** Every run is described (`A_DESCRIPTION_REQUIRED` without a published `morphobricks.toml`, contract §0); the run derives `experiment_design.csv` from the description and writes it with the table (`experiment_design_sha256` in the manifest). The diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d8db432..31a64af0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -264,6 +264,34 @@ already removed is not repeated here. **Breaking** where it says so. fetch, went in PP-C; `test_schema_v2_equivalence.py` was renamed `test_histogram_goldens.py` there.) +- **`experiment_design.csv`: an output only, nothing reads it back; the notebook's pairing card + rebuilt on the draft** (inventory PP-02, owner answers 3 and 4; Spec ADR 0002 AR11 (3)). + **Breaking.** Removed: `read_confirmed_design`, `ConfirmedDesign`, `DesignCheckError` (its + codes `missing_experiment_design`, `unreadable_experiment_design`, + `changed_experiment_design`), `RunInputs.design()`, the lenient `read_experiment_design`, + `load_design`, `DesignChanges`, `default_design`, the editor model `ExperimentDesign` (its + `with_group`, `with_confirmed`, `with_control`, `dated`), `design_edit.py` + (`DesignEdit`, `DesignEditError`, `apply_design_edit`, `parse_repeat_group`), and the unused + `report.design_summary`/`design_table`. The derived rows are now a read-only `DesignTable` + (`design.py`): the run's check against the folders (`reconcile_design`, which returns the + rows only; `design_issues`), `write_experiment_design`, `DESIGN_COLUMNS`, `design_derive` + and the manifest's `experiment_design_sha256` stay, and `RunInputs.open` still checks the + file's SHA-256. `parse_design_bytes` (strict) stays for the draft's experiment-v1 §7 + migration of a legacy file. A brick takes the repeat groups from the description + (`Fov.repeat_group`, `resolve_request`). *Check and save* compares the candidate's derived + rows with those of the published description, not with the file in the output folder + (`check_candidate` lost its `output` argument). **Notebook:** the pairing card shows one row + per folder of the draft's base, with the repeat it is paired with + (`PreprocessingNotebook.pairing`, replacing `.design`); a change is a draft answer at once. + Removed with the retired card parts: the hidden comparability questions + (`group_confirmations`, `groups_box`), the hidden control boxes (`controls_box`, + `control_checks`, `_sync_controls`), the *Change the experiment design* button, and the dead + `_design_pending`/`_design_items`/`_design_confirmed` gates. The shipped notebook file is + unchanged (it installs its `RELEASE_REF`). **Downstream** (pins hold until moved): + CellPatcher (`normalization.py`, `config.py`, `setting_checks.py`, `file_runs.py`, CP-1) and + MorphoFeature (`experiment_design.py`, `fits.py`, MF-4) must read `repeat_group` from the + description before their pin passes this change. + ## 0.5.0 - 2026-11-09 Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`): diff --git a/docs/conformance.md b/docs/conformance.md index 1e01ee8f..b4cfccec 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -143,8 +143,8 @@ GS1: a scenario that names fixtures is shown on those fixtures; further fixtures `v13`, `v33`, `v34`; - `test_the_derived_file_equals_the_golden_file_byte_for_byte`: `v13`, `v33`, `v34`, byte for byte; - `test_the_derived_file_equals_the_oracles_on_every_valid_fixture` and - `test_the_derived_file_is_read_back_by_the_strict_reader`: every valid fixture, against - `refval.derive_csv` and through `read_confirmed_design`; + `test_the_derived_file_is_read_back_by_the_strict_parser`: every valid fixture, against + `refval.derive_csv` and through `parse_design_bytes`; - `test_the_file_names_folders_never_ids`: `v33` carries the folder label `control`, never `ctrl`. Legacy parity with the `v0.4.0` tag (`tests/test_parity_v0_4_0.py`, 24 passed, the tag's `src` diff --git a/docs/contract.md b/docs/contract.md index 12f12e9d..86a75889 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -331,11 +331,11 @@ It gives: (v0.4.0, plan §12 question 54). A `z` or `t` outside the image (`Fov.z_count`, `Fov.t_count`) is an `IndexError` for both reads (audit I5). There is no read by `plane_id`: a row gives its `z` and `t`; -- `histograms()` (the table as read when opened) and `design()`. `design()` - reads `experiment_design.csv` with the strict `read_confirmed_design` (§16) and the - manifest's `experiment_design_sha256` (issue #83, plan §12 question 62; it used the lenient - reader before): a design that reader refuses raises `RunInputsError`, with the refusal - as its `__cause__`; +- `histograms()` (the table as read when opened). There is no `design()` (removed in v0.5.1, + SWEEP-PP; Spec ADR 0002 AR11 (3)): `experiment_design.csv` is an output only, its SHA-256 + checked at `open` against the manifest's `experiment_design_sha256` but never read back; a + brick takes the repeat groups from the description (`Fov.repeat_group`, + `resolve_request`); - **semantics of a published description** (Stage 1 PR 12; every run has one since v0.5.1, §0: an output folder without a published `morphobricks.toml` is refused at `open` with `DescriptionRequiredError`, `A_DESCRIPTION_REQUIRED`, and an older manifest with @@ -518,11 +518,15 @@ biological repeats, confirms that each repeat group is comparable (N35), and may control conditions (N36). Since v0.5.1 (§0, ADR 0002 AR10/AR11) these answers are part of the published description, and `experiment_design.csv` is **derived** from it (experiment-v1 §11, `design_derive.derive_design`): written by the publication and by every run, never read back -by preprocessing as an input. The rows marked *(removed in v0.5.1)* below describe what is gone; -the lenient reader `read_experiment_design` stays only for the one-way draft migration of a -legacy file (`draft.migrate_legacy_design`) until SWEEP-PP, and the strict -`read_confirmed_design` (`RunInputs.design`) only until CellPatcher and MorphoFeature read -`repeat_group` from the description (owner answer 3). +by preprocessing or any other brick (owner answer 3, Spec ADR 0002 AR11 (3)). The rows marked +*(removed in v0.5.1)* below describe what is gone: the lenient reader `read_experiment_design`, +`load_design`, the strict `read_confirmed_design` with `ConfirmedDesign`/`DesignCheckError` and +`RunInputs.design` (SWEEP-PP), and the notebook's editor model (`ExperimentDesign`, +`default_design`, `apply_design_edit`/`DesignEdit`, owner answer 4). What stays: +`write_experiment_design`, `DESIGN_COLUMNS`, `design_derive`, the manifest's +`experiment_design_sha256`, the rows as `DesignTable` (read-only) with the run's check +`reconcile_design`/`design_issues`, and the strict `parse_design_bytes` of the draft's +experiment-v1 §7 migration (`draft.migrate_legacy_design`). | Column | Meaning | |---|---| @@ -540,12 +544,12 @@ legacy file (`draft.migrate_legacy_design`) until SWEEP-PP, and the strict | Pairing | By default repeats with the same folder name in different conditions are one group, named after the folder. The user may move a folder into another group. A group holds at most one folder per condition (`repeat_group_conflict`, an error). A group not found in every condition is used as it is, with the warning `repeat_missing_conditions` (§11). | | Confirmation | Per repeat group: "Were these images prepared and acquired under the same circumstances, so they are comparable with each other?". A group is confirmed when all its rows are. Without biological-repeat folders (every condition `single`), the whole experiment is one group and the question is asked once (N31). Missing technical-repeat folders need no question. | | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | -| Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. | -| Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A header that names a column twice (compared ignoring case, outer whitespace and invisible format characters such as a byte-order mark, so ` Repeat_Group` repeats `repeat_group`) or has a column without a name (issue #83, plan §12 question 62: no reader chooses between two values; both readers refuse it with the column and its positions named, and nothing is rewritten), a missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). `load_design(output_folder, repeats)` reads the file with it and brings it up to date with the folders (Reruns), returning `(design, changes)`, the default design when there is no file; a file that cannot be read raises `DesignCheckError` (`unreadable_experiment_design`, §16) and is left as it is (owner decision of 31 October 2026, plan §12 question 43). | -| Notebook | The pairing is answered on the design card and kept in the draft (`morphobricks.draft.json`), confirmed with the rest on *Check and save*; the notebook never writes or reads `experiment_design.csv` (v0.5.1). The card's model is rebuilt on the draft in SWEEP-PP (owner answer 4). | +| Run's check | The run brings the derived rows up to date with the folders found (`reconcile_design`): folders in them keep their group, control mark and confirmation (added or removed images within a folder keep it too); a folder found but not described joins the group named after it, inherits its condition's control mark, and that group is refused as not confirmed (`design_not_confirmed`); folders no longer found are dropped. Nothing is read from the output folder. | +| Reading | *(removed in v0.5.1, SWEEP-PP)* No reader of the file for its answers remains: the lenient `read_experiment_design`, `load_design` and the strict `read_confirmed_design` are gone. `parse_design_bytes` parses the file exactly as `write_experiment_design` writes it (UTF-8 without a byte-order mark, the columns in order, `true`/`false`, whole counts, `YYYY-MM-DD`, identities as §16, a header naming a column twice or not at all refused naming it, issue #83), only for the draft's experiment-v1 §7 migration of a legacy file; it raises `DesignFormatError` naming the line and column. `design_issues` gives `invalid_experiment_design` for an unusable repeat-group name or date in rows built in code, and `write_experiment_design` refuses to write them. `DesignTable.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping. | +| Notebook | The pairing is answered on the design card and kept in the draft (`morphobricks.draft.json`), confirmed with the rest on *Check and save*; the notebook never writes or reads `experiment_design.csv` (v0.5.1). The card is built on the draft (SWEEP-PP, owner answer 4): one row per folder of the draft's base with the repeat it is paired with (`PreprocessingNotebook.pairing`); a change is a draft answer at once, and a replicate whose folders it changes is asked "same way?" again. There are no comparability questions or control boxes on the card (they are on *Check and save* and the references). | | Command line | *(removed in v0.5.1)* `run --design FILE` and the template `run` wrote: `run` refuses an output folder without a published description (§0). | | Template from `check` | *(removed in v0.5.1)* `check` writes only its report. | -| `design` command | *(removed in v0.5.1)*: the pairing, confirmations and references are answered in the notebook's describe steps. `apply_design_edit` and `DesignEdit` stay for the notebook's pairing card until SWEEP-PP. | +| `design` command | *(removed in v0.5.1)*: the pairing, confirmations and references are answered in the notebook's describe steps. `apply_design_edit` and `DesignEdit` were removed in SWEEP-PP. | | `describe` command (Stage 1, PR 17) | `describe EXPERIMENT [--output FOLDER] [--exclude PATH] [--publish]` prints the draft experiment description (experiment-v1 §12) in plain language: the proposed ID of each condition, technical-repeat folder and biological repeat (no proposal, with the rule's code, for a folder that cannot be an ID: never repaired), each repeat with its folders and image counts and whether it is confirmed (and the date), the references, the notes of a migration with several dates, and every open question with what the user has to do. The draft is the legacy `experiment_design.csv` of the output folder migrated (`draft.migrate_legacy_design`; one that cannot be read or breaks a rule is exit status 1 and left as it is) when there is one and no published `morphobricks.toml`, else the draft of the folders (`draft.draft_from_discovery`); when `morphobricks.toml` is published (`run.description_run.load_description`), the folders are compared with it (`description_reconcile.reconcile`: unchanged, new, gone, image counts, repeats to confirm again), reading the folder structure only, so the channels are not compared. It reads no image and writes nothing: no output folder, draft, description, design or records file. **It confirms nothing**: an ID, the comparability of a repeat (N35), a reference, a removal and every date are the user's answers (notebook), never given by the command or read from the clock. `--publish` publishes a draft only when no question is open and every decision is confirmed; otherwise it prints `Not published. What is missing:` with every item and exits with status 1. This version holds no answers outside the notebook (the draft file is not in the package), so `--publish` refuses every draft built from folders and publishes nothing; for a published description that matches the folders it says `Nothing to publish` (exit 0); a new, gone or changed folder is listed and the file is left as it is. An error (experiment folder not found, discovery errors, unreadable description or records) is exit status 1; the command takes no lock. | | Run | The run derives the design from the published description and writes it with the table, published before the manifest, which records its SHA-256 (§9). `run_experiment` takes no design. `plane_histograms.csv` does not depend on the design. | @@ -735,7 +739,7 @@ instead of keeping copies. | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_signature` (`inventory`), `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`); from v0.4.0 file runs read through `RunInputs` (§9.1), which also checks masks. Only a current manifest is read at all (§9, version 4). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_signature`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`, also for an entry without a `position_id`). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`; an entry without roles fits no axes, nothing is resolved from the header in their place); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. A size that cannot be converted to a positive finite length in micrometres (`1e308 m`) is unknown, never infinite, and an infinite or NaN value is never the same as any other (issue #93; before, `inf` matched every size). The folder check's `different_pixel_sizes` uses both (§11). | -| Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order (a column named twice or without a name is refused naming it, as by the lenient reader, §12); one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking in v0.4.0:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | +| Experiment design | *(removed in v0.5.1, SWEEP-PP)* `read_confirmed_design`, `ConfirmedDesign`, `DesignCheckError`, `load_design` | `experiment_design.csv` is an output only (Spec ADR 0002 AR11 (3)); no brick reads it for its repeat groups. A brick takes `repeat_group` from the published description through `RunInputs` (`Fov.repeat_group`, `resolve_request`). The codes `missing_experiment_design`, `unreadable_experiment_design` and `changed_experiment_design` are no longer raised by this package (`RunInputs.open` still refuses a design file whose SHA-256 is not the manifest's). | | Analysis refusals (Stage 2) | `ChannelNotFoundError` (`A_SCOPE_INCOMPLETE`), `MaskNotInstancesError` (`A_MASK_NOT_INSTANCES`), `RgbNotSupportedError` (`A_RGB_NOT_SUPPORTED`), `DraftRefusedError` (`A_DRAFT_REFUSED`), `MaskUncheckedError` (`A_MASK_UNCHECKED`), `RunInputsError.code` (`run_inputs_error`, exported at the package top level) | The Spec's analysis-layer codes (experiment-v1 §8) as one class each, all `RunInputsError` subclasses (a `ValueError`) with the code as the class attribute `code` (`None` on `RunInputsError` itself), so a brick matches the class or the code and keeps no copy. Preprocessing raises `A_DRAFT_REFUSED`, `A_MASK_UNCHECKED` and `A_SCOPE_INCOMPLETE` itself (§9.1); `A_MASK_NOT_INSTANCES` (a mask that is not `instances` used as an object set) and `A_RGB_NOT_SUPPORTED` (`rgb` units given to a brick that does not declare RGB support) are found by `resolve_request` (§9.1) against the consumer's declared capability, and a brick raises them through `ResolvedRequest.require()` (or reports its `refusals`). A brick writes the code in its message text, as preprocessing does. The classes pickle, so a worker process can raise them. `RunInputs.section_digests()` (§9.1) gives the description's section digests for cache keys. `Fov.image_type` (§9.1) is each position's declared `rgb` or `channels`; there is no run-wide image type. `RunInputs.resolve_request()` (§9.1) is the one interpreter of a described run for every route: a brick builds a `Request` and reads through the `FovSelection`s it returns, so channel-by-ID, scope, mask-representation and RGB rules have one implementation; `VolumeNotSupportedError` (`A_VOLUME_NOT_SUPPORTED`) and `DescriptionRequiredError` (`A_DESCRIPTION_REQUIRED`, raised by `RunInputs.open`) are the resolver contract's own codes (MorphoBricks-Spec ADR 0002), not experiment-v1 §8's. | | Experiment description (Stage 1) | `read_description(path)`, `description_consume.read_published_description(path)` (the published `morphobricks.toml` only), `run.description_file.publish_description` (the only writer), `draft.draft_from_discovery`, `draft.migrate_legacy_design`, `description_reconcile.reconcile` | A brick reads the **published** `morphobricks.toml` through `RunInputs` (§9.1) and never a draft: a draft is internal to preprocessing, is never named `morphobricks.toml` and is refused with `A_DRAFT_REFUSED` (experiment-v1 §12). The command line shows the draft (`describe`, §12) and confirms nothing: the user's answers are the only way an ID, a repeat or a removal becomes confirmed. The owner's answers Q1 to Q8 of the Stage 1 design are in plan §12, question 78. | diff --git a/docs/user_guide.md b/docs/user_guide.md index f8ecacc8..f16e0a15 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -985,9 +985,9 @@ The default output folder is next to the experiment folder: run; later bricks ask you to run this brick again on it. - **`experiment_design.csv`:** one row per biological-repeat folder of each condition: its repeat group, whether its condition is a control, its image and technical-repeat counts, - and whether and when the repeat group was confirmed. Written in step 1 and by every run. - Later bricks read the pairing from it (`read_experiment_design`). Format in - [contract.md](contract.md) §12. + and whether and when the repeat group was confirmed. Derived from the saved description and + written by every run (and by *Check and save*); a record for people, never read back: later + bricks take the pairing from the description. Format in [contract.md](contract.md) §12. - **`run_manifest.json`:** what the run read and with which settings: labels, axis roles, the image type (and whether the channels are RGB colour samples), bin count, histogram range, memory limit, excluded images, the limits of the intensity checks, the planes with a @@ -1202,10 +1202,7 @@ continues. | `lab_links_repeat_missing` | Error | The lab-tool repeat codes leave out a biological repeat of the description; a mapping names every repeat. Add its code, or leave `repeat_codes` out. | | `rgb_not_possible` | Error | The image type RGB was chosen, but the images are not three channels of a whole-number pixel type. Choose fluorescence, or check the axis roles: the RGB samples must have the role C. | | `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another replicate on the *Replicates* page and save the description again (*Check and save*). | -| `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, ``, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. Correct `experiment_design.csv` (the message names the line and column). | -| `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a column named twice or without a name, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. | -| `unreadable_experiment_design` | Error (`load_design`, notebook step 1, `check`) | The saved `experiment_design.csv` in the output folder cannot be read even leniently (for example a date a spreadsheet program rewrote as 26/10/2026, a repeat group `NA`, or a header that names a column twice, such as two `repeat_group` columns, or has a column without a name); the message names the line and column. No reader chooses between two columns of one name: delete one. The file is left as it is. Correct it, or answer the questions again in step 1, which replaces it. | -| `changed_experiment_design` | Error (other bricks) | `experiment_design.csv` is not the file the preprocessing run that wrote the statistics used (its SHA-256 differs from the one in the run manifest). Give that run's design, or run the preprocessing brick again. | +| `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, ``, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. It comes from a description edited by hand or rows built in code; correct the description (the message names the folder). | | `image_intensity_outlier` | Warning (intensity check) | An image's median or high percentile in a channel is far from the other images of its condition and biological repeat. Look at it: it may be out of focus, contain debris or bubbles, or have been acquired differently. Exclude it if it is faulty. | | `repeat_intensity_shift` | Warning (intensity check) | A biological repeat is brighter or darker than the other repeats of its condition. Check whether it was prepared or acquired differently; normalising per biological repeat in later bricks removes such differences. | | `saturated_image` | Warning (intensity check) | More than the limit (default 1 %) of an image's pixels in a channel are at the largest value of its pixel type, so their true intensity is lost. Check the exposure, gain or laser power. | diff --git a/pyproject.toml b/pyproject.toml index d2b9e18e..81b56876 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -257,7 +257,6 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.records_file" = "pure" "morphobricks_preprocessing.description_rules" = "pure" "morphobricks_preprocessing.description_structure" = "pure" -"morphobricks_preprocessing.design_edit" = "pure" "morphobricks_preprocessing.discovery" = "io" "morphobricks_preprocessing.links" = "io" "morphobricks_preprocessing.mounts" = "io" diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 52305528..eae330a2 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -65,21 +65,14 @@ DESIGN_COLUMNS, DESIGN_FILENAME, DESIGN_SCHEMA_VERSION, - ConfirmedDesign, - DesignChanges, - DesignCheckError, DesignFormatError, DesignRow, - ExperimentDesign, - default_design, + DesignTable, design_issues, - load_design, - read_confirmed_design, - read_experiment_design, + parse_design_bytes, reconcile_design, write_experiment_design, ) -from .design_edit import DesignEdit, DesignEditError, apply_design_edit from .discovery import ( FOV_KEY, MISSING_TEXT, @@ -289,20 +282,15 @@ "CalibrationEntry", "CalibrationFormatError", "CalibrationReview", - "ConfirmedDesign", "ConversionRecord", "ConversionRecordError", "ConversionPreview", "ConversionResult", "DatasetReport", - "DesignChanges", - "DesignCheckError", - "DesignEdit", - "DesignEditError", "DesignFormatError", "DesignRow", + "DesignTable", "DiscoveryResult", - "ExperimentDesign", "FileEntry", "GroupSummary", "IdentityError", @@ -366,7 +354,6 @@ "ValidationIssue", "XycPlane", "ZarrUnavailableError", - "apply_design_edit", "axis_disagreements", "axis_length_ranges", "balance_table", @@ -388,10 +375,10 @@ "convert_experiment", "preview_conversion", "declared_mapping", - "default_design", "default_output_folder", "default_repeat_groups", "design_issues", + "parse_design_bytes", "discover_experiment", "file_mapping_for", "file_sha256", @@ -411,7 +398,6 @@ "iter_planes", "iter_xyc_planes", "likely_axis_cause", - "load_design", "images_named_as_masks", "mask_issues", "mask_name_issues", @@ -432,8 +418,6 @@ "preflight", "read_calibration", "read_conversion_record", - "read_confirmed_design", - "read_experiment_design", "read_lab_links", "read_manifest", "read_mask", diff --git a/src/morphobricks_preprocessing/description_candidate.py b/src/morphobricks_preprocessing/description_candidate.py index 551d5d98..8983de6b 100644 --- a/src/morphobricks_preprocessing/description_candidate.py +++ b/src/morphobricks_preprocessing/description_candidate.py @@ -8,25 +8,25 @@ candidate is also compared with the published file by the same transition checks publication runs (:func:`.run.description_file.transition_refusals`: experiment-v1 sections 4.4a, 4.6, 4.9 and 10), so Save is never offered for a candidate that publication would refuse. The derived -``experiment_design.csv`` (experiment-v1 section 11) is compared with the file in the output -folder now. Nothing is written: publication repeats the reader on the same bytes under the lock. +``experiment_design.csv`` (experiment-v1 section 11) is compared with the one the published file +derives (the file in the output folder is an output only and is not read back, Spec ADR 0002 +AR11 (3)). Nothing is written: publication repeats the reader on the same bytes under the lock. """ from __future__ import annotations from collections.abc import Mapping from dataclasses import dataclass -from pathlib import Path from .description import Description, DescriptionError, description_from_bytes -from .description_file import read_file_bytes, sha256_hex +from .description_file import sha256_hex from .description_report import DescriptionIssue from .description_review import design_changes from .description_rules import Refusal from .description_run_text import reader_issue from .description_transition import Reconfirmation from .description_write import canonical_bytes -from .design import DESIGN_FILENAME, DesignFormatError, ExperimentDesign, parse_design_bytes +from .design import DesignFormatError from .design_derive import derive_design from .physical_check import HETEROGENEOUS_LAYOUT from .run.description_file import transition_refusals @@ -75,17 +75,6 @@ def can_run(self) -> bool: return self.can_save and self.physical_run and not self.run_refusals -def _current_design(output: Path | None) -> tuple[ExperimentDesign | None, bool]: - """Return the ``experiment_design.csv`` in the output folder now, and whether it was read.""" - if output is None: - return None, True - try: - data = read_file_bytes(output / DESIGN_FILENAME) - return (None if data is None else parse_design_bytes(data)), True - except (DescriptionError, DesignFormatError, OSError): - return None, False - - def transition_issue(issue: DescriptionIssue) -> ValidationIssue: """Return one publication refusal of the candidate as an error naming its code and path.""" code, path = issue.code.value, issue.path @@ -103,7 +92,6 @@ def check_candidate( candidate: Description, pre: Preflight, roles: Mapping | None, - output: Path | None, published: Description | None = None, confirmations: tuple[Reconfirmation, ...] = (), ) -> CandidateCheck: @@ -118,11 +106,10 @@ def check_candidate( roles The confirmed roles by relative path (``AxisReview.file_mappings``), or None when the axis roles are not confirmed (the physical checks then wait, never pass). - output - The output folder, whose ``experiment_design.csv`` is compared; None when unknown. published The published description the candidate replaces (None when there is none): its - transition refusals are refusals here, exactly as publication raises them. + transition refusals are refusals here, exactly as publication raises them, and its + derived ``experiment_design.csv`` is the one the candidate's is compared with. confirmations The user's recorded re-confirmations that publication will be given. """ @@ -136,13 +123,11 @@ def check_candidate( ) found = RunDescription(parsed, sha256_hex(data), ()) checked, problems = check_description(found, pre, roles) - before, readable = _current_design(output) try: + before = None if published is None else derive_design(published) changes = design_changes(before, derive_design(parsed)) except DesignFormatError as exc: changes = [f"cannot be derived: {exc}"] - if not readable: - changes = ["the file in the output folder cannot be read; it is replaced", *changes] physical = checked.physical is not None return CandidateCheck(parsed, refused, tuple(problems), physical, tuple(changes)) diff --git a/src/morphobricks_preprocessing/description_review.py b/src/morphobricks_preprocessing/description_review.py index 7b095df1..ffadd88c 100644 --- a/src/morphobricks_preprocessing/description_review.py +++ b/src/morphobricks_preprocessing/description_review.py @@ -14,7 +14,7 @@ from collections.abc import Mapping from dataclasses import dataclass, fields -from .design import DesignRow, ExperimentDesign +from .design import DesignRow, DesignTable from .draft import Draft from .draft_answers import ConditionAnswer, DraftAnswers, RepeatAnswer, UnitKey from .notebook_values import natural_key @@ -164,11 +164,11 @@ def _differences(old: DesignRow, new: DesignRow) -> list[str]: ] -def design_changes(before: ExperimentDesign | None, after: ExperimentDesign) -> list[str]: +def design_changes(before: DesignTable | None, after: DesignTable) -> list[str]: """Return how ``experiment_design.csv`` changes, one line per row added, removed or changed. - ``before`` is the file as it is now (None when there is none). An empty list means the file - is written with the same rows. + ``before`` is the design the published description derives (None when there is none). An + empty list means the file is written with the same rows. """ new: Mapping[tuple[str, str], DesignRow] = {r.key: r for r in after.rows} if before is None: diff --git a/src/morphobricks_preprocessing/design.py b/src/morphobricks_preprocessing/design.py index 7dd98f41..dc50bed4 100644 --- a/src/morphobricks_preprocessing/design.py +++ b/src/morphobricks_preprocessing/design.py @@ -1,31 +1,29 @@ -"""The experiment design: repeat groups, their confirmation, and control conditions. - -A repeat group is one biological repeat across all conditions (normalization -plan N30). By default, biological-repeat folders with the same name are one -group; the user may move a folder into another group when a repeat was named -differently in some condition. Before processing, the user confirms for each -group that its images were prepared and acquired under the same circumstances -(N35); without biological-repeat folders the whole experiment is one group -(N31). Conditions may be marked as controls, for the record only (N36). - -The answers are kept in ``experiment_design.csv`` in the output folder -(``docs/contract.md`` §12), so a rerun with the same folders does not ask -again. Later bricks read the pairing with :func:`read_confirmed_design`, the one -strict reader every brick uses (contract §16); :func:`read_experiment_design` is -this brick's lenient reader of a file the user may still be editing. Nothing here -reads images. +"""The rows of ``experiment_design.csv``: repeat groups, their confirmation, and controls. + +A repeat group is one biological repeat across all conditions (normalization plan N30); each +row is one biological-repeat folder of a condition, with its group, whether the condition is a +control (recorded only, N36), its counts and the confirmation that the group's images are +comparable (N35). The file is an **output only** (Spec ADR 0002 AR10 and AR11 (3)): the run +derives it from the published description (:mod:`.design_derive`) and writes it with +:func:`write_experiment_design`; no brick reads it back as an input to grouping, and the +notebook's pairing is an answer of the draft (:mod:`.notebook_design`). :class:`DesignTable` +holds rows and answers questions about them; it has no editing methods. + +:func:`reconcile_design` and :func:`design_issues` are the run's check of the derived rows +against the folders found; :func:`parse_design_bytes` is the strict parser the draft's +experiment-v1 section 7 migration of a legacy file uses (:func:`.draft.migrate_legacy_design`). +The lenient reader, the default pairing, the editor model and the strict reader for other +bricks (``read_confirmed_design``) were removed in v0.5.1 (SWEEP-PP). Nothing here reads images. """ from __future__ import annotations import csv -import hashlib import io import os from collections.abc import Iterable, Mapping from dataclasses import dataclass, replace from pathlib import Path -from types import MappingProxyType from morphobricks_normalization.records import SINGLE @@ -57,26 +55,6 @@ ) -class DesignCheckError(ValueError): - """Raised by :func:`read_confirmed_design` when a design cannot be used. - - Also raised by :func:`load_design`, with the one error - ``unreadable_experiment_design``, when a saved file cannot be read. - - Attributes - ---------- - issues - The errors, each with its code (``missing_experiment_design``, - ``unreadable_experiment_design``, ``repeat_group_conflict``, - ``design_not_confirmed`` or ``changed_experiment_design``), message, path and - suggested action. The exception's text joins their messages. - """ - - def __init__(self, issues: Iterable[ValidationIssue]) -> None: - self.issues = tuple(issues) - super().__init__(" ".join(f"{issue.code}: {issue.message}" for issue in self.issues)) - - @dataclass(frozen=True) class DesignRow: """One biological-repeat folder of a condition, and the answers about it. @@ -114,35 +92,10 @@ def key(self) -> tuple[str, str]: @dataclass(frozen=True) -class DesignChanges: - """How the folders differ from an earlier experiment design. - - Attributes - ---------- - added - Biological-repeat folders not in the earlier design. - removed - Folders of the earlier design no longer found. - reopened - Repeat groups that were confirmed but are asked again, because a folder - was added to them. - """ - - added: tuple[tuple[str, str], ...] = () - removed: tuple[tuple[str, str], ...] = () - reopened: tuple[str, ...] = () - - @property - def any(self) -> bool: - """True if the folders changed since the earlier design.""" - return bool(self.added or self.removed) +class DesignTable: + """The rows of an ``experiment_design.csv``, sorted, and what they say. - -@dataclass(frozen=True) -class ExperimentDesign: - """The pairing of biological repeats, their confirmation and the controls. - - Every method returns a new design; a design is never changed in place. + Read-only: there is no editing method (the notebook's pairing is a draft answer). Attributes ---------- @@ -230,181 +183,29 @@ def repeat_groups(self) -> dict[tuple[str, str], str]: """ return {row.key: row.repeat_group for row in self.rows} - def with_group(self, key: tuple[str, str], group: str) -> ExperimentDesign: - """Move a folder into another repeat group. - - Both groups changed, so neither stays confirmed. - """ - old = self.row(key) - if old is None: - raise KeyError(f"{key!r} is not a folder of this design.") - if old.repeat_group == group: - return self - touched = {old.repeat_group, group} - rows = [] - for row in self.rows: - if row.key == key: - row = replace(row, repeat_group=group) - if row.repeat_group in touched or row.key == key: - row = replace(row, confirmed=False, confirmed_on="") - rows.append(row) - return ExperimentDesign(tuple(rows)) - - def with_confirmed( - self, - group: str, - confirmed: bool, - on: str = "", - repeats: Iterable[BiologicalRepeat] = (), - ) -> ExperimentDesign: - """Confirm a repeat group, or take the confirmation back. - - Parameters - ---------- - group - The repeat group. - confirmed - The answer. - on - Date of the confirmation (ISO 8601). - repeats - The folders as found now; confirming records their counts. - """ - counts = {r.key: (r.image_count, r.technical_repeat_count) for r in repeats} - rows = [] - for row in self.rows: - if row.repeat_group == group: - images, technical = counts.get( - row.key, (row.image_count, row.technical_repeat_count) - ) - row = replace( - row, - confirmed=confirmed, - confirmed_on=on if confirmed else "", - image_count=images, - technical_repeat_count=technical, - ) - rows.append(row) - return ExperimentDesign(tuple(rows)) - - def with_control(self, condition: str, control: bool) -> ExperimentDesign: - """Mark a condition as a control, or not.""" - return ExperimentDesign( - tuple( - replace(row, control=control) if row.condition == condition else row - for row in self.rows - ) - ) - - def dated(self, on: str) -> ExperimentDesign: - """Fill the date of confirmed rows that have none (e.g. confirmed by hand in the file).""" - return ExperimentDesign( - tuple( - replace(row, confirmed_on=on) if row.confirmed and not row.confirmed_on else row - for row in self.rows - ) - ) +def reconcile_design(design: DesignTable, repeats: Iterable[BiologicalRepeat]) -> DesignTable: + """Bring derived rows up to date with the folders found now (the run's check). -@dataclass(frozen=True) -class ConfirmedDesign: - """An ``experiment_design.csv`` checked for use by another brick. - - Build it with :func:`read_confirmed_design`. Every repeat group is confirmed and holds - at most one folder per condition. - - Attributes - ---------- - path - The file that was read. - sha256 - SHA-256 of exactly the bytes that were parsed (the manifest's - ``experiment_design_sha256``). - design - The design itself. - repeat_groups - Read-only ``{(condition, biological_repeat): repeat_group}``, the pairing the - normalization brick's ``biological_repeat`` scope takes. - """ - - path: Path - sha256: str - design: ExperimentDesign - repeat_groups: Mapping[tuple[str, str], str] - - @property - def conditions(self) -> tuple[str, ...]: - """Condition labels, sorted.""" - return self.design.conditions - - @property - def controls(self) -> tuple[str, ...]: - """Conditions marked as controls, sorted (recorded only, contract §12).""" - return self.design.controls - - @property - def groups(self) -> tuple[str, ...]: - """Repeat-group labels, sorted.""" - return self.design.groups - - def group_members(self) -> Mapping[str, Mapping[str, str]]: - """Return ``{repeat_group: {condition: biological_repeat}}``, read-only. - - One biological-repeat folder per condition and group, which the design's checks - guarantee. - """ - members: dict[str, dict[str, str]] = {} - for (condition, repeat), group in self.repeat_groups.items(): - members.setdefault(group, {})[condition] = repeat - return MappingProxyType( - { - group: MappingProxyType(dict(sorted(m.items()))) - for group, m in sorted(members.items()) - } - ) - - -def default_design(repeats: Iterable[BiologicalRepeat]) -> ExperimentDesign: - """Build the design before any answer: repeats with the same folder name are one group. - - Nothing is confirmed and no condition is a control. - """ - return ExperimentDesign(tuple(_new_row(repeat, repeat.biological_repeat) for repeat in repeats)) - - -def reconcile_design( - previous: ExperimentDesign | None, repeats: Iterable[BiologicalRepeat] -) -> tuple[ExperimentDesign, DesignChanges]: - """Bring an earlier design up to date with the folders found now. - - Folders of the earlier design keep their repeat group, control mark and - confirmation; added or removed images within a folder keep the - confirmation. A new folder joins the group named after it (and the - control mark of its condition), and that group is asked again. Folders no - longer found are dropped. + Folders of ``design`` keep their repeat group, control mark and confirmation; added or + removed images within a folder keep the confirmation. A folder found now but not in + ``design`` joins the group named after it (and the control mark of its condition) + unconfirmed, so that group is not confirmed (:func:`design_issues`). Folders no longer + found are dropped. Parameters ---------- - previous - The earlier design, or None. + design + The rows derived from the published description. repeats - The biological-repeat folders found now - (:func:`.dataset.biological_repeats`). - - Returns - ------- - tuple of ExperimentDesign and DesignChanges + The biological-repeat folders found now (:func:`.dataset.biological_repeats`). """ - repeats = list(repeats) - if previous is None: - return default_design(repeats), DesignChanges() - controls = set(previous.controls) - current = {repeat.key for repeat in repeats} - rows, added = [], [] + controls = set(design.controls) + rows, added = [], set() for repeat in repeats: - row = previous.row(repeat.key) + row = design.row(repeat.key) if row is None: - added.append(repeat.key) + added.add(repeat.key) row = replace( _new_row(repeat, repeat.biological_repeat), control=repeat.condition in controls ) @@ -416,18 +217,16 @@ def reconcile_design( technical_repeat_count=repeat.technical_repeat_count, ) rows.append(row) - reopened = sorted({row.repeat_group for row in rows if row.key in added}) - was_confirmed = [g for g in reopened if previous.group_confirmed(g)] - rows = [ - replace(row, confirmed=False, confirmed_on="") if row.repeat_group in reopened else row - for row in rows - ] - removed = sorted(row.key for row in previous.rows if row.key not in current) - changes = DesignChanges(tuple(sorted(added)), tuple(removed), tuple(was_confirmed)) - return ExperimentDesign(tuple(rows)), changes - - -def design_issues(design: ExperimentDesign) -> list[ValidationIssue]: + reopened = {row.repeat_group for row in rows if row.key in added} + return DesignTable( + tuple( + replace(row, confirmed=False, confirmed_on="") if row.repeat_group in reopened else row + for row in rows + ) + ) + + +def design_issues(design: DesignTable) -> list[ValidationIssue]: """Check that a design can be used: every repeat group confirmed, one folder per condition. Returns @@ -491,153 +290,7 @@ def design_issues(design: ExperimentDesign) -> list[ValidationIssue]: return issues -def read_experiment_design(path: str | os.PathLike) -> ExperimentDesign: - """Read ``experiment_design.csv``. - - Labels are read as text. ``control`` and ``confirmed`` accept ``true`` or - ``false`` in any case (also ``yes``/``no``, ``1``/``0``; empty is false), so - a file saved by a spreadsheet program still reads. The user's own edits are - checked (owner decision, 31 October 2026): ``repeat_group`` follows the identity - rules of decision D-e (not empty, ``NA``, ``nan``, ``none``, ``null``, ``n/a`` or - ```` in any case, no leading or trailing whitespace), and ``confirmed_on`` is - empty or a date in :data:`DATE_FORMAT`; the error names the line and column. - - Raises - ------ - DesignFormatError - If the file cannot be read, the header names a column twice (ignoring case, - outer whitespace and invisible characters) or has a column without a name, a - column is missing, a value is not valid - (including a repeat-group name or confirmation date as above), a folder is - listed twice, the rows of a condition disagree about ``control``, or the - schema version is not 1. - """ - path = Path(path) - try: - text = path.read_text(encoding="utf-8-sig") - except (OSError, UnicodeDecodeError) as exc: - raise DesignFormatError(f"{path} cannot be read: {exc}") from exc - reader = csv.DictReader(io.StringIO(text)) - check_header(reader.fieldnames or (), path.name) - missing = [c for c in DESIGN_COLUMNS if c not in (reader.fieldnames or ())] - if missing: - raise DesignFormatError( - f"{path.name} is not an experiment design: the column(s) {', '.join(missing)} " - "are missing." - ) - rows: list[DesignRow] = [] - numbers: list[int] = [] - for number, record in enumerate(reader, start=2): - if not any((value or "").strip() for value in record.values() if isinstance(value, str)): - continue - rows.append(_parse_row(record, f"{path.name}, line {number}")) - numbers.append(number) - _check_group_case(zip(numbers, rows, strict=True), path.name) - _check_rows(rows, path.name) - return ExperimentDesign(tuple(rows)) - - -def read_confirmed_design( - path: str | os.PathLike, *, expected_sha256: str | None = None -) -> ConfirmedDesign: - """Read ``experiment_design.csv`` for use by another brick, refusing anything doubtful. - - The one reader of the experiment design for every MorphoBricks brick (contract §16), - replacing CellPatcher's ``load_experiment_design`` and MorphoFeature's - ``ExperimentDesign.read``. It takes the file as this brick writes it - (:func:`write_experiment_design`) and fills nothing in, where - :func:`read_experiment_design`, for the user's own edits in step 1, is lenient about - the file's form: - - - the file is UTF-8 without a byte-order mark, its header is exactly - :data:`DESIGN_COLUMNS` in that order, and every row has one value per column - (an unclosed quote, a missing or extra value, or a blank line is refused); - - no value has leading or trailing whitespace; ``control`` and ``confirmed`` are - ``true`` or ``false``; ``image_count`` and ``technical_repeat_count`` are whole - numbers of at least 0 written as such; ``confirmed_on`` is empty or a date in - :data:`DATE_FORMAT` (as :func:`read_experiment_design` requires too); - ``schema_version`` is ``1``; - - ``condition``, ``biological_repeat`` and ``repeat_group`` pass - :func:`.discovery.check_identity_text` (the repeat group as - :func:`read_experiment_design` requires too), and labels of one level differing - only in case are refused (:func:`.discovery.check_identity_case`); - - a folder listed twice, rows of a condition disagreeing on ``control``, or a file - without rows are refused; - - every repeat group is confirmed and holds one folder per condition - (:func:`design_issues`); - - with ``expected_sha256`` (a run manifest's ``experiment_design_sha256``), the file - must be that one. - - The SHA-256 is of the bytes that were parsed, so the file cannot change between the - two. - - Parameters - ---------- - path - The ``experiment_design.csv`` file. - expected_sha256 - The SHA-256 the preprocessing run that wrote the statistics recorded, if any. - - Returns - ------- - ConfirmedDesign - - Raises - ------ - DesignCheckError - With the errors ``missing_experiment_design`` (no such file), - ``unreadable_experiment_design`` (not a design this version reads strictly), - ``repeat_group_conflict`` and ``design_not_confirmed`` (:func:`design_issues`), or - ``changed_experiment_design`` (not the file the run recorded). - """ - path = Path(path) - rerun = ( - "Check this experiment in the preprocessing notebook (it pairs and confirms the " - "biological repeats), then run the preprocessing brick again." - ) - if not path.is_file(): - raise DesignCheckError( - [ - error( - "missing_experiment_design", - f"The experiment design {path} does not exist.", - [str(path)], - rerun, - ) - ] - ) - try: - data = path.read_bytes() - design = parse_design_bytes(data, path.name) - except (OSError, DesignFormatError) as exc: - raise _unreadable(path, exc, rerun) from exc - issues = design_issues(design) - if issues: - raise DesignCheckError(issues) - sha256 = hashlib.sha256(data).hexdigest() - if expected_sha256 is not None and sha256 != expected_sha256: - raise DesignCheckError( - [ - error( - "changed_experiment_design", - f"{path.name} is not the experiment design the preprocessing run used: its " - "SHA-256 differs from the one in the run manifest, so the repeat groups may " - "not be the ones the statistics were checked with.", - [str(path)], - "Give the experiment design of that run, or run the preprocessing brick " - "again on this experiment.", - ) - ] - ) - return ConfirmedDesign( - path=path, - sha256=sha256, - design=design, - repeat_groups=MappingProxyType(design.repeat_groups()), - ) - - -def write_experiment_design(design: ExperimentDesign, path: str | os.PathLike) -> Path: +def write_experiment_design(design: DesignTable, path: str | os.PathLike) -> Path: """Write ``experiment_design.csv``, replacing the file only once writing succeeded. The file is UTF-8 with LF line endings; the same design always gives the @@ -647,7 +300,7 @@ def write_experiment_design(design: ExperimentDesign, path: str | os.PathLike) - Raises ------ DesignFormatError - If a row has a repeat-group name or confirmation date the readers refuse + If a row has a repeat-group name or confirmation date the parser refuses (:func:`design_issues`, ``invalid_experiment_design``); nothing is written. """ path = Path(path) @@ -678,70 +331,19 @@ def write_experiment_design(design: ExperimentDesign, path: str | os.PathLike) - return write_text_atomic(path, buffer.getvalue(), parents=True) -def load_design( - output_folder: str | os.PathLike, repeats: Iterable[BiologicalRepeat] -) -> tuple[ExperimentDesign, DesignChanges]: - """Read the design in an output folder, if any, and bring it up to date with the folders. +def parse_design_bytes(data: bytes, name: str = DESIGN_FILENAME) -> DesignTable: + """Parse a design as :func:`write_experiment_design` writes it; confirmation not required. - The file is read with the lenient :func:`read_experiment_design`. A file that - cannot be read is refused, never replaced by the default design (owner decision of - 31 October 2026, plan §12 question 43): the file is left as it is. - - Parameters - ---------- - output_folder - The output folder that holds ``experiment_design.csv``, if it was written. - repeats - The biological-repeat folders found now - (:func:`.dataset.biological_repeats`). - - Returns - ------- - tuple of ExperimentDesign and DesignChanges - The saved design brought up to date with the folders (:func:`reconcile_design`), - or the default design when there is no file yet, and how the folders changed - since it was written. + Strict (experiment-v1 section 7): UTF-8 without a byte-order mark, exactly + :data:`DESIGN_COLUMNS` in that order, one value per column, no outer whitespace, ``true`` or + ``false``, whole counts, ISO dates, identities as :func:`.discovery.check_identity_text` + takes them and no case collision. Only the draft's legacy migration uses it. Raises ------ - DesignCheckError - With the one error ``unreadable_experiment_design`` when a saved file cannot be - read; its message is :func:`read_experiment_design`'s refusal, naming the line and - column where it can. The :class:`DesignFormatError` is its ``__cause__``. + DesignFormatError + Naming the line and column of the first value refused. """ - path = Path(output_folder) / DESIGN_FILENAME - previous = None - if path.exists(): - try: - previous = read_experiment_design(path) - except DesignFormatError as exc: - raise _unreadable( - path, - exc, - "Correct the file where the message says (a spreadsheet program may have " - f"rewritten a date such as 2026-10-26 as 26/10/2026; dates are {DATE_FORMAT}). " - "Answering the questions again in the notebook also replaces it; " - "nothing replaces it without an answer.", - ) from exc - return reconcile_design(previous, repeats) - - -def _unreadable(path: Path, exc: Exception, action: str) -> DesignCheckError: - """Return the refusal ``unreadable_experiment_design`` of a design file.""" - return DesignCheckError( - [ - error( - "unreadable_experiment_design", - f"The experiment design cannot be read: {exc}", - [str(path)], - action, - ) - ] - ) - - -def parse_design_bytes(data: bytes, name: str = DESIGN_FILENAME) -> ExperimentDesign: - """Parse a design as :func:`write_experiment_design` writes it; confirmation not required.""" if data.startswith(b"\xef\xbb\xbf"): raise DesignFormatError( f"{name} starts with a byte-order mark: it was saved by another program, not " @@ -782,7 +384,7 @@ def parse_design_bytes(data: bytes, name: str = DESIGN_FILENAME) -> ExperimentDe except IdentityError as exc: raise DesignFormatError(f"{name}: {exc}") from None _check_group_case(zip(numbers, rows, strict=True), name) - return ExperimentDesign(tuple(rows)) + return DesignTable(tuple(rows)) def _check_rows(rows: list[DesignRow], name: str) -> None: @@ -872,8 +474,8 @@ def text(column: str) -> str: ) -def _design_problems(design: ExperimentDesign) -> list[tuple[DesignRow, str]]: - """Return the rows whose repeat group or date the readers refuse, with the reason. +def _design_problems(design: DesignTable) -> list[tuple[DesignRow, str]]: + """Return the rows whose repeat group or date the parser refuses, with the reason. Each row's own values (:func:`_row_problems`), and a repeat group spelled differently from an earlier row's only in case (decision D-e: one spelling per name). diff --git a/src/morphobricks_preprocessing/design_derive.py b/src/morphobricks_preprocessing/design_derive.py index e5b9c36f..280502b3 100644 --- a/src/morphobricks_preprocessing/design_derive.py +++ b/src/morphobricks_preprocessing/design_derive.py @@ -23,10 +23,10 @@ from morphobricks_normalization.records import SINGLE from .description import Description -from .design import DesignRow, ExperimentDesign, write_experiment_design +from .design import DesignRow, DesignTable, write_experiment_design -def derive_design(description: Description) -> ExperimentDesign: +def derive_design(description: Description) -> DesignTable: """Return the design the description derives, one row per member (section 11). Raises @@ -53,7 +53,7 @@ def derive_design(description: Description) -> ExperimentDesign: confirmed_on=repeat.confirmed_on, ) ) - return ExperimentDesign(tuple(rows)) + return DesignTable(tuple(rows)) def write_derived_design(description: Description, path: str | os.PathLike) -> Path: diff --git a/src/morphobricks_preprocessing/design_edit.py b/src/morphobricks_preprocessing/design_edit.py deleted file mode 100644 index 3b78ae17..00000000 --- a/src/morphobricks_preprocessing/design_edit.py +++ /dev/null @@ -1,195 +0,0 @@ -"""Change an experiment design without editing the CSV: groups, controls, confirmations. - -The notebook's pairing card applies a :class:`DesignEdit` to the design it shows (the -``design`` command and ``edit_design`` were removed in v0.5.1, PP-C): move folders to other repeat -groups, mark conditions as controls or not, and confirm repeat groups. Each change uses the -design's own rules (:class:`.design.ExperimentDesign`), and a repeat group is confirmed only -when its rows pass the run's validator (:func:`.design.design_issues`). Nothing is confirmed -by default: the confirmation stays an explicit act of the user. Anything unknown is refused -with nothing changed. -""" - -from __future__ import annotations - -from collections.abc import Iterable, Mapping -from dataclasses import dataclass, field -from types import MappingProxyType - -from .design import ExperimentDesign, design_issues - -_UNCONFIRMED = "design_not_confirmed" -"""The one design issue a repeat group about to be confirmed may have.""" - - -class DesignEditError(ValueError): - """Raised when a change cannot be made to the design; the design is left as it is.""" - - -@dataclass(frozen=True) -class DesignEdit: - """The changes to make to an experiment design, applied in this order. - - Attributes - ---------- - repeat_groups - ``{(condition, biological_repeat): repeat group}``: move each folder into that - group; both groups it leaves and joins are no longer confirmed. - control, not_control - Conditions to mark as controls, and to unmark. - confirm - Repeat groups to confirm (after the moves). - confirm_all - Confirm every repeat group not confirmed yet. - unconfirm - Repeat groups whose confirmation is taken back (after the moves, before the - confirmations); the notebook's "Undo" (PP-10). Python only. It never confirms - anything: a group in it cannot also be confirmed, and it - cannot be combined with ``confirm_all``. - """ - - repeat_groups: Mapping[tuple[str, str], str] = field(default_factory=dict) - control: tuple[str, ...] = () - not_control: tuple[str, ...] = () - confirm: tuple[str, ...] = () - confirm_all: bool = False - unconfirm: tuple[str, ...] = () - - def __post_init__(self) -> None: - object.__setattr__(self, "repeat_groups", MappingProxyType(dict(self.repeat_groups))) - for name in ("control", "not_control", "confirm", "unconfirm"): - object.__setattr__(self, name, tuple(getattr(self, name))) - - -def parse_repeat_group(text: str) -> tuple[tuple[str, str], str]: - """Read ``CONDITION/REPEAT=GROUP``, the command line's way to move a folder. - - Parameters - ---------- - text - The folder's condition and biological repeat (``single`` for none), ``=``, and the - repeat group; the group is the text after the last ``=``. - - Returns - ------- - tuple - ``((condition, biological_repeat), group)``. - - Raises - ------ - DesignEditError - If the text is not in that form. - """ - folder, equals, group = text.rpartition("=") - condition, slash, repeat = folder.partition("/") - if not equals or not slash or not condition or not repeat or "/" in repeat: - raise DesignEditError( - f"{text!r} is not CONDITION/REPEAT=GROUP (for example ctrl/rep1=day1; the " - "biological repeat is single when the condition has no repeat folders)" - ) - return (condition, repeat), group - - -def apply_design_edit(design: ExperimentDesign, edit: DesignEdit, on: str) -> ExperimentDesign: - """Return the design with the changes made, refusing any that cannot be made. - - Parameters - ---------- - design - The design as read from ``experiment_design.csv``. - edit - The changes. - on - The date of the confirmations, ``YYYY-MM-DD``. - - Returns - ------- - ExperimentDesign - The changed design; a repeat group confirmed before keeps its date. - - Raises - ------ - DesignEditError - For an unknown folder, condition or repeat group, a condition both marked and - unmarked, a repeat group to confirm whose rows break the design rules - (``repeat_group_conflict``, ``invalid_experiment_design``), or a design whose values - cannot be written. The message says which and why. - """ - design = _moved(design, edit.repeat_groups) - design = _controls(design, edit.control, edit.not_control) - design = _unconfirmed(design, edit.unconfirm, edit.confirm, edit.confirm_all) - groups = design.unconfirmed_groups if edit.confirm_all else edit.confirm - _known("repeat group", groups, design.groups) - for group in dict.fromkeys(groups): - if not design.group_confirmed(group): - _check_group(design, group) - design = design.with_confirmed(group, True, on) - invalid = [i for i in design_issues(design) if i.code == "invalid_experiment_design"] - if invalid: - raise DesignEditError(invalid[0].message) - return design - - -def _moved(design: ExperimentDesign, moves: Mapping[tuple[str, str], str]) -> ExperimentDesign: - """Move folders into other repeat groups, refusing folders the design does not have.""" - folders = {row.key for row in design.rows} - unknown = [f"{c}/{r}" for c, r in moves if (c, r) not in folders] - if unknown: - known = ", ".join(f"{c}/{r}" for c, r in sorted(folders)) - raise DesignEditError( - f"unknown folder {', '.join(unknown)}: the design's folders are {known}" - ) - for key, group in moves.items(): - design = design.with_group(key, group) - return design - - -def _controls( - design: ExperimentDesign, control: tuple[str, ...], not_control: tuple[str, ...] -) -> ExperimentDesign: - """Mark and unmark controls, refusing unknown conditions and contradictions.""" - _known("condition", (*control, *not_control), design.conditions) - both = sorted(set(control) & set(not_control)) - if both: - raise DesignEditError(f"{', '.join(both)} cannot be marked a control and not one") - for condition in control: - design = design.with_control(condition, True) - for condition in not_control: - design = design.with_control(condition, False) - return design - - -def _unconfirmed( - design: ExperimentDesign, - unconfirm: tuple[str, ...], - confirm: tuple[str, ...], - confirm_all: bool, -) -> ExperimentDesign: - """Take back confirmations, refusing unknown groups and a group both confirmed and not.""" - _known("repeat group", unconfirm, design.groups) - both = sorted(set(unconfirm) & set(confirm)) if not confirm_all else sorted(set(unconfirm)) - if both: - raise DesignEditError(f"{', '.join(both)} cannot be confirmed and unconfirmed at once") - for group in dict.fromkeys(unconfirm): - design = design.with_confirmed(group, False) - return design - - -def _known(what: str, given: Iterable[str], known: tuple[str, ...]) -> None: - """Refuse names the design does not have, listing those it has.""" - unknown = [name for name in dict.fromkeys(given) if name not in known] - if unknown: - raise DesignEditError( - f"unknown {what} {', '.join(map(repr, unknown))}: the design's {what}s are " - + (", ".join(known) or "none") - ) - - -def _check_group(design: ExperimentDesign, group: str) -> None: - """Refuse to confirm a repeat group whose rows break the design rules.""" - members = ExperimentDesign(design.members(group)) - problems = [issue for issue in design_issues(members) if issue.code != _UNCONFIRMED] - if problems: - raise DesignEditError( - f"repeat group {group!r} cannot be confirmed: " - + " ".join(issue.message for issue in problems) - ) diff --git a/src/morphobricks_preprocessing/discovery_report.py b/src/morphobricks_preprocessing/discovery_report.py index 1e4a8871..294397c7 100644 --- a/src/morphobricks_preprocessing/discovery_report.py +++ b/src/morphobricks_preprocessing/discovery_report.py @@ -32,7 +32,7 @@ if TYPE_CHECKING: from .axes import AxisReview - from .design import ExperimentDesign + from .design import DesignTable from .run.preflight import Preflight from .run.recorded import InputChange @@ -164,7 +164,7 @@ def build_report( issues: Sequence[ValidationIssue], *, review: AxisReview | None = None, - design: ExperimentDesign | None = None, + design: DesignTable | None = None, image_type: str | None = None, outcome: RunOutcome | None = None, unit_records: Sequence[Mapping[str, object]] | None = None, @@ -344,7 +344,7 @@ def _calibration(pre: Preflight) -> dict: } -def _design(design: ExperimentDesign | None) -> list[dict] | None: +def _design(design: DesignTable | None) -> list[dict] | None: """Return the design's rows, or None without a design.""" if design is None: return None diff --git a/src/morphobricks_preprocessing/draft.py b/src/morphobricks_preprocessing/draft.py index 237c01ab..1d6989d0 100644 --- a/src/morphobricks_preprocessing/draft.py +++ b/src/morphobricks_preprocessing/draft.py @@ -25,7 +25,7 @@ from .description_read import PHASES from .description_report import DescriptionIssue from .description_rules import SCHEMA_ID -from .design import ExperimentDesign, design_issues, parse_design_bytes +from .design import DesignTable, design_issues, parse_design_bytes from .discovery import DiscoveryResult from .draft_ids import IdKind, id_problems, propose_id @@ -370,8 +370,8 @@ def draft_from_discovery( ) -> Draft: """Build the draft of a folder before any answer: proposed IDs, units, counts, pairing. - Biological-repeat folders with the same name are one repeat, as in the default design - (:func:`.design.default_design`); nothing is confirmed. IDs are proposed as the folder + Biological-repeat folders with the same name are one repeat (the default pairing, which the + notebook's pairing card starts from); nothing is confirmed. IDs are proposed as the folder labels where allowed (Q6: the same rule for conditions, technical repeats and repeats), and ``used_ids`` (the IDs a published file of this lineage lists) are never proposed again. No image is opened, no folder renamed and no file written. @@ -380,7 +380,7 @@ def draft_from_discovery( return _draft(_members(discovery, set(excluded)), {}, used) -def _legacy_groups(legacy: ExperimentDesign) -> dict[tuple[str, str], str]: +def _legacy_groups(legacy: DesignTable) -> dict[tuple[str, str], str]: issues = [issue for issue in design_issues(legacy) if issue.code == "repeat_group_conflict"] if issues: raise LegacyMigrationRefused(issues[0].message) @@ -461,7 +461,7 @@ def _check_legacy_rules(draft: Draft) -> None: def _migrated_repeat( - group: str, repeat: DraftRepeat, legacy: ExperimentDesign + group: str, repeat: DraftRepeat, legacy: DesignTable ) -> tuple[DraftRepeat, MigrationNote | None]: """Carry a legacy group's confirmation: only if every folder is confirmed, latest date.""" rows = [legacy.row(member.key) for member in repeat.members] diff --git a/src/morphobricks_preprocessing/notebook_axis_preview.py b/src/morphobricks_preprocessing/notebook_axis_preview.py index cccfec0e..3c35d2a1 100644 --- a/src/morphobricks_preprocessing/notebook_axis_preview.py +++ b/src/morphobricks_preprocessing/notebook_axis_preview.py @@ -304,11 +304,6 @@ def _refresh_axis_next(self) -> None: "Go to Step 1 and confirm the folder labels.", "Review the labels and click Yes, the labels are correct before running Step 3.", ) - elif self._design_pending(): - self.axis_next.value = ui.next_steps( - "Go to Step 1 and confirm the biological repeats.", - "Under Experiment design, answer each yellow question before running Step 3.", - ) else: self.axis_next.value = ui.next_steps( "Go to Step 3 and compute the histograms.", diff --git a/src/morphobricks_preprocessing/notebook_check_save.py b/src/morphobricks_preprocessing/notebook_check_save.py index 5f1e4954..4c584d90 100644 --- a/src/morphobricks_preprocessing/notebook_check_save.py +++ b/src/morphobricks_preprocessing/notebook_check_save.py @@ -186,7 +186,7 @@ def _published_check(self) -> CandidateCheck | None: if getattr(self, "_published_check_key", None) != key: self._published_check_key = key self._published_checked = check_candidate( - state.description, self.preflight, roles, self._design_folder() + state.description, self.preflight, roles, state.description ) return self._published_checked @@ -297,9 +297,7 @@ def _candidate_check(self, answers: object) -> CandidateCheck | None: state = self._published() published = state.description if state is not None else None _, confirmations = self._publication_terms(state, candidate) - return check_candidate( - candidate, self.preflight, roles, self._design_folder(), published, confirmations - ) + return check_candidate(candidate, self.preflight, roles, published, confirmations) def _show_replicates(self, applied: object, answers: object, labels: list[str]) -> None: """One box per replicate: folders, pictures, channels, and its own question.""" diff --git a/src/morphobricks_preprocessing/notebook_described.py b/src/morphobricks_preprocessing/notebook_described.py index 07c49728..f9715e86 100644 --- a/src/morphobricks_preprocessing/notebook_described.py +++ b/src/morphobricks_preprocessing/notebook_described.py @@ -9,10 +9,10 @@ What the described run changes, as overrides of the step mixins: - **Step 1.** The "Yes, the labels are correct" toggle is retired: the labels are shown and - restated on *Check and save*. The check writes no design template, the design card keeps only - the pairing (in the draft, never written to ``experiment_design.csv`` before the save) and its - "comparable" questions move to *Check and save*, one per replicate. The names card's "Yes, use - these IDs" is retired too: the save click adopts every short name. + restated on *Check and save*. The check writes no design template; the pairing card + (:mod:`.notebook_design`) is an answer of the draft, and the "comparable" questions are on + *Check and save*, one per replicate. The names card's "Yes, use these IDs" is retired too: the + save click adopts every short name. - **Step 2.** The global *Image type* card is retired: the image type is answered per unit on *Your channels*; the run is given the type the saved description states. - **Reopening.** The published file is the answers: the folders found now decide what still @@ -24,15 +24,11 @@ from __future__ import annotations from collections.abc import Mapping -from dataclasses import replace -from datetime import date from pathlib import Path from . import notebook_style as ui from .description_file import DESCRIPTION_FILENAME from .description_reopen import PublishedState, published_state -from .design import default_design -from .design_edit import DesignEdit, DesignEditError, apply_design_edit from .draft import Draft, DraftError, draft_from_discovery from .draft_answers import DraftAnswers, UnitKey, apply_answers, carry_over from .fingerprint import current_fingerprint @@ -48,13 +44,6 @@ from .run import Preflight -def _units(draft: Draft, label: str) -> frozenset: - repeat = draft.repeats.get(label) - if repeat is None: - return frozenset() - return frozenset((m.key, u.source_folder) for m in repeat.members for u in m.units) - - class _DescribedRun: """The overrides of a described run (see the module docstring).""" @@ -103,59 +92,6 @@ def _build_description_cards(self) -> None: "Optional. For a treated", "Answer for each condition. For a treated" ).replace("Nothing is chosen for you.", "Choose no comparison where none.") - def _save_design(self) -> None: - """Keep the design card's answers in the draft (never written to a design file).""" - self._design_saved = self._design_waiting = False - - def _build_group_questions(self) -> None: - super()._build_group_questions() - ui.set_visible(self.groups_box, False) - - def _design_pending(self) -> bool: - return False - - def _open_design(self, *, output_changed: bool = False) -> None: - """Open the pairing from the draft (or the published file).""" - self._design_output = self._design_folder() - self.design = default_design(self.preflight.dataset.repeats) - self._design_unreadable, self._design_saved = False, False - self._design_note = self._design_refusal = "" - self._forget_published() - self._load_description() - self._folded = self._saved_current() - self._take_pairing() - ui.set_visible(self.design_editor, True) - ui.set_visible(self.design_change, False) - ui.set_visible(self.design_card, True) - self._design_changed() - - def _take_pairing(self) -> None: - """Set the design card's pairing to the one the answers hold.""" - pairing = dict(self.draft_answers.pairing) if self.draft_answers else {} - if pairing: - try: - self.design = apply_design_edit( - self.design, DesignEdit(repeat_groups=pairing), date.today().isoformat() - ) - except DesignEditError as exc: - self._design_note = ui.status("warn", "The saved pairing was not used.", [str(exc)]) - self._build_design_editor() - - def _refresh_design_status(self) -> None: - """Say how the pairing is kept (no confirmation on this card).""" - if self.design is None: - self.design_status.value = "" - return - note = self._design_note + self._design_refusal - self.design_status.value = note + ui.status( - "info", - "Folders with the same name are the same replicate.", - [ - "Choose another replicate for a folder that was named differently. Your pairing " - "is saved with your other answers and confirmed on Check and save." - ], - ) - # Answers ----------------------------------------------------------------------------- def _base_draft(self) -> Draft: @@ -222,25 +158,6 @@ def _still_confirmed( """ return resumed(self._draft_base, saved, kept) - def _sync_description(self) -> None: - """Take the design card's pairing into the answers; changed replicates are asked again.""" - answers = self.draft_answers - if answers is None or self.design is None: - return - pairing = self._design_pairing() - if pairing == dict(answers.pairing): - self._refresh_description() - return - old, new = self._draft_base.paired(answers.pairing), self._draft_base.paired(pairing) - repeats = { - label: answer - if _units(old, label) == _units(new, label) - else replace(answer, confirmed_on=None) - for label, answer in answers.repeats.items() - if label in new.repeats - } - self._answer(replace(answers, pairing=pairing, repeats=repeats)) - def _id_typed(self, entry: object, state: object, text: str) -> None: super()._id_typed(entry, state, text) if not self._refreshing: @@ -260,7 +177,7 @@ def _refresh_channels(self) -> None: """Refresh *Your channels*; the pairing card gets its pictures once they can be read.""" super()._refresh_channels() thumbs = self._channel_thumbs - if self.design is not None and thumbs is not self._pictures_for: + if self.pairing is not None and thumbs is not self._pictures_for: self._pictures_for = thumbs self._build_design_editor() @@ -291,9 +208,6 @@ def _refresh_axis_next(self) -> None: # The run ----------------------------------------------------------------------------- - def _design_items(self) -> list[tuple[bool, str]]: - return [] - def _gate_items(self) -> list[tuple[bool, str]]: return [(self._run_allowed(), "Save the experiment description (Check and save)")] diff --git a/src/morphobricks_preprocessing/notebook_description.py b/src/morphobricks_preprocessing/notebook_description.py index d6d65575..1dbad565 100644 --- a/src/morphobricks_preprocessing/notebook_description.py +++ b/src/morphobricks_preprocessing/notebook_description.py @@ -128,8 +128,6 @@ def _build_description_cards(self) -> None: ], visible=False, ) - # A control is a condition others are compared against: the references say it. - ui.set_visible(self.controls_box, False) # Opening, resuming and closing ------------------------------------------------------- @@ -139,10 +137,6 @@ def _hide_description(self) -> None: ui.set_visible(self.references_card, False) self._hide_channels() - def _design_pairing(self) -> dict[tuple[str, str], str]: - """Return the folders the design pairs under another name than their own.""" - return {k: g for k, g in self.design.repeat_groups().items() if g != k[1]} - def _load_description(self) -> None: """Derive the draft from the folders and resume the saved answers, if any are usable. @@ -171,33 +165,6 @@ def _base_draft(self) -> Draft: # Answers ----------------------------------------------------------------------------- - def _sync_description(self) -> None: - """Take the design card's pairing and comparability dates into the answers.""" - answers = self.draft_answers - if answers is None or self.design is None: - return - repeats = {} - for group in self.design.groups: - day = self.design.confirmed_on(group) if self.design.group_confirmed(group) else "" - kept = replace(answers.repeats.get(group, RepeatAnswer()), confirmed_on=day or None) - if kept != RepeatAnswer(): - repeats[group] = kept - lost = [g for g, a in answers.repeats.items() if g not in repeats and a.id is not None] - new = replace(answers, pairing=self._design_pairing(), repeats=repeats) - if new != answers: - if lost: - self._draft_note = ui.status( - "info", - "The pairing changed.", - [ - f"No longer a repeat of its own: {', '.join(lost)}. Its name was dropped; " - "a repeat that is new is named below." - ], - ) - self._answer(new) - else: - self._refresh_description() - def _answer(self, answers: DraftAnswers, *, rebuild: bool = True) -> bool: """Apply new answers to the draft and save them; False when they are refused. @@ -444,7 +411,6 @@ def _reference_chosen(self, condition: str, target: str | None) -> None: new = _updated(answers, "conditions", condition, reference=target) if self._answer(new): self._references_refusal = "" - self._sync_controls() else: self._references_refusal = refusal_card( "This answer was not taken.", @@ -453,17 +419,6 @@ def _reference_chosen(self, condition: str, target: str | None) -> None: self._refresh_description() self._refresh_design_status() - def _sync_controls(self) -> None: - """Mark as controls (design card, ``experiment_design.csv``) the conditions referenced. - - The checkboxes are the design card's own, hidden; setting one saves the design as - ticking it did. - """ - referenced = set(self.draft.references.values()) - for condition, check in self.control_checks.items(): - if check.value != (condition in referenced): - check.value = condition in referenced - def _with_ids(answers: DraftAnswers, ids: dict[tuple[str, str], str]) -> DraftAnswers: """Return the answers with exactly these IDs (any ID not in ``ids`` is taken back).""" diff --git a/src/morphobricks_preprocessing/notebook_design.py b/src/morphobricks_preprocessing/notebook_design.py index e7f656d8..5d5d3458 100644 --- a/src/morphobricks_preprocessing/notebook_design.py +++ b/src/morphobricks_preprocessing/notebook_design.py @@ -1,98 +1,93 @@ -"""Step 1 of the notebook: the experiment design card (the pairing of the repeats). - -The pairing is kept in the draft and confirmed on *Check and save* (:mod:`.notebook_described`, -which opens the card and reports its status); nothing here reads or writes -``experiment_design.csv``: the run derives it from the published description. The card's model -(:class:`.design.ExperimentDesign`, :func:`.design_edit.apply_design_edit`) is rebuilt on the -draft in a later change (SWEEP-PP, owner answer 4). +"""Step 1 of the notebook: the pairing card (which folders are the same biological repeat). + +The pairing is an answer of the draft (``DraftAnswers.pairing``, rebuilt on the draft in v0.5.1, +SWEEP-PP, owner answer 4): the card shows one row per biological-repeat folder of the draft's +base (the folders found, paired by name), with a dropdown of the repeat it belongs to. A change +is saved in the draft at once; the replicates it changes are asked "same way?" again on *Check +and save*, where the comparability of every replicate is confirmed. Nothing here reads or writes +``experiment_design.csv``: the run derives it from the published description. There is no other +model behind the card: the shown pairing is :attr:`pairing`, the draft's pairing filled with each +folder's own name. """ from __future__ import annotations import html -from datetime import date +from dataclasses import replace from pathlib import Path import ipywidgets as widgets +from morphobricks_normalization.records import SINGLE from . import notebook_style as ui from .dataset import check_dataset -from .design_edit import DesignEdit, DesignEditError, apply_design_edit -from .discovery import check_output_folder, default_output_folder +from .discovery import ( + IdentityError, + check_identity_text, + check_output_folder, + default_output_folder, +) +from .draft import DraftMember from .notebook_options import _PAIRING_COLUMNS -from .notebook_render import _group_explanation, _group_question from .notebook_row_fold import RowFold, row_status from .validation import plural +def _units(draft: object, label: str) -> frozenset: + """Return the units of one repeat of a draft (to tell whether a pairing changed it).""" + repeat = draft.repeats.get(label) + if repeat is None: + return frozenset() + return frozenset((m.key, u.source_folder) for m in repeat.members for u in m.units) + + class _DesignCard: - """Step 1's experiment design card: repeat groups, comparability and controls.""" + """Step 1's pairing card: one repeat per biological-repeat folder, kept in the draft.""" - # Experiment design (step 1) ------------------------------------------------------------- + # Building ---------------------------------------------------------------------------- def _build_design_card(self) -> None: - """Build the experiment design card, hidden until the labels are confirmed.""" + """Build the pairing card, hidden until the folder is checked.""" + self.pairing: dict[tuple[str, str], str] | None = None + self._design_output: Path | None = None + self._design_note = "" self.design_status = widgets.HTML() - self.design_change = ui.button( - " Change the experiment design", - style="warning", - icon="pencil", - tooltip="Show the pairing, the confirmations and the controls again", - ) - ui.set_visible(self.design_change, False) self.pairing_grid = widgets.VBox() - self.pairing_detail = widgets.VBox() # a picture's Enlarge (described run) + self.pairing_detail = widgets.VBox() # a picture's Enlarge self.pairing_box = widgets.VBox( [ - widgets.HTML("1. Pair the biological repeats across conditions"), + widgets.HTML("Pair the biological repeats across conditions"), ui.help_text( "A repeat group is one biological repeat in every condition: the same " "preparation, imaged for each condition. Folders with the same name are " "paired by default. If a repeat has another folder name in some " "condition (acquired a day later, for example), choose its group in " "the dropdown. Each group may hold one folder per condition. The dropdown " - "offers " - "folder names only; the repeat's own ID is set with the names below." + "offers folder names only; the repeat's own ID is set with the names below." ), self.pairing_grid, self.pairing_detail, ] ) - self.groups_box = widgets.VBox() - self.control_checks: dict[str, widgets.Checkbox] = {} - self.controls_grid = widgets.VBox() - self.controls_box = widgets.VBox( - [ - widgets.HTML("Controls (optional)"), - ui.help_text( - "Tick the control conditions, if any: none, one or several. This is " - "recorded for later bricks; nothing uses it yet." - ), - self.controls_grid, - ] - ) - self.design_editor = widgets.VBox([self.pairing_box, self.groups_box, self.controls_box]) self.design_card = ui.card( "Experiment design", [ ui.help_text( "Later bricks normalise each biological repeat over all conditions " "together, so the brick needs to know which folders are the same " - "biological repeat, and that their images are comparable. Your pairing is " - "saved with your other answers in the draft; experiment_design.csv " - "is written only when you save the experiment description (Check and " - "save)." + "biological repeat. Your pairing is saved with your other answers in the " + "draft; whether each replicate's images are comparable is confirmed on " + "Check and save, and experiment_design.csv is written only when you " + "save the experiment description." ), self.design_status, - self.design_change, - self.design_editor, + self.pairing_box, ], visible=False, ) - self.design_change.on_click(lambda _: self._reopen_design()) def _design_folder(self) -> Path | None: - """Return the output folder the design is saved in, or None if it may not be written.""" + """Return the output folder the answers are saved in, or None if it may not be written.""" folder = ui.clean_path(self.folder.value) if not folder: return None @@ -101,25 +96,67 @@ def _design_folder(self) -> Path | None: return None return output + # Opening and closing ----------------------------------------------------------------- + + def _open_design(self, *, output_changed: bool = False) -> None: + """Open the card on the draft of the checked folder (or of the published file).""" + self._design_output = self._design_folder() + self._design_note = "" + self._forget_published() + self._load_description() + self._folded = self._saved_current() + self.pairing = self._pairing_of( + dict(self.draft_answers.pairing) if self.draft_answers else {} + ) + self._build_design_editor() + ui.set_visible(self.design_card, True) + self._design_changed() + def _close_design(self) -> None: - self.design = None + self.pairing = None ui.set_visible(self.design_card, False) self._hide_description() if self.preflight is not None: self.dataset = self.preflight.dataset self._show_issue_summary() - def _reopen_design(self) -> None: - ui.set_visible(self.design_editor, True) - ui.set_visible(self.design_change, False) - self._refresh_design_status() + def _members(self) -> tuple[DraftMember, ...]: + """Return the folders of the draft's base, sorted by condition and folder.""" + base = self._draft_base + found = () if base is None else (m for r in base.repeats.values() for m in r.members) + return tuple(sorted(found, key=lambda m: m.key)) + + def _pairing_of(self, saved: dict[tuple[str, str], str]) -> dict[tuple[str, str], str]: + """Return every folder's repeat: its saved pairing, else its own name. + + A saved pairing that names a folder the draft does not have, or a repeat that is not a + name (empty, ``NA``-like or with outer spaces), is not used: the folders' own names are + shown and a note says why. Nothing is repaired. + """ + members = {m.key: m.source_repeat for m in self._members()} + problems = [ + f"{'/'.join(key)} is not a folder of the draft" for key in saved if key not in members + ] + for key, label in saved.items(): + try: + check_identity_text(label, "replicate") + except IdentityError as exc: + problems.append(f"{'/'.join(key)}: {exc}") + if problems: + self._design_note = ui.status("warn", "The saved pairing was not used.", problems) + saved = {} + return {key: saved.get(key, own) for key, own in members.items()} + + # The pairing grid -------------------------------------------------------------------- def _build_design_editor(self) -> None: - """Build the pairing dropdowns and the control checkboxes of the design.""" - design = self.design - pairing = len(design.conditions) > 1 and not design.single_repeat - ui.set_visible(self.pairing_box, pairing) - options = sorted({row.biological_repeat for row in design.rows} | set(design.groups)) + """Build one row per folder with the dropdown of its repeat.""" + members = self._members() + conditions = {m.condition for m in members} + shown = len(conditions) > 1 and any(m.source_repeat != SINGLE for m in members) + ui.set_visible(self.pairing_box, shown) + pairing = self.pairing or {} + options = sorted({m.source_repeat for m in members} | set(pairing.values())) rows = [ widgets.GridBox( [widgets.HTML(f"{h}") for h in ("Condition", "Folder", "Repeat group")], @@ -128,22 +165,24 @@ def _build_design_editor(self) -> None: ] units = self._reopen_units() folded: list[tuple[widgets.Widget, str]] = [] - for row in design.rows if pairing else (): + for member in members if shown else (): dropdown = widgets.Dropdown( - options=options, value=row.repeat_group, layout=widgets.Layout(width="99%") + options=options, + value=pairing.get(member.key, member.source_repeat), + layout=widgets.Layout(width="99%"), ) dropdown.observe( - lambda change, key=row.key: self._pair(key, change["new"]), names="value" + lambda change, key=member.key: self._pair(key, change["new"]), names="value" ) - keys = [k for k in units or () if k[:2] == row.key] + keys = [k for k in units or () if k[:2] == member.key] folded.append(( widgets.GridBox( [ - widgets.HTML(html.escape(row.condition)), + widgets.HTML(html.escape(member.condition)), self._folder_cell( - row.key, - f"{html.escape(row.biological_repeat)} " - f"{plural(row.image_count, 'image')}", + member.key, + f"{html.escape(member.source_repeat)} " + f"{plural(member.image_count, 'image')}", ), dropdown, ], @@ -154,96 +193,21 @@ def _build_design_editor(self) -> None: if not hasattr(self, "_pairing_fold"): self._pairing_fold = RowFold("folder") self.pairing_grid.children = rows + self._pairing_fold.arrange(folded) - self.control_checks = {} - for condition in design.conditions: - check = widgets.Checkbox( - value=condition in design.controls, - description=condition, - indent=False, - layout=widgets.Layout(width="auto"), - ) - check.observe( - lambda change, c=condition: self._set_control(c, change["new"]), names="value" - ) - self.control_checks[condition] = check - self.controls_grid.children = [ - widgets.HBox( - list(self.control_checks.values()), layout=widgets.Layout(flex_flow="wrap") - ) - ] - self._build_group_questions() - - def _build_group_questions(self) -> None: - """One yellow question per repeat group; rebuilt when the pairing changes.""" - design = self.design - self.group_confirmations: dict[str, ui.ConfirmToggle] = {} - boxes = [ - widgets.HTML( - "2. Confirm that the biological repeats are comparable" - if len(design.conditions) > 1 and not design.single_repeat - else "Confirm that the images are comparable" - ) - ] - for group in design.groups: - toggle = ui.ConfirmToggle( - _group_question(design, group), - _group_explanation(design, group), - "Yes, they are comparable", - "Confirmed: these images are comparable.", - "Undo: they are not comparable", - ) - toggle.set(design.group_confirmed(group), notify=False) - toggle.disabled = group in design.conflicts - toggle.observe(lambda value, g=group: self._confirm_group(g, value)) - self.group_confirmations[group] = toggle - boxes.append(toggle.widget) - self.groups_box.children = boxes def _pair(self, key: tuple[str, str], group: str) -> None: - if self.design is None: + if self.pairing is None or self.pairing.get(key) == group: return - if self._edit_design(DesignEdit(repeat_groups={key: group})): - self._build_group_questions() - self._save_design() + self.pairing = {**self.pairing, key: group} self._design_changed() - def _confirm_group(self, group: str, confirmed: bool) -> None: - if self.design is None: - return - edit = DesignEdit(confirm=(group,)) if confirmed else DesignEdit(unconfirm=(group,)) - if not self._edit_design(edit): - self.group_confirmations[group].set(False, notify=False) - self._save_design() - self._design_changed() - - def _set_control(self, condition: str, control: bool) -> None: - if self.design is None: - return - edit = DesignEdit(control=(condition,)) if control else DesignEdit(not_control=(condition,)) - self._edit_design(edit) - self._save_design() - self._refresh_design_status() - - def _edit_design(self, edit: DesignEdit) -> bool: - """Apply an answer with the design's own rules; False when it is refused. - - :func:`.design_edit.apply_design_edit` confirms a repeat group only when its rows - pass the run's validator; a refused answer changes nothing and is shown. - """ - try: - self.design = apply_design_edit(self.design, edit, date.today().isoformat()) - except DesignEditError as exc: - self._design_refusal = ui.status("error", "This answer was not taken.", [str(exc)]) - return False - self._design_refusal = "" - return True + def _design_pairing(self) -> dict[tuple[str, str], str]: + """Return the folders paired under another name than their own (the draft's answer).""" + return {k: g for k, g in (self.pairing or {}).items() if g != k[1]} def _design_changed(self) -> None: """Update the balance, the messages, the status and the later steps after an answer.""" self.dataset = check_dataset( - self.preflight.discovery, - self.design.repeat_groups() if self.design is not None else None, - excluded=self.preflight.excluded, + self.preflight.discovery, self.pairing, excluded=self.preflight.excluded ) self._show_issue_summary() self._refresh_design_status() @@ -252,5 +216,35 @@ def _design_changed(self) -> None: self._refresh_axis_next() self._refresh_run_step() - def _design_confirmed(self) -> bool: - return self.design is not None and self.design.confirmed + def _refresh_design_status(self) -> None: + """Say how the pairing is kept (no confirmation on this card).""" + if self.pairing is None: + self.design_status.value = "" + return + self.design_status.value = self._design_note + ui.status( + "info", + "Folders with the same name are the same replicate.", + [ + "Choose another replicate for a folder that was named differently. Your pairing " + "is saved with your other answers and confirmed on Check and save." + ], + ) + + def _sync_description(self) -> None: + """Take the pairing into the answers; a replicate whose folders changed is asked again.""" + answers = self.draft_answers + if answers is None or self.pairing is None: + return + pairing = self._design_pairing() + if pairing == dict(answers.pairing): + self._refresh_description() + return + old, new = self._draft_base.paired(answers.pairing), self._draft_base.paired(pairing) + repeats = { + label: answer + if _units(old, label) == _units(new, label) + else replace(answer, confirmed_on=None) + for label, answer in answers.repeats.items() + if label in new.repeats + } + self._answer(replace(answers, pairing=pairing, repeats=repeats)) diff --git a/src/morphobricks_preprocessing/notebook_folder.py b/src/morphobricks_preprocessing/notebook_folder.py index a7db2122..b9de0aa6 100644 --- a/src/morphobricks_preprocessing/notebook_folder.py +++ b/src/morphobricks_preprocessing/notebook_folder.py @@ -238,8 +238,8 @@ def check_folder(self) -> Preflight | None: return pre def _show_folder_results(self, pre: Preflight) -> None: - """Show the labels, messages and pixel sizes of a new check; the design starts over.""" - self.design = None + """Show the labels, messages and pixel sizes of a new check; the pairing starts over.""" + self.pairing = None self.dataset = pre.dataset ui.set_visible(self.design_card, False) self._hide_description() @@ -328,13 +328,7 @@ def _refresh_folder_next(self) -> None: self.folder_next.value = "" if self.preflight is None or not self.preflight.ok: return - if self.labels_confirmed.value and self._design_pending(): - self.folder_next.value = ui.next_steps( - "Confirm the biological repeats in Step 1.", - "Under Experiment design, pair the repeats if needed and answer each yellow " - "question with Yes, they are comparable.", - ) - elif self.labels_confirmed.value: + if self.labels_confirmed.value: self.folder_next.value = ui.next_steps( "Go to Step 2 and review the axis roles.", "Run its cell, check the preview, then click Confirm axis roles.", @@ -361,7 +355,7 @@ def _folder_edited(self) -> None: self.folder_tiles.value = "" self.folder_next.value = "" self.labels_confirmed.set(False, notify=False) - self.design = None + self.pairing = None ui.set_visible(self.design_card, False) self._hide_description() ui.set_visible(self.labels_card, False) @@ -379,9 +373,9 @@ def _output_edited(self) -> None: self._update_output_hint() # The inputs start from the earlier run in the new output folder (audit I4). self._prefill_from_output() - # A design belongs to its output folder (issue #98): another folder's design is + # The answers belong to their output folder (issue #98): another folder's draft is # read, or the questions asked again; nothing is carried over or written here. - if self.design is not None and self._design_folder() != self._design_output: + if self.pairing is not None and self._design_folder() != self._design_output: self._open_design(output_changed=True) # "What will happen" and the free disk space depend on the output folder. if hasattr(self, "readiness"): diff --git a/src/morphobricks_preprocessing/notebook_render.py b/src/morphobricks_preprocessing/notebook_render.py index 78559b69..48f3ce4b 100644 --- a/src/morphobricks_preprocessing/notebook_render.py +++ b/src/morphobricks_preprocessing/notebook_render.py @@ -10,7 +10,6 @@ from . import notebook_style as ui from .axes import AxisMapping from .dataset import BalanceTable -from .design import ExperimentDesign from .discovery import LeafFolder from .notebook_options import _SKIPPED_CODES from .notebook_values import plane_counts, value_range @@ -212,43 +211,6 @@ def _labels_explanation(show_design: bool) -> str: return (text + " Step 3 can only run once the labels are confirmed.") if show_design else text -def _group_question(design: ExperimentDesign, group: str) -> str: - """Return the question asked for a repeat group, or for the whole experiment.""" - if design.single_repeat: - return ( - "All images of every condition come from one biological repeat: were they " - "prepared and acquired under the same circumstances?" - ) - return ( - f"Repeat group {group}: were these images prepared and acquired under the same " - "circumstances, so they are comparable with each other?" - ) - - -def _group_explanation(design: ExperimentDesign, group: str) -> str: - """Describe the folders of a repeat group and anything unusual about it (HTML).""" - items = [ - f"{html.escape(row.condition)}: {html.escape(row.biological_repeat)} " - f"({plural(row.image_count, 'image')})" - for row in design.members(group) - ] - text = "Folders: " + "; ".join(items) + "." - missing = design.missing_conditions(group) - if missing and not design.single_repeat: - text += ( - "
Not found in " - + ", ".join(html.escape(c) for c in missing) - + ": it is still used, with the conditions that have it." - ) - if group in design.conflicts: - text += ( - "
This group holds more than one folder of " - + ", ".join(html.escape(c) for c in design.conflicts[group]) - + ". Move one of them into another group above before confirming." - ) - return text + " If they are not comparable, process them as separate experiments." - - def _folder_tiles(pre: Preflight, issues: Sequence[ValidationIssue]) -> list[tuple[str, str, str]]: """Return the headline numbers of a folder check.""" skipped = sum(len(i.paths) for i in pre.discovery.issues if i.code in _SKIPPED_CODES) diff --git a/src/morphobricks_preprocessing/notebook_run.py b/src/morphobricks_preprocessing/notebook_run.py index 085e746c..c4a46aae 100644 --- a/src/morphobricks_preprocessing/notebook_run.py +++ b/src/morphobricks_preprocessing/notebook_run.py @@ -247,7 +247,6 @@ def _refresh_run_step(self) -> None: ) items = [ (folder_ok, "Check the experiment folder and fix any errors (step 1)"), - *self._design_items(), (self.confirmed_mapping is not None, "Confirm the axis roles (step 2)"), *self._gate_items(), ] @@ -271,11 +270,6 @@ def _refresh_run_step(self) -> None: self._refresh_plan_card() self._refresh_mask_step() - def _design_items(self) -> list[tuple[bool, str]]: - """Return the experiment design answer the run waits for (step 1), if shown.""" - design = (self._design_confirmed(), "Confirm the biological repeats (step 1)") - return [design] if self._show_design else [] - def _gate_items(self) -> list[tuple[bool, str]]: """Return the answers after the axis roles that the run waits for (the image type).""" return [(self.confirmed_image_type is not None, "Confirm the image type (step 2)")] diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index f0785bf2..26c76285 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -47,7 +47,6 @@ from . import notebook_style as ui from .axes import AxisMapping from .dataset import DatasetReport -from .design import ExperimentDesign from .image_type import ImageTypeReview from .notebook_axis_preview import _AxisPreview from .notebook_axis_step import _AxisStep @@ -181,10 +180,9 @@ class PreprocessingNotebook( def __init__(self, experiment_folder: str = "", output_folder: str = "") -> None: self.preflight: Preflight | None = None - self.design: ExperimentDesign | None = None - # True while the saved design cannot be read and has not been replaced by an answer. - self._design_unreadable = False - # The output folder the design was read for: a design belongs to it (issue #98). + # The pairing card's folders and their repeats (None while the card is closed), and the + # output folder its answers were read for (issue #98): see notebook_design. + self.pairing: dict[tuple[str, str], str] | None = None self._design_output: Path | None = None self.dataset = DatasetReport() self.mapping: AxisMapping | None = None diff --git a/src/morphobricks_preprocessing/report.py b/src/morphobricks_preprocessing/report.py index fc485934..2acfa6af 100644 --- a/src/morphobricks_preprocessing/report.py +++ b/src/morphobricks_preprocessing/report.py @@ -17,7 +17,6 @@ from .axes import AxisMapping, AxisReview, axis_length_ranges, axis_name from .dataset import BalanceTable -from .design import ExperimentDesign from .discovery import DiscoveryResult from .discovery_report import REPORT_FILENAME, REPORT_TEXT_FILENAME from .run import Preflight, RunResult @@ -116,74 +115,6 @@ def balance_summary(balance: BalanceTable) -> str: return "\n".join(lines) -def design_summary(design: ExperimentDesign) -> str: - """Describe the repeat groups, their folders and confirmation, and the controls.""" - lines = [] - for group in design.groups: - folders = ", ".join( - f"{row.condition}/{row.biological_repeat}" for row in design.members(group) - ) - if group in design.conflicts: - state = "more than one folder of " + ", ".join(design.conflicts[group]) - elif design.group_confirmed(group): - on = design.confirmed_on(group) - state = f"confirmed{f' on {on}' if on else ''}" - else: - state = "not confirmed" - lines.append(f" {group}: {folders} ({state})") - controls = ", ".join(design.controls) or "none marked" - lines.append(f" Controls: {controls}") - return "\n".join(lines) - - -def design_table(design: ExperimentDesign) -> str: - """Show the design's rows as a table, then the repeat groups still to confirm. - - Parameters - ---------- - design - The experiment design. - - Returns - ------- - str - One row per biological-repeat folder of each condition (``experiment_design.csv`` - without its schema column), then which repeat groups are not confirmed yet. - """ - rows = [ - [ - "condition", - "biological repeat", - "repeat group", - "control", - "images", - "technical repeats", - "confirmed", - ] - ] - for row in design.rows: - on = f", {row.confirmed_on}" if row.confirmed_on else "" - confirmed = f"yes{on}" if row.confirmed else "no" - rows.append( - [ - row.condition, - row.biological_repeat, - row.repeat_group, - "yes" if row.control else "no", - str(row.image_count), - str(row.technical_repeat_count), - confirmed, - ] - ) - waiting = design.unconfirmed_groups - last = ( - f"Not confirmed yet: {', '.join(waiting)}" - if waiting - else "Every repeat group is confirmed." - ) - return format_table(rows) + "\n" + last - - def mapping_summary(mapping: AxisMapping, files: Iterable = ()) -> str: """Describe each axis on one line: its label, its length across files and its role.""" ranges = axis_length_ranges(mapping, files) diff --git a/src/morphobricks_preprocessing/run/result.py b/src/morphobricks_preprocessing/run/result.py index d1c1745c..d7adfca9 100644 --- a/src/morphobricks_preprocessing/run/result.py +++ b/src/morphobricks_preprocessing/run/result.py @@ -11,7 +11,7 @@ from ..design import ( DESIGN_FILENAME, - ExperimentDesign, + DesignTable, ) from ..discovery_report import REPORT_FILENAME from ..intensity import IntensityReport @@ -83,7 +83,7 @@ class RunResult: excluded: tuple[str, ...] = () rebuild_reason: str | None = None plane_count: int = 0 - design: ExperimentDesign | None = None + design: DesignTable | None = None image_type: str | None = None intensity: IntensityReport | None = None mode: str = RESTART diff --git a/src/morphobricks_preprocessing/run/steps.py b/src/morphobricks_preprocessing/run/steps.py index c976fc9e..532d0016 100644 --- a/src/morphobricks_preprocessing/run/steps.py +++ b/src/morphobricks_preprocessing/run/steps.py @@ -10,7 +10,6 @@ from collections.abc import Callable, Mapping from dataclasses import asdict, dataclass, field, replace -from datetime import datetime from pathlib import Path from morphobricks_normalization.records import PlaneRecord @@ -18,7 +17,7 @@ from ..axes import AxisReview from ..axis_review import review_axes from ..dataset import check_dataset -from ..design import ExperimentDesign, design_issues, reconcile_design +from ..design import DesignTable, design_issues, reconcile_design from ..discovery_report import RunOutcome from ..fingerprint import unchanged_confirmation from ..image_type import ImageTypeReview, image_type_issues, review_image_type @@ -45,7 +44,7 @@ class Checked: review, type_review The axis and image-type reviews; None when no axis roles were given or recorded. design - The design brought up to date with the folders, or None. + The derived design brought up to date with the folders, or None. issues Every issue found so far. bins @@ -54,7 +53,7 @@ class Checked: review: AxisReview | None type_review: ImageTypeReview | None - design: ExperimentDesign | None + design: DesignTable | None issues: list[ValidationIssue] = field(default_factory=list) bins: int | None = None @@ -62,7 +61,7 @@ class Checked: def header_checks( pre: Preflight, inputs: ResolvedInputs, - design: ExperimentDesign | None, + design: DesignTable | None, memory_limit: int, ) -> Checked: """Check everything that needs no pixel: folders, axes, image type, design, masks. @@ -74,7 +73,7 @@ def header_checks( inputs The run's resolved inputs. design - The design given or recorded, or None. + The design derived from the published description, or None. memory_limit Working-memory limit per plane, in bytes. @@ -91,8 +90,7 @@ def header_checks( type_review = review_image_type(pre.metadata, review) dataset, design_problems = pre.dataset, [] if design is not None: - design, _ = reconcile_design(design, pre.dataset.repeats) - design = design.dated(datetime.now().date().isoformat()) + design = reconcile_design(design, pre.dataset.repeats) design_problems = design_issues(design) dataset = check_dataset(pre.discovery, design.repeat_groups(), excluded=pre.excluded) issues = [ diff --git a/src/morphobricks_preprocessing/run_inputs.py b/src/morphobricks_preprocessing/run_inputs.py index 616a6865..2c3a10d9 100644 --- a/src/morphobricks_preprocessing/run_inputs.py +++ b/src/morphobricks_preprocessing/run_inputs.py @@ -1,7 +1,7 @@ """What a clean preprocessing run gives the other bricks: :class:`RunInputs` (design v0.4 §6). A brick that works on files reads them only through a clean run: the images, their axis -roles, pixel sizes, masks, histograms, design and each unit's declared image type +roles, pixel sizes, masks, histograms and each unit's declared image type (``Fov.image_type``), all as the run recorded them. Nothing is parsed from names, guessed or transposed, and no file is read once it changed: @@ -27,6 +27,11 @@ A :class:`RunInputs` is picklable: it holds paths and the records read when it opened (its snapshot of the table), and opens files only to read them. + +The derived ``experiment_design.csv`` is an output only (Spec ADR 0002 AR11 (3)): its SHA-256 +is checked at open with the other outputs, but it is not read back. A brick takes the repeat +groups from the published description (``Fov.repeat_group``, :meth:`RunInputs.resolve_request`); +``RunInputs.design()`` was removed in v0.5.1 (SWEEP-PP). """ from __future__ import annotations @@ -44,7 +49,7 @@ from .calibration import CALIBRATION_FILENAME from .conversion_check import conversion_problem from .description import Description -from .design import DESIGN_FILENAME, DesignCheckError, ExperimentDesign, read_confirmed_design +from .design import DESIGN_FILENAME from .fingerprint import Fingerprint from .folder_rules import masks_folder_issues from .links import LinkRefusedError @@ -382,30 +387,6 @@ def histograms(self) -> tuple[PlaneRecord, ...]: """Return the table's rows, as read when the run was opened.""" return self.records - def design(self) -> ExperimentDesign | None: - """Return the run's experiment design, or None when it had none. - - Read with the strict :func:`.design.read_confirmed_design` (issue #83), checked - against the manifest's ``experiment_design_sha256`` on the bytes it parses. - - Raises - ------ - RunInputsError - If the design is missing, changed since the run, or not one the strict reader - takes (its refusal is the ``__cause__``). - """ - expected = self.manifest.experiment_design_sha256 - if expected is None: - return None - path = published_path(self.output_folder, DESIGN_FILENAME, expected) - try: - return read_confirmed_design(path, expected_sha256=expected).design - except DesignCheckError as exc: - raise RunInputsError( - f"The design {DESIGN_FILENAME} cannot be used: {exc.issues[0].message} " - "Run preprocessing again." - ) from exc - def _check_files(self) -> None: """Refuse changed or gone files, masks without axes or shared, unrecorded files.""" changed = [e.relative_path for e in self.manifest.files if not self._unchanged(e)] diff --git a/tests/test_axis_check.py b/tests/test_axis_check.py index b27d673d..4d9fb9b1 100644 --- a/tests/test_axis_check.py +++ b/tests/test_axis_check.py @@ -27,7 +27,6 @@ review_axes, ) from morphobricks_preprocessing.axis_check import axis_disagreements -from morphobricks_preprocessing.design import default_design, write_experiment_design from morphobricks_preprocessing.discovery import default_output_folder from run_files import only_the_report from tiff_tree import write_data @@ -361,12 +360,7 @@ def test_plain_stack_masks_are_not_compared_before_the_roles(tmp_path): def confirm(root): - design = default_design(preflight(root).dataset.repeats) - for group in design.groups: - design = design.with_confirmed(group, True, "2026-11-01") - output = default_output_folder(root) - output.mkdir(parents=True, exist_ok=True) - write_experiment_design(design, output / "experiment_design.csv") + default_output_folder(root).mkdir(parents=True, exist_ok=True) def test_cli_check_reports_plain_stacks(tmp_path, capsys): diff --git a/tests/test_brick_api_surface.py b/tests/test_brick_api_surface.py index a50a282c..96256dc7 100644 --- a/tests/test_brick_api_surface.py +++ b/tests/test_brick_api_surface.py @@ -55,7 +55,6 @@ "t: 'int | None' = None) -> 'np.ndarray'" ), "histograms": "(self) -> 'tuple[PlaneRecord, ...]'", - "design": "(self) -> 'ExperimentDesign | None'", "description": "(self) -> 'Description'", "section_digests": "(self) -> 'dict[str, str]'", "images_without_header_channel_names": "(self) -> 'dict[str, tuple[int, ...]]'", diff --git a/tests/test_cli_describe.py b/tests/test_cli_describe.py index 3501df62..e1ba20f5 100644 --- a/tests/test_cli_describe.py +++ b/tests/test_cli_describe.py @@ -18,7 +18,7 @@ from morphobricks_preprocessing.cli import main from morphobricks_preprocessing.description import description_from_bytes from morphobricks_preprocessing.description_file import sha256_hex -from morphobricks_preprocessing.design import DesignRow, ExperimentDesign, write_experiment_design +from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design from morphobricks_preprocessing.discovery import discover_experiment from morphobricks_preprocessing.draft import draft_from_discovery from morphobricks_preprocessing.run.description_file import publish_description @@ -90,7 +90,7 @@ def test_describe_reads_the_legacy_design_without_confirming_anything(experiment row("DrugA", "rep2", control=True, confirmed=False, on=""), ] output.mkdir() - write_experiment_design(ExperimentDesign(tuple(rows)), output / DESIGN) + write_experiment_design(DesignTable(tuple(rows)), output / DESIGN) before = listing(tmp_path) assert main(["describe", str(experiment), "--output", str(output)]) == 0 out = capsys.readouterr().out @@ -119,7 +119,7 @@ def test_publish_never_confirms_for_the_user(experiment, tmp_path, capsys): output = tmp_path / "out" output.mkdir() rows = [row(c, r) for c in ("CTRL", "DrugA") for r in ("rep1", "rep2")] - write_experiment_design(ExperimentDesign(tuple(rows)), output / DESIGN) + write_experiment_design(DesignTable(tuple(rows)), output / DESIGN) before = listing(tmp_path) assert main(["describe", str(experiment), "--output", str(output), "--publish"]) == 1 out = capsys.readouterr().out diff --git a/tests/test_dataset.py b/tests/test_dataset.py index 148693ed..1e712631 100644 --- a/tests/test_dataset.py +++ b/tests/test_dataset.py @@ -197,15 +197,14 @@ def test_a_technical_repeat_folder_named_single_is_the_level_value(tmp_path): def test_the_design_written_by_a_run_counts_only_analysed_technical_repeats(tmp_path): - from morphobricks_preprocessing import read_confirmed_design - from morphobricks_preprocessing.design import DESIGN_FILENAME + from morphobricks_preprocessing.design import DESIGN_FILENAME, parse_design_bytes root = tree(tmp_path / "exp", {"ctrl/rep1/techA": 1, "ctrl/rep1/techB": 1}) excluded = ["ctrl/rep1/techB/img_00.tif"] output = tmp_path / "out" # the description declares the analysed unit only result = run_described(root, prefill_mapping("YX"), output_folder=output, exclude=excluded) assert result.written - (row,) = read_confirmed_design(output / DESIGN_FILENAME).design.rows + (row,) = parse_design_bytes((output / DESIGN_FILENAME).read_bytes()).rows assert (row.image_count, row.technical_repeat_count) == (1, 1) diff --git a/tests/test_described_unit_checks.py b/tests/test_described_unit_checks.py index 78edddf4..79086a2e 100644 --- a/tests/test_described_unit_checks.py +++ b/tests/test_described_unit_checks.py @@ -84,7 +84,7 @@ def test_the_described_folder_check_of_a_mixed_experiment_passes(tmp_path): def test_the_check_and_save_screen_explains_the_refusal_in_words(tmp_path): root, out = mixed_experiment(tmp_path) pre = unit_checked(preflight(root, out)) - check = check_candidate(description_from_bytes(VALID["v29"].data), pre, None, out) + check = check_candidate(description_from_bytes(VALID["v29"].data), pre, None) assert check.can_save assert check.heterogeneous is not None and not check.can_run text = checks_html(check) diff --git a/tests/test_design.py b/tests/test_design.py index 62755315..0bf56abe 100644 --- a/tests/test_design.py +++ b/tests/test_design.py @@ -1,18 +1,28 @@ +"""The rows of experiment_design.csv: the run's check, the writer and the strict parser. + +The file is an output only (v0.5.1, Spec ADR 0002 AR11 (3)): the run derives it from the +published description and checks it against the folders found (:func:`reconcile_design`, +:func:`design_issues`); :func:`parse_design_bytes` is the strict parser of the draft's legacy +migration. The editor model, the default pairing, the lenient reader, ``load_design`` and the +strict reader for other bricks were removed (SWEEP-PP). +""" + +from dataclasses import replace + import pytest from described_tree import run_described from morphobricks_preprocessing import ( DESIGN_FILENAME, - DesignCheckError, DesignFormatError, + DesignRow, + DesignTable, biological_repeats, - default_design, default_output_folder, design_issues, discover_experiment, - load_design, + parse_design_bytes, preflight, - read_experiment_design, read_manifest, reconcile_design, run_experiment, @@ -43,10 +53,24 @@ def repeats_of(root, excluded=()): return biological_repeats(discover_experiment(root), excluded) -def confirm_all(design, repeats=()): - for group in design.groups: - design = design.with_confirmed(group, True, TODAY, repeats) - return design +def table(root, groups=None, *, confirmed=True, controls=()): + """The rows a description of ``root`` derives: each folder its own repeat unless moved.""" + groups = groups or {} + return DesignTable( + tuple( + DesignRow( + r.condition, + r.biological_repeat, + groups.get(r.key, r.biological_repeat), + control=r.condition in controls, + image_count=r.image_count, + technical_repeat_count=r.technical_repeat_count, + confirmed=confirmed, + confirmed_on=TODAY if confirmed else "", + ) + for r in repeats_of(root) + ) + ) PAIRED = { @@ -58,11 +82,11 @@ def confirm_all(design, repeats=()): } -# Pairing and confirmation ---------------------------------------------------------------- +# What the rows say ----------------------------------------------------------------------- -def test_same_named_repeats_pair_by_default(tmp_path): - design = default_design(repeats_of(tree(tmp_path / "exp", PAIRED))) +def test_the_rows_answer_groups_controls_and_members(tmp_path): + design = table(tree(tmp_path / "exp", PAIRED), controls=("control",)) assert design.groups == ("rep1", "rep2", "rep2_day2") assert design.repeat_groups() == { ("control", "rep1"): "rep1", @@ -70,28 +94,14 @@ def test_same_named_repeats_pair_by_default(tmp_path): ("treated", "rep1"): "rep1", ("treated", "rep2_day2"): "rep2_day2", } - assert design.missing_conditions("rep2") == ("treated",) row = design.row(("control", "rep1")) assert (row.image_count, row.technical_repeat_count) == (3, 2) - assert not design.confirmed and design.unconfirmed_groups == design.groups - assert design.controls == () and not design.single_repeat - - -def test_moving_a_folder_pairs_it_and_asks_again(tmp_path): - design = confirm_all(default_design(repeats_of(tree(tmp_path / "exp", PAIRED)))) - assert design.confirmed and design.confirmed_on("rep1") == TODAY - moved = design.with_group(("treated", "rep2_day2"), "rep2") - assert moved.groups == ("rep1", "rep2") - assert moved.missing_conditions("rep2") == () - # The group that changed is asked again; the others keep their answer. - assert moved.unconfirmed_groups == ("rep2",) - assert moved.group_confirmed("rep1") - assert moved.with_group(("treated", "rep2_day2"), "rep2") == moved + assert design.confirmed and design.controls == ("control",) and not design.single_repeat + assert [r.key for r in design.members("rep1")] == [("control", "rep1"), ("treated", "rep1")] def test_a_group_cannot_hold_two_folders_of_a_condition(tmp_path): - design = default_design(repeats_of(tree(tmp_path / "exp", PAIRED))) - clash = confirm_all(design.with_group(("control", "rep2"), "rep1")) + clash = table(tree(tmp_path / "exp", PAIRED), {("control", "rep2"): "rep1"}) assert clash.conflicts == {"rep1": ("control",)} assert not clash.confirmed found = design_issues(clash) @@ -100,13 +110,18 @@ def test_a_group_cannot_hold_two_folders_of_a_condition(tmp_path): def test_unconfirmed_groups_are_errors(tmp_path): - design = default_design(repeats_of(tree(tmp_path / "exp", PAIRED))) - design = design.with_confirmed("rep1", True, TODAY) + root = tree(tmp_path / "exp", PAIRED) + design = DesignTable( + tuple( + row if row.repeat_group == "rep1" else replace(row, confirmed=False, confirmed_on="") + for row in table(root).rows + ) + ) (found,) = design_issues(design) assert found.code == "design_not_confirmed" and found.is_error assert "2 repeat groups (rep2, rep2_day2) are not confirmed" in found.message assert found.paths == ("control/rep2", "treated/rep2_day2") - assert design_issues(confirm_all(design)) == [] + assert design_issues(table(root)) == [] REMOVED_FIXES = ("design command", "--confirm", "experiment_design.csv", "repeat_group column") @@ -115,9 +130,9 @@ def test_unconfirmed_groups_are_errors(tmp_path): def test_the_design_refusals_name_the_fix_of_a_described_run(tmp_path): # Review of PP-C: the actions named the removed design command and editing # experiment_design.csv, which is now an output only (derived from the description). - design = default_design(repeats_of(tree(tmp_path / "exp", PAIRED))) - clash = design.with_group(("control", "rep2"), "rep1") - found = [*design_issues(design), *design_issues(clash)] + root = tree(tmp_path / "exp", PAIRED) + clash = table(root, {("control", "rep2"): "rep1"}) + found = [*design_issues(table(root, confirmed=False)), *design_issues(clash)] unpaired = preflight(tree(tmp_path / "unpaired", {"a/r1": 1, "b/r2": 1})).issues found += [issue for issue in unpaired if issue.code == "repeat_missing_conditions"] assert {i.code for i in found} == { @@ -131,73 +146,63 @@ def test_the_design_refusals_name_the_fix_of_a_described_run(tmp_path): def test_without_biological_repeat_folders_one_question(tmp_path): - design = default_design(repeats_of(tree(tmp_path / "exp", {"control": 2, "treated": 3}))) + root = tree(tmp_path / "exp", {"control": 2, "treated": 3}) + design = table(root, confirmed=False) assert design.single_repeat and design.groups == ("single",) (found,) = design_issues(design) assert found.message.startswith("The biological repeat of this experiment is not confirmed") - assert design_issues(confirm_all(design)) == [] - - -def test_controls_are_recorded(tmp_path): - design = default_design(repeats_of(tree(tmp_path / "exp", PAIRED))) - design = design.with_control("control", True) - assert design.controls == ("control",) - assert all(row.control for row in design.rows if row.condition == "control") - assert design.with_control("control", False).controls == () + assert design_issues(table(root)) == [] -# Reruns ------------------------------------------------------------------------------------ +# The run's check against the folders found ----------------------------------------------- def test_unchanged_folders_keep_the_answers(tmp_path): root = tree(tmp_path / "exp", PAIRED) - design = confirm_all(default_design(repeats_of(root)).with_control("control", True)) + design = table(root, controls=("control",)) # Images added to a confirmed folder keep the confirmation and its counts. make_tree(root, ["control/rep1/t1/img_99.tif"]) - again, changes = reconcile_design(design, repeats_of(root)) - assert again == design and not changes.any + again = reconcile_design(design, repeats_of(root)) + assert again == design assert again.row(("control", "rep1")).image_count == 3 def test_a_new_repeat_asks_only_for_its_group(tmp_path): root = tree(tmp_path / "exp", PAIRED) - design = confirm_all(default_design(repeats_of(root)).with_control("treated", True)) + design = table(root, controls=("treated",)) make_tree(root, ["treated/rep2/img_00.tif", "treated/rep3/img_00.tif"]) - again, changes = reconcile_design(design, repeats_of(root)) - assert changes.added == (("treated", "rep2"), ("treated", "rep3")) - assert changes.reopened == ("rep2",) + again = reconcile_design(design, repeats_of(root)) assert again.unconfirmed_groups == ("rep2", "rep3") assert again.group_confirmed("rep1") and again.group_confirmed("rep2_day2") # A new folder of a control condition is a control too. assert again.row(("treated", "rep3")).control + assert "design_not_confirmed" in codes(design_issues(again)) def test_removed_folders_are_dropped(tmp_path): root = tree(tmp_path / "exp", PAIRED) - design = confirm_all(default_design(repeats_of(root))) - again, changes = reconcile_design( - design, repeats_of(root, excluded=["control/rep2/img_00.tif", "control/rep2/img_01.tif"]) + again = reconcile_design( + table(root), + repeats_of(root, excluded=["control/rep2/img_00.tif", "control/rep2/img_01.tif"]), ) - assert changes.removed == (("control", "rep2"),) and changes.added == () assert again.confirmed and again.groups == ("rep1", "rep2_day2") -def test_manual_pairing_survives_a_rerun(tmp_path): +def test_a_pairing_of_the_rows_is_kept(tmp_path): root = tree(tmp_path / "exp", PAIRED) - design = default_design(repeats_of(root)).with_group(("treated", "rep2_day2"), "rep2") - path = write_experiment_design(confirm_all(design), tmp_path / DESIGN_FILENAME) - again, _ = reconcile_design(read_experiment_design(path), repeats_of(root)) + design = table(root, {("treated", "rep2_day2"): "rep2"}) + again = reconcile_design(design, repeats_of(root)) assert again.repeat_groups()[("treated", "rep2_day2")] == "rep2" and again.confirmed -# The file ---------------------------------------------------------------------------------- +# The file: written, and parsed strictly --------------------------------------------------- def test_file_round_trip_is_byte_identical(tmp_path): # Labels a table reader could mistake for other values stay text ("NA" and "null" # themselves are refused by discovery since decision D-e). root = tree(tmp_path / "exp", {"FALSE/001/t1": 1, "FALSE/TRUE": 2}) - design = confirm_all(default_design(repeats_of(root)).with_control("FALSE", True)) + design = table(root, controls=("FALSE",)) path = write_experiment_design(design, tmp_path / DESIGN_FILENAME) data = path.read_bytes() assert b"\r\n" not in data @@ -207,53 +212,44 @@ def test_file_round_trip_is_byte_identical(tmp_path): "FALSE,001,001,true,1,1,true,2026-10-26,1", "FALSE,TRUE,TRUE,true,2,0,true,2026-10-26,1", ] - read = read_experiment_design(path) + read = parse_design_bytes(data) assert read == design assert write_experiment_design(read, tmp_path / "again.csv").read_bytes() == data -def test_files_edited_in_a_spreadsheet_still_read(tmp_path): - path = tmp_path / DESIGN_FILENAME - text = ( - "condition,biological_repeat,repeat_group,control,image_count," - "technical_repeat_count,confirmed,confirmed_on,schema_version\r\n" - "ctrl,rep1,rep1,TRUE,2,0,Yes,,1\r\n" - "drug,rep1b,rep1,FALSE,2,0,1,2026-10-25,1\r\n" - ",,,,,,,,\r\n" - ) - path.write_bytes(text.encode("utf-8")) - design = read_experiment_design(path) - assert design.repeat_groups() == {("ctrl", "rep1"): "rep1", ("drug", "rep1b"): "rep1"} - assert design.controls == ("ctrl",) and design.confirmed - # A confirmation without a date gets one when it is used. - assert design.dated(TODAY).row(("ctrl", "rep1")).confirmed_on == TODAY - assert design.dated(TODAY).row(("drug", "rep1b")).confirmed_on == "2026-10-25" - - -# Typed edits (owner decision, 31 October 2026) -------------------------------------------- +@pytest.mark.parametrize( + "text", + [ + "\ufeff" + HEADER + "ctrl,rep1,rep1,true,2,0,true,,1\n", # a byte-order mark + HEADER.replace("\n", "\r\n") + "ctrl,rep1,rep1,TRUE,2,0,true,,1\r\n", # TRUE + HEADER + "ctrl,rep1,rep1,true,2,0,Yes,,1\n", + HEADER + "ctrl,rep1,rep1,true,2,0,1,,1\n", + HEADER + "ctrl,rep1,rep1,true,2,0,true,,1\n,,,,,,,,\n", # an empty row + ], +) +def test_a_file_saved_by_a_spreadsheet_is_refused(text): + with pytest.raises(DesignFormatError): + parse_design_bytes(text.encode("utf-8")) @pytest.mark.parametrize("group", ["", "NA", "na", "NaN", "none", "NULL", "n/a", "", ""]) -def test_a_repeat_group_that_reads_as_missing_is_refused(tmp_path, group): - path = tmp_path / DESIGN_FILENAME - path.write_bytes((HEADER + f"ctrl,rep1,{group},false,1,0,true,,1\n").encode("utf-8")) +def test_a_repeat_group_that_reads_as_missing_is_refused(group): + data = (HEADER + f"ctrl,rep1,{group},false,1,0,true,,1\n").encode("utf-8") with pytest.raises(DesignFormatError, match=r"line 2, column repeat_group: .*missing"): - read_experiment_design(path) + parse_design_bytes(data) @pytest.mark.parametrize("group", [" rep1", "rep1 ", "\trep1"]) -def test_a_repeat_group_with_outer_whitespace_is_refused(tmp_path, group): - path = tmp_path / DESIGN_FILENAME - path.write_bytes((HEADER + f'ctrl,rep1,"{group}",false,1,0,true,,1\n').encode("utf-8")) - with pytest.raises(DesignFormatError, match="line 2, column repeat_group: .*whitespace"): - read_experiment_design(path) +def test_a_repeat_group_with_outer_whitespace_is_refused(group): + data = (HEADER + f'ctrl,rep1,"{group}",false,1,0,true,,1\n').encode("utf-8") + with pytest.raises(DesignFormatError, match="line 2: repeat_group .*whitespace"): + parse_design_bytes(data) -def test_a_confirmation_date_in_another_form_is_refused(tmp_path): - path = tmp_path / DESIGN_FILENAME - path.write_bytes((HEADER + "ctrl,rep1,rep1,false,1,0,true,26/10/2026,1\n").encode("utf-8")) +def test_a_confirmation_date_in_another_form_is_refused(): + data = (HEADER + "ctrl,rep1,rep1,false,1,0,true,26/10/2026,1\n").encode("utf-8") with pytest.raises(DesignFormatError) as caught: - read_experiment_design(path) + parse_design_bytes(data) assert str(caught.value) == ( "experiment_design.csv, line 2, column confirmed_on: '26/10/2026' is not a date in the " "form YYYY-MM-DD (for example 2026-10-26)." @@ -261,27 +257,25 @@ def test_a_confirmation_date_in_another_form_is_refused(tmp_path): @pytest.mark.parametrize("on", ["2026-10-1", "20261026", "2026-02-30", "2026/10/26", "today"]) -def test_only_iso_dates_are_confirmation_dates(tmp_path, on): - path = tmp_path / DESIGN_FILENAME - path.write_bytes((HEADER + f"ctrl,rep1,rep1,false,1,0,true,{on},1\n").encode("utf-8")) +def test_only_iso_dates_are_confirmation_dates(on): + data = (HEADER + f"ctrl,rep1,rep1,false,1,0,true,{on},1\n").encode("utf-8") with pytest.raises(DesignFormatError, match="column confirmed_on: .*YYYY-MM-DD"): - read_experiment_design(path) + parse_design_bytes(data) def test_repeat_groups_differing_only_in_case_are_refused(tmp_path): # Decision D-e: a name is spelled one way; Rep1 and rep1 are not two repeat groups. - path = tmp_path / DESIGN_FILENAME - path.write_bytes( - (HEADER + "ctrl,rep1,Rep1,false,1,0,true,,1\ndrug,rep1,rep1,false,1,0,true,,1\n").encode() - ) + data = ( + HEADER + "ctrl,rep1,Rep1,false,1,0,true,,1\ndrug,rep1,rep1,false,1,0,true,,1\n" + ).encode() with pytest.raises(DesignFormatError) as caught: - read_experiment_design(path) + parse_design_bytes(data) assert str(caught.value).startswith( "experiment_design.csv, line 3, column repeat_group: 'rep1' differs only in case from " "'Rep1' on line 2." ) root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - clash = default_design(repeats_of(root)).with_group(("treated", "rep1"), "REP1") + clash = table(root, {("treated", "rep1"): "REP1"}) assert "invalid_experiment_design" in codes(design_issues(clash)) with pytest.raises(DesignFormatError, match="only in case"): write_experiment_design(clash, tmp_path / "out.csv") @@ -289,19 +283,18 @@ def test_repeat_groups_differing_only_in_case_are_refused(tmp_path): def test_the_writer_writes_only_iso_dates_and_valid_groups(tmp_path): root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - design = default_design(repeats_of(root)) + design = table(root) path = tmp_path / DESIGN_FILENAME for bad in ( - design.with_confirmed("rep1", True, "26/10/2026"), - design.with_group(("treated", "rep1"), "NA"), - design.with_group(("treated", "rep1"), " rep2"), + DesignTable(tuple(replace(r, confirmed_on="26/10/2026") for r in design.rows)), + table(root, {("treated", "rep1"): "NA"}), + table(root, {("treated", "rep1"): " rep2"}), ): assert "invalid_experiment_design" in codes(design_issues(bad)) with pytest.raises(DesignFormatError, match="cannot be written"): write_experiment_design(bad, path) assert not path.exists() - dated = confirm_all(design).dated("2026-10-31") - assert read_experiment_design(write_experiment_design(dated, path)) == dated + assert parse_design_bytes(write_experiment_design(design, path).read_bytes()) == design @pytest.mark.parametrize( @@ -312,23 +305,19 @@ def test_the_writer_writes_only_iso_dates_and_valid_groups(tmp_path): ("ctrl,rep1,rep1,false,1,0,true,,2\n", "schema_version '2' is not supported"), ("ctrl,rep1,rep1,false,1,0,true,,1\nctrl,rep1,rep1,false,1,0,true,,1\n", "more than once"), ("ctrl,rep1,rep1,true,1,0,true,,1\nctrl,rep2,rep2,false,1,0,true,,1\n", "control must be"), - (",rep1,rep1,false,1,0,true,,1\n", "must not be empty"), + (",rep1,rep1,false,1,0,true,,1\n", "condition"), ], ) -def test_invalid_files_are_refused(tmp_path, body, message): - path = tmp_path / DESIGN_FILENAME - path.write_bytes((HEADER + body).encode("utf-8")) +def test_invalid_files_are_refused(body, message): with pytest.raises(DesignFormatError, match=message): - read_experiment_design(path) + parse_design_bytes((HEADER + body).encode("utf-8")) -def test_files_without_the_columns_are_refused(tmp_path): - path = tmp_path / DESIGN_FILENAME - path.write_bytes(b"condition,biological_repeat\nctrl,rep1\n") - with pytest.raises(DesignFormatError, match="repeat_group, control"): - read_experiment_design(path) - with pytest.raises(DesignFormatError, match="cannot be read"): - read_experiment_design(tmp_path / "missing.csv") +def test_files_without_the_columns_are_refused(): + with pytest.raises(DesignFormatError, match="must have the columns"): + parse_design_bytes(b"condition,biological_repeat\nctrl,rep1\n") + with pytest.raises(DesignFormatError, match="lists no folder"): + parse_design_bytes(HEADER.encode("utf-8")) # The run ----------------------------------------------------------------------------------- @@ -386,7 +375,7 @@ def test_the_run_derives_the_design_from_the_description_and_records_its_hash(tm assert result.written # Paired by the description, the repeat is found in every condition. assert "repeat_missing_conditions" not in codes(result.issues) - design = read_experiment_design(result.design_path) + design = parse_design_bytes(result.design_path.read_bytes()) assert design.repeat_groups() == {("control", "rep1"): "rep1", ("treated", "rep1b"): "rep1"} assert design.controls == ("control",) and design.confirmed manifest = read_manifest(result.manifest_path) @@ -395,67 +384,6 @@ def test_the_run_derives_the_design_from_the_description_and_records_its_hash(tm assert result.csv_path.read_bytes() == plain.csv_path.read_bytes() -# Command line ------------------------------------------------------------------------------ - - -# load_design (owner decision of 31 October 2026, plan §12 question 43) ------------------ - - -def test_load_design_without_a_file_gives_the_default_design(tmp_path): - root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - design, changes = load_design(tmp_path / "out", repeats_of(root)) - assert design == default_design(repeats_of(root)) - assert not changes.any - assert not (tmp_path / "out").exists() - - -def test_load_design_reads_and_reconciles_a_saved_design(tmp_path): - root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - saved = confirm_all(default_design(repeats_of(root))).with_control("control", True) - write_experiment_design(saved, tmp_path / "out" / DESIGN_FILENAME) - make_tree(root, ["treated/rep2/img_00.tif"]) - design, changes = load_design(tmp_path / "out", repeats_of(root)) - assert design.group_confirmed("rep1") and design.controls == ("control",) - assert changes.added == (("treated", "rep2"),) - - -@pytest.mark.parametrize( - ("edit", "where"), - [ - (lambda text: text.replace(TODAY, "26/10/2026"), "line 2, column confirmed_on"), - (lambda text: text.replace("rep1,rep1,", "rep1,NA,", 1), "line 2, column repeat_group"), - (lambda text: "not a design", "column(s)"), - ], -) -def test_load_design_raises_for_an_unreadable_file(tmp_path, edit, where): - # Owner decision of 31 October 2026: no default design and no problem string; the - # refusal names the line and column, and the file is left as it is. - root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - path = tmp_path / "out" / DESIGN_FILENAME - write_experiment_design(confirm_all(default_design(repeats_of(root))), path) - path.write_text(edit(path.read_text(encoding="utf-8")), encoding="utf-8") - unreadable = path.read_bytes() - with pytest.raises(DesignCheckError) as caught: - load_design(tmp_path / "out", repeats_of(root)) - (issue,) = caught.value.issues - assert issue.code == "unreadable_experiment_design" - assert issue.is_error - assert issue.message.startswith("The experiment design cannot be read: ") - assert where in issue.message - assert issue.paths == (str(path),) - assert issue.action - assert isinstance(caught.value.__cause__, DesignFormatError) - assert isinstance(caught.value, ValueError) - assert path.read_bytes() == unreadable - - -def test_load_design_raises_when_the_design_path_is_a_folder(tmp_path): - root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) - (tmp_path / "out" / DESIGN_FILENAME).mkdir(parents=True) - with pytest.raises(DesignCheckError, match="unreadable_experiment_design"): - load_design(tmp_path / "out", repeats_of(root)) - - def test_preflight_repeats_feed_the_design(tmp_path): root = tree(tmp_path / "exp", PAIRED) assert preflight(root).dataset.repeats == tuple(repeats_of(root)) diff --git a/tests/test_confirmed_design.py b/tests/test_design_file.py similarity index 57% rename from tests/test_confirmed_design.py rename to tests/test_design_file.py index 089acdda..9a11d9cb 100644 --- a/tests/test_confirmed_design.py +++ b/tests/test_design_file.py @@ -1,22 +1,26 @@ -"""The shared strict reader of experiment_design.csv (contract §16, plan question 40).""" +"""The derived experiment_design.csv as a file: the run's output, and the strict parser. -import hashlib -import re +The file is an output only (v0.5.1, Spec ADR 0002 AR11 (3)): no brick reads it back for its +repeat groups (they come from the description through ``RunInputs``). Its golden here is parsed +in the test with the standard ``csv`` module; :func:`parse_design_bytes` is the strict parser of +the draft's experiment-v1 section 7 migration, which refuses anything doubtful. The strict +reader for other bricks (``read_confirmed_design``, ``ConfirmedDesign``, ``DesignCheckError``) +was removed in SWEEP-PP. +""" + +import csv +import io import pytest from morphobricks_preprocessing import ( DESIGN_FILENAME, - ConfirmedDesign, - DesignCheckError, DesignFormatError, - biological_repeats, - default_design, + DesignRow, + DesignTable, default_output_folder, - discover_experiment, - load_design, - read_confirmed_design, - read_experiment_design, + design_issues, + parse_design_bytes, read_manifest, run_experiment, write_experiment_design, @@ -25,7 +29,6 @@ from morphobricks_preprocessing.manifest import file_sha256 from tiff_tree import make_tree -TODAY = "2026-10-31" HEADER = ( "condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count," "confirmed,confirmed_on,schema_version\n" @@ -38,28 +41,7 @@ def tree(root, counts): return root -def confirm_all(design): - for group in design.groups: - design = design.with_confirmed(group, True, TODAY) - return design - - -def write(tmp_path, text): - path = tmp_path / DESIGN_FILENAME - path.write_bytes(text.encode("utf-8") if isinstance(text, str) else text) - return path - - -def refusal(path, **kwargs): - with pytest.raises(DesignCheckError) as caught: - read_confirmed_design(path, **kwargs) - (issue,) = caught.value.issues - assert issue.is_error and issue.paths == (str(path),) - assert issue.code in str(caught.value) - return issue - - -# What both bricks need --------------------------------------------------------------------- +# The run's output --------------------------------------------------------------------------- def _unit(repeat, condition, folder): @@ -85,51 +67,42 @@ def _unit(repeat, condition, folder): ) -def test_a_design_written_by_a_run_reads_with_everything_both_bricks_need(tmp_path): +def test_the_run_writes_the_derived_file_and_records_its_hash(tmp_path): # The run derives the design from the published description (v0.5.1): the pairing and - # the control are the description's, and the strict reader gives both bricks all of it. + # the control are the description's. Parsed here with the csv module only. root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1b": 1, "treated/rep2": 1}) output = default_output_folder(root) output.mkdir(parents=True) (output / "morphobricks.toml").write_text(PAIRED, encoding="utf-8") result = run_experiment(root, prefill_mapping("YX")) assert result.written, result.issues - recorded = read_manifest(result.manifest_path).experiment_design_sha256 - - read = read_confirmed_design(result.design_path, expected_sha256=recorded) - assert isinstance(read, ConfirmedDesign) - assert read.path == result.design_path - assert read.sha256 == recorded == file_sha256(result.design_path) - assert dict(read.repeat_groups) == { + data = result.design_path.read_bytes() + assert data == ( + HEADER + "control,rep1,rep1,true,1,0,true,2026-10-26,1\n" + "treated,rep1b,rep1,false,1,0,true,2026-10-26,1\n" + "treated,rep2,rep2,false,1,0,true,2026-10-26,1\n" + ).encode("utf-8") + rows = list(csv.DictReader(io.StringIO(data.decode("utf-8")))) + assert {(r["condition"], r["biological_repeat"]): r["repeat_group"] for r in rows} == { ("control", "rep1"): "rep1", ("treated", "rep1b"): "rep1", ("treated", "rep2"): "rep2", } - assert {g: dict(m) for g, m in read.group_members().items()} == { - "rep1": {"control": "rep1", "treated": "rep1b"}, - "rep2": {"treated": "rep2"}, - } - assert read.conditions == ("control", "treated") and read.controls == ("control",) - assert read.groups == ("rep1", "rep2") - with pytest.raises(TypeError): - read.repeat_groups[("control", "rep1")] = "other" # read-only - - -def test_the_hash_is_of_the_bytes_read(tmp_path): - path = write(tmp_path, HEADER + GOOD) - read = read_confirmed_design(path) - assert read.sha256 == hashlib.sha256((HEADER + GOOD).encode()).hexdigest() - # Windows line endings are the same rows (the hash differs, as the bytes do). - crlf = write(tmp_path, (HEADER + GOOD).replace("\n", "\r\n")) - assert read_confirmed_design(crlf).repeat_groups == read.repeat_groups + recorded = read_manifest(result.manifest_path).experiment_design_sha256 + assert recorded == file_sha256(result.design_path) -# Refusals ---------------------------------------------------------------------------------- +def test_every_design_the_brick_writes_is_parsed_back(tmp_path): + rows = ( + DesignRow("FALSE", "001", "001", True, 1, 1, True, "2026-10-31"), + DesignRow("FALSE", "TRUE", "TRUE", True, 2, 0, True, "2026-10-31"), + DesignRow("single", "single", "single", False, 1, 0, True, "2026-10-31"), + ) + path = write_experiment_design(DesignTable(rows), tmp_path / DESIGN_FILENAME) + assert parse_design_bytes(path.read_bytes()) == DesignTable(rows) -def test_a_missing_file_is_refused(tmp_path): - assert refusal(tmp_path / DESIGN_FILENAME).code == "missing_experiment_design" - assert refusal(tmp_path).code == "missing_experiment_design" # a folder +# The strict parser: anything doubtful is refused --------------------------------------------- @pytest.mark.parametrize( @@ -178,18 +151,10 @@ def test_a_missing_file_is_refused(tmp_path): ), ], ) -def test_anything_doubtful_is_refused(tmp_path, text, message): - issue = refusal(write(tmp_path, text)) - assert issue.code == "unreadable_experiment_design" - assert re.search(message, issue.message), issue.message - - -def test_the_lenient_reader_takes_what_the_shared_reader_refuses(tmp_path): - # The copies in CellPatcher and MorphoFeature used the lenient reader: spreadsheet - # spellings of booleans and spaces around labels were read. - path = write(tmp_path, HEADER + "ctrl ,rep1,rep1,TRUE,2,0,yes,,1\n") - assert read_experiment_design(path).repeat_groups() == {("ctrl", "rep1"): "rep1"} - assert refusal(path).code == "unreadable_experiment_design" +def test_anything_doubtful_is_refused(text, message): + data = text.encode("utf-8") if isinstance(text, str) else text + with pytest.raises(DesignFormatError, match=message): + parse_design_bytes(data) DUPLICATED = HEADER.replace("\n", ",repeat_group\n") @@ -229,50 +194,19 @@ def test_the_lenient_reader_takes_what_the_shared_reader_refuses(tmp_path): ), ], ) -def test_every_reader_refuses_a_column_named_twice_or_not_at_all(tmp_path, text, message): - path = write(tmp_path, text) +def test_a_column_named_twice_or_not_at_all_is_refused(text, message): with pytest.raises(DesignFormatError, match=message): - read_experiment_design(path) - issue = refusal(path) - assert issue.code == "unreadable_experiment_design" - assert re.search(message, issue.message), issue.message - # The step-1 reader refuses it too, and leaves the file as it is. - with pytest.raises(DesignCheckError, match=message): - load_design(tmp_path, []) - assert path.read_text(encoding="utf-8") == text + parse_design_bytes(text.encode("utf-8")) -def test_an_unconfirmed_group_is_refused(tmp_path): - path = write(tmp_path, HEADER + "ctrl,rep1,rep1,true,2,0,false,,1\n") - with pytest.raises(DesignCheckError) as caught: - read_confirmed_design(path) - # design_issues names the folders of the group, as in the preprocessing run. - (issue,) = caught.value.issues +def test_an_unconfirmed_group_is_parsed_and_refused_by_the_check(): + design = parse_design_bytes((HEADER + "ctrl,rep1,rep1,true,2,0,false,,1\n").encode()) + (issue,) = design_issues(design) assert issue.code == "design_not_confirmed" and issue.paths == ("ctrl/rep1",) -def test_a_group_with_two_folders_of_a_condition_is_refused(tmp_path): - path = write( - tmp_path, - HEADER + "ctrl,rep1,g,true,2,0,true,,1\nctrl,rep2,g,true,2,0,true,,1\n", - ) - with pytest.raises(DesignCheckError) as caught: - read_confirmed_design(path) - assert "repeat_group_conflict" in [issue.code for issue in caught.value.issues] - - -def test_a_design_other_than_the_runs_is_refused(tmp_path): - path = write(tmp_path, HEADER + GOOD) - assert refusal(path, expected_sha256="0" * 64).code == "changed_experiment_design" - assert read_confirmed_design(path, expected_sha256=file_sha256(path)).sha256 == file_sha256( - path - ) - - -def test_every_design_the_brick_writes_is_read(tmp_path): - root = tree(tmp_path / "exp", {"FALSE/001/t1": 1, "FALSE/TRUE": 2, "single": 1}) - design = confirm_all( - default_design(biological_repeats(discover_experiment(root))).with_control("FALSE", True) - ) - path = write_experiment_design(design, tmp_path / DESIGN_FILENAME) - assert read_confirmed_design(path).design == design +def test_a_group_with_two_folders_of_a_condition_is_refused_by_the_check(): + text = HEADER + "ctrl,rep1,g,true,2,0,true,,1\nctrl,rep2,g,true,2,0,true,,1\n" + assert "repeat_group_conflict" in [ + i.code for i in design_issues(parse_design_bytes(text.encode())) + ] diff --git a/tests/test_draft.py b/tests/test_draft.py index e90581a0..bc6f07c5 100644 --- a/tests/test_draft.py +++ b/tests/test_draft.py @@ -9,7 +9,7 @@ import pytest from morphobricks_preprocessing import DesignFormatError, discover_experiment -from morphobricks_preprocessing.design import DesignRow, ExperimentDesign, write_experiment_design +from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design from morphobricks_preprocessing.draft import ( DraftError, LegacyMigrationRefused, @@ -33,7 +33,7 @@ def folders(root, *paths): def legacy(tmp_path, rows): """Return the bytes of a legacy design with ``rows`` written by the brick's own writer.""" path = tmp_path / "legacy.csv" - write_experiment_design(ExperimentDesign(tuple(rows)), path) + write_experiment_design(DesignTable(tuple(rows)), path) return path.read_bytes() diff --git a/tests/test_draft_answers.py b/tests/test_draft_answers.py index 588f1509..6ccfc1ff 100644 --- a/tests/test_draft_answers.py +++ b/tests/test_draft_answers.py @@ -10,7 +10,7 @@ ) from morphobricks_preprocessing import discover_experiment from morphobricks_preprocessing.description import Channel, Condition -from morphobricks_preprocessing.design import DesignRow, ExperimentDesign, write_experiment_design +from morphobricks_preprocessing.design import DesignRow, DesignTable, write_experiment_design from morphobricks_preprocessing.draft import DraftError, draft_from_discovery, migrate_legacy_design from morphobricks_preprocessing.draft_answers import ( ConditionAnswer, @@ -162,7 +162,7 @@ def test_a_legacy_confirmation_is_carried_not_asked_again(tmp_path): for r in ("rep1", "rep2") ] csv = tmp_path / "legacy.csv" - write_experiment_design(ExperimentDesign(tuple(rows)), csv) + write_experiment_design(DesignTable(tuple(rows)), csv) draft = migrate_legacy_design(csv.read_bytes(), discover_experiment(tmp_path / "exp")) answers = complete() only_ids = DraftAnswers( diff --git a/tests/test_heterogeneous_refusal_first.py b/tests/test_heterogeneous_refusal_first.py index e7e5c376..6e53db5d 100644 --- a/tests/test_heterogeneous_refusal_first.py +++ b/tests/test_heterogeneous_refusal_first.py @@ -65,7 +65,7 @@ def test_the_check_and_save_list_leads_with_the_layout_refusal(tmp_path, confirm root, out = mixed_experiment(tmp_path) pre = unit_checked(preflight(root, out)) roles = review_axes(pre.metadata, prefill_mapping("CYX")).file_mappings if confirmed else None - check = check_candidate(description_from_bytes(VALID["v29"].data), pre, roles, out) + check = check_candidate(description_from_bytes(VALID["v29"].data), pre, roles) assert check.run_refusals[0].code == LAYOUT assert check.heterogeneous is check.run_refusals[0] text = checks_html(check) diff --git a/tests/test_histogram_range.py b/tests/test_histogram_range.py index 8ec002ff..1a4dc634 100644 --- a/tests/test_histogram_range.py +++ b/tests/test_histogram_range.py @@ -195,14 +195,8 @@ def test_a_table_of_schema_version_3_is_recomputed(hot_experiment, reads): def test_the_command_line_takes_the_range(hot_experiment, tmp_path, capsys): - from morphobricks_preprocessing import default_design, preflight, write_experiment_design - output = tmp_path / "cli" - design = default_design(preflight(hot_experiment).dataset.repeats) - for group in design.groups: - design = design.with_confirmed(group, True, "2026-10-28") output.mkdir() - write_experiment_design(design, output / "experiment_design.csv") args = ["run", str(hot_experiment), "--axes", "YX", "--output", str(output)] assert ( main_described([*args, "--histogram-range", "robust", "--robust-quantiles", "0.1", "99.9"]) diff --git a/tests/test_make_described_run.py b/tests/test_make_described_run.py index 6bc44c59..a27bd6d4 100644 --- a/tests/test_make_described_run.py +++ b/tests/test_make_described_run.py @@ -8,6 +8,8 @@ from __future__ import annotations +import csv +import io from collections import Counter import pytest @@ -172,5 +174,10 @@ def test_make_clean_run_without_a_description_makes_a_described_run_of_its_tree( ("drug", "rep1", ("ch1", "ch2")), } assert inputs.mask_check_status("Cells", fovs[0]) == "checked" # the mask step ran - assert inputs.design().repeat_groups() == {("ctrl", "rep1"): "rep1", ("drug", "rep1"): "rep1"} - assert inputs.design().controls == ("ctrl",) + # The derived experiment_design.csv is an output only: parsed here, never by RunInputs. + rows = list(csv.DictReader(io.StringIO((run.output / "experiment_design.csv").read_text()))) + assert {(r["condition"], r["biological_repeat"]): r["repeat_group"] for r in rows} == { + ("ctrl", "rep1"): "rep1", + ("drug", "rep1"): "rep1", + } + assert [r["condition"] for r in rows if r["control"] == "true"] == ["ctrl"] diff --git a/tests/test_modes_lock_report.py b/tests/test_modes_lock_report.py index 36c4138c..9dea5fa8 100644 --- a/tests/test_modes_lock_report.py +++ b/tests/test_modes_lock_report.py @@ -18,12 +18,9 @@ from morphobricks_preprocessing import ( AxisMapping, RunMode, - default_design, default_output_folder, prefill_mapping, - preflight, read_manifest, - write_experiment_design, ) from morphobricks_preprocessing.discovery_report import REPORT_FILENAME, REPORT_TEXT_FILENAME from morphobricks_preprocessing.manifest import file_sha256 @@ -431,11 +428,8 @@ def cli_run(*args): def confirmed_design(root, output): - """Confirm every repeat group, as the user does before 'run' (contract §12).""" - design = default_design(preflight(root).dataset.repeats) - for group in design.groups: - design = design.with_confirmed(group, True, "2026-11-02") - write_experiment_design(design, output / "experiment_design.csv") + """Make the output folder; the replicates are confirmed in the description (contract §12).""" + output.mkdir(parents=True, exist_ok=True) def test_no_exclude_clears_the_recorded_exclusions(experiment, tmp_path, capsys): diff --git a/tests/test_notebook_cards.py b/tests/test_notebook_cards.py index 1a9c3f94..8943e995 100644 --- a/tests/test_notebook_cards.py +++ b/tests/test_notebook_cards.py @@ -1,9 +1,9 @@ """PP-10: the notebook's cards for calibration, design, recorded inputs, report and lock. The notebook catches up with the command line and API of PP-7, PP-8, PP-8b and PP-8c, with -the same functions: the check writes its report and the design template; pixel sizes are -shown with their source and stated through ``update_calibration``; design answers go -through ``apply_design_edit`` and are saved under the lock; the run step shows the mode +the same functions: the check writes its report; pixel sizes are shown with their source and +stated through ``update_calibration``; the pairing is a draft answer (SWEEP-PP); the run step +shows the mode and the input changes; a run that stops says what it wrote; a stale lock is removed only on an explicit click, never one of a process still running here. """ diff --git a/tests/test_notebook_check_save.py b/tests/test_notebook_check_save.py index 6598603e..58e3bdf8 100644 --- a/tests/test_notebook_check_save.py +++ b/tests/test_notebook_check_save.py @@ -131,7 +131,7 @@ def test_a_described_run_retires_the_labels_toggle_the_comparable_questions_and_ root, masks, out = exp nb = opened(root, masks, out) assert nb.labels_confirmed.value and not ui.is_visible(nb.labels_confirmed.widget) - assert ui.is_visible(nb.design_card) and not ui.is_visible(nb.groups_box) + assert ui.is_visible(nb.design_card) assert not ui.is_visible(nb.ids_confirmed.widget) assert not ui.is_visible(nb.image_type_card) and nb.confirmed_image_type is None assert "Optional" not in nb.references_card.children[1].value @@ -288,7 +288,7 @@ def test_the_draft_carries_the_pairing_across_a_reopen(tmp_path): dropdown.value = "rep2" assert json.loads((out / DRAFT_FILENAME).read_text())["pairing"] == {"drug": {"day2": "rep2"}} again = opened(root, masks, out) - assert again.design.repeat_groups()[("drug", "day2")] == "rep2" + assert again.pairing[("drug", "day2")] == "rep2" assert not (out / DESIGN_FILENAME).exists() diff --git a/tests/test_notebook_description.py b/tests/test_notebook_description.py index 0573574d..dbdab17c 100644 --- a/tests/test_notebook_description.py +++ b/tests/test_notebook_description.py @@ -170,14 +170,12 @@ def test_two_differently_named_folders_are_paired_into_one_repeat(root, out): def test_a_reference_is_chosen_saved_and_marks_the_control(root, out): nb = labelled(root, out) assert nb.reference_boxes["drug"].value is None # nothing is chosen for the user - assert not ui.is_visible(nb.controls_box) # the design card's tick boxes are replaced nb.reference_boxes["drug"].value = "ctrl" assert dict(nb.draft.references) == {"drug": "ctrl"} assert saved(out).conditions["drug"].reference == "ctrl" - assert nb.design.controls == ("ctrl",) # experiment_design.csv follows the references assert "the reference of drug" in nb.references_grid.children[1].children[2].value nb.reference_boxes["drug"].value = None - assert saved(out).conditions == {} and nb.design.controls == () + assert saved(out).conditions == {} def test_a_loop_of_references_is_refused_at_once(root, out): @@ -300,7 +298,7 @@ def test_an_unanswered_reference_and_no_comparison_look_different(root, out): assert box.value is None and not (out / DRAFT_FILENAME).exists() box.value = "" # the user answered: compared against nothing assert box.value == "" and saved(out).conditions["drug"].reference == "" - assert dict(nb.draft.references) == {} and nb.design.controls == () + assert dict(nb.draft.references) == {} again = labelled(root, out) assert again.reference_boxes["drug"].value == "" # resumed as answered, not as unanswered assert again.reference_boxes["ctrl"].value is None diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index c5d97cb1..5aef2e52 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -11,7 +11,7 @@ from morphobricks_normalization.csv_io import read_plane_histograms from described_tree import describe_tree # noqa: E402 -from morphobricks_preprocessing import Plane, notebook_ui, read_experiment_design, read_manifest +from morphobricks_preprocessing import Plane, notebook_ui, read_manifest from morphobricks_preprocessing import notebook_style as ui # noqa: E402 from morphobricks_preprocessing.axes import AxisMapping # noqa: E402 from morphobricks_preprocessing.notebook_ui import ( # noqa: E402 @@ -48,13 +48,6 @@ def checked(experiment, **kwargs): return nb -def confirm_repeats(nb): - """Answer every experiment-design question of step 1 with yes.""" - for toggle in nb.group_confirmations.values(): - if not toggle.value: - toggle.button.click() - - def confirmed(experiment, **kwargs): """Check the described folder and confirm what step 3 still needs: the axis roles.""" return roles_confirmed(experiment, **kwargs) @@ -190,7 +183,6 @@ def test_different_channel_counts_are_an_error_in_the_consistency_panel(experime assert "different channel counts" not in nb.folder_issues.value assert "Action required" in nb.folder_status.value # Confirming the axis roles does not lift it: the run stays locked. - confirm_repeats(nb) nb.preview_button.click() nb.preview_checked.button.click() nb.confirm_button.click() @@ -222,24 +214,11 @@ def labelled(root, tmp_path): return nb -def test_without_repeat_folders_one_question_is_asked(tmp_path): +def test_without_repeat_folders_nothing_is_paired(tmp_path): root = design_tree(tmp_path / "exp", ["ctrl", "drug"]) nb = labelled(root, tmp_path) assert not ui.is_visible(nb.pairing_box) - (toggle,) = nb.group_confirmations.values() - assert "All images of every condition come from one biological repeat" in toggle.prompt.value - toggle.button.click() - assert nb.design.confirmed - - -def spreadsheet_saved(path): - """Rewrite the ISO dates of a saved design as a spreadsheet program may (26/10/2026).""" - text = path.read_text(encoding="utf-8") - for on in {row.confirmed_on for row in read_experiment_design(path).rows} - {""}: - year, month, day = on.split("-") - text = text.replace(on, f"{day}/{month}/{year}") - path.write_text(text, encoding="utf-8") - return path.read_bytes() + assert nb.pairing == {("ctrl", "single"): "single", ("drug", "single"): "single"} def test_changing_the_folder_resets_later_steps(experiment): @@ -607,7 +586,6 @@ def test_folder_step_with_its_own_words_and_no_output_card(experiment): # The check works without the card, with the default output folder. nb.check_button.click() assert nb.preflight is not None and nb.preflight.ok - confirm_repeats(nb) # embedded steps keep the experiment-design gating (P3) assert "Go to Step 2" in nb.folder_next.value and not ui.is_visible(nb.folder_next) @@ -770,7 +748,6 @@ def test_plan_names_images_read_in_blocks(tmp_path, monkeypatch): def fail(*args, **kwargs): raise AssertionError("pixel data was read") - confirm_repeats(nb) nb.preview_button.click() nb.preview_checked.button.click() for cls in (tifffile.TiffFile, tifffile.TiffPage, tifffile.TiffFrame, tifffile.TiffPageSeries): @@ -821,7 +798,6 @@ def spy(path, *args, **kwargs): nb = checked(root) nb.recommended.value = False assert calls == [] # nothing is opened before the axis roles are confirmed - confirm_repeats(nb) nb.preview_button.click() nb.preview_checked.button.click() nb.confirm_button.click() @@ -845,7 +821,6 @@ def recomputed(): nb.confirm_button.click() assert recomputed() == 0 # the same roles and settings as before nb.check_button.click() - confirm_repeats(nb) nb.preview_button.click() nb.preview_checked.button.click() nb.confirm_button.click() @@ -1009,7 +984,6 @@ def test_statistics_need_a_table_and_valid_percentiles(experiment): nb.csv_file.value = str(experiment / "missing.csv") nb.show_statistics() assert "does not exist" in nb.stats_status.value - confirm_repeats(nb) nb.preview_button.click() nb.preview_checked.button.click() nb.confirm_button.click() diff --git a/tests/test_pp10_api.py b/tests/test_pp10_api.py index 311341ec..f20527ab 100644 --- a/tests/test_pp10_api.py +++ b/tests/test_pp10_api.py @@ -6,8 +6,6 @@ from described_tree import run_described from morphobricks_preprocessing import prefill_mapping, preflight from morphobricks_preprocessing.calibration import Calibration, calibration_entry -from morphobricks_preprocessing.design import default_design -from morphobricks_preprocessing.design_edit import DesignEdit, DesignEditError, apply_design_edit from morphobricks_preprocessing.run import write_calibration from morphobricks_preprocessing.run.check_step import suggested_review from morphobricks_preprocessing.run.planned import planned_changes @@ -26,25 +24,6 @@ def experiment(tmp_path): return root -def test_unconfirm_takes_a_confirmation_back_and_refuses_contradictions(experiment): - pre = preflight(experiment, experiment.parent / "out") - design = apply_design_edit( - default_design(pre.dataset.repeats), DesignEdit(confirm_all=True), "2026-11-02" - ) - assert design.confirmed - undone = apply_design_edit(design, DesignEdit(unconfirm=("rep1",)), "2026-11-02") - assert not undone.group_confirmed("rep1") and undone.group_confirmed("rep2") - with pytest.raises(DesignEditError, match="unknown repeat group 'nope'"): - apply_design_edit(design, DesignEdit(unconfirm=("nope",)), "2026-11-02") - with pytest.raises(DesignEditError, match="cannot be confirmed and unconfirmed"): - apply_design_edit(design, DesignEdit(confirm=("rep1",), unconfirm=("rep1",)), "2026-11-02") - with pytest.raises(DesignEditError, match="cannot be confirmed and unconfirmed"): - apply_design_edit(design, DesignEdit(unconfirm=("rep1",), confirm_all=True), "2026-11-02") - # Taking back a confirmation never confirms anything. - fresh = default_design(pre.dataset.repeats) - assert not apply_design_edit(fresh, DesignEdit(unconfirm=fresh.groups), "2026-11-02").confirmed - - def test_recalibrated_reads_calibration_again_and_replaces_its_issues(experiment, tmp_path): output = tmp_path / "out" pre = preflight(experiment, output) diff --git a/tests/test_publication.py b/tests/test_publication.py index fb7d40a0..123c1e2b 100644 --- a/tests/test_publication.py +++ b/tests/test_publication.py @@ -76,8 +76,7 @@ def consistent(out): """Return the generation's table as RunInputs reads it, checking it is one generation.""" inputs = RunInputs.open(out) manifest = inputs.manifest - design = inputs.design() - assert design is not None + assert manifest.experiment_design_sha256 is not None return manifest.csv_sha256, inputs.histograms() @@ -388,5 +387,5 @@ def fresh(name): def consistent_at(out, root): inputs = RunInputs.open(out, experiment_root=root) - assert inputs.design() is not None + assert inputs.manifest.experiment_design_sha256 is not None return inputs.manifest.csv_sha256, inputs.histograms() diff --git a/tests/test_published_answers.py b/tests/test_published_answers.py index adfdc0e2..3b731ca6 100644 --- a/tests/test_published_answers.py +++ b/tests/test_published_answers.py @@ -190,7 +190,7 @@ def test_the_csv_changes_are_listed_by_row(exp): def test_the_candidate_is_checked_on_its_exact_bytes_and_waits_for_the_roles(exp, tmp_path): _, published = described(exp) - check = check_candidate(published, preflight(exp), None, tmp_path / "out") + check = check_candidate(published, preflight(exp), None) assert check.can_save and not check.physical_run and not check.can_run assert check.changes == ("written for the first time (4 rows)",) diff --git a/tests/test_run_inputs.py b/tests/test_run_inputs.py index a1476d1d..e9631de1 100644 --- a/tests/test_run_inputs.py +++ b/tests/test_run_inputs.py @@ -66,14 +66,14 @@ def test_masks_by_the_recorded_pairing(clean): inputs.read_mask("drug/rep1/b.ome.tif", "Nuclei") -def test_histograms_design_and_image_type(clean): +def test_histograms_and_image_type(clean): inputs = RunInputs.open(clean.output) assert len(inputs.histograms()) == 4 assert {r.source_relative_path for r in inputs.histograms()} == { f.relative_path for f in inputs.fovs() } - # The design derived from the description (an output of the run, v0.5.1). - assert inputs.design().confirmed + # The derived experiment_design.csv is an output only: RunInputs does not read it back. + assert not hasattr(inputs, "design") # The image type is each unit's declaration (PP-1): no run-wide value hides it. assert not hasattr(inputs, "image_type") assert {f.image_type for f in inputs.fovs()} == {"channels"} @@ -200,21 +200,19 @@ def mask_of(image): return image.replace(".ome.tif", ".tif") -def test_the_design_of_a_run_with_one(tmp_path): +def test_the_repeat_groups_come_from_the_description(tmp_path): clean = make_clean_run(tmp_path, design=True) - design = RunInputs.open(clean.output).design() - assert design.repeat_groups() == {("ctrl", "rep1"): "rep1", ("drug", "rep1"): "rep1"} - assert {row.condition: row.control for row in design.rows} == {"ctrl": True, "drug": False} - assert all(row.confirmed for row in design.rows) + fovs = RunInputs.open(clean.output).fovs() + assert {(f.key[0], f.key[1], f.repeat_group) for f in fovs} == { + ("ctrl", "rep1", "rep1"), + ("drug", "rep1", "rep1"), + } def test_a_changed_or_missing_design_is_refused(tmp_path): clean = make_clean_run(tmp_path, design=True) path = clean.output / DESIGN_FILENAME - inputs = RunInputs.open(clean.output) path.write_text(path.read_text(encoding="utf-8").replace("true", "false"), encoding="utf-8") - with pytest.raises(RunInputsError, match="design"): - inputs.design() with pytest.raises(RunInputsError, match="design"): RunInputs.open(clean.output) path.unlink() @@ -222,26 +220,6 @@ def test_a_changed_or_missing_design_is_refused(tmp_path): RunInputs.open(clean.output) -def test_the_design_is_read_with_the_strict_reader(tmp_path): - # Issue #83: RunInputs.design() used the lenient reader, so a design the strict - # reader refuses was given to bricks once its SHA-256 matched the manifest. - from morphobricks_preprocessing.manifest import file_sha256 - - clean = make_clean_run(tmp_path, design=True) - path = clean.output / DESIGN_FILENAME - text = path.read_text(encoding="utf-8") - header, rest = text.split("\n", 1) - path.write_text(f"{header},repeat_group\n" + rest.replace(",1\n", ",1,other\n"), "utf-8") - manifest_path = clean.output / MANIFEST_FILENAME - manifest = json.loads(manifest_path.read_text(encoding="utf-8")) - manifest["experiment_design_sha256"] = file_sha256(path) - manifest_path.write_text(json.dumps(manifest), encoding="utf-8") - inputs = RunInputs.open(clean.output) - with pytest.raises(RunInputsError, match="'repeat_group' twice") as caught: - inputs.design() - assert caught.value.__cause__ is not None - - def test_a_header_naming_x_before_y_is_read_y_before_x(tmp_path): # Owner rule (design v0.4 change log): the header names the axes, so reordering is exact. stored = np.arange(2 * 2 * 5 * 7, dtype=np.uint16).reshape(2, 2, 5, 7) # T, C, X, Y diff --git a/tests/test_spec_derived_design.py b/tests/test_spec_derived_design.py index bd75f44f..546a5932 100644 --- a/tests/test_spec_derived_design.py +++ b/tests/test_spec_derived_design.py @@ -12,7 +12,7 @@ import pytest from frozen_clock import FrozenClock -from morphobricks_preprocessing import discover_experiment, read_confirmed_design +from morphobricks_preprocessing import discover_experiment, parse_design_bytes from morphobricks_preprocessing.description import description_from_bytes from morphobricks_preprocessing.design_derive import derive_design, write_derived_design from morphobricks_preprocessing.draft import migrate_legacy_design @@ -53,10 +53,10 @@ def test_the_derived_file_equals_the_oracles_on_every_valid_fixture(item, tmp_pa @pytest.mark.parametrize("item", VALID.values(), ids=lambda item: item.name.split("_")[0]) -def test_the_derived_file_is_read_back_by_the_strict_reader(item, tmp_path): +def test_the_derived_file_is_read_back_by_the_strict_parser(item, tmp_path): description = description_from_bytes(item.data) path = write_derived_design(description, tmp_path / "design.csv") - design = read_confirmed_design(path).design + design = parse_design_bytes(path.read_bytes()) assert design == derive_design(description) assert all(row.confirmed for row in design.rows) diff --git a/tools/code_rules_baseline.json b/tools/code_rules_baseline.json index 3f4d1ca8..7596a5f5 100644 --- a/tools/code_rules_baseline.json +++ b/tools/code_rules_baseline.json @@ -14,7 +14,6 @@ "src/morphobricks_preprocessing/dataset.py _unbalanced_issues: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/design.py _check_strict_values: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/design.py _parse_row: private function of # lines without a docstring": 36, - "src/morphobricks_preprocessing/design.py: # lines (max #)": 926, "src/morphobricks_preprocessing/discovery.py _check_names: private function of # lines without a docstring": 14, "src/morphobricks_preprocessing/discovery.py finish: # executable lines (max #)": 145, "src/morphobricks_preprocessing/discovery.py: # lines (max #)": 609, @@ -37,7 +36,7 @@ "tests/test_discovery.py: # lines (max #)": 826, "tests/test_masks.py: # lines (max #)": 612, "tests/test_metadata.py: # lines (max #)": 629, - "tests/test_notebook_ui.py: # lines (max #)": 1159, + "tests/test_notebook_ui.py: # lines (max #)": 1133, "tests/test_planes.py: # lines (max #)": 818, "tests/test_run.py: # lines (max #)": 860, "tests/test_zarr_input.py: # lines (max #)": 850,