Experiment design: one strict reader for other bricks (v0.3.4)

AGENTS.md

@@ -75,7 +75,8 @@ src/morphobricks_preprocessing/
   dataset.py        dataset consistency from the folders: biological repeats per condition, balance
                     table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2)
   design.py         experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with
-                    the folders on reruns, design_issues, read/write experiment_design.csv, load_design
+                    the folders on reruns, design_issues, read/write experiment_design.csv, load_design;
+                    read_confirmed_design/ConfirmedDesign (the strict reader other bricks use, §16)
   image_type.py     image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from
                     TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the
                     brick's check_rgb), image_type_issues (rgb_not_possible)
@@ -398,7 +399,10 @@ functions the tests use.
   get one whole-experiment question. Controls are recorded only.
 - **`experiment_design.csv`** (schema 1): one row per condition x biological-repeat folder,
   sorted; counts at confirmation; `confirmed_on` ISO date. Lenient reader (any-case
-  booleans, BOM, empty group = folder name); `DesignFormatError` otherwise.
+  booleans, BOM); `DesignFormatError` otherwise, also for a repeat group that is not a
+  D-e identity or a `confirmed_on` that is not `YYYY-MM-DD` (naming line and column).
+  Other bricks read it with the strict `read_confirmed_design` (contract §16,
+  question 40): nothing filled in, every group confirmed, SHA-256 of the bytes parsed.
 - **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that
   gain a folder, drops removed folders.
 - **Notebook** writes the file after every answer once the labels are confirmed; a

docs/contract.md

@@ -258,7 +258,7 @@ folder.
 | Confirmation | Per repeat group: "Were these images prepared and acquired under the same circumstances, so they are comparable with each other?". A group is confirmed when all its rows are. Without biological-repeat folders (every condition `single`), the whole experiment is one group and the question is asked once (N31). Missing technical-repeat folders need no question. |
 | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. |
 | Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. |
-| Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; an empty `repeat_group` means the folder name; a byte-order mark is ignored. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. |
+| Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `<NA>` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). |
 | Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. |
 | Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1. Confirmed rows without a date get the date of the run. |
 | Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. |
@@ -413,4 +413,5 @@ instead of keeping copies.
 | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. |
 | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). |
 | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. The folder check's `different_pixel_sizes` uses both (§11). |
+| Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order; one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). |
 

docs/plan.md

@@ -958,3 +958,24 @@ Raised during review; to be decided before the milestones that need them:
     names differing only in case anywhere in the experiment; pixel sizes are compared
     within 0.1 % instead of after rounding. The normalization brick stays at v0.3.2. The
     version in `pyproject.toml` is 0.3.3 before tagging. Every brick pins this tag (D79).*
+40. Which experiment-design reader do the other bricks use (core fixes for v0.3.4)?
+    *Decided (31 October 2026): one strict reader here, `read_confirmed_design(path,
+    expected_sha256=)` returning `ConfirmedDesign` (contract §16), replacing CellPatcher's
+    `load_experiment_design` and MorphoFeature's `ExperimentDesign.read`. Both copies ran
+    this brick's lenient `read_experiment_design` and `design_issues` and compared the
+    file's SHA-256 with the manifest's; where they differed, the stricter behaviour is
+    kept: CellPatcher's error codes and its separate `missing_experiment_design`, and the
+    read-only repeat groups of MorphoFeature. Beyond both, nothing is filled in or
+    converted (no outer whitespace trimmed, `true`/`false` only, no empty repeat group or
+    count, no blank line), the CSV structure is checked (exact header, one value per
+    column, quotes closed), labels follow D-e, an empty design is refused, and the SHA-256
+    is of the bytes parsed. Every file this brick writes is read; a file edited in a
+    spreadsheet is read once a preprocessing run has written it back.
+    `read_experiment_design` stays lenient about the form of the user's own edits in step
+    1 and the CLI, but (owner decision, 31 October 2026) repeat-group names follow D-e
+    (no NA-like or empty value, no outer whitespace, no two names differing only in case;
+    an empty group no longer means the folder name) and `confirmed_on` is empty or
+    `YYYY-MM-DD`, in both readers; a bad edit is refused naming the line and column.
+    `design_issues` reports such values in a design built in code
+    (`invalid_experiment_design`), and the writer refuses them, so the brick only ever
+    writes ISO dates.*

docs/user_guide.md

@@ -573,6 +573,9 @@ continues.
 | `design_not_confirmed` | Error | Some repeat groups are not confirmed as comparable. Confirm them in step 1, or set `confirmed` to `true` in `experiment_design.csv`. |
 | `rgb_not_possible` | Error | The image type RGB was chosen, but the images are not three channels of a whole-number pixel type. Choose fluorescence, or check the axis roles: the RGB samples must have the role C. |
 | `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another group. |
+| `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, `<NA>`, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. Correct `experiment_design.csv` (the message names the line and column). |
+| `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. |
+| `changed_experiment_design` | Error (other bricks) | `experiment_design.csv` is not the file the preprocessing run that wrote the statistics used (its SHA-256 differs from the one in the run manifest). Give that run's design, or run the preprocessing brick again. |
 | `image_intensity_outlier` | Warning (intensity check) | An image's median or high percentile in a channel is far from the other images of its condition and biological repeat. Look at it: it may be out of focus, contain debris or bubbles, or have been acquired differently. Exclude it if it is faulty. |
 | `repeat_intensity_shift` | Warning (intensity check) | A biological repeat is brighter or darker than the other repeats of its condition. Check whether it was prepared or acquired differently; normalising per biological repeat in later bricks removes such differences. |
 | `saturated_image` | Warning (intensity check) | More than the limit (default 1 %) of an image's pixels in a channel are at the largest value of its pixel type, so their true intensity is lost. Check the exposure, gain or laser power. |

src/morphobricks_preprocessing/__init__.py

@@ -40,13 +40,16 @@
     DESIGN_COLUMNS,
     DESIGN_FILENAME,
     DESIGN_SCHEMA_VERSION,
+    ConfirmedDesign,
     DesignChanges,
+    DesignCheckError,
     DesignFormatError,
     DesignRow,
     ExperimentDesign,
     default_design,
     design_issues,
     load_design,
+    read_confirmed_design,
     read_experiment_design,
     reconcile_design,
     write_experiment_design,
@@ -235,10 +238,12 @@ def __dir__() -> list[str]:
     "BiologicalRepeat",
     "Block",
     "BlockReader",
+    "ConfirmedDesign",
     "ConversionEstimate",
     "ConversionSettings",
     "DatasetReport",
     "DesignChanges",
+    "DesignCheckError",
     "DesignFormatError",
     "DesignRow",
     "DiscoveryResult",
@@ -350,6 +355,7 @@ def __dir__() -> list[str]:
     "prefill_experiment_mapping",
     "prefill_mapping",
     "preflight",
+    "read_confirmed_design",
     "read_experiment_design",
     "read_manifest",
     "read_mask",

src/morphobricks_preprocessing/cli.py

@@ -552,7 +552,10 @@ def _design_for_run(pre, args: argparse.Namespace) -> tuple[ExperimentDesign | N
         "or a text editor, one row per biological-repeat folder of each condition:",
         "  - repeat_group: rows with the same value are the same biological repeat across "
         "conditions (by default the folder name). Change it to pair repeats whose folders "
-        "are named differently; each group may hold one folder per condition.",
+        "are named differently; each group may hold one folder per condition. It must not "
+        "be empty or NA.",
+        "  - confirmed_on: the date of the confirmation as YYYY-MM-DD, or empty (the run "
+        "fills it in).",
         "  - confirmed: set it to true in every row of a repeat group once you are sure its "
         "images were prepared and acquired under the same circumstances, so they are "
         "comparable.",