diff --git a/AGENTS.md b/AGENTS.md index fc651204..87fb90c9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,8 @@ src/morphobricks_preprocessing/ dataset.py dataset consistency from the folders: biological repeats per condition, balance table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2) design.py experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with - the folders on reruns, design_issues, read/write experiment_design.csv, load_design + the folders on reruns, design_issues, read/write experiment_design.csv, load_design; + read_confirmed_design/ConfirmedDesign (the strict reader other bricks use, §16) image_type.py image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the brick's check_rgb), image_type_issues (rgb_not_possible) @@ -398,7 +399,10 @@ functions the tests use. get one whole-experiment question. Controls are recorded only. - **`experiment_design.csv`** (schema 1): one row per condition x biological-repeat folder, sorted; counts at confirmation; `confirmed_on` ISO date. Lenient reader (any-case - booleans, BOM, empty group = folder name); `DesignFormatError` otherwise. + booleans, BOM); `DesignFormatError` otherwise, also for a repeat group that is not a + D-e identity or a `confirmed_on` that is not `YYYY-MM-DD` (naming line and column). + Other bricks read it with the strict `read_confirmed_design` (contract §16, + question 40): nothing filled in, every group confirmed, SHA-256 of the bytes parsed. - **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that gain a folder, drops removed folders. - **Notebook** writes the file after every answer once the labels are confirmed; a diff --git a/docs/contract.md b/docs/contract.md index db878017..9fe9969a 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -258,7 +258,7 @@ folder. | Confirmation | Per repeat group: "Were these images prepared and acquired under the same circumstances, so they are comparable with each other?". A group is confirmed when all its rows are. Without biological-repeat folders (every condition `single`), the whole experiment is one group and the question is asked once (N31). Missing technical-repeat folders need no question. | | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | | Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. | -| Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; an empty `repeat_group` means the folder name; a byte-order mark is ignored. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. | +| Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). | | Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | | Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1. Confirmed rows without a date get the date of the run. | | Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. | @@ -413,4 +413,5 @@ instead of keeping copies. | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. The folder check's `different_pixel_sizes` uses both (§11). | +| Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order; one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). | diff --git a/docs/plan.md b/docs/plan.md index eba3aa4b..810dfdf2 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -958,3 +958,24 @@ Raised during review; to be decided before the milestones that need them: names differing only in case anywhere in the experiment; pixel sizes are compared within 0.1 % instead of after rounding. The normalization brick stays at v0.3.2. The version in `pyproject.toml` is 0.3.3 before tagging. Every brick pins this tag (D79).* +40. Which experiment-design reader do the other bricks use (core fixes for v0.3.4)? + *Decided (31 October 2026): one strict reader here, `read_confirmed_design(path, + expected_sha256=)` returning `ConfirmedDesign` (contract §16), replacing CellPatcher's + `load_experiment_design` and MorphoFeature's `ExperimentDesign.read`. Both copies ran + this brick's lenient `read_experiment_design` and `design_issues` and compared the + file's SHA-256 with the manifest's; where they differed, the stricter behaviour is + kept: CellPatcher's error codes and its separate `missing_experiment_design`, and the + read-only repeat groups of MorphoFeature. Beyond both, nothing is filled in or + converted (no outer whitespace trimmed, `true`/`false` only, no empty repeat group or + count, no blank line), the CSV structure is checked (exact header, one value per + column, quotes closed), labels follow D-e, an empty design is refused, and the SHA-256 + is of the bytes parsed. Every file this brick writes is read; a file edited in a + spreadsheet is read once a preprocessing run has written it back. + `read_experiment_design` stays lenient about the form of the user's own edits in step + 1 and the CLI, but (owner decision, 31 October 2026) repeat-group names follow D-e + (no NA-like or empty value, no outer whitespace, no two names differing only in case; + an empty group no longer means the folder name) and `confirmed_on` is empty or + `YYYY-MM-DD`, in both readers; a bad edit is refused naming the line and column. + `design_issues` reports such values in a design built in code + (`invalid_experiment_design`), and the writer refuses them, so the brick only ever + writes ISO dates.* diff --git a/docs/user_guide.md b/docs/user_guide.md index 57daaaf0..ef992f7d 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -573,6 +573,9 @@ continues. | `design_not_confirmed` | Error | Some repeat groups are not confirmed as comparable. Confirm them in step 1, or set `confirmed` to `true` in `experiment_design.csv`. | | `rgb_not_possible` | Error | The image type RGB was chosen, but the images are not three channels of a whole-number pixel type. Choose fluorescence, or check the axis roles: the RGB samples must have the role C. | | `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another group. | +| `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, ``, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. Correct `experiment_design.csv` (the message names the line and column). | +| `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. | +| `changed_experiment_design` | Error (other bricks) | `experiment_design.csv` is not the file the preprocessing run that wrote the statistics used (its SHA-256 differs from the one in the run manifest). Give that run's design, or run the preprocessing brick again. | | `image_intensity_outlier` | Warning (intensity check) | An image's median or high percentile in a channel is far from the other images of its condition and biological repeat. Look at it: it may be out of focus, contain debris or bubbles, or have been acquired differently. Exclude it if it is faulty. | | `repeat_intensity_shift` | Warning (intensity check) | A biological repeat is brighter or darker than the other repeats of its condition. Check whether it was prepared or acquired differently; normalising per biological repeat in later bricks removes such differences. | | `saturated_image` | Warning (intensity check) | More than the limit (default 1 %) of an image's pixels in a channel are at the largest value of its pixel type, so their true intensity is lost. Check the exposure, gain or laser power. | diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 36dfed6d..4c3ab833 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -40,13 +40,16 @@ DESIGN_COLUMNS, DESIGN_FILENAME, DESIGN_SCHEMA_VERSION, + ConfirmedDesign, DesignChanges, + DesignCheckError, DesignFormatError, DesignRow, ExperimentDesign, default_design, design_issues, load_design, + read_confirmed_design, read_experiment_design, reconcile_design, write_experiment_design, @@ -235,10 +238,12 @@ def __dir__() -> list[str]: "BiologicalRepeat", "Block", "BlockReader", + "ConfirmedDesign", "ConversionEstimate", "ConversionSettings", "DatasetReport", "DesignChanges", + "DesignCheckError", "DesignFormatError", "DesignRow", "DiscoveryResult", @@ -350,6 +355,7 @@ def __dir__() -> list[str]: "prefill_experiment_mapping", "prefill_mapping", "preflight", + "read_confirmed_design", "read_experiment_design", "read_manifest", "read_mask", diff --git a/src/morphobricks_preprocessing/cli.py b/src/morphobricks_preprocessing/cli.py index 9b808428..ccf03962 100644 --- a/src/morphobricks_preprocessing/cli.py +++ b/src/morphobricks_preprocessing/cli.py @@ -552,7 +552,10 @@ def _design_for_run(pre, args: argparse.Namespace) -> tuple[ExperimentDesign | N "or a text editor, one row per biological-repeat folder of each condition:", " - repeat_group: rows with the same value are the same biological repeat across " "conditions (by default the folder name). Change it to pair repeats whose folders " - "are named differently; each group may hold one folder per condition.", + "are named differently; each group may hold one folder per condition. It must not " + "be empty or NA.", + " - confirmed_on: the date of the confirmation as YYYY-MM-DD, or empty (the run " + "fills it in).", " - confirmed: set it to true in every row of a repeat group once you are sure its " "images were prepared and acquired under the same circumstances, so they are " "comparable.", diff --git a/src/morphobricks_preprocessing/design.py b/src/morphobricks_preprocessing/design.py index 733739b8..3577d91e 100644 --- a/src/morphobricks_preprocessing/design.py +++ b/src/morphobricks_preprocessing/design.py @@ -10,23 +10,29 @@ The answers are kept in ``experiment_design.csv`` in the output folder (``docs/contract.md`` §12), so a rerun with the same folders does not ask -again. Later bricks read the pairing with :func:`read_experiment_design`. -Nothing here reads images. +again. Later bricks read the pairing with :func:`read_confirmed_design`, the one +strict reader every brick uses (contract §16); :func:`read_experiment_design` is +this brick's lenient reader of a file the user may still be editing. Nothing here +reads images. """ from __future__ import annotations import csv +import datetime +import hashlib import io import os import tempfile from collections.abc import Iterable, Mapping from dataclasses import dataclass, replace from pathlib import Path +from types import MappingProxyType from morphobricks_normalization.records import SINGLE from .dataset import BiologicalRepeat +from .discovery import IdentityError, check_identity_case, check_identity_text from .validation import ValidationIssue, error, plural DESIGN_FILENAME = "experiment_design.csv" @@ -43,6 +49,9 @@ "schema_version", ) +DATE_FORMAT = "YYYY-MM-DD" +"""The form of ``confirmed_on``: an ISO 8601 calendar date, e.g. ``2026-10-26``.""" + _TRUE = frozenset({"true", "yes", "1"}) _FALSE = frozenset({"false", "no", "0", ""}) @@ -51,6 +60,23 @@ class DesignFormatError(ValueError): """Raised when a file is not an experiment design this version can read.""" +class DesignCheckError(ValueError): + """Raised by :func:`read_confirmed_design` when a design cannot be used. + + Attributes + ---------- + issues + The errors, each with its code (``missing_experiment_design``, + ``unreadable_experiment_design``, ``repeat_group_conflict``, + ``design_not_confirmed`` or ``changed_experiment_design``), message, path and + suggested action. The exception's text joins their messages. + """ + + def __init__(self, issues: Iterable[ValidationIssue]) -> None: + self.issues = tuple(issues) + super().__init__(" ".join(f"{issue.code}: {issue.message}" for issue in self.issues)) + + @dataclass(frozen=True) class DesignRow: """One biological-repeat folder of a condition, and the answers about it. @@ -280,6 +306,64 @@ def dated(self, on: str) -> ExperimentDesign: ) +@dataclass(frozen=True) +class ConfirmedDesign: + """An ``experiment_design.csv`` checked for use by another brick. + + Build it with :func:`read_confirmed_design`. Every repeat group is confirmed and holds + at most one folder per condition. + + Attributes + ---------- + path + The file that was read. + sha256 + SHA-256 of exactly the bytes that were parsed (the manifest's + ``experiment_design_sha256``). + design + The design itself. + repeat_groups + Read-only ``{(condition, biological_repeat): repeat_group}``, the pairing the + normalization brick's ``biological_repeat`` scope takes. + """ + + path: Path + sha256: str + design: ExperimentDesign + repeat_groups: Mapping[tuple[str, str], str] + + @property + def conditions(self) -> tuple[str, ...]: + """Condition labels, sorted.""" + return self.design.conditions + + @property + def controls(self) -> tuple[str, ...]: + """Conditions marked as controls, sorted (recorded only, contract §12).""" + return self.design.controls + + @property + def groups(self) -> tuple[str, ...]: + """Repeat-group labels, sorted.""" + return self.design.groups + + def group_members(self) -> Mapping[str, Mapping[str, str]]: + """Return ``{repeat_group: {condition: biological_repeat}}``, read-only. + + One biological-repeat folder per condition and group, which the design's checks + guarantee. + """ + members: dict[str, dict[str, str]] = {} + for (condition, repeat), group in self.repeat_groups.items(): + members.setdefault(group, {})[condition] = repeat + return MappingProxyType( + { + group: MappingProxyType(dict(sorted(m.items()))) + for group, m in sorted(members.items()) + } + ) + + def default_design(repeats: Iterable[BiologicalRepeat]) -> ExperimentDesign: """Build the design before any answer: repeats with the same folder name are one group. @@ -349,11 +433,27 @@ def design_issues(design: ExperimentDesign) -> list[ValidationIssue]: Returns ------- list of ValidationIssue - ``repeat_group_conflict`` for a group with two folders of one - condition and ``design_not_confirmed`` for groups not confirmed yet; - both are errors. + ``invalid_experiment_design`` for a repeat-group name that is not an identity + (decision D-e: NA-like, empty, or with outer whitespace) or a ``confirmed_on`` that + is not empty or a date in :data:`DATE_FORMAT`; ``repeat_group_conflict`` for a + group with two folders of one condition; ``design_not_confirmed`` for groups not + confirmed yet. All are errors. """ issues = [] + invalid = _design_problems(design) + if invalid: + issues.append( + error( + "invalid_experiment_design", + "The experiment design has values that cannot be used: " + + "; ".join(f"{_folder(row)}: {problem}" for row, problem in invalid) + + ".", + sorted({_folder(row) for row, _ in invalid}), + "Give every repeat group a name (not NA, nan, none, null, n/a or empty, " + "without spaces around it, spelled the same way in every row) and every " + f"confirmation date as {DATE_FORMAT}.", + ) + ) if design.conflicts: described = "; ".join( f"{group} has more than one folder of {', '.join(conditions)}" @@ -396,15 +496,19 @@ def read_experiment_design(path: str | os.PathLike) -> ExperimentDesign: Labels are read as text. ``control`` and ``confirmed`` accept ``true`` or ``false`` in any case (also ``yes``/``no``, ``1``/``0``; empty is false), so - a file saved by a spreadsheet program still reads. An empty - ``repeat_group`` means the group named after the folder. + a file saved by a spreadsheet program still reads. The user's own edits are + checked (owner decision, 31 October 2026): ``repeat_group`` follows the identity + rules of decision D-e (not empty, ``NA``, ``nan``, ``none``, ``null``, ``n/a`` or + ```` in any case, no leading or trailing whitespace), and ``confirmed_on`` is + empty or a date in :data:`DATE_FORMAT`; the error names the line and column. Raises ------ DesignFormatError - If the file cannot be read, a column is missing, a value is not valid, - a folder is listed twice, the rows of a condition disagree about - ``control``, or the schema version is not 1. + If the file cannot be read, a column is missing, a value is not valid + (including a repeat-group name or confirmation date as above), a folder is + listed twice, the rows of a condition disagree about ``control``, or the + schema version is not 1. """ path = Path(path) try: @@ -419,10 +523,13 @@ def read_experiment_design(path: str | os.PathLike) -> ExperimentDesign: "are missing." ) rows: list[DesignRow] = [] + numbers: list[int] = [] for number, record in enumerate(reader, start=2): if not any((value or "").strip() for value in record.values() if isinstance(value, str)): continue rows.append(_parse_row(record, f"{path.name}, line {number}")) + numbers.append(number) + _check_group_case(zip(numbers, rows, strict=True), path.name) keys = [row.key for row in rows] twice = sorted({key for key in keys if keys.count(key) > 1}) if twice: @@ -438,13 +545,136 @@ def read_experiment_design(path: str | os.PathLike) -> ExperimentDesign: return ExperimentDesign(tuple(rows)) +def read_confirmed_design( + path: str | os.PathLike, *, expected_sha256: str | None = None +) -> ConfirmedDesign: + """Read ``experiment_design.csv`` for use by another brick, refusing anything doubtful. + + The one reader of the experiment design for every MorphoBricks brick (contract §16), + replacing CellPatcher's ``load_experiment_design`` and MorphoFeature's + ``ExperimentDesign.read``. It takes the file as this brick writes it + (:func:`write_experiment_design`) and fills nothing in, where + :func:`read_experiment_design`, for the user's own edits in step 1, is lenient about + the file's form: + + - the file is UTF-8 without a byte-order mark, its header is exactly + :data:`DESIGN_COLUMNS` in that order, and every row has one value per column + (an unclosed quote, a missing or extra value, or a blank line is refused); + - no value has leading or trailing whitespace; ``control`` and ``confirmed`` are + ``true`` or ``false``; ``image_count`` and ``technical_repeat_count`` are whole + numbers of at least 0 written as such; ``confirmed_on`` is empty or a date in + :data:`DATE_FORMAT` (as :func:`read_experiment_design` requires too); + ``schema_version`` is ``1``; + - ``condition``, ``biological_repeat`` and ``repeat_group`` pass + :func:`.discovery.check_identity_text` (the repeat group as + :func:`read_experiment_design` requires too), and labels of one level differing + only in case are refused (:func:`.discovery.check_identity_case`); + - a folder listed twice, rows of a condition disagreeing on ``control``, or a file + without rows are refused; + - every repeat group is confirmed and holds one folder per condition + (:func:`design_issues`); + - with ``expected_sha256`` (a run manifest's ``experiment_design_sha256``), the file + must be that one. + + The SHA-256 is of the bytes that were parsed, so the file cannot change between the + two. + + Parameters + ---------- + path + The ``experiment_design.csv`` file. + expected_sha256 + The SHA-256 the preprocessing run that wrote the statistics recorded, if any. + + Returns + ------- + ConfirmedDesign + + Raises + ------ + DesignCheckError + With the errors ``missing_experiment_design`` (no such file), + ``unreadable_experiment_design`` (not a design this version reads strictly), + ``repeat_group_conflict`` and ``design_not_confirmed`` (:func:`design_issues`), or + ``changed_experiment_design`` (not the file the run recorded). + """ + path = Path(path) + rerun = ( + "Run step 1 of the preprocessing brick on this experiment (it pairs and confirms the " + "biological repeats), then run the preprocessing brick again." + ) + if not path.is_file(): + raise DesignCheckError( + [ + error( + "missing_experiment_design", + f"The experiment design {path} does not exist.", + [str(path)], + rerun, + ) + ] + ) + try: + data = path.read_bytes() + design = _parse_strict(data, path.name) + except (OSError, DesignFormatError) as exc: + raise DesignCheckError( + [ + error( + "unreadable_experiment_design", + f"The experiment design cannot be read: {exc}", + [str(path)], + rerun, + ) + ] + ) from exc + issues = design_issues(design) + if issues: + raise DesignCheckError(issues) + sha256 = hashlib.sha256(data).hexdigest() + if expected_sha256 is not None and sha256 != expected_sha256: + raise DesignCheckError( + [ + error( + "changed_experiment_design", + f"{path.name} is not the experiment design the preprocessing run used: its " + "SHA-256 differs from the one in the run manifest, so the repeat groups may " + "not be the ones the statistics were checked with.", + [str(path)], + "Give the experiment design of that run, or run the preprocessing brick " + "again on this experiment.", + ) + ] + ) + return ConfirmedDesign( + path=path, + sha256=sha256, + design=design, + repeat_groups=MappingProxyType(design.repeat_groups()), + ) + + def write_experiment_design(design: ExperimentDesign, path: str | os.PathLike) -> Path: """Write ``experiment_design.csv``, replacing the file only once writing succeeded. The file is UTF-8 with LF line endings; the same design always gives the - same bytes. + same bytes. Every ``confirmed_on`` it writes is empty or a date in + :data:`DATE_FORMAT`, and every repeat group a valid name. + + Raises + ------ + DesignFormatError + If a row has a repeat-group name or confirmation date the readers refuse + (:func:`design_issues`, ``invalid_experiment_design``); nothing is written. """ path = Path(path) + invalid = _design_problems(design) + if invalid: + raise DesignFormatError( + "The experiment design cannot be written: " + + "; ".join(f"{_folder(row)}: {problem}" for row, problem in invalid) + + "." + ) buffer = io.StringIO() writer = csv.writer(buffer, lineterminator="\n") writer.writerow(DESIGN_COLUMNS) @@ -497,6 +727,82 @@ def load_design( return design, changes, problem +def _parse_strict(data: bytes, name: str) -> ExperimentDesign: + """Parse a design as :func:`write_experiment_design` writes it, refusing anything else.""" + if data.startswith(b"\xef\xbb\xbf"): + raise DesignFormatError( + f"{name} starts with a byte-order mark: it was saved by another program, not " + "written by the preprocessing brick." + ) + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + raise DesignFormatError(f"{name} is not UTF-8 text: {exc}") from None + try: + lines = list(csv.reader(io.StringIO(text, newline=""), strict=True)) + except csv.Error as exc: + raise DesignFormatError(f"{name} is not a well-formed CSV file: {exc}") from None + if not lines or tuple(lines[0]) != DESIGN_COLUMNS: + found = ", ".join(lines[0]) if lines else "nothing" + raise DesignFormatError( + f"{name} must have the columns {', '.join(DESIGN_COLUMNS)} in this order, not {found}." + ) + rows: list[DesignRow] = [] + numbers: list[int] = [] + for number, values in enumerate(lines[1:], start=2): + where = f"{name}, line {number}" + if len(values) != len(DESIGN_COLUMNS): + raise DesignFormatError( + f"{where} has {plural(len(values), 'value')} for " + f"{plural(len(DESIGN_COLUMNS), 'column')}." + ) + record = dict(zip(DESIGN_COLUMNS, values, strict=True)) + _check_strict_values(record, where) + rows.append(_parse_row(record, where)) + numbers.append(number) + if not rows: + raise DesignFormatError(f"{name} lists no folder.") + keys = [row.key for row in rows] + twice = sorted({key for key in keys if keys.count(key) > 1}) + if twice: + raise DesignFormatError( + f"{name} lists a folder more than once: " + "; ".join(f"{c} / {b}" for c, b in twice) + ) + for condition in sorted({row.condition for row in rows}): + if len({row.control for row in rows if row.condition == condition}) > 1: + raise DesignFormatError( + f"{name}: control must be the same in every row of condition {condition}." + ) + try: + check_identity_case((row.condition, row.biological_repeat, SINGLE) for row in rows) + except IdentityError as exc: + raise DesignFormatError(f"{name}: {exc}") from None + _check_group_case(zip(numbers, rows, strict=True), name) + return ExperimentDesign(tuple(rows)) + + +def _check_strict_values(record: Mapping[str, str], where: str) -> None: + for column, value in record.items(): + if value != value.strip(): + raise DesignFormatError(f"{where}: {column} {value!r} starts or ends with whitespace.") + for column in ("condition", "biological_repeat"): + try: + check_identity_text(record[column], column) + except IdentityError as exc: + raise DesignFormatError(f"{where}: {exc}.") from None + for column in ("control", "confirmed"): + if record[column] not in ("true", "false"): + raise DesignFormatError( + f"{where}: {column} must be true or false, not {record[column]!r}." + ) + for column in ("image_count", "technical_repeat_count"): + value = record[column] + if not (value.isascii() and value.isdigit() and str(int(value)) == value): + raise DesignFormatError( + f"{where}: {column} must be a whole number of at least 0, not {value!r}." + ) + + def _new_row(repeat: BiologicalRepeat, group: str) -> DesignRow: return DesignRow( repeat.condition, @@ -519,20 +825,89 @@ def text(column: str) -> str: condition, repeat = text("condition"), text("biological_repeat") if not condition or not repeat: raise DesignFormatError(f"{where}: condition and biological_repeat must not be empty.") + group = record.get("repeat_group") or "" + try: + check_identity_text(group, "repeat_group") + except IdentityError as exc: + raise DesignFormatError( + f"{where}, column repeat_group: {exc}. Give the repeat group a name, for example " + f"the folder name {repeat!r}." + ) from None + on = text("confirmed_on") + if on and not _is_date(on): + raise DesignFormatError( + f"{where}, column confirmed_on: {on!r} is not a date in the form {DATE_FORMAT} " + "(for example 2026-10-26)." + ) return DesignRow( condition=condition, biological_repeat=repeat, - repeat_group=text("repeat_group") or repeat, + repeat_group=group, control=_bool(text("control"), "control", where), image_count=_count(text("image_count"), "image_count", where), technical_repeat_count=_count( text("technical_repeat_count"), "technical_repeat_count", where ), confirmed=_bool(text("confirmed"), "confirmed", where), - confirmed_on=text("confirmed_on"), + confirmed_on=on, ) +def _is_date(value: str) -> bool: + if len(value) != len(DATE_FORMAT) or value[4] != "-" or value[7] != "-": + return False + try: + datetime.date.fromisoformat(value) + except ValueError: + return False + return True + + +def _design_problems(design: ExperimentDesign) -> list[tuple[DesignRow, str]]: + """Return the rows whose repeat group or date the readers refuse, with the reason. + + Each row's own values (:func:`_row_problems`), and a repeat group spelled differently + from an earlier row's only in case (decision D-e: one spelling per name). + """ + problems = [(row, problem) for row in design.rows for problem in _row_problems(row)] + spellings: dict[str, str] = {} + for row in design.rows: + first = spellings.setdefault(row.repeat_group.casefold(), row.repeat_group) + if first != row.repeat_group: + problems.append( + (row, f"repeat_group {row.repeat_group!r} differs from {first!r} only in case") + ) + return problems + + +def _check_group_case(numbered: Iterable[tuple[int, DesignRow]], name: str) -> None: + """Refuse repeat groups whose names differ only in case (decision D-e), naming lines.""" + spellings: dict[str, tuple[int, str]] = {} + for number, row in numbered: + first_line, first = spellings.setdefault( + row.repeat_group.casefold(), (number, row.repeat_group) + ) + if first != row.repeat_group: + raise DesignFormatError( + f"{name}, line {number}, column repeat_group: {row.repeat_group!r} differs " + f"only in case from {first!r} on line {first_line}. Spell a repeat group the " + "same way in every row." + ) + + +def _row_problems(row: DesignRow) -> list[str]: + problems = [] + try: + check_identity_text(row.repeat_group, "repeat_group") + except IdentityError as exc: + problems.append(str(exc)) + if row.confirmed_on and not _is_date(row.confirmed_on): + problems.append( + f"confirmed_on {row.confirmed_on!r} is not a date in the form {DATE_FORMAT}" + ) + return problems + + def _bool(value: str, column: str, where: str) -> bool: lowered = value.lower() if lowered in _TRUE: diff --git a/tests/test_confirmed_design.py b/tests/test_confirmed_design.py new file mode 100644 index 00000000..4143b9b0 --- /dev/null +++ b/tests/test_confirmed_design.py @@ -0,0 +1,233 @@ +"""The shared strict reader of experiment_design.csv (contract §16, plan question 40).""" + +import hashlib +import re + +import pytest + +from morphobricks_preprocessing import ( + DESIGN_FILENAME, + ConfirmedDesign, + DesignCheckError, + biological_repeats, + default_design, + default_output_folder, + discover_experiment, + read_confirmed_design, + read_experiment_design, + read_manifest, + run_experiment, + write_experiment_design, +) +from morphobricks_preprocessing.axes import prefill_mapping +from morphobricks_preprocessing.cli import main +from morphobricks_preprocessing.manifest import file_sha256 +from tiff_tree import make_tree + +TODAY = "2026-10-31" +HEADER = ( + "condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count," + "confirmed,confirmed_on,schema_version\n" +) +GOOD = "ctrl,rep1,rep1,true,2,0,true,2026-10-31,1\ndrug,rep1b,rep1,false,2,1,true,2026-10-31,1\n" + + +def tree(root, counts): + make_tree(root, [f"{folder}/img_{i:02d}.tif" for folder, n in counts.items() for i in range(n)]) + return root + + +def confirm_all(design): + for group in design.groups: + design = design.with_confirmed(group, True, TODAY) + return design + + +def write(tmp_path, text): + path = tmp_path / DESIGN_FILENAME + path.write_bytes(text.encode("utf-8") if isinstance(text, str) else text) + return path + + +def refusal(path, **kwargs): + with pytest.raises(DesignCheckError) as caught: + read_confirmed_design(path, **kwargs) + (issue,) = caught.value.issues + assert issue.is_error and issue.paths == (str(path),) + assert issue.code in str(caught.value) + return issue + + +# What both bricks need --------------------------------------------------------------------- + + +def test_a_design_written_by_a_run_reads_with_everything_both_bricks_need(tmp_path): + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1b": 1, "treated/rep2": 1}) + repeats = biological_repeats(discover_experiment(root)) + design = default_design(repeats).with_group(("treated", "rep1b"), "rep1") + design = confirm_all(design).with_control("control", True) + result = run_experiment(root, prefill_mapping("YX"), design=design) + assert result.written + recorded = read_manifest(result.manifest_path).experiment_design_sha256 + + read = read_confirmed_design(result.design_path, expected_sha256=recorded) + assert isinstance(read, ConfirmedDesign) + assert read.path == result.design_path and read.design == design + assert read.sha256 == recorded == file_sha256(result.design_path) + assert dict(read.repeat_groups) == { + ("control", "rep1"): "rep1", + ("treated", "rep1b"): "rep1", + ("treated", "rep2"): "rep2", + } + assert {g: dict(m) for g, m in read.group_members().items()} == { + "rep1": {"control": "rep1", "treated": "rep1b"}, + "rep2": {"treated": "rep2"}, + } + assert read.conditions == ("control", "treated") and read.controls == ("control",) + assert read.groups == ("rep1", "rep2") + with pytest.raises(TypeError): + read.repeat_groups[("control", "rep1")] = "other" # read-only + + +def test_a_spreadsheet_edit_is_read_strictly_once_the_run_has_written_it(tmp_path, capsys): + # The lenient reader takes the user's own edits; the run writes them back as the brick + # writes every design, and that file is what other bricks read. + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1b": 1}) + output = default_output_folder(root) + output.mkdir(parents=True) + edited = write( + output, + "" + HEADER.replace("\n", "\r\n") + "control ,rep1,rep1,TRUE,1,0,Yes,2026-10-26,1\r\n" + "treated,rep1b,rep1,FALSE,1,0,1,,1\r\n", + ) + with pytest.raises(DesignCheckError, match="byte-order mark"): + read_confirmed_design(edited) + assert main(["run", str(root), "--axes", "YX"]) == 0 + capsys.readouterr() + read = read_confirmed_design( + edited, expected_sha256=read_manifest(output / "run_manifest.json").experiment_design_sha256 + ) + assert dict(read.repeat_groups) == {("control", "rep1"): "rep1", ("treated", "rep1b"): "rep1"} + # A confirmation without a date got the run's date. + assert read.design.row(("treated", "rep1b")).confirmed_on != "" + assert read.design.row(("control", "rep1")).confirmed_on == "2026-10-26" + + +def test_the_hash_is_of_the_bytes_read(tmp_path): + path = write(tmp_path, HEADER + GOOD) + read = read_confirmed_design(path) + assert read.sha256 == hashlib.sha256((HEADER + GOOD).encode()).hexdigest() + # Windows line endings are the same rows (the hash differs, as the bytes do). + crlf = write(tmp_path, (HEADER + GOOD).replace("\n", "\r\n")) + assert read_confirmed_design(crlf).repeat_groups == read.repeat_groups + + +# Refusals ---------------------------------------------------------------------------------- + + +def test_a_missing_file_is_refused(tmp_path): + assert refusal(tmp_path / DESIGN_FILENAME).code == "missing_experiment_design" + assert refusal(tmp_path).code == "missing_experiment_design" # a folder + + +@pytest.mark.parametrize( + ("text", "message"), + [ + # Structure + (b"\xef\xbb\xbf" + (HEADER + GOOD).encode(), "byte-order mark"), + ((HEADER + GOOD).encode("utf-16"), "byte-order mark|not UTF-8"), + ("caf\xe9".encode("latin-1"), "not UTF-8"), + ("", "must have the columns"), + (HEADER.replace("repeat_group,control", "control,repeat_group") + GOOD, "in this order"), + (HEADER.replace("\n", ",notes\n") + GOOD, "in this order"), + (HEADER.replace("schema_version", "condition") + GOOD, "in this order"), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,2026-10-31\n", "8 values for 9 columns"), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,2026-10-31,1,x\n", "10 values for 9 columns"), + (HEADER + '"ctrl,rep1,rep1,true,2,0,true,2026-10-31,1\n', "well-formed CSV"), + (HEADER + GOOD + "\n", "0 values for 9 columns"), + (HEADER, "lists no folder"), + # Values: nothing is trimmed, converted or filled in + (HEADER + "ctrl ,rep1,rep1,true,2,0,true,,1\n", "whitespace"), + (HEADER + "ctrl,rep1, rep1,true,2,0,true,,1\n", "whitespace"), + (HEADER + "ctrl,rep1,rep1,TRUE,2,0,true,,1\n", "control must be true or false"), + (HEADER + "ctrl,rep1,rep1,true,2,0,yes,,1\n", "confirmed must be true or false"), + (HEADER + "ctrl,rep1,rep1,,2,0,true,,1\n", "control must be true or false"), + (HEADER + "ctrl,rep1,,true,2,0,true,,1\n", "column repeat_group: repeat_group '' reads"), + (HEADER + "ctrl,rep1,n/a,true,2,0,true,,1\n", "column repeat_group: repeat_group 'n/a'"), + ( + HEADER + "ctrl,rep1,rep1,true,2,0,true,26/10/2026,1\n", + "line 2, column confirmed_on: '26/10/2026' is not a date in the form YYYY-MM-DD", + ), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,2026-10-1,1\n", "column confirmed_on"), + (HEADER + "ctrl,rep1,rep1,true,,0,true,,1\n", "image_count must be a whole number"), + (HEADER + "ctrl,rep1,rep1,true,+2,0,true,,1\n", "image_count must be a whole number"), + (HEADER + "ctrl,rep1,rep1,true,2,-1,true,,1\n", "technical_repeat_count must be"), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,,2\n", "schema_version '2'"), + (HEADER + "NA,rep1,rep1,true,2,0,true,,1\n", "condition 'NA' reads as a missing"), + (HEADER + "ctrl,,rep1,true,2,0,true,,1\n", "biological_repeat '' reads as a missing"), + # Rows + (HEADER + GOOD + GOOD.splitlines()[0] + "\n", "more than once"), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,,1\nctrl,rep2,rep2,false,2,0,true,,1\n", "control"), + (HEADER + "ctrl,rep1,rep1,true,2,0,true,,1\nCtrl,rep2,rep2,true,2,0,true,,1\n", "case"), + (HEADER + "a,Rep1,g1,true,2,0,true,,1\nb,rep1,g2,true,2,0,true,,1\n", "case"), + ( + HEADER + "a,r1,Rep1,true,2,0,true,,1\nb,r1,rep1,true,2,0,true,,1\n", + "line 3, column repeat_group: 'rep1' differs only in case from 'Rep1' on line 2", + ), + ], +) +def test_anything_doubtful_is_refused(tmp_path, text, message): + issue = refusal(write(tmp_path, text)) + assert issue.code == "unreadable_experiment_design" + assert re.search(message, issue.message), issue.message + + +def test_the_lenient_reader_takes_what_the_shared_reader_refuses(tmp_path): + # The copies in CellPatcher and MorphoFeature used the lenient reader: spreadsheet + # spellings of booleans and spaces around labels were read. + path = write(tmp_path, HEADER + "ctrl ,rep1,rep1,TRUE,2,0,yes,,1\n") + assert read_experiment_design(path).repeat_groups() == {("ctrl", "rep1"): "rep1"} + assert refusal(path).code == "unreadable_experiment_design" + # A duplicated column: the lenient reader silently kept the last one. + path = write( + tmp_path, + HEADER.replace("\n", ",repeat_group\n") + "ctrl,rep1,rep1,true,2,0,true,,1,other\n", + ) + assert read_experiment_design(path).repeat_groups() == {("ctrl", "rep1"): "other"} + assert refusal(path).code == "unreadable_experiment_design" + + +def test_an_unconfirmed_group_is_refused(tmp_path): + path = write(tmp_path, HEADER + "ctrl,rep1,rep1,true,2,0,false,,1\n") + with pytest.raises(DesignCheckError) as caught: + read_confirmed_design(path) + # design_issues names the folders of the group, as in the preprocessing run. + (issue,) = caught.value.issues + assert issue.code == "design_not_confirmed" and issue.paths == ("ctrl/rep1",) + + +def test_a_group_with_two_folders_of_a_condition_is_refused(tmp_path): + path = write( + tmp_path, + HEADER + "ctrl,rep1,g,true,2,0,true,,1\nctrl,rep2,g,true,2,0,true,,1\n", + ) + with pytest.raises(DesignCheckError) as caught: + read_confirmed_design(path) + assert "repeat_group_conflict" in [issue.code for issue in caught.value.issues] + + +def test_a_design_other_than_the_runs_is_refused(tmp_path): + path = write(tmp_path, HEADER + GOOD) + assert refusal(path, expected_sha256="0" * 64).code == "changed_experiment_design" + assert read_confirmed_design(path, expected_sha256=file_sha256(path)).sha256 == file_sha256( + path + ) + + +def test_every_design_the_brick_writes_is_read(tmp_path): + root = tree(tmp_path / "exp", {"FALSE/001/t1": 1, "FALSE/TRUE": 2, "single": 1}) + design = confirm_all( + default_design(biological_repeats(discover_experiment(root))).with_control("FALSE", True) + ) + path = write_experiment_design(design, tmp_path / DESIGN_FILENAME) + assert read_confirmed_design(path).design == design diff --git a/tests/test_design.py b/tests/test_design.py index 122d9057..b0b97119 100644 --- a/tests/test_design.py +++ b/tests/test_design.py @@ -21,6 +21,10 @@ from tiff_tree import make_tree TODAY = "2026-10-26" +HEADER = ( + "condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count," + "confirmed,confirmed_on,schema_version\n" +) def codes(issues): @@ -190,8 +194,8 @@ def test_files_edited_in_a_spreadsheet_still_read(tmp_path): text = ( "condition,biological_repeat,repeat_group,control,image_count," "technical_repeat_count,confirmed,confirmed_on,schema_version\r\n" - "ctrl,rep1,,TRUE,2,0,Yes,,1\r\n" - "drug,rep1b,rep1,FALSE,2,0,1,26/10/2026,1\r\n" + "ctrl,rep1,rep1,TRUE,2,0,Yes,,1\r\n" + "drug,rep1b,rep1,FALSE,2,0,1,2026-10-25,1\r\n" ",,,,,,,,\r\n" ) path.write_bytes(text.encode("utf-8")) @@ -200,13 +204,105 @@ def test_files_edited_in_a_spreadsheet_still_read(tmp_path): assert design.controls == ("ctrl",) and design.confirmed # A confirmation without a date gets one when it is used. assert design.dated(TODAY).row(("ctrl", "rep1")).confirmed_on == TODAY - assert design.dated(TODAY).row(("drug", "rep1b")).confirmed_on == "26/10/2026" + assert design.dated(TODAY).row(("drug", "rep1b")).confirmed_on == "2026-10-25" -HEADER = ( - "condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count," - "confirmed,confirmed_on,schema_version\n" -) +# Typed edits (owner decision, 31 October 2026) -------------------------------------------- + + +@pytest.mark.parametrize("group", ["", "NA", "na", "NaN", "none", "NULL", "n/a", "", ""]) +def test_a_repeat_group_that_reads_as_missing_is_refused(tmp_path, group): + path = tmp_path / DESIGN_FILENAME + path.write_bytes((HEADER + f"ctrl,rep1,{group},false,1,0,true,,1\n").encode("utf-8")) + with pytest.raises(DesignFormatError, match=r"line 2, column repeat_group: .*missing"): + read_experiment_design(path) + + +@pytest.mark.parametrize("group", [" rep1", "rep1 ", "\trep1"]) +def test_a_repeat_group_with_outer_whitespace_is_refused(tmp_path, group): + path = tmp_path / DESIGN_FILENAME + path.write_bytes((HEADER + f'ctrl,rep1,"{group}",false,1,0,true,,1\n').encode("utf-8")) + with pytest.raises(DesignFormatError, match="line 2, column repeat_group: .*whitespace"): + read_experiment_design(path) + + +def test_a_confirmation_date_in_another_form_is_refused(tmp_path): + path = tmp_path / DESIGN_FILENAME + path.write_bytes((HEADER + "ctrl,rep1,rep1,false,1,0,true,26/10/2026,1\n").encode("utf-8")) + with pytest.raises(DesignFormatError) as caught: + read_experiment_design(path) + assert str(caught.value) == ( + "experiment_design.csv, line 2, column confirmed_on: '26/10/2026' is not a date in the " + "form YYYY-MM-DD (for example 2026-10-26)." + ) + + +@pytest.mark.parametrize("on", ["2026-10-1", "20261026", "2026-02-30", "2026/10/26", "today"]) +def test_only_iso_dates_are_confirmation_dates(tmp_path, on): + path = tmp_path / DESIGN_FILENAME + path.write_bytes((HEADER + f"ctrl,rep1,rep1,false,1,0,true,{on},1\n").encode("utf-8")) + with pytest.raises(DesignFormatError, match="column confirmed_on: .*YYYY-MM-DD"): + read_experiment_design(path) + + +def test_repeat_groups_differing_only_in_case_are_refused(tmp_path): + # Decision D-e: a name is spelled one way; Rep1 and rep1 are not two repeat groups. + path = tmp_path / DESIGN_FILENAME + path.write_bytes( + (HEADER + "ctrl,rep1,Rep1,false,1,0,true,,1\ndrug,rep1,rep1,false,1,0,true,,1\n").encode() + ) + with pytest.raises(DesignFormatError) as caught: + read_experiment_design(path) + assert str(caught.value).startswith( + "experiment_design.csv, line 3, column repeat_group: 'rep1' differs only in case from " + "'Rep1' on line 2." + ) + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) + clash = default_design(repeats_of(root)).with_group(("treated", "rep1"), "REP1") + assert "invalid_experiment_design" in codes(design_issues(clash)) + with pytest.raises(DesignFormatError, match="only in case"): + write_experiment_design(clash, tmp_path / "out.csv") + + +def test_the_writer_writes_only_iso_dates_and_valid_groups(tmp_path): + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) + design = default_design(repeats_of(root)) + path = tmp_path / DESIGN_FILENAME + for bad in ( + design.with_confirmed("rep1", True, "26/10/2026"), + design.with_group(("treated", "rep1"), "NA"), + design.with_group(("treated", "rep1"), " rep2"), + ): + assert "invalid_experiment_design" in codes(design_issues(bad)) + with pytest.raises(DesignFormatError, match="cannot be written"): + write_experiment_design(bad, path) + assert not path.exists() + dated = confirm_all(design).dated("2026-10-31") + assert read_experiment_design(write_experiment_design(dated, path)) == dated + + +def test_a_run_with_an_invalid_date_stops_before_writing(tmp_path): + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) + design = default_design(repeats_of(root)).with_confirmed("rep1", True, "26/10/2026") + result = run_experiment(root, prefill_mapping("YX"), design=design) + assert not result.written and "invalid_experiment_design" in codes(result.issues) + assert not result.output_folder.exists() + + +def test_the_cli_dates_a_hand_confirmation_as_an_iso_date(tmp_path, capsys): + root = tree(tmp_path / "exp", {"control/rep1": 1, "treated/rep1": 1}) + output = default_output_folder(root) + output.mkdir(parents=True) + (output / DESIGN_FILENAME).write_bytes( + ( + HEADER + "control,rep1,rep1,false,1,0,true,,1\ntreated,rep1,rep1,false,1,0,true,,1\n" + ).encode("utf-8") + ) + assert main(["run", str(root), "--axes", "YX"]) == 0 + capsys.readouterr() + dates = {row.confirmed_on for row in read_experiment_design(output / DESIGN_FILENAME).rows} + (on,) = dates + assert len(on) == 10 and on[4] == on[7] == "-" @pytest.mark.parametrize(