diff --git a/AGENTS.md b/AGENTS.md index 4deb5344..b063d769 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -168,6 +168,9 @@ src/morphobricks_preprocessing/ preprocessing's discovery and pairing; RunInputs.open raises UnrecordedFilesError for them (PP-8b, contract §9.1) run_inputs_text.py the texts RunInputs refuses a run with (moved from run_inputs.py in PP-12) + required_description.py every run needs a published morphobricks.toml (v0.5.1, ADR 0002 + AR10): is_published, the A_DESCRIPTION_REQUIRED issue and its fix, shared by + the run, the mask step, the CLI and RunInputs mask_record.py the manifest's masks entry with fingerprints and roles (manifest v2); the fingerprint is the one bound to the mask's header (issue #160) mask_binding.py masks bound to their fingerprints (issues #160, #165): changed_masks, @@ -217,7 +220,7 @@ src/morphobricks_preprocessing/ moved from report.py in PP-10d discovery_report.py the run report (discovery_report.json and .txt, design v0.4 §4): status, manifest SHA-256, what was found, reused, recomputed and replaced - cli.py, __main__ `morphobricks-preprocessing check|run|masks|calibrate|design|convert` (also `python -m morphobricks_preprocessing`); + cli.py, __main__ `morphobricks-preprocessing check|run|masks|calibrate|describe|convert` (also `python -m morphobricks_preprocessing`); cli_run.py holds `run`'s options and output; cli_calibrate.py `calibrate`; cli_convert.py `convert`; log_target.py where `--log` may write (#99), by write_target.py's write_target_refusal (the shared rule other bricks @@ -287,8 +290,8 @@ src/morphobricks_preprocessing/ mask_answers.py (pure) the mask answers of the draft; mask_evidence.py the evidence beside the question (images with the mask, dtype, values in one image, the mask on that image) - notebook_described.py a described run (PreprocessingNotebook(describe=True), the shipped - notebook): the overrides that retire the labels toggle, the design card's + notebook_described.py the described run (every PreprocessingNotebook; there is no + undescribed notebook since v0.5.1): the overrides that retire the labels toggle, the design card's comparable questions and CSV writes, the IDs toggle and the Image type card, and reopen from the published file notebook_check_save.py the Check and save screen (sentences, one "same way?" per replicate, @@ -331,7 +334,8 @@ src/morphobricks_preprocessing/ notebooks/MorphoBricks_Preprocessing.ipynb thin notebook: every code cell calls notebook_ui only notebooks/notebook_latest_versions.yaml the notebook's latest version, for LabConstrictor's version check tests/ pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py); the - histogram science goldens (tests/histogram_goldens.py, tests/data) + histogram science goldens (tests/histogram_goldens.py, tests/data); + described test runs of a tree (tests/described_tree.py) experiments/binning_study/ the study behind the binning decisions (not part of the package) ``` @@ -658,12 +662,11 @@ functions the tests use. `RunInputs.design()` uses it too (#83). - **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that gain a folder, drops removed folders. -- **Notebook** writes the file after every answer once the labels are confirmed; a - confirmed design for the same folders is folded away behind **Change the experiment - design**; step 3 is gated on it. `folder_step(show_design=False)` (embedding, Q31) asks - and writes nothing, and step 3 runs without a design. **CLI** `run` reads it (or `--design`), else writes the - reconciled template and exits 1. **API** `run_experiment(design=None)` is optional; with - a design it is written with the table and `experiment_design_sha256` goes in the manifest. +- **v0.5.1: an output only.** Every run is described (`A_DESCRIPTION_REQUIRED` without a + published `morphobricks.toml`, contract §0); the run derives `experiment_design.csv` from the + description and writes it with the table (`experiment_design_sha256` in the manifest). The + notebook keeps the pairing in the draft; there is no `design=`, `--design`, `design` + command or `check` template any more. ### Image type (contract §13, follow-up P4) @@ -1033,7 +1036,8 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb input read directly (TIFF to OME-Zarr conversion and the `_moved.py` re-exports were removed); optional masks, the mask step and the notebook's **Describe the masks**; named-axis copies (`convert`, notebook card); calibration; resumable, crash-consistent runs. -- **Command line:** `check`, `run`, `masks`, `calibrate`, `design`, `convert`. +- **Command line:** `check`, `run`, `masks`, `calibrate`, `describe`, `convert` (`check`, + `run` and `masks` need a published description, v0.5.1). - **Next:** [docs/roadmap.md](docs/roadmap.md). - **Not yet verified:** - the exact look of the notebook in a real JupyterLab; diff --git a/CHANGELOG.md b/CHANGELOG.md index 43a06d6c..ebe27636 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,92 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature, on b are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the [migration guide](docs/migration-0.3-to-0.4.md). +## Unreleased (v0.5.1) + +### PP-C: every run is described; undescribed runs and their compatibility code removed + +No backward compatibility before the first pipeline release (owner policy of 10 November 2026, +MorphoBricks-Spec ADR 0002 AR10 and AR11, Stage 2 plan §0.1 and §0.2). **Breaking.** + +- **Every run needs a published experiment description.** `run_experiment`, + `prepare_mask_step`, the command line's `check`, `run` and `masks`, and `RunInputs.open` refuse + an output folder without `morphobricks.toml` with the one code `A_DESCRIPTION_REQUIRED` + (owner ruling 1), before anything is read or written, naming the fix: the notebook's describe + steps (*Check and save*), or the command line's `describe` to see what is open. The refusal + comes before the recorded run or any image header is read (also for an output folder inside + the experiment or the masks, which is said beside it) and, on the command line, before + `--log` is opened or `--remove-stale-lock` applied. New: + `required_description` (`is_published`, `description_required_issue`, `undescribed_issues`) and + `DescriptionRequiredError` (a `RunInputsError`, exported). Contract §0. +- **Removed: the undescribed ("v0.4") run.** Its `NO_DESCRIPTION` warning; the + `with_description` argument of `preflight`, `remasked` and `masks.find_masks` and + `run.preflight.description_in_use` (the name refusals `R_FOLDER_NOT_NFC` and + `R_MASK_TYPE_NAME_INVALID` always apply); the `None` answers of `RunInputs.description()`, + `section_digests()`, `images_without_header_channel_names()`, `references()` and + `mask_semantics()`; `make_clean_run`'s undescribed run (below). +- **Removed: `experiment_design.csv` as an input to grouping.** `run_experiment(design=...)`, + `run --design FILE`, the `design` command (`cli_design.py`), `run.design_file` + (`write_design_template`, `write_design`, `edit_design`, `design_for_run`), `check`'s design + template (`run.write_check_outputs`, `CheckOutputs`; `write_check_report` stays), + `recorded.design_change` and `planned_changes(design=...)`, and the notebook's writes and reads + of the file. The run now derives the design from the published description + (`design_derive.derive_design`) and writes it with the table (`experiment_design_sha256` in + the manifest): an output only, for every run. Stays until SWEEP-PP: the lenient reader + `read_experiment_design` (only the one-way draft migration `migrate_legacy_design` uses it), + `ExperimentDesign`/`default_design`/`apply_design_edit` behind the notebook's pairing card + (owner answer 4), and the strict `read_confirmed_design`/`RunInputs.design` until CellPatcher + and MorphoFeature read `repeat_group` from the description (owner answer 3). +- **Identities.** No identity is taken from a folder name in place of the description: the + run's grouping comes from the description's repeats. `position_id` stays the image file stem, + always together with the canonical condition, repeat and technical-repeat IDs (owner ruling 2; + no change). +- **Run manifest: version `"4"` only** (owner ruling 16). Every run writes version 4 (with + `experiment_description`; content hashes exactly when `verify_content`), and only version 4 + is read: `"1"`, `"2"`, `"3"` (and any later version) are refused with "Run preprocessing + again", never converted; the next run reads every image again. Removed: + `MANIFEST_VERSION = "2"`, `STRICT_MANIFEST_VERSION`, `DESCRIBED_MANIFEST_VERSION`, + `LEGACY_MANIFEST_VERSION`, `version_for`, the `manifest_before_v040` source issue and the + version-1 branches of row reuse, recorded inputs and `RunInputs`. The tolerant reading of + version-1 file fields (`manifest_fields`' `strict=False`) goes in SWEEP-PP. +- **Notebook: always described.** `PreprocessingNotebook` has no `describe` argument and every + `_describe` branch is gone: the labels toggle and the global *Image type* card are never + shown (the type comes from the saved description), and the design card no longer reads or + writes `experiment_design.csv` (its undescribed `_open_design`, `_save_design`, + `_design_load_note`, `_refresh_design_status`, `_design_pending` and the check's design + template are removed). The widgets of the retired cards stay until SWEEP-PP. The embedding + API keeps `folder_step`, `axis_step`, `run_step` and `show_design`; an embedding brick must + now let the user describe and save before the run (below). **The shipped notebook is + unchanged** until the v0.5.1 release: it installs `RELEASE_REF` (v0.5.0, `fd7afd3`) and calls + that code's `PreprocessingNotebook(describe=True)`; the release PR moves `RELEASE_REF` and the + call to `PreprocessingNotebook()` together (`test_notebook_file.RELEASE_CALLS`, the release + check list). +- **Refusal texts.** `design_not_confirmed`, `repeat_group_conflict` and + `repeat_missing_conditions` name the notebook's describe steps / *Replicates* page and + *Check and save*, not the removed `design` command or an edit of `experiment_design.csv`. +- **`testing.make_clean_run` makes a described run.** Same tree and FOV keys as before + (`ctrl`/`drug`, `rep1`, `a`/`b`); it publishes `testing.clean_description(...)` (channels + `ch1`..`chN`, one `instances` mask per mask type, `design=True` makes `drug` reference + `ctrl`), writes the derived `experiment_design.csv` every time, and runs the mask step when + there are masks, so every mask is `checked`. New: `testing_described.publish_and_run`. +- **Tests.** The undescribed-run tests now run described trees (`tests/described_tree.py`: + `run_described`, `main_described`, `describe_tree`, the description the user would confirm + for a tree the test wrote, every proposal accepted; never part of the package). Deleted as + undescribed- or legacy-only: `test_parity_v0_4_0.py` and `parity_datasets.py`, + `test_removed_in_v040.py`, `test_design_command.py`, `test_strict_manifest.py` and + `test_described_manifest_version.py` (replaced by `test_manifest_version.py`), + `released_code.py`, `tests/data/rc4_run_manifest.json` and `make_rc4_manifest.py`, and the + single tests of other files that checked only undescribed or legacy behaviour (listed in the + pull request). Kept as science goldens and renamed (owner ruling 6): + `test_schema_v2_equivalence.py` is `test_histogram_goldens.py` (`histogram_goldens.py`, + `tests/data/golden_*`, `make_histogram_goldens.py`), its version-2 manifest test deleted. CI + no longer fetches the whole history (`fetch-depth: 0` was only for the old-code runs). + +**Downstream** (each brick is unaffected until its pin moves past this change): every brick's +test runs must be described (`make_clean_run` does it; a direct `run_experiment` call needs a +published `morphobricks.toml` first), `run_experiment(design=...)` and the `describe` argument +of `PreprocessingNotebook` are gone, and a run written by v0.5.0 or earlier is refused until it +is run again. + ## 0.5.0 - 2026-11-09 Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`): diff --git a/README.md b/README.md index 44c18fd3..5919ab74 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,10 @@ Pre-alpha; the current release is v0.5.0 (what is planned next is in - Reading a run from another brick: `RunInputs.open(output_folder)` - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb)) and a command line (`morphobricks-preprocessing check` / `run` / `masks` / `calibrate` / - `design` / `describe` / `convert`) + `describe` / `convert`) +- **Every run is described** (v0.5.1): the output folder must hold a published experiment + description, `morphobricks.toml`, written by the notebook's describe steps (*Check and + save*). A run without one is refused with `A_DESCRIPTION_REQUIRED` on every route. Every feature, message code and output file is described in the [user guide](docs/user_guide.md). @@ -128,10 +131,10 @@ reuses every input the earlier run recorded in the output folder (axis roles, ma exclusions, design, image type, settings); `mode="full"` recomputes everything, and `exclude=[...]` leaves out damaged files. One run at a time holds the output folder's lock, and every run, also one that stops, writes `discovery_report.json` and a text version. The notebook and the -command line also ask for the experiment design (which biological repeats pair across -conditions, and that they are comparable) and write `experiment_design.csv`; in Python, -pass it with `design=` (see the [user guide](docs/user_guide.md) §9). Later bricks read it -with `read_experiment_design`. They also ask for the image type, `fluorescence` or `rgb` +notebook asks for the experiment description (conditions, which biological repeats pair +across conditions and that they are comparable, channels, masks) and publishes +`morphobricks.toml`; every run needs it, and writes the `experiment_design.csv` derived from +it (an output only). They also ask for the image type, `fluorescence` or `rgb` (suggested by `review_image_type`), and record it in the manifest; in Python, pass it with `image_type=`. After writing, the run checks the images' intensities from the table (`result.intensity`); `check_table(path)` checks any table. @@ -139,8 +142,8 @@ with `read_experiment_design`. They also ask for the image type, `fluorescence` ## Command line ```bash -morphobricks-preprocessing check /data/exp1 # report, suggested --axes, design template -morphobricks-preprocessing design --output /data/exp1_MorphoBricks_Preprocessing --confirm-all +morphobricks-preprocessing describe /data/exp1 # the draft description, what is open +morphobricks-preprocessing check /data/exp1 # report, suggested --axes morphobricks-preprocessing run /data/exp1 --axes ZCYX morphobricks-preprocessing run --output /data/exp1_MorphoBricks_Preprocessing # later runs morphobricks-preprocessing masks --output /data/exp1_MorphoBricks_Preprocessing # mask step @@ -148,11 +151,10 @@ morphobricks-preprocessing calibrate --output /data/exp1_MorphoBricks_Preprocess morphobricks-preprocessing convert /data/exp1 --to /data/exp1_named_axes --axes ZXY # axis errors ``` -`check` writes the report and, when the output folder holds no design, an -`experiment_design.csv` prefilled from the folders. `design` prints it and confirms repeat -groups (`--confirm GROUP`, `--confirm-all`), marks controls (`--control`, `--no-control`) -and pairs repeats (`--repeat-group CONDITION/REPEAT=GROUP`) without editing the file; -nothing is confirmed unless asked. `run` and `masks` reuse every input the run in the +`check`, `run` and `masks` need the experiment description published in the output folder +(the notebook's describe steps); without it they stop with `A_DESCRIPTION_REQUIRED` and write +nothing. `describe` shows the draft and what is still to be answered. `check` writes the +report. `run` and `masks` reuse every input the run in the output folder recorded. A run that stops writes its report and says so. Every output file gets the permissions a file you save yourself would (your umask). `convert` saves the images whose metadata name their axes wrongly or not at all as named-axis OME-TIFF copies diff --git a/docs/contract.md b/docs/contract.md index c84cd460..944e619e 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -11,6 +11,32 @@ whole experiment; any change to them goes through the normalization brick. [normalization contract]: https://github.com/CellMigrationLab/morphobricks-normalization/blob/main/docs/contract.md +## 0. Every run is described (v0.5.1, Spec ADR 0002 AR10/AR11) + +There is no backward compatibility before the first pipeline release (owner policy of 10 +November 2026). **Every run needs a published experiment description** (`morphobricks.toml` in +its output folder, experiment-v1). Without one every route refuses the run with the one code +`A_DESCRIPTION_REQUIRED`, before anything is read or written, naming the fix (the notebook's +describe steps and *Check and save*; the command line's `describe` shows what is still open): + +| Route | Refusal | +|---|---| +| `run_experiment` | `RunResult(written=False)` whose first issue is the error `A_DESCRIPTION_REQUIRED`, with `output_inside_experiment` / `output_inside_masks` when the folders given say so (their paths are compared; no image header and no recorded run is read); no report, no lock, nothing in the output folder (`required_description.undescribed_issues`). | +| `prepare_mask_step` (and the command line's `masks`) | `MaskStep.refusal` is the error; nothing is read. | +| Command line `check`, `run`, `masks` | The error is printed, exit status 1, before the recorded run is read, `--log` opened or `--remove-stale-lock` applied; nothing written. | +| `RunInputs.open` | `DescriptionRequiredError` (a `RunInputsError`, `code == "A_DESCRIPTION_REQUIRED"`). | +| Notebook | The run waits for *Check and save*; there is no undescribed notebook (`PreprocessingNotebook` has no `describe` argument). | + +Removed in v0.5.1 (PP-C), each refused rather than read: the undescribed ("v0.4") run and its +warning; the `with_description` switch (the name refusals `R_FOLDER_NOT_NFC` and +`R_MASK_TYPE_NAME_INVALID` always apply); `experiment_design.csv` as an input (`run_experiment` +has no `design` argument, the command line no `--design` option and no `design` command, `check` +writes no template; the run derives the file from the description, an output only, §12); run +manifests of versions `"1"`, `"2"` and `"3"` (§9: refused, "run preprocessing again"). The FOV +key and `position_id` (the image file stem) are unchanged (owner ruling 2): a position is always +read together with the canonical condition, repeat and technical-repeat IDs (§9.1), never as an +identity alone. + ## 1. Folder hierarchy and labels ``` @@ -73,7 +99,7 @@ planes of an experiment use the same bin count. In a described run (a published `morphobricks.toml`) the pixel type is compared among the units that declare the same `image_type`, and the notebook's folder check compares images inside each unit; units of different image types (RGB and fluorescence) are refused as `R_HETEROGENEOUS_LAYOUT_NOT_ENABLED` -(Stages 1-2), not as `mixed_dtypes`. An undescribed run compares every image of the experiment. The +(Stages 1-2), not as `mixed_dtypes`. Every run is described (§0). The invariant is that a previously blocking input stays blocked; the diagnostic codes may change (the per-type checks and the Spec's `R_*` codes replace the global `mixed_dtypes` and `inconsistent_dimensions` of a described run). @@ -147,8 +173,7 @@ Every message states what happened in plain language, lists the affected paths, suggests an action. Technical details go to the diagnostic log. The command line's log (`--log FILE`, issue #99) is checked before it is opened, and opened only after the other arguments: it is refused inside the experiment or masks folder (given, or recorded by the -earlier run in the output folder), inside `convert`'s folder of copies, as the design file -given with `--design` (an input of the run), as a folder or in a +earlier run in the output folder), inside `convert`'s folder of copies, as a folder or in a folder that does not exist, and as or inside an output artefact (`plane_histograms.csv`, `run_manifest.json`, the report files, `experiment_design.csv`, `calibration.csv`, `mask_relationships.json`, the lock, `.morphobricks-conversion.json`, the publication and @@ -184,7 +209,7 @@ whether processing is blocked. The consistency issues of §11 are shown in their **Dataset consistency** panel, next to the balance table, and not repeated under the other messages. -## 9. Run manifest (`run_manifest.json`, manifest versions 2 and 3) +## 9. Run manifest (`run_manifest.json`, manifest version 4) UTF-8 JSON next to `plane_histograms.csv`, written only when the run succeeded. Its replacement is the commit of the run's outputs: they are one **generation**, published @@ -206,7 +231,7 @@ interpretation. | Key | Meaning | |---|---| -| `manifest_version`, `schema_version` | Manifest format and CSV format (`"4"`), written as JSON **strings**, not numbers. The manifest format is `"3"` (v0.4.0rc5, release review) for a content-verification run (`verify_content`, below), which every run is unless it opts out (#159), and `"2"` (since v0.4.0) for a run that opted out: version 3 is version 2 with `settings.verify_content` `true` and a `content_sha256` in every file entry (`manifest_versions`). The version, not only those keys, marks a strict run, because readers of v0.4.0rc4 and earlier ignore keys they do not know but refuse a version they do not know (`ManifestFormatError: Manifest version '3' is not supported`, tested by running v0.4.0rc4's code on a strict manifest): they never open a strict run as a fingerprint-only one. The reader and the writer refuse a version 3 manifest without `verify_content` or without a file's `content_sha256`, and a version 1 or 2 manifest that holds either (a strict manifest written as version 2, only by unreleased rc5 builds before this rule). A run that opted out writes version 2, as before except for `settings.verify_content: false` (tested against the code of the commit before version 3). Switching between a strict and a default run recomputes every row as before. A manifest whose `schema_version` is not the current one makes the next run recompute every row. A run with an `experiment_description` writes version `"4"` (v0.5.0, Stage 2 design question 9): version 2 or 3 as `settings.verify_content` says, plus the description; v0.4.0 readers skip keys they do not know and would have ignored the description, so they refuse version 4 (tested with the code of the `v0.4.0` tag). A version 4 manifest must carry `experiment_description` and a version 1, 2 or 3 manifest must not (reader and writer refuse the mismatch); version 4 holds a `content_sha256` in every entry exactly when `verify_content` is true. A run without a description keeps version 2 or 3 byte for byte. This version reads `"1"` to `"4"`; `RunInputs`, `restart` reuse and the recorded inputs treat 2, 3 and 4 alike. | +| `manifest_version`, `schema_version` | Manifest format (`"4"`) and CSV format (`"4"`), written as JSON **strings**, not numbers. Version 4 is the only manifest version written and read (v0.5.1, owner ruling 16, ADR 0002 AR11): every manifest carries `experiment_description`, and holds a `content_sha256` in every file entry exactly when `settings.verify_content` is true (a content-verification run, which every run is unless it opts out, #159). The reader and the writer refuse a manifest without `experiment_description` or with content hashes that do not match `verify_content`. Any other version (`"1"` before v0.4.0; `"2"` and `"3"`, the undescribed runs of v0.4.0 and v0.5.0; a later one) is refused with `ManifestFormatError: Manifest version '…' is not supported … Run preprocessing again …`: by `read_manifest`, `RunInputs.open` and `check_sources`; the next run in that output folder reads every image again and replaces it. Nothing is converted. Switching between a strict and a default run recomputes every row. A manifest whose `schema_version` is not the current one makes the next run recompute every row. | | `written_at` | UTC time of writing, ISO 8601. A rerun whose manifest would differ from the earlier one only in `written_at` keeps the earlier time, so the manifest's bytes and SHA-256, which later bricks pin, stay the same (cross-brick test F10, 3 November 2026); the report still records the rerun. | | `experiment_root` | Absolute path of the experiment folder, for information. | | `dtype` | Pixel type of the experiment. | @@ -215,7 +240,7 @@ interpretation. | `labels` | One entry per leaf folder: `folder` (relative path, `.` for the root) and its three labels. | | `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1: its name without the extension, never edited; required in version 2, where an entry without it, with null, or with another value than `position_id(relative_path)` is refused naming the entry, by the reader and the writer, round-2 audit A-m5; null when absent in a version 1 manifest), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files, `mtime_ns` the time of its newest file, and `inventory_sha256` (v0.4.0, PP-12, issue #86; only for OME-Zarr images) the SHA-256 of its inventory: one line per file (its path relative to the image with `/`, a NUL, its size, a NUL, its `mtime_ns`, a newline), sorted by path, its root `labels` group left out; links to folders inside the image are followed, as the reader follows them, and links that reach one folder twice are refused. Adding, removing, renaming, resizing or touching any chunk or metadata file changes it; an OME-Zarr image recorded without it (v0.4.0rc3 and earlier) counts as changed (recomputed; `RunInputs` refuses it). Like a TIFF file's size and time it is metadata only (design v0.4 §1): a file rewritten with the same size and its time set back is not seen, only hashing the content would. **Version 2 (v0.4.0):** `roles` is required, with `axes_from` (required): `header` when the metadata names every axis and the roles agree, `confirmed` for a plain stack whose roles the user confirmed (§2); and `pixel_sizes`, one entry per role `X`, `Y`, `Z`, `T` with `value`, `unit` (OME-NGFF name) and `source`: `header` (the image's metadata) or `stated` (`calibration.csv`, §9.3) with a value, or `pixels` (stated as deliberately uncalibrated) with `value` and `unit` null; `value` and `source` null for an axis nothing gives a size (never guessed). A `header` or `stated` size without a value, or a value with no source or with `pixels`, is refused. The reader refuses a version 2 entry without `roles` or `axes_from`, and the writer refuses to write one. `content_sha256` (only in a `verify_content` run): the SHA-256 of the image's bytes, or for an OME-Zarr image of one line per file (its path relative to the image with `/`, a NUL, the file's SHA-256, a newline) over the files its inventory covers, in path order (`source_content.source_content_sha256`). | | `csv_sha256` | SHA-256 of the table written with this manifest. | -| `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. | +| `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` the run derived from its description and wrote with the table (§12), so later bricks can check that the design they read is the one the run wrote. | | `masks` | The masks paired with the images (§15), when the run was given a masks folder or its OME-Zarr images have label images (always used): `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes other than Y and X, `YX` for a 2D mask (`[1, 512]` for a mask one pixel high), with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes. **Version 2:** `axes` are the mask's axes as stored (its header's letters: `IYX` for a plain stack), and each mask entry also has `size` and `mtime_ns` (its fingerprint, as `source_fingerprint` gives it: the mask file, or the label image's folder), `roles` (one per axis of `axes`, what each is read as: `TYX` for that plain stack stated `--mask-axes Cells=TYX`), `axes_from` (`header`, or `stated` when the axes were stated for its mask type). The release candidates up to v0.4.0rc6 also wrote `shared` (true for a 2-D mask stated `YX` for its type: one plane shared by every plane of its image); the option was removed by the owner decision of 5 November 2026, the key is no longer written, and `RunInputs.open` refuses a manifest that records `shared: true` (stated or not), naming the masks and asking for one mask plane per time point and a new run. A mask is read as stored, never reordered. | | `image_type` | `fluorescence` or `rgb`, the image type the user confirmed (§13); null when the run was given none, or absent in manifests written before it. Optional key: the manifest stays version 1. Changing it recomputes nothing. | | `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. | @@ -223,9 +248,9 @@ interpretation. | `narrow_histogram_ranges` | The channel planes whose range check found the central range too narrow (§4), sorted by plane and channel: `relative_path`, `plane_id`, `channel` and the fields of the normalization brick's `RangeCheck` (`histogram_range`, `central_percentiles`, `central_bins`, `robust_quantiles`, `robust_limits`, `outside_robust_count`, `robust_central_bins`, `min_central_bins`). Empty when there are none; absent in manifests written before follow-up P6. Optional key: the manifest stays version 1. | | `conversion_record_sha256` | SHA-256 of the conversion record (`.morphobricks-conversion.json`) in the experiment root when the experiment is a tree of named-axis copies (design v0.4 §5, §17); null otherwise. `RunInputs` checks it (owner decision of 2 November 2026, plan question 59): a record changed or gone since the run is refused, as for `calibration_sha256`. A changed record changes no row (the copies are checked by their fingerprints like any image). | | `calibration_sha256` | SHA-256 of the output folder's `calibration.csv` as the run read it (§9.3); null when there was none (or absent in manifests written before PP-8). `RunInputs` checks it. | -| `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation; and, only for a unit that has such images, `images_without_header_channel_names`: a list sorted by `image` (relative path) of objects `image`, `header_names` (`none` when the header names no channel, `partial` when only some) and `channel_positions` (the zero-based C positions without a header name). Such an image passes the header checks, needs no evidence and the confirmed layout is used (§4.3); it is listed, not warned about, and the key is omitted for a unit whose images are all fully named; RGB units list nothing), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). | +| `experiment_description` | Always present (v0.5.1: every run is described, §0; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation; and, only for a unit that has such images, `images_without_header_channel_names`: a list sorted by `image` (relative path) of objects `image`, `header_names` (`none` when the header names no channel, `partial` when only some) and `channel_positions` (the zero-based C positions without a header name). Such an image passes the header checks, needs no evidence and the confirmed layout is used (§4.3); it is listed, not warned about, and the key is omitted for a unit whose images are all fully named; RGB units list nothing), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). | | `projection` | Only for a run on a tree of Z-projected copies (the key is left out otherwise, so a run without projection is byte for byte as before; issue #176, run side). An object with exactly `method` (`max`, `mean` or `sum`), `z_range` (`[start, stop]` requested for the experiment, or null for every slice), `conversion_record` (`{file, sha256}`: the record these facts come from, the same SHA-256 as `conversion_record_sha256`), `source_root`, `images` (one per analysed image, by path: `relative_path`, `source_relative_path`, `source_content_sha256`, `source_z_size`, `z_range` (the slices used), `source_dtype`, `output_dtype`, `dtype_rule`, `rounding`, `significant_bits_dropped`), `masks_paired` (the masks paired with projected images, sorted) and `warnings` (`["masks_on_projected_images"]` exactly when `masks_paired` is not empty). A reader that knows nothing of the key ignores it, but it also refuses the conversion record version 3 the run names, so projected copies are never read by it as plain images; `manifest_version` is not changed. | -| Version 1 | A manifest written before v0.4.0 (`"1"`) is still read, for reporting, but no row of it is reused: its axes were never checked. The next run recomputes every row ("the earlier run was written before v0.4.0, without the axis check"). Bricks refuse it. | +| Older versions | Refused (v0.5.1, see `manifest_version`): no row of an older run is reused and nothing is taken from it; the next run reads every image again. | | Row reuse | A file's rows are reused only when its row key is the earlier run's (`fingerprint.RowKey`, one rule for "the same file, read the same way", audit D5): its fingerprint (`fingerprint.Fingerprint`: size and modification time, and for an OME-Zarr image the inventory digest), labels, axes, roles, `singleton_roles`, `axes_from` and the path it is read from (so moving or converting the experiment recomputes it), and the settings that change rows (bins, histogram range, robust quantiles, range-check limit). **The identity rule is the fingerprint, which is metadata** (design v0.4 §1): its known blind spot is a file rewritten with the same size whose modification time (for an OME-Zarr image, every changed file's) is set back to its earlier value; its earlier rows are then reused, its staged rows resumed, and `RunInputs` accepts it. | | Content verification (on by default, owner 7 November 2026, #159; opt-in before) | Every run does it unless it opts out with `RunMode(name, verify_content=False)` (`run_experiment(mode=...)`, CLI `--no-verify-content`, the notebook's "Verify each image's content" box unticked); the opt-out is recorded as `settings.verify_content: false`. It closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (true without one: a first run, or a version 1 manifest); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the cost is in the pull request that made it the default (about 3 seconds per 1.5 GB read, measured at 1 GB/s of SHA-256). A run that verifies writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). | | Removed keys | `settings.convert_tiffs`, `settings.conversion` and `files[].ome_zarr_path` were written by v0.3 runs that converted TIFF images to OME-Zarr in the output folder (removed in v0.4.0, §10); a manifest that holds them is still read, and they are ignored. The copies such a run left in `/ome_zarr` are not touched. | @@ -310,10 +335,10 @@ It gives: manifest's `experiment_design_sha256` (issue #83, plan §12 question 62; it used the lenient reader before): a design that reader refuses raises `RunInputsError`, with the refusal as its `__cause__`; -- **semantics of a published description** (Stage 1 PR 12; only for a run whose manifest has - `experiment_description`, §9; a run without one is unchanged: the fields below are `None`, - `description()`, `references()` and `mask_semantics()` return `None`, and `read_mask` is as - before). `Fov.repeat_group`, `Fov.condition_id`, `Fov.technical_repeat_id` (the canonical +- **semantics of a published description** (Stage 1 PR 12; every run has one since v0.5.1, + §0: an output folder without a published `morphobricks.toml` is refused at `open` with + `DescriptionRequiredError`, `A_DESCRIPTION_REQUIRED`, and an older manifest with + `ManifestFormatError`, "run preprocessing again"). `Fov.repeat_group`, `Fov.condition_id`, `Fov.technical_repeat_id` (the canonical IDs, added beside the unchanged `Fov.key`, whose condition, repeat and technical repeat stay the source folder labels: owner decision Q8, experiment-v1 §3.4) and `Fov.channel_ids` (the canonical channel ID of each C index, as the unit declares them; a unit whose channel count @@ -328,12 +353,15 @@ It gives: `morphobricks.toml` (`description_consume.read_published_description`) or a published file with an unconfirmed repeat is refused with `DraftRefusedError` (`A_DRAFT_REFUSED`). `read_mask` of a mask file the mask step has not checked (§15; `mask_representation.json`) is refused with - `MaskUncheckedError` (`A_MASK_UNCHECKED`); the error classes carry the Spec code as `code`. `ChannelNotFoundError` is `A_SCOPE_INCOMPLETE` (Stage 2, owner answer 2): one rule and one code for a channel that no image has and for one that some images lack, and its message names the image. `section_digests()` returns a copy of the manifest's `experiment_description.section_digests` (SHA-256 of the `conditions`, `channels`, `repeats` and `masks` sections; the published file is not read again), or `None` for a run without a description. + `MaskUncheckedError` (`A_MASK_UNCHECKED`); the error classes carry the Spec code as `code`. `ChannelNotFoundError` is `A_SCOPE_INCOMPLETE` (Stage 2, owner answer 2): one rule and one code for a channel that no image has and for one that some images lack, and its message names the image. `section_digests()` returns a copy of the manifest's `experiment_description.section_digests` (SHA-256 of the `conditions`, `channels`, `repeats` and `masks` sections; the published file is not read again). A `RunInputs` is picklable, so worker processes can use it. `morphobricks_preprocessing.testing.make_clean_run(folder, ...)` writes a tiny experiment of -named-axis OME-TIFFs (optionally with masks and a confirmed design) and its clean run, for -the bricks' tests. +named-axis OME-TIFFs (optionally with masks), publishes its description +(`testing.clean_description`: conditions `ctrl` and `drug`, one repeat `rep1`, channels `ch1`.., +one `instances` mask per mask type; `design=True` makes `ctrl` the reference of `drug`), and +runs preprocessing and, with masks, the mask step: a described run, for the bricks' tests +(v0.5.1; before, it was an undescribed run). `make_clean_run(folder, description=)` (also `testing.make_described_run`) writes instead the tree a published experiment description declares (`image_count` tiny images per member, `rgb` units as RGB colour, one mask per image for each declared mask ID in `mask_types`), @@ -353,9 +381,9 @@ raise `ValueError`. Tested for the Spec fixtures `v03 v08 v13 v28 v30 v33 v34`. | Axis roles | `mapping=None` uses the recorded roles; without any, the run stops with `axes_not_confirmed`. CLI `--axes` is needed for the first run of an experiment with plain stacks and for plain stacks that are new or changed; when every readable image names all its axes (`header_axes`) and the suggested mapping gives each a role, the first run uses those roles (design v0.4 §3; cross-brick test F9). `convert` always needs `--axes` (§17). | | Lock | `.morphobricks-preprocessing.lock` in the output folder, created with `O_CREAT \| O_EXCL`, JSON `host`, `pid`, `started_at`; held by `run`, by `check` while it writes its report (and the design template), by `masks` from before its first mask read to the write of `mask_relationships.json` (a locked folder is refused before any mask is read), and by `design` and `calibrate` while they change `experiment_design.csv` and `calibration.csv` (CLI `run` also holds it to write the design template after a run stopped on the design), removed on every exit, and removed when the lock file cannot be written. A second run is refused with `output_locked`, naming the holder, and writes nothing; when the holder is a process on this computer that is no longer running (POSIX), the message says the lock looks stale and how to remove it, and the lock is kept (cross-brick test F11). `remove_stale_lock(output)` / `--remove-stale-lock` (every command) removes a lock only on that request, never one of a process still running on the same computer (checked on POSIX). Not written inside the experiment. The lock is not part of the manifest. | | Report | `discovery_report.json` (report version 1) and `discovery_report.txt`, written on every run that holds the lock, also when it stops: `step` (`run` or `check`), `status` (`stopped` or `written`; `check` is always `stopped`), `provisional` (true for the report written after the header checks, before any pixel is read), `manifest_sha256` (the manifest published with it, `written` only), `experiment_root` and `experiment_root_changed`, `conversion` (for a tree of named-axis copies, §17: `record`, `sha256`, `source_root`; null otherwise), `output_folder`, `masks_folder`, `excluded`, `groups` (labels, position identifiers, image counts), `channels`, `channel_names`, `images` (per image `axes`, `roles`, `axes_from`, `pixel_sizes` by role with unit and source, `header`, `stated` or `pixels`, or `"uncalibrated"` without an X and a Y entry), `calibration` (`file`, `sha256` and the `statements`), `design`, `image_type`, `masks` (per type paired masks and images without one; unpaired and ambiguous masks), `mode`, `reused`, `recomputed` (file: reason), `removed`, `resumed` (files whose rows an interrupted run had staged, reused without reading their pixels, §9.4), `staged_discarded` (file: why its staged rows were not reused), `replaced` (file: what happened), `input_changes`, `issues` (code, severity, message, paths, action). For people only: bricks never read it. A published report is part of the run's generation (§9.4). | -| `check` | The same step stopped after the report; it writes the report, and the design template when the output folder holds no design (§12; owner decision of 2 November 2026, plan §12 question 57, which amends design v0.4 §1 "it writes only the report"). The notebook's **Check folder** does the same with the same functions (`run.check_step.suggested_review`, `write_check_outputs`; v0.4.0, PP-10): the report, and the template with the design card; nothing when an embedding brick hides the output folder (`folder_step(show_output=False)`). | -| Exit status (v0.4.0) | Every command (`check`, `run`, `masks`, `calibrate`, `design`, `convert`) exits with status 1 when it prints an error and 0 otherwise; warnings never change it, and options argparse cannot read exit with 2. `check` exits with 1 on every error it reports: the preflight's, the axis check's, a refused write of its outputs (`output_locked`) and an experiment design it cannot read (`unreadable_experiment_design`; found in the review of #130, it exited with 0) or one with `repeat_group_conflict` or `invalid_experiment_design`, printed after its summary (found in the review of #133: `run` stopped on it, `check` exited with 0); `design_not_confirmed` does not count, since `check` writes the template to confirm. `design` exits with 1 when the design it prints has `repeat_group_conflict` or `invalid_experiment_design`, also after a change it wrote (it exited with 0); `design_not_confirmed` is not printed by `design` and does not count. | -| Notebook (v0.4.0, PP-10) | Step 3 offers the mode (`restart`, `full`) with its effect, and lists before the run the inputs that differ from the recorded ones with their effect (`run.planned.planned_changes(given, output_folder, design=, calibration=)`, from `resolve_inputs`, `design_for_run` and `calibration_change`: the changes the run then reports). After a run it names the mode, the files replaced and the input changes, and shows the report; a stopped run says what it wrote (question 55). A refusal `output_locked` (check, run, a design answer or a stated size) shows the holder (computer, process ID, start time) and a **Remove the lock of a stopped run** button: `remove_stale_lock` only on that click, never the lock of a process still running on the same computer; on Google Drive one session per output folder (question 58). | +| `check` | The same step stopped after the report; it writes the report only (`run.check_step.write_check_report`; v0.5.1: no design template, `experiment_design.csv` is derived by the run, §12). The command line's `check` refuses an output folder without a published description first (§0). The notebook's **Check folder** writes the same report with the same functions (`run.check_step.suggested_review`, `write_check_report`): it is the first step of describing the experiment, so it runs before *Check and save*; nothing when an embedding brick hides the output folder (`folder_step(show_output=False)`). | +| Exit status (v0.4.0; v0.5.1) | Every command (`check`, `run`, `masks`, `calibrate`, `describe`, `convert`) exits with status 1 when it prints an error and 0 otherwise; warnings never change it, and options argparse cannot read exit with 2. `check`, `run` and `masks` exit with 1 on `A_DESCRIPTION_REQUIRED` (§0). `check` exits with 1 on every error it reports: the preflight's, the axis check's and a refused write of its report (`output_locked`, `output_inside_masks`; found in the review of #130, it exited with 0). The `design` command and the design checks of `check` were removed in v0.5.1 (PP-C). | +| Notebook (v0.4.0, PP-10) | Step 3 offers the mode (`restart`, `full`) with its effect, and lists before the run the inputs that differ from the recorded ones with their effect (`run.planned.planned_changes(given, output_folder, calibration=)`, from `resolve_inputs` and `calibration_change`: the changes the run then reports). After a run it names the mode, the files replaced and the input changes, and shows the report; a stopped run says what it wrote (question 55). A refusal `output_locked` (check, run or a stated size) shows the holder (computer, process ID, start time) and a **Remove the lock of a stopped run** button: `remove_stale_lock` only on that click, never the lock of a process still running on the same computer; on Google Drive one session per output folder (question 58). | | Notebook masks (v0.4.0, PP-10c) | Step 1 has an optional masks folder; **Check folder** pairs it with `preflight(masks_folder=, mask_axes=)`, the masks folder and statements being those the run would use (`run.planned.planned_masks`, from `resolve_inputs`: given, else recorded). The **Masks** card shows each mask type (`mask_review.mask_type_rows`), the images without one and the masks' issues (`masks.issues`, and `mask_checks.mask_issues` once the roles are confirmed, as `run`). Mask axes are asked per type only where needed (`mask_review.mask_axes_questions`: `mask_not_2d`, `invalid_mask_axes`, `mask_axes_contradict_header`, and `mask_planes_mismatch` of 2-D masks, answered `YX`), with only the statements `masks.stated.mask_roles` accepts for every mask of the type; nothing is preselected or applied until **Check the masks** (`run.preflight.remasked`, which keeps the other answers). The run is given the same `masks_folder` and `mask_axes`; a masks folder not given reuses the recorded one, and **Drop the recorded masks** gives `masks_folder=False`. Step 3 is gated on masks without errors and on a pairing that matches the masks folder and statements chosen. `folder_step(show_output=False)` shows neither and pairs no masks folder. The mask step (`mask_relationships.json`) is step 3's **Describe the masks** (next row). | | Notebook mask step (v0.4.0, PP-10d) | Step 3's **Describe the masks (optional)** card. Its button is enabled when step 1's output folder holds a v2 manifest that recorded masks (`run.mask_step.recorded_masks(output_folder)`, a `RecordedMasks` from the manifest's `experiment_root`, `written_at` and `masks` keys): after a run in this session, or for an earlier run in a reopened notebook. The help line names the run it describes (output folder, when written, experiment folder, masks folder and mask types). The output folder is the one step 1 shows (default: next to the experiment folder); the experiment folder typed in step 1 only locates it, and the recorded one is read. A click runs the `masks` command's step with the run's recorded inputs (`prepare_mask_step(None, output_folder)`, the path the CLI's `masks --output` takes: experiment folder, masks folder, mask-axis statements, exclusions) and the default thresholds (the notebook offers no threshold; the command line's options change them), with a progress bar per image, then `describe_masks`, which reads the masks and writes `mask_relationships.json` holding the lock throughout. The card shows the mask types, the relationships with their counts (`mask_text.mask_type_rows`, `mask_relationship_rows`), masks of the same objects and the hierarchy (`mask_hierarchy_lines`), labelled as suggestions to check, and the path written. A refused step changes nothing: `experiment_not_recorded`, errors found since the run (an experiment or masks folder moved or removed, a mask changed; shown with their messages), no masks, masks that cannot be used (a red card), and `output_locked` (the lock card with **Remove the lock of a stopped run**). A new run clears the card. `folder_step(show_output=False)` hides the card; the pinned embedding API is unchanged (the click handler is private). | @@ -418,13 +446,19 @@ row (follow-up P3b, normalization plan N43). | Panel | The notebook shows the balance table with the issues `mixed_dtypes`, `inconsistent_dimensions`, `missing_axes`, `inconsistent_mapping`, `different_sizes`, `different_channel_counts`, `different_plane_counts`, `channel_names_differ`, `different_pixel_sizes`, `missing_pixel_size`, `unsupported_pixel_size_unit`, `invalid_physical_calibration`, `partial_pixel_size`, `non_square_pixels`, `unbalanced_images` and `repeat_missing_conditions`. The command line `check` prints the balance table and every issue; `run` reports the same warnings, and stops on `different_channel_counts` and `channel_names_differ`. | | Outputs | Unchanged: the checks write nothing, and `plane_histograms.csv` and `run_manifest.json` do not change. | -## 12. Experiment design (`experiment_design.csv`, schema version 1) +## 12. Experiment design (`experiment_design.csv`, schema version 1): an output only A **repeat group** is one biological repeat across all conditions (normalization plan N30): later bricks pool it over the conditions. Before processing, the user pairs the biological repeats, confirms that each repeat group is comparable (N35), and may mark -control conditions (N36). The answers are written to `experiment_design.csv` in the output -folder. +control conditions (N36). Since v0.5.1 (§0, ADR 0002 AR10/AR11) these answers are part of the +published description, and `experiment_design.csv` is **derived** from it (experiment-v1 §11, +`design_derive.derive_design`): written by the publication and by every run, never read back +by preprocessing as an input. The rows marked *(removed in v0.5.1)* below describe what is gone; +the lenient reader `read_experiment_design` stays only for the one-way draft migration of a +legacy file (`draft.migrate_legacy_design`) until SWEEP-PP, and the strict +`read_confirmed_design` (`RunInputs.design`) only until CellPatcher and MorphoFeature read +`repeat_group` from the description (owner answer 3). | Column | Meaning | |---|---| @@ -444,12 +478,12 @@ folder. | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | | Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. | | Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A header that names a column twice (compared ignoring case, outer whitespace and invisible format characters such as a byte-order mark, so ` Repeat_Group` repeats `repeat_group`) or has a column without a name (issue #83, plan §12 question 62: no reader chooses between two values; both readers refuse it with the column and its positions named, and nothing is rewritten), a missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). `load_design(output_folder, repeats)` reads the file with it and brings it up to date with the folders (Reruns), returning `(design, changes)`, the default design when there is no file; a file that cannot be read raises `DesignCheckError` (`unreadable_experiment_design`, §16) and is left as it is (owner decision of 31 October 2026, plan §12 question 43). | -| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. A saved file that cannot be read (`load_design` raises `DesignCheckError`, `unreadable_experiment_design`, e.g. for a `confirmed_on` a spreadsheet program rewrote as `26/10/2026`) is refused and left unchanged (owner decision of 31 October 2026, plan §12 question 42): step 1 shows the refusal, which names the line and column, starts the questions unanswered and writes nothing until the user corrects the file and confirms the labels again, or answers a question, which replaces the file. Nothing falls back to a default design silently, and step 3 does not run until every group is confirmed again. The design belongs to its output folder (issue #98, plan §12 question 63): changing the output folder reads the design from the new folder with `load_design` (an unreadable file refused and left as it is, as above), or starts the questions unanswered when it holds none; the earlier folder's confirmations and controls are never carried over, nothing is written until an answer, and step 3 runs on the new folder's design. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | -| Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1; the run stops on `design_not_confirmed` first and writes its report (§9.2). Confirmed rows without a date get the date of the run. | -| Template from `check` (v0.4.0, plan §12 question 57) | `check` (CLI; `run.write_check_outputs` in Python) also writes the design prefilled from the folders, under the output folder's lock: one row per condition and biological-repeat folder, `repeat_group` the folder name, the current counts, `confirmed=false`, `control=false`. Only when the output folder holds no `experiment_design.csv` (a design there, edited or unreadable, is never replaced) and the folders have no error. | -| `design` command (v0.4.0, plan §12 question 57) | `design --output FOLDER` prints the design as a table and the repeat groups not confirmed yet, and any `repeat_group_conflict` or `invalid_experiment_design`. Options change it without editing the file, in this order: `--repeat-group CONDITION/REPEAT=GROUP` (repeatable; moves a folder, and the groups it leaves and joins are no longer confirmed, as `ExperimentDesign.with_group`), `--control CONDITION` and `--no-control CONDITION` (repeatable), `--confirm GROUP` (repeatable) and `--confirm-all` (every group not confirmed yet). It reads, changes and writes the file under the lock (`output_locked` while a run holds it), with the run's reader (`read_experiment_design`) and validator (`design_issues`); a confirmation fills `confirmed_on` with today's date and the counts are those in the file; a group already confirmed keeps its date. An unknown folder, condition or group, a condition both marked and unmarked, a group to confirm whose rows break the rules (`repeat_group_conflict`, `invalid_experiment_design`) or a design that cannot be written is refused, and nothing is changed (exit status 1). Nothing is confirmed by default. Python: `edit_design(output_folder, DesignEdit(...), on=None)` returns the design and the file written (None when nothing changed), raising `DesignEditError`; `apply_design_edit(design, edit, on)` is the same change without files. `DesignEdit.unconfirm` (v0.4.0, PP-10; not a CLI option) takes confirmations back, after the moves and before the confirmations (an unknown group, or one both confirmed and unconfirmed, also through `confirm_all`, is refused; it never confirms anything). The notebook applies every design answer with `apply_design_edit` (the "Undo" with `unconfirm`) and saves it with `run.design_file.write_design(design, output_folder)`, under the lock. The run reconciles the design with the folders as before, so a group that gained a folder since is asked again. | +| Notebook | The pairing is answered on the design card and kept in the draft (`morphobricks.draft.json`), confirmed with the rest on *Check and save*; the notebook never writes or reads `experiment_design.csv` (v0.5.1). The card's model is rebuilt on the draft in SWEEP-PP (owner answer 4). | +| Command line | *(removed in v0.5.1)* `run --design FILE` and the template `run` wrote: `run` refuses an output folder without a published description (§0). | +| Template from `check` | *(removed in v0.5.1)* `check` writes only its report. | +| `design` command | *(removed in v0.5.1)*: the pairing, confirmations and references are answered in the notebook's describe steps. `apply_design_edit` and `DesignEdit` stay for the notebook's pairing card until SWEEP-PP. | | `describe` command (Stage 1, PR 17) | `describe EXPERIMENT [--output FOLDER] [--exclude PATH] [--publish]` prints the draft experiment description (experiment-v1 §12) in plain language: the proposed ID of each condition, technical-repeat folder and biological repeat (no proposal, with the rule's code, for a folder that cannot be an ID: never repaired), each repeat with its folders and image counts and whether it is confirmed (and the date), the references, the notes of a migration with several dates, and every open question with what the user has to do. The draft is the legacy `experiment_design.csv` of the output folder migrated (`draft.migrate_legacy_design`; one that cannot be read or breaks a rule is exit status 1 and left as it is) when there is one and no published `morphobricks.toml`, else the draft of the folders (`draft.draft_from_discovery`); when `morphobricks.toml` is published (`run.description_run.load_description`), the folders are compared with it (`description_reconcile.reconcile`: unchanged, new, gone, image counts, repeats to confirm again), reading the folder structure only, so the channels are not compared. It reads no image and writes nothing: no output folder, draft, description, design or records file. **It confirms nothing**: an ID, the comparability of a repeat (N35), a reference, a removal and every date are the user's answers (notebook), never given by the command or read from the clock. `--publish` publishes a draft only when no question is open and every decision is confirmed; otherwise it prints `Not published. What is missing:` with every item and exits with status 1. This version holds no answers outside the notebook (the draft file is not in the package), so `--publish` refuses every draft built from folders and publishes nothing; for a published description that matches the folders it says `Nothing to publish` (exit 0); a new, gone or changed folder is listed and the file is left as it is. An error (experiment folder not found, discovery errors, unreadable description or records) is exit status 1; the command takes no lock. | -| Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. | +| Run | The run derives the design from the published description and writes it with the table, published before the manifest, which records its SHA-256 (§9). `run_experiment` takes no design. `plane_histograms.csv` does not depend on the design. | ## 13. Image type diff --git a/docs/user_guide.md b/docs/user_guide.md index bdd88268..08066a28 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -323,10 +323,11 @@ results or comparing another grouping. confirmed; if images are not comparable, process them as separate experiments. - **Controls (optional).** Tick the control conditions. This is recorded for later bricks; nothing uses it yet. - - The answers are saved in **experiment_design.csv** in the output folder (§6) after - every answer, holding the output folder's lock, with the rules of the `design` - command (§5): a repeat group is confirmed only when its folders pass the run's - checks, and a refused answer is shown with the reason and changes nothing. Nothing + - The answers are saved in the draft (`morphobricks.draft.json`) after every answer and + published with the description on *Check and save*; `experiment_design.csv` is + derived from the published description: a repeat group is confirmed only when its + folders pass the run's checks, and a refused answer is shown with the reason and + changes nothing. Nothing is confirmed for you. Next time, if the folders are the same, they are loaded and the questions are folded away; **Change the experiment design** shows them again. A new biological-repeat folder asks again only for the group it joins. @@ -627,36 +628,18 @@ was suggested and how to choose the other one; `check` prints the suggestion too for images that are not three channels of a whole-number pixel type stops the run (`rgb_not_possible`). -`run` also needs the experiment design (§4, step 1). It reads `experiment_design.csv` from -the output folder (or the file given with `--design`). `check` writes that file, prefilled -from the folders, when the output folder holds none (one row per biological-repeat folder of -each condition, the repeat group named after the folder, nothing confirmed, no control); it -never replaces a design that is there. `run` does the same the first time, or when folders -were added, and stops. Fill it in with a spreadsheet program or a text editor: give rows the -same `repeat_group` to pair biological repeats named differently, set `confirmed` to `true` -in every row of a repeat group whose images are comparable, and optionally `control` to -`true` for control conditions. Or use the `design` command, which changes the file for you: - -```bash -morphobricks-preprocessing design --output /data/exp1_MorphoBricks_Preprocessing -morphobricks-preprocessing design --output /data/exp1_MorphoBricks_Preprocessing \ - --repeat-group drug/d2=day2 --control ctrl --confirm day1 --confirm day2 -``` - -The first prints the design as a table and the repeat groups not confirmed yet; nothing is -confirmed unless you ask. The second moves the folder `drug/d2` into the repeat group -`day2` (pairing it with the other conditions' `day2`), marks `ctrl` as a control and -confirms the two groups, filling in today's date. `--confirm-all` confirms every group not -confirmed yet, and `--no-control` unmarks a control. A group whose folders break the rules -(two folders of one condition, a name such as `NA`) is not confirmed, an unknown name is -refused, and the file is then left as it was. Then run again. `check` also says whether the -design is confirmed, or why a saved one cannot be read (naming the line and column). Both -commands write the report, `discovery_report.json` and `discovery_report.txt`, to the -output folder; `run` writes it on every run, also one that -stops (`status` `stopped` or `written`, with the SHA-256 of the manifest it describes). -A run that stops says so, says that no table or manifest was written, and names what it -did write: the report, and the design template when it wrote one. Only a run refused before -it starts (another run holds the lock) writes nothing. +`check`, `run` and `masks` need the experiment description published in the output folder +(`morphobricks.toml`, v0.5.1): the conditions, which biological repeats pair across conditions +and that they are comparable, the channels and the masks, answered in the notebook's describe +steps and saved with *Check and save*. Without it they print `A_DESCRIPTION_REQUIRED` with what +to do, write nothing and exit with status 1. `describe` shows the draft description of the +folders and every question still open. The run writes `experiment_design.csv`, derived from the +description (an output for later bricks; nothing reads it back). `check` and `run` write the +report, `discovery_report.json` and `discovery_report.txt`, to the output folder; `run` writes +it on every run, also one that stops (`status` `stopped` or `written`, with the SHA-256 of the +manifest it describes). A run that stops says so, says that no table or manifest was written, +and names what it did write: the report. Only a run refused before it starts (another run +holds the lock, or no description is published) writes nothing. A later `run` in the same output folder reuses everything the earlier run recorded: the axis roles (`--axes` is needed again only for plain stacks that are new or changed), the @@ -675,14 +658,13 @@ SHA-256 still accept it; the report records the rerun. Other options: | `--memory-limit SIZE` | Working-memory limit per plane, e.g. `2GiB` or `1.5GiB`: a whole number of bytes above 0 (`1.5` or `0.5B` are refused before anything is read) | | `--missing-as-singleton`, `--no-missing-as-singleton` | Confirm that images lacking some dimensions have one plane along them (`check` and `run`; default: the earlier run's answer, else no). `check` then reports what `run` will: an experiment of 2D images and Z-stacks passes `check` once confirmed, and the suggested option says `--axes ZCYX --missing-as-singleton` | | `--image-type fluorescence\|rgb` | The image type (default: the earlier run's, else the suggested one, printed) | -| `--design FILE` | The confirmed experiment design (default: `experiment_design.csv` in the output folder); it is copied to the output folder | | `--mode restart\|full` | `restart` (default) reuses the rows of unchanged images; `full` recomputes every row and check. Both keep your confirmations and replace the results only when the run succeeds | | `--verify-content`, `--no-verify-content` | Reuse rows (earlier or staged) only when each image's content SHA-256 is the one recorded with them, not only its size and modification time; every image is read once more. Default: the earlier run's choice, else on; `--no-verify-content` opts out and the manifest records it. Turning it on or off recomputes every row. With `masks`: also compare each mask's content SHA-256 before and after it is read (same default) | | `--no-masks` | Drop the masks folder an earlier run recorded, with the mask axes stated for its masks (those of OME-Zarr label images are kept) | | `--no-exclude` | Clear the exclusions an earlier run recorded, so every image is analysed again (not together with `--exclude`) | -| `--remove-stale-lock` | Remove the lock left by a run that was stopped or whose computer restarted (`check`, `run`, `masks`, `design` and `calibrate`); give it only when no other run uses the output folder | +| `--remove-stale-lock` | Remove the lock left by a run that was stopped or whose computer restarted (`check`, `run`, `masks` and `calibrate`); give it only when no other run uses the output folder | | `--outlier-limit N`, `--saturation-limit FRACTION`, `--min-images N` | Limits of the intensity checks printed after the run (default 3, 0.01 and 5) | -| `--log FILE` | Write technical details to a file (`run`, `masks`, `convert`). It is checked before anything is written and opened only once the other options are checked: a file inside the experiment or masks folder (given or recorded by the earlier run), inside the folder of the copies (`convert`), the design file given with `--design`, or named as one of the run's files in the output folder (`plane_histograms.csv`, `run_manifest.json`, the reports, `experiment_design.csv`, `calibration.csv`, `mask_relationships.json`, the lock) is refused and nothing is written; so is a file that is a hard link (it has another name). Any other name in the output folder is fine, for example `run.log` | +| `--log FILE` | Write technical details to a file (`run`, `masks`, `convert`). It is checked before anything is written and opened only once the other options are checked: a file inside the experiment or masks folder (given or recorded by the earlier run), inside the folder of the copies (`convert`), or named as one of the run's files in the output folder (`plane_histograms.csv`, `run_manifest.json`, the reports, `experiment_design.csv`, `calibration.csv`, `mask_relationships.json`, the lock) is refused and nothing is written; so is a file that is a hard link (it has another name). Any other name in the output folder is fine, for example `run.log` | | `--masks FOLDER` | The masks folder (§2). `check` lists the mask types found and checks every mask from its header; `run` stops on a mask error and records the masks in `run_manifest.json` | | `--mask-axes TYPE=ROLES` | The axes of a mask type, one role (Y, X, Z or T) per axis as stored, stated once per type: `Cells=TYX` for plain stacks of time points whose files do not name their axes; a 2D mask needs none, and `Cells=YX` (the removed shared 2D mask) is the error `shared_mask_not_supported` (repeatable; `check`, `run`, `masks`). Masks are never reordered. | @@ -690,14 +672,10 @@ After writing, `run` prints the intensity checks of the table, grouped by kind, the channel, the image or biological repeat and the values compared, and what to do once per kind; it draws no plots (the notebook does). -**Exit status.** Every command (`check`, `run`, `masks`, `calibrate`, `design`, `describe` and +**Exit status.** Every command (`check`, `run`, `masks`, `calibrate`, `describe` and `convert`) exits with status 1 when it prints an error, and with 0 otherwise: warnings, intensity warnings included, never change it. So `check` exits with 1 on every error it -reports, also an experiment design it cannot read (it never replaces that file) or one -with a `repeat_group_conflict` or `invalid_experiment_design`, which `run` stops on; and -`design` exits with 1 when the design it prints has a `repeat_group_conflict` or -`invalid_experiment_design`, also after a change it wrote (since v0.4.0; both exited with 0 -before). Options that cannot be read exit with 2. Scripts can therefore stop on any +reports, `A_DESCRIPTION_REQUIRED` included. Options that cannot be read exit with 2. Scripts can therefore stop on any non-zero status. The optional `masks` command reads the masks and tells how they relate: @@ -1056,7 +1034,7 @@ first (`--remove-stale-lock`). Once a run has written its results, `.staging` is written into the hidden folder `.publish`; replacing `run_manifest.json` then makes them the current results in one step, and the other files are moved into place. A run stopped before that step leaves the earlier results exactly as they were; one stopped after it is -finished by the next run (or `check`, `masks`, `design`, `calibrate`). Later bricks always +finished by the next run (or `check`, `masks`, `calibrate`). Later bricks always read one whole set of results, old or new. If a file cannot be moved into place (for example because another program has the table open on Windows), the run says so: close that program and run again. This needs temporary disk space for the new outputs. @@ -1084,7 +1062,7 @@ this computer that is no longer running, the message says the lock looks stale; never removed without your request. `check` holds the lock while it writes its report, `masks` (and **Describe the masks**) from before the first mask is read until `mask_relationships.json` is written (a locked folder is refused before -any mask is read), `design` and `calibrate` while they change their files, +any mask is read), and `calibrate` while it changes its file, and are refused the same way. A stopped run leaves the last good results in place; rerunning checks the table against the manifest and rebuilds a mismatched pair. `mask_relationships.json` is removed by a `full` run and when the masks changed since it @@ -1216,9 +1194,9 @@ continues. | `calibration_unreadable` | Error | `calibration.csv` cannot be read; the message names the line and what is wrong. The file is left as it is: correct it, or remove it to use the sizes in the metadata. | | `unbalanced_images` | Warning | A condition and biological repeat has fewer than half or more than twice the median number of images. Check for missing, misplaced or duplicated images. | | `repeat_missing_conditions` | Warning | A biological repeat is not found in every condition. It is still used. If it has another folder name in some condition, pair it in the experiment design (step 1). | -| `design_not_confirmed` | Error | Some repeat groups are not confirmed as comparable. Confirm them in step 1, with the `design` command (`--confirm GROUP`, `--confirm-all`), or set `confirmed` to `true` in `experiment_design.csv`. | +| `design_not_confirmed` | Error | Some repeat groups are not confirmed as comparable. Confirm them in the notebook's describe steps and save the description again. | | `rgb_not_possible` | Error | The image type RGB was chosen, but the images are not three channels of a whole-number pixel type. Choose fluorescence, or check the axis roles: the RGB samples must have the role C. | -| `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another group. | +| `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another replicate on the *Replicates* page and save the description again (*Check and save*). | | `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, ``, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. Correct `experiment_design.csv` (the message names the line and column). | | `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a column named twice or without a name, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. | | `unreadable_experiment_design` | Error (`load_design`, notebook step 1, `check`) | The saved `experiment_design.csv` in the output folder cannot be read even leniently (for example a date a spreadsheet program rewrote as 26/10/2026, a repeat group `NA`, or a header that names a column twice, such as two `repeat_group` columns, or has a column without a name); the message names the line and column. No reader chooses between two columns of one name: delete one. The file is left as it is. Correct it, or answer the questions again in step 1, which replaces it. | @@ -1271,43 +1249,12 @@ result = run_experiment("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/data/exp1", mapping) records = read_plane_histograms(result.csv_path) ``` -The experiment design is optional in Python: without `design=`, the run neither asks for -nor writes one. To record it as the notebook does (`load_design` returns the saved design -brought up to date with the folders, or the default design when there is no file yet; a -saved file that cannot be read raises `DesignCheckError` instead, never a default design; -breaking in v0.4.0, plan §12 question 43: until v0.3.4 it returned a -third value, `problem`, with the default design): - -```python -from morphobricks_preprocessing import load_design - -# An existing file that cannot be read raises DesignCheckError (unreadable_experiment_design, -# naming the line and column) and is left as it is: correct it, do not overwrite it. -design, changes = load_design(result.output_folder, check.dataset.repeats) -design = design.with_group(("treated", "rep2_day2"), "rep2") # pair a repeat -for group in design.groups: - design = design.with_confirmed(group, True, "2026-10-26") -result = run_experiment("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/data/exp1", mapping, design=design) -pairing = design.repeat_groups() # {(condition, biological_repeat): repeat_group} -``` - -The `design` command's Python equivalent changes `experiment_design.csv` in the output -folder under its lock (refusing what the command refuses, with `DesignEditError`): - -```python -from morphobricks_preprocessing import DesignEdit, edit_design - -design, written = edit_design( - "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/data/exp1_MorphoBricks_Preprocessing", - DesignEdit( - repeat_groups={("treated", "rep2_day2"): "rep2"}, - control=("ctrl",), - confirm=("rep1", "rep2"), - ), -) # written is None when nothing changed -``` - -Later bricks read the file with `read_experiment_design(path)`. +The output folder must hold the published experiment description (`morphobricks.toml`, +written by the notebook's *Check and save*); without it `run_experiment` returns +`written=False` with the one error `A_DESCRIPTION_REQUIRED` and writes nothing. The run writes +the `experiment_design.csv` derived from the description; `run_experiment` takes no design +(v0.5.1). Later bricks read the pairing from the description (`RunInputs.fovs()`: +`repeat_group`, `condition_id`, `technical_repeat_id`). Stated pixel sizes (§5) are built with `calibration_entry` and written with `write_calibration`; the next run records them, and later bricks read every image's sizes