PP-C3: document that every run is described (v0.5.1, 3/3)

AGENTS.md

@@ -168,6 +168,9 @@ src/morphobricks_preprocessing/
                     preprocessing's discovery and pairing; RunInputs.open raises
                     UnrecordedFilesError for them (PP-8b, contract §9.1)
   run_inputs_text.py the texts RunInputs refuses a run with (moved from run_inputs.py in PP-12)
+  required_description.py every run needs a published morphobricks.toml (v0.5.1, ADR 0002
+                    AR10): is_published, the A_DESCRIPTION_REQUIRED issue and its fix, shared by
+                    the run, the mask step, the CLI and RunInputs
   mask_record.py    the manifest's masks entry with fingerprints and roles (manifest v2); the
                     fingerprint is the one bound to the mask's header (issue #160)
   mask_binding.py   masks bound to their fingerprints (issues #160, #165): changed_masks,
@@ -217,7 +220,7 @@ src/morphobricks_preprocessing/
                     moved from report.py in PP-10d
   discovery_report.py the run report (discovery_report.json and .txt, design v0.4 §4): status,
                     manifest SHA-256, what was found, reused, recomputed and replaced
-  cli.py, __main__  `morphobricks-preprocessing check|run|masks|calibrate|design|convert` (also `python -m morphobricks_preprocessing`);
+  cli.py, __main__  `morphobricks-preprocessing check|run|masks|calibrate|describe|convert` (also `python -m morphobricks_preprocessing`);
                     cli_run.py holds `run`'s options and output; cli_calibrate.py `calibrate`;
                     cli_convert.py `convert`; log_target.py where `--log` may write (#99),
                     by write_target.py's write_target_refusal (the shared rule other bricks
@@ -287,8 +290,8 @@ src/morphobricks_preprocessing/
                     mask_answers.py (pure) the mask answers of the draft; mask_evidence.py the
                     evidence beside the question (images with the mask, dtype, values in one
                     image, the mask on that image)
-  notebook_described.py a described run (PreprocessingNotebook(describe=True), the shipped
-                    notebook): the overrides that retire the labels toggle, the design card's
+  notebook_described.py the described run (every PreprocessingNotebook; there is no
+                    undescribed notebook since v0.5.1): the overrides that retire the labels toggle, the design card's
                     comparable questions and CSV writes, the IDs toggle and the Image type card,
                     and reopen from the published file
   notebook_check_save.py the Check and save screen (sentences, one "same way?" per replicate,
@@ -331,7 +334,8 @@ src/morphobricks_preprocessing/
 notebooks/MorphoBricks_Preprocessing.ipynb   thin notebook: every code cell calls notebook_ui only
 notebooks/notebook_latest_versions.yaml      the notebook's latest version, for LabConstrictor's version check
 tests/              pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py); the
-                    histogram science goldens (tests/histogram_goldens.py, tests/data)
+                    histogram science goldens (tests/histogram_goldens.py, tests/data);
+                    described test runs of a tree (tests/described_tree.py)
 experiments/binning_study/      the study behind the binning decisions (not part of the package)
 ```
 
@@ -658,12 +662,11 @@ functions the tests use.
   `RunInputs.design()` uses it too (#83).
 - **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that
   gain a folder, drops removed folders.
-- **Notebook** writes the file after every answer once the labels are confirmed; a
-  confirmed design for the same folders is folded away behind **Change the experiment
-  design**; step 3 is gated on it. `folder_step(show_design=False)` (embedding, Q31) asks
-  and writes nothing, and step 3 runs without a design. **CLI** `run` reads it (or `--design`), else writes the
-  reconciled template and exits 1. **API** `run_experiment(design=None)` is optional; with
-  a design it is written with the table and `experiment_design_sha256` goes in the manifest.
+- **v0.5.1: an output only.** Every run is described (`A_DESCRIPTION_REQUIRED` without a
+  published `morphobricks.toml`, contract §0); the run derives `experiment_design.csv` from the
+  description and writes it with the table (`experiment_design_sha256` in the manifest). The
+  notebook keeps the pairing in the draft; there is no `design=`, `--design`, `design`
+  command or `check` template any more.
 
 ### Image type (contract §13, follow-up P4)
 
@@ -1033,7 +1036,8 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb
   input read directly (TIFF to OME-Zarr conversion and the `_moved.py` re-exports were removed);
   optional masks, the mask step and the notebook's **Describe the masks**; named-axis copies
   (`convert`, notebook card); calibration; resumable, crash-consistent runs.
-- **Command line:** `check`, `run`, `masks`, `calibrate`, `design`, `convert`.
+- **Command line:** `check`, `run`, `masks`, `calibrate`, `describe`, `convert` (`check`,
+  `run` and `masks` need a published description, v0.5.1).
 - **Next:** [docs/roadmap.md](docs/roadmap.md).
 - **Not yet verified:**
   - the exact look of the notebook in a real JupyterLab;

CHANGELOG.md

@@ -6,6 +6,92 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature, on b
 are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 [migration guide](docs/migration-0.3-to-0.4.md).
 
+## Unreleased (v0.5.1)
+
+### PP-C: every run is described; undescribed runs and their compatibility code removed
+
+No backward compatibility before the first pipeline release (owner policy of 10 November 2026,
+MorphoBricks-Spec ADR 0002 AR10 and AR11, Stage 2 plan §0.1 and §0.2). **Breaking.**
+
+- **Every run needs a published experiment description.** `run_experiment`,
+  `prepare_mask_step`, the command line's `check`, `run` and `masks`, and `RunInputs.open` refuse
+  an output folder without `morphobricks.toml` with the one code `A_DESCRIPTION_REQUIRED`
+  (owner ruling 1), before anything is read or written, naming the fix: the notebook's describe
+  steps (*Check and save*), or the command line's `describe` to see what is open. The refusal
+  comes before the recorded run or any image header is read (also for an output folder inside
+  the experiment or the masks, which is said beside it) and, on the command line, before
+  `--log` is opened or `--remove-stale-lock` applied. New:
+  `required_description` (`is_published`, `description_required_issue`, `undescribed_issues`) and
+  `DescriptionRequiredError` (a `RunInputsError`, exported). Contract §0.
+- **Removed: the undescribed ("v0.4") run.** Its `NO_DESCRIPTION` warning; the
+  `with_description` argument of `preflight`, `remasked` and `masks.find_masks` and
+  `run.preflight.description_in_use` (the name refusals `R_FOLDER_NOT_NFC` and
+  `R_MASK_TYPE_NAME_INVALID` always apply); the `None` answers of `RunInputs.description()`,
+  `section_digests()`, `images_without_header_channel_names()`, `references()` and
+  `mask_semantics()`; `make_clean_run`'s undescribed run (below).
+- **Removed: `experiment_design.csv` as an input to grouping.** `run_experiment(design=...)`,
+  `run --design FILE`, the `design` command (`cli_design.py`), `run.design_file`
+  (`write_design_template`, `write_design`, `edit_design`, `design_for_run`), `check`'s design
+  template (`run.write_check_outputs`, `CheckOutputs`; `write_check_report` stays),
+  `recorded.design_change` and `planned_changes(design=...)`, and the notebook's writes and reads
+  of the file. The run now derives the design from the published description
+  (`design_derive.derive_design`) and writes it with the table (`experiment_design_sha256` in
+  the manifest): an output only, for every run. Stays until SWEEP-PP: the lenient reader
+  `read_experiment_design` (only the one-way draft migration `migrate_legacy_design` uses it),
+  `ExperimentDesign`/`default_design`/`apply_design_edit` behind the notebook's pairing card
+  (owner answer 4), and the strict `read_confirmed_design`/`RunInputs.design` until CellPatcher
+  and MorphoFeature read `repeat_group` from the description (owner answer 3).
+- **Identities.** No identity is taken from a folder name in place of the description: the
+  run's grouping comes from the description's repeats. `position_id` stays the image file stem,
+  always together with the canonical condition, repeat and technical-repeat IDs (owner ruling 2;
+  no change).
+- **Run manifest: version `"4"` only** (owner ruling 16). Every run writes version 4 (with
+  `experiment_description`; content hashes exactly when `verify_content`), and only version 4
+  is read: `"1"`, `"2"`, `"3"` (and any later version) are refused with "Run preprocessing
+  again", never converted; the next run reads every image again. Removed:
+  `MANIFEST_VERSION = "2"`, `STRICT_MANIFEST_VERSION`, `DESCRIBED_MANIFEST_VERSION`,
+  `LEGACY_MANIFEST_VERSION`, `version_for`, the `manifest_before_v040` source issue and the
+  version-1 branches of row reuse, recorded inputs and `RunInputs`. The tolerant reading of
+  version-1 file fields (`manifest_fields`' `strict=False`) goes in SWEEP-PP.
+- **Notebook: always described.** `PreprocessingNotebook` has no `describe` argument and every
+  `_describe` branch is gone: the labels toggle and the global *Image type* card are never
+  shown (the type comes from the saved description), and the design card no longer reads or
+  writes `experiment_design.csv` (its undescribed `_open_design`, `_save_design`,
+  `_design_load_note`, `_refresh_design_status`, `_design_pending` and the check's design
+  template are removed). The widgets of the retired cards stay until SWEEP-PP. The embedding
+  API keeps `folder_step`, `axis_step`, `run_step` and `show_design`; an embedding brick must
+  now let the user describe and save before the run (below). **The shipped notebook is
+  unchanged** until the v0.5.1 release: it installs `RELEASE_REF` (v0.5.0, `fd7afd3`) and calls
+  that code's `PreprocessingNotebook(describe=True)`; the release PR moves `RELEASE_REF` and the
+  call to `PreprocessingNotebook()` together (`test_notebook_file.RELEASE_CALLS`, the release
+  check list).
+- **Refusal texts.** `design_not_confirmed`, `repeat_group_conflict` and
+  `repeat_missing_conditions` name the notebook's describe steps / *Replicates* page and
+  *Check and save*, not the removed `design` command or an edit of `experiment_design.csv`.
+- **`testing.make_clean_run` makes a described run.** Same tree and FOV keys as before
+  (`ctrl`/`drug`, `rep1`, `a`/`b`); it publishes `testing.clean_description(...)` (channels
+  `ch1`..`chN`, one `instances` mask per mask type, `design=True` makes `drug` reference
+  `ctrl`), writes the derived `experiment_design.csv` every time, and runs the mask step when
+  there are masks, so every mask is `checked`. New: `testing_described.publish_and_run`.
+- **Tests.** The undescribed-run tests now run described trees (`tests/described_tree.py`:
+  `run_described`, `main_described`, `describe_tree`, the description the user would confirm
+  for a tree the test wrote, every proposal accepted; never part of the package). Deleted as
+  undescribed- or legacy-only: `test_parity_v0_4_0.py` and `parity_datasets.py`,
+  `test_removed_in_v040.py`, `test_design_command.py`, `test_strict_manifest.py` and
+  `test_described_manifest_version.py` (replaced by `test_manifest_version.py`),
+  `released_code.py`, `tests/data/rc4_run_manifest.json` and `make_rc4_manifest.py`, and the
+  single tests of other files that checked only undescribed or legacy behaviour (listed in the
+  pull request). Kept as science goldens and renamed (owner ruling 6):
+  `test_schema_v2_equivalence.py` is `test_histogram_goldens.py` (`histogram_goldens.py`,
+  `tests/data/golden_*`, `make_histogram_goldens.py`), its version-2 manifest test deleted. CI
+  no longer fetches the whole history (`fetch-depth: 0` was only for the old-code runs).
+
+**Downstream** (each brick is unaffected until its pin moves past this change): every brick's
+test runs must be described (`make_clean_run` does it; a direct `run_experiment` call needs a
+published `morphobricks.toml` first), `run_experiment(design=...)` and the `describe` argument
+of `PreprocessingNotebook` are gone, and a run written by v0.5.0 or earlier is refused until it
+is run again.
+
 ## 0.5.0 - 2026-11-09
 
 Stage 1 of the experiment description (MorphoBricks-Spec experiment-v1, pinned at `1bed024`):

README.md

@@ -33,7 +33,10 @@ Pre-alpha; the current release is v0.5.0 (what is planned next is in
 - Reading a run from another brick: `RunInputs.open(output_folder)`
 - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb))
   and a command line (`morphobricks-preprocessing check` / `run` / `masks` / `calibrate` /
-  `design` / `describe` / `convert`)
+  `describe` / `convert`)
+- **Every run is described** (v0.5.1): the output folder must hold a published experiment
+  description, `morphobricks.toml`, written by the notebook's describe steps (*Check and
+  save*). A run without one is refused with `A_DESCRIPTION_REQUIRED` on every route.
 
 Every feature, message code and output file is described in the [user guide](docs/user_guide.md).
 
@@ -128,31 +131,30 @@ reuses every input the earlier run recorded in the output folder (axis roles, ma
 exclusions, design, image type, settings); `mode="full"` recomputes everything, and
 `exclude=[...]` leaves out damaged files. One run at a time holds the output folder's lock,
 and every run, also one that stops, writes `discovery_report.json` and a text version. The notebook and the
-command line also ask for the experiment design (which biological repeats pair across
-conditions, and that they are comparable) and write `experiment_design.csv`; in Python,
-pass it with `design=` (see the [user guide](docs/user_guide.md) §9). Later bricks read it
-with `read_experiment_design`. They also ask for the image type, `fluorescence` or `rgb`
+notebook asks for the experiment description (conditions, which biological repeats pair
+across conditions and that they are comparable, channels, masks) and publishes
+`morphobricks.toml`; every run needs it, and writes the `experiment_design.csv` derived from
+it (an output only). They also ask for the image type, `fluorescence` or `rgb`
 (suggested by `review_image_type`), and record it in the manifest; in Python, pass it with
 `image_type=`. After writing, the run checks the images' intensities from the table
 (`result.intensity`); `check_table(path)` checks any table.
 
 ## Command line
 
 ```bash
-morphobricks-preprocessing check /data/exp1          # report, suggested --axes, design template
-morphobricks-preprocessing design --output /data/exp1_MorphoBricks_Preprocessing --confirm-all
+morphobricks-preprocessing describe /data/exp1       # the draft description, what is open
+morphobricks-preprocessing check /data/exp1          # report, suggested --axes
 morphobricks-preprocessing run /data/exp1 --axes ZCYX
 morphobricks-preprocessing run --output /data/exp1_MorphoBricks_Preprocessing   # later runs
 morphobricks-preprocessing masks --output /data/exp1_MorphoBricks_Preprocessing # mask step
 morphobricks-preprocessing calibrate --output /data/exp1_MorphoBricks_Preprocessing
 morphobricks-preprocessing convert /data/exp1 --to /data/exp1_named_axes --axes ZXY  # axis errors
 ```
 
-`check` writes the report and, when the output folder holds no design, an
-`experiment_design.csv` prefilled from the folders. `design` prints it and confirms repeat
-groups (`--confirm GROUP`, `--confirm-all`), marks controls (`--control`, `--no-control`)
-and pairs repeats (`--repeat-group CONDITION/REPEAT=GROUP`) without editing the file;
-nothing is confirmed unless asked. `run` and `masks` reuse every input the run in the
+`check`, `run` and `masks` need the experiment description published in the output folder
+(the notebook's describe steps); without it they stop with `A_DESCRIPTION_REQUIRED` and write
+nothing. `describe` shows the draft and what is still to be answered. `check` writes the
+report. `run` and `masks` reuse every input the run in the
 output folder recorded. A run that stops writes its report and says so. Every output file
 gets the permissions a file you save yourself would (your umask). `convert` saves the
 images whose metadata name their axes wrongly or not at all as named-axis OME-TIFF copies