PP-13: migration guide, CHANGELOG, Colab release check, docs pass; version 0.4.0rc4

CHANGELOG.md

@@ -0,0 +1,143 @@
+# Changelog
+
+All notable changes to `morphobricks-preprocessing`. Numbers are pull requests (PR) and
+issues of [CellMigrationLab/morphobricks-preprocessing](https://github.com/CellMigrationLab/morphobricks-preprocessing).
+The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature. The decisions
+are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
+[migration guide](docs/migration-0.3-to-0.4.md).
+
+## 0.4.0 (unreleased; release candidates rc1 to rc4)
+
+Preprocessing becomes the one step that discovers, validates, records and serves an
+experiment. Every later brick reads the run through `RunInputs`. Nothing is guessed: axes,
+calibration, masks and design come from the headers or from the user's recorded answers.
+
+Release candidates: **v0.4.0rc1** (after PP-6, #112), **v0.4.0rc2** (after PP-8d, #118),
+**v0.4.0rc3** (#125, after PP-10d) and **v0.4.0rc4** (PP-13, for the round-2 data-flow audit
+and the bricks' migration).
+
+### Breaking
+
+- **Manifest version 2** (PP-4, #110). It records per file `roles`, `axes_from` and
+  `pixel_sizes` by role with unit and source, and per mask `size`, `mtime_ns`, `roles`,
+  `axes_from` and `shared`. A v1 manifest gives no reusable row and no recorded input:
+  **every v0.3 run must be run again.** A row is reused only when its `RowKey` matches.
+- **The axis check** (PP-1, #64). Roles that swap the header's Y and X, contradict the
+  header, or put X before Y in a plain stack are errors (`axes_yx_swapped`,
+  `axes_contradict_header`, `axes_x_before_y`). Nothing is read transposed. A plain stack's
+  confirmation covers only the files it was given for (`plain_stack_not_confirmed`, PP-4).
+- **Modes replace `force_rebuild`** (PP-7, #113): `mode="restart"` (default) and
+  `mode="full"`, CLI `--mode`. A later run reuses every recorded input. A value given that
+  differs is reported with its effect. `--no-masks` and `--no-exclude` drop the recorded
+  masks and exclusions.
+- **Lock** (PP-7, #113): an `O_EXCL` lock file in the output folder. A second run is refused
+  with `output_locked`. `--remove-stale-lock` / `remove_stale_lock()` removes a stale lock
+  only on request.
+- **`image_named_as_mask` is an error** (PP-5, #111). Masks are checked as stored and never
+  converted. OME-Zarr `labels/` are always used. Plain-stack masks need `--mask-axes` per
+  type. A 2-D mask of a multi-plane image needs `--mask-axes TYPE=YX` (shared). A mask whose
+  header names X before Y is refused (`mask_layout`, PP-8b, #115).
+- **Bricks read runs through `RunInputs`** (PP-6, #112), which refuses v1 manifests,
+  changed files and checksums, and files the run did not record (`UnrecordedFilesError`,
+  PP-8b, #115).
+- **OME-Zarr**: OME-NGFF 0.1 and 0.2 stores, and stores without `axes`, are
+  `unreadable_ome_zarr`. Every OME-Zarr image is matched by its declared roles
+  (`ome_zarr_axes_mismatch`) (#129; issues #72, #73). An OME-Zarr image's fingerprint
+  includes its file inventory: images recorded by v0.4.0rc3 or earlier count as changed, and
+  `RunInputs` refuses such runs (PP-12, #132; issue #86).
+- **Links** in the experiment and masks folders are followed only inside them
+  (`link_outside_folder`, `linked_twice`, `link_loop`) (#128; issue #81).
+- **Exit status**: every command exits with 1 when it prints an error. `check` and `design`
+  exited with 0 on some errors (#133; issue #93).
+- **Experiment design**: every design reader refuses a header with a duplicate or unnamed
+  column. `RunInputs.design()` uses the strict reader (#130; issue #83). `load_design`
+  returns `(design, changes)` and raises `DesignCheckError` for an unreadable file (#62).
+- **Conversion records** are version 2, with each copy's SHA-256. Version 1 records
+  (v0.4.0rc3 or earlier) are refused (PP-11c, #126; issue #87).
+- `copy_changed_after_conversion` is an error that stops the run (PP-12, #132).
+- `--memory-limit` must be a whole number of bytes above 0 (PP-11c, #126; issue #80).
+
+### Added
+
+- `RunInputs.open(output, experiment_root=None, masks_root=None)`, with `fovs()`,
+  `pixel_sizes()`, `read_plane()`, `read_image()` (lazy), `mask_types()`, `read_mask()`
+  (labels checked), `histograms()`, `design()` and `image_type()`. It is picklable.
+  `testing.make_clean_run` writes a clean run for the bricks' tests (PP-6, #112).
+  `unrecorded_files()` (PP-8b, #115).
+- `discovery_report.json` and `.txt` on every run, also when it stops, with what was
+  reused, recomputed, resumed and replaced (PP-7, #113).
+- Calibration: `calibration.csv` in the output folder (a size or `pixels` per experiment,
+  folder or image), recorded per axis with its source. CLI `calibrate`, `write_calibration`,
+  `read_calibration`, `review_calibration` (PP-8, #114). No Z or T size is recorded for an
+  image without that axis (`calibration_axis_absent`, PP-8c, #117).
+- CLI `design` (`--repeat-group`, `--control`, `--no-control`, `--confirm`,
+  `--confirm-all`) and `edit_design` / `DesignEdit`. `check` writes a prefilled design
+  template (PP-8c, #117).
+- Named-axis copies: CLI `convert EXPERIMENT --to FOLDER --axes ROLES` and
+  `convert_experiment`. Images with axis errors are written as OME-TIFF with the confirmed
+  roles; other images are copied byte for byte; the conversion record
+  `.morphobricks-conversion.json` is written last (PP-11, #124). Copy SHA-256 and a single
+  metadata epoch (PP-11c, #126; issues #87, #88, #92, #96). `preview_conversion` (PP-11b,
+  #127).
+- Staging and resume: each image's rows are staged in `.staging` and reused by the next run
+  when their key and fingerprint still match. Crash-consistent publication through
+  `.publish`, committed by the manifest's `os.replace` (PP-12, #132). Images that change
+  during the run are `source_changed_during_run` (issue #90).
+- Notebook: Google Colab and JupyterLab support, pinned to `RELEASE_TAG` (PP-8d, #118). The
+  **Pixel sizes** and **Experiment design** cards, the **Mode** choice, the report, and
+  **Remove the lock of a stopped run** (PP-10a, #120). The **Masks folder** field with mask
+  axes per type (PP-10c, #122). **Describe the masks** (PP-10d, #123). **Make named-axis
+  copies** (PP-11b, #127).
+- `_cp_masks` and `_masks` mask suffixes (PP-5, #111).
+- Code standards (design v0.4 §9) enforced by `tools/check.py --rules` with a shrink-only
+  baseline (PP-0, #65). The pinned embedding API for MorphoRelate (PP-9, #119).
+- Docs: the [migration guide](docs/migration-0.3-to-0.4.md), this changelog, and the
+  [Colab hand check](docs/colab-release-check.md) for the major release (PP-13). The roadmap
+  (#116).
+
+### Fixed
+
+- Output files get the permissions of a plain `open()` (your umask) instead of `0600`
+  (PP-8c, #117).
+- A stopped run says what it wrote (PP-8c, #117).
+- `read_plane` and `read_mask` raise `TypeError` for a `z` or `t` that is not an integer
+  (PP-8c, #117).
+- The `masks` command uses the recorded inputs (PP-8c, #117).
+- `--no-masks` after a run with stated mask axes was refused (PP-10c, #122).
+- Malformed OME-Zarr calibrations, and header sizes that overflow or are not positive
+  finite, give unknown sizes with `invalid_physical_calibration` (#129, #133; issues #94,
+  #93). Every conversion to micrometres is positive and finite (`to_micrometers`).
+- Technical-repeat folders whose images are all excluded are not counted (#130; issue #82).
+- Changing the notebook's output folder reloads that folder's design (#130; issue #98).
+- `--log` is checked before it is opened, and refused inside the inputs or on a protected
+  output (#128; issue #99).
+- `same_objects` is a one-to-one maximum matching; `mask_relationships.json` is report
+  version 2 (#131; issue #67).
+- `different_sizes` and `different_plane_counts` follow the confirmed roles (#131; issue
+  #74).
+- The conversion checks memory and free space before writing and refuses 64-bit integer
+  images (PP-11, #124; issues #75, #76). The run reads the copies themselves (issue #79).
+- Masks one pixel high or wide are read as 2-D `YX` masks (#63).
+- Step 1 keeps an unreadable `experiment_design.csv` untouched (#61).
+
+### Removed
+
+- `convert_tiffs` and `zarr_export` (OME-Zarr copies in `<output>/ome_zarr`):
+  `ConversionSettings`, `ConversionEstimate`, `estimate_conversion`, `OmeZarrWriter`,
+  `ome_zarr_relative_path`. Also the CLI `--convert-tiff`, `--chunk-size`, `--compression`,
+  `--compression-level` and `--pyramid-levels`, and the notebook's conversion card (PP-2,
+  #66).
+- The deprecated stand-ins of the code moved to `morphobricks_normalization` (`_moved`,
+  `csv_io`, `dtypes`, `histogram`, `normalization`, `pooling`, `records` and their
+  top-level names) (PP-2, #66).
+- `check_declared_axes` and `declared_axes_transposed` (PP-2, #66).
+- `find_masks(ome_zarr_labels=)` (PP-5, #111).
+- `run_experiment(force_rebuild=)` and `--force-rebuild` (PP-7, #113).
+- `notebook_options.PYRAMID_OPTIONS` (PP-10b, #121).
+- The backup-and-restore publication protocol (PP-12, #132).
+- Compatibility of `PhysicalSizes` pickles across versions is not guaranteed (owner
+  decision, design v0.4 change log).
+
+## 0.3.4 and earlier
+
+See the release notes of the tags and [docs/plan.md](docs/plan.md) §12, questions 33 to 41.

README.md

@@ -51,9 +51,15 @@ Pre-alpha. Implemented and tested:
   and tells how the kinds relate (inside, grouped by, same objects, with the counts behind
   it, such as cells with two nuclei) and their hierarchy, in `mask_relationships.json`
   (`mask_scan`, command `masks`). A mask is 2D, or has depth (Z) and/or time (T) planes
-  named in its own metadata, one per image plane; a 2D mask of a Z-stack or time-lapse is
-  refused, projections included (`mask_planes_mismatch`), and the manifest records each
-  mask's `axes` and `shape`
+  named in its own metadata (or stated once per mask type), one per image plane; a 2D mask
+  of a Z-stack or time-lapse is refused (`mask_planes_mismatch`) unless its type is stated
+  as one mask shared by every plane (`--mask-axes Cells=YX`); masks are checked as stored
+  and never converted, and the manifest records each mask's axes, roles and size
+- Reading a run from another brick: `RunInputs.open(output_folder)` gives the FOVs, pixel
+  sizes, planes, masks, design and image type the run recorded, and refuses a run whose
+  files changed or that misses files (`run_inputs`)
+- Interrupted runs resume, and results are published in one crash-consistent step
+  (`run.staging`, `run.resume`, `run.generation`)
 - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb))
   and a command line (`morphobricks-preprocessing check` / `run` / `masks` / `calibrate` /
   `design` / `convert`)
@@ -68,6 +74,10 @@ removed in v0.4.0).
 Optional (the `zarr` extra): OME-Zarr images as input (`zarr_input`). **New users: start with the
 [user guide](docs/user_guide.md).**
 
+**Upgrading from v0.3:** v0.4.0 is a breaking release. Every v0.3 run must be run again, and
+bricks read runs through `RunInputs`. See the [migration guide](docs/migration-0.3-to-0.4.md)
+and the [CHANGELOG](CHANGELOG.md).
+
 ## Installation
 
 ```bash
@@ -100,16 +110,17 @@ LabConstrictor's
    needs ipywidgets 8). Locally Step 0 prints the Python version, platform and working folder.
    No shell or line magics are used; pip runs with the kernel's Python.
 3. When the package is missing (always on a fresh Colab runtime), Step 0 installs the release
-   named by `RELEASE_TAG` (currently `v0.4.0rc3`), never the moving `main` branch, and prints
+   named by `RELEASE_TAG` (currently `v0.4.0rc4`), never the moving `main` branch, and prints
    which source and version it uses. A checkout around the notebook, or an installed package,
    is used first.
 4. `current_version` and `notebook_name` in Step 0 are the names LabConstrictor's version check
    reads; [`notebooks/notebook_latest_versions.yaml`](notebooks/notebook_latest_versions.yaml)
    lists the latest version. At each release, `RELEASE_TAG`, `current_version`, the index and
    the tag named in the Step 0 Colab callout move to the new tag before it is tagged
    ([plan](docs/plan.md) §10, release checklist); `tests/test_notebook_file.py` checks that
-   they agree, that the rules above hold, and runs the notebook in a local kernel. The Colab
-   run itself is checked by hand.
+   they agree, that the rules above hold, and runs the notebook in a local kernel. Until the
+   major release the notebook is verified headless only; the Colab run is checked by hand on
+   the major release with [docs/colab-release-check.md](docs/colab-release-check.md).
 5. On Colab, Step 0 installs the `notebook` and `zarr` extras; locally it installs `notebook`.
 6. Steps 0 and 1 have *Colab users* / *JupyterLab users* callouts. On Colab, keep the data and
    the output folder on Google Drive; the [user guide](docs/user_guide.md) §1 explains why, and

docs/colab-release-check.md

@@ -0,0 +1,100 @@
+# Colab hand check for the major release
+
+**Until this check is done, the notebook is verified headless only.** The tests
+(`tests/test_notebook_file.py`, `tests/test_notebook_*.py`) run the whole notebook in a local
+kernel and drive its widgets without a browser, and they fake `google.colab`. No release
+candidate of v0.4.0 has been run by hand on Google Colab.
+
+The owner runs this check by hand on Google Colab **once, for the major release (v0.4.0)**,
+after the round-2 data-flow audit and the matrix run (owner, 2 November 2026). Release
+candidates are not checked on Colab. The list collects the hand-check lists of PR #118
+(PP-8d, Colab readiness), PR #123 (PP-10d, Describe the masks) and PR #127 (PP-11b, Make
+named-axis copies). Items were updated for later changes: PP-10a replaced the stale-lock code
+cell with a button, and PP-12 replaced the `.run-*` backup folders with `.publish` and
+`.staging`.
+
+Use a dataset on Google Drive with a masks folder, and a second small experiment of
+Y/X-swapped OME-TIFFs for section F. Note the tag, the date and any failure next to each item.
+Report a failure as an issue before the final tag.
+
+## A. Step 0 on a fresh runtime (PR #118)
+
+1. Open the notebook from the release tag on a fresh runtime and run Step 0.
+   - [ ] The Drive authorisation window appears and the mount succeeds.
+   - [ ] Step 0 prints "Using release v0.4.0 from GitHub" and installs
+     `morphobricks-preprocessing[notebook,zarr]`.
+   - [ ] The green ready card shows the expected package version.
+   - [ ] No "Restart session" request appears. If one does, restart, run Step 0 again and
+     note it.
+2. - [ ] Run Steps 1–4. Every widget renders and reacts: buttons, dropdowns, accordions, the
+     preview image, sliders, progress bars and matplotlib plots. If anything is blank, try
+     without `enable_custom_widget_manager()` and report it.
+
+## B. Step 1 (PR #118)
+
+3. - [ ] **Browse…** shows the "could not be opened … Copy path" message.
+   - [ ] A pasted **/content/drive/MyDrive/…** path checks fine.
+   - [ ] The default output folder lands on Drive next to the experiment.
+
+## C. Step 3 on the full dataset (PR #118)
+
+4. - [ ] The run finishes.
+   - [ ] On Drive, the output folder has `plane_histograms.csv`, `run_manifest.json`,
+     `experiment_design.csv` and the report (`discovery_report.txt` and `.json`).
+   - [ ] There are no "(1)" duplicate files, and no leftover `.publish`, `.staging` or
+     `.<name>.<random>.tmp` files.
+   - [ ] There is no lock file (`.morphobricks-preprocessing.lock`) after the run.
+5. - [ ] Run Step 3 again in the same session. Everything is unchanged and nothing is read.
+6. - [ ] Disconnect the runtime (Runtime › Disconnect and delete runtime) and open a new one.
+     Run Steps 0–3 again on the same Drive folders. The rows are reused (unchanged). This
+     shows the Drive times are stable across mounts.
+7. Optional: stop a run part-way (interrupt, or disconnect the runtime).
+   - [ ] The next check names the lock holder (`output_locked`), and **Remove the lock of a
+     stopped run** clears it.
+   - [ ] The next run resumes: the report lists the images read before the stop as
+     `resumed`.
+8. - [ ] Optional: an experiment with OME-Zarr images on Drive checks and runs (the `zarr`
+     extra is installed).
+
+## D. Describe the masks (PR #123)
+
+9. - [ ] In Steps 1–3 with an experiment and a masks folder on Drive, before **Compute
+     histograms**, the card shows a greyed button: "Available when the output folder holds a
+     run that recorded masks".
+10. - [ ] After the run, the help line says "Describes the run in … written …, of …: N mask
+    types from …", and the button is enabled.
+11. - [ ] Click it: a spinner, a progress bar per image, then a green bar. The tables are
+    labelled as suggestions, and `mask_relationships.json` appears in the output folder.
+12. - [ ] Run Step 3 again: the card clears.
+13. - [ ] **Reopen** the notebook (Runtime › Restart session) and choose the same folders in
+    Step 1 without running. The button is enabled, names the earlier run, and works.
+14. - [ ] Move the masks folder on Drive and click: a red card says the masks folder does not
+    exist, and nothing is written.
+15. - [ ] The tables scroll inside the card, with no horizontal page scrollbar.
+
+## E. Make named-axis copies (PR #127)
+
+16. - [ ] Run Step 0, then Step 1 on an experiment of Y/X-swapped OME-TIFFs on Drive, and
+    press **Check folder**.
+17. - [ ] In Step 2, swap the Y and X roles. The red axis check appears, with the **Make
+    named-axis copies (optional)** card under it. Its button is disabled.
+18. - [ ] Press **Show preview** and **Yes, the preview looks right**, then **Yes, write the
+    copies with these roles**.
+19. - [ ] Press **Browse…**: the amber card names **Copy path**. Paste a Drive folder inside
+    the experiment and check that it is refused. Then paste one next to the experiment.
+20. - [ ] Check the plan lists and the space, then press **Make named-axis copies**. The bar
+    moves, the green card shows the record path, and the pixel-size note lists the copies.
+21. - [ ] Press **Check the copies in step 1**. Step 1 shows the copy folder, the same output
+    folder and a clean check. The roles are suggested from the copies.
+22. - [ ] Confirm the labels, preview, roles and image type, then run Step 3. The report says
+    "named-axis copies of …".
+23. - [ ] Optional: disconnect the runtime during a conversion, reconnect, and convert into
+    the same folder. The lock card appears. Press **Remove the lock of a stopped run**, then
+    convert again.
+
+## After the check
+
+- Record the result (tag, date, items passed, failures with their issue numbers) in the
+  release PR of v0.4.0.
+- Then update this file's first paragraph, the README's "Notebook" section and plan §10
+  (release checklist) to say which release was checked on Colab.