diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..e716cddd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,143 @@ +# Changelog + +All notable changes to `morphobricks-preprocessing`. Numbers are pull requests (PR) and +issues of [CellMigrationLab/morphobricks-preprocessing](https://github.com/CellMigrationLab/morphobricks-preprocessing). +The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature. The decisions +are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the +[migration guide](docs/migration-0.3-to-0.4.md). + +## 0.4.0 (unreleased; release candidates rc1 to rc4) + +Preprocessing becomes the one step that discovers, validates, records and serves an +experiment. Every later brick reads the run through `RunInputs`. Nothing is guessed: axes, +calibration, masks and design come from the headers or from the user's recorded answers. + +Release candidates: **v0.4.0rc1** (after PP-6, #112), **v0.4.0rc2** (after PP-8d, #118), +**v0.4.0rc3** (#125, after PP-10d) and **v0.4.0rc4** (PP-13, for the round-2 data-flow audit +and the bricks' migration). + +### Breaking + +- **Manifest version 2** (PP-4, #110). It records per file `roles`, `axes_from` and + `pixel_sizes` by role with unit and source, and per mask `size`, `mtime_ns`, `roles`, + `axes_from` and `shared`. A v1 manifest gives no reusable row and no recorded input: + **every v0.3 run must be run again.** A row is reused only when its `RowKey` matches. +- **The axis check** (PP-1, #64). Roles that swap the header's Y and X, contradict the + header, or put X before Y in a plain stack are errors (`axes_yx_swapped`, + `axes_contradict_header`, `axes_x_before_y`). Nothing is read transposed. A plain stack's + confirmation covers only the files it was given for (`plain_stack_not_confirmed`, PP-4). +- **Modes replace `force_rebuild`** (PP-7, #113): `mode="restart"` (default) and + `mode="full"`, CLI `--mode`. A later run reuses every recorded input. A value given that + differs is reported with its effect. `--no-masks` and `--no-exclude` drop the recorded + masks and exclusions. +- **Lock** (PP-7, #113): an `O_EXCL` lock file in the output folder. A second run is refused + with `output_locked`. `--remove-stale-lock` / `remove_stale_lock()` removes a stale lock + only on request. +- **`image_named_as_mask` is an error** (PP-5, #111). Masks are checked as stored and never + converted. OME-Zarr `labels/` are always used. Plain-stack masks need `--mask-axes` per + type. A 2-D mask of a multi-plane image needs `--mask-axes TYPE=YX` (shared). A mask whose + header names X before Y is refused (`mask_layout`, PP-8b, #115). +- **Bricks read runs through `RunInputs`** (PP-6, #112), which refuses v1 manifests, + changed files and checksums, and files the run did not record (`UnrecordedFilesError`, + PP-8b, #115). +- **OME-Zarr**: OME-NGFF 0.1 and 0.2 stores, and stores without `axes`, are + `unreadable_ome_zarr`. Every OME-Zarr image is matched by its declared roles + (`ome_zarr_axes_mismatch`) (#129; issues #72, #73). An OME-Zarr image's fingerprint + includes its file inventory: images recorded by v0.4.0rc3 or earlier count as changed, and + `RunInputs` refuses such runs (PP-12, #132; issue #86). +- **Links** in the experiment and masks folders are followed only inside them + (`link_outside_folder`, `linked_twice`, `link_loop`) (#128; issue #81). +- **Exit status**: every command exits with 1 when it prints an error. `check` and `design` + exited with 0 on some errors (#133; issue #93). +- **Experiment design**: every design reader refuses a header with a duplicate or unnamed + column. `RunInputs.design()` uses the strict reader (#130; issue #83). `load_design` + returns `(design, changes)` and raises `DesignCheckError` for an unreadable file (#62). +- **Conversion records** are version 2, with each copy's SHA-256. Version 1 records + (v0.4.0rc3 or earlier) are refused (PP-11c, #126; issue #87). +- `copy_changed_after_conversion` is an error that stops the run (PP-12, #132). +- `--memory-limit` must be a whole number of bytes above 0 (PP-11c, #126; issue #80). + +### Added + +- `RunInputs.open(output, experiment_root=None, masks_root=None)`, with `fovs()`, + `pixel_sizes()`, `read_plane()`, `read_image()` (lazy), `mask_types()`, `read_mask()` + (labels checked), `histograms()`, `design()` and `image_type()`. It is picklable. + `testing.make_clean_run` writes a clean run for the bricks' tests (PP-6, #112). + `unrecorded_files()` (PP-8b, #115). +- `discovery_report.json` and `.txt` on every run, also when it stops, with what was + reused, recomputed, resumed and replaced (PP-7, #113). +- Calibration: `calibration.csv` in the output folder (a size or `pixels` per experiment, + folder or image), recorded per axis with its source. CLI `calibrate`, `write_calibration`, + `read_calibration`, `review_calibration` (PP-8, #114). No Z or T size is recorded for an + image without that axis (`calibration_axis_absent`, PP-8c, #117). +- CLI `design` (`--repeat-group`, `--control`, `--no-control`, `--confirm`, + `--confirm-all`) and `edit_design` / `DesignEdit`. `check` writes a prefilled design + template (PP-8c, #117). +- Named-axis copies: CLI `convert EXPERIMENT --to FOLDER --axes ROLES` and + `convert_experiment`. Images with axis errors are written as OME-TIFF with the confirmed + roles; other images are copied byte for byte; the conversion record + `.morphobricks-conversion.json` is written last (PP-11, #124). Copy SHA-256 and a single + metadata epoch (PP-11c, #126; issues #87, #88, #92, #96). `preview_conversion` (PP-11b, + #127). +- Staging and resume: each image's rows are staged in `.staging` and reused by the next run + when their key and fingerprint still match. Crash-consistent publication through + `.publish`, committed by the manifest's `os.replace` (PP-12, #132). Images that change + during the run are `source_changed_during_run` (issue #90). +- Notebook: Google Colab and JupyterLab support, pinned to `RELEASE_TAG` (PP-8d, #118). The + **Pixel sizes** and **Experiment design** cards, the **Mode** choice, the report, and + **Remove the lock of a stopped run** (PP-10a, #120). The **Masks folder** field with mask + axes per type (PP-10c, #122). **Describe the masks** (PP-10d, #123). **Make named-axis + copies** (PP-11b, #127). +- `_cp_masks` and `_masks` mask suffixes (PP-5, #111). +- Code standards (design v0.4 §9) enforced by `tools/check.py --rules` with a shrink-only + baseline (PP-0, #65). The pinned embedding API for MorphoRelate (PP-9, #119). +- Docs: the [migration guide](docs/migration-0.3-to-0.4.md), this changelog, and the + [Colab hand check](docs/colab-release-check.md) for the major release (PP-13). The roadmap + (#116). + +### Fixed + +- Output files get the permissions of a plain `open()` (your umask) instead of `0600` + (PP-8c, #117). +- A stopped run says what it wrote (PP-8c, #117). +- `read_plane` and `read_mask` raise `TypeError` for a `z` or `t` that is not an integer + (PP-8c, #117). +- The `masks` command uses the recorded inputs (PP-8c, #117). +- `--no-masks` after a run with stated mask axes was refused (PP-10c, #122). +- Malformed OME-Zarr calibrations, and header sizes that overflow or are not positive + finite, give unknown sizes with `invalid_physical_calibration` (#129, #133; issues #94, + #93). Every conversion to micrometres is positive and finite (`to_micrometers`). +- Technical-repeat folders whose images are all excluded are not counted (#130; issue #82). +- Changing the notebook's output folder reloads that folder's design (#130; issue #98). +- `--log` is checked before it is opened, and refused inside the inputs or on a protected + output (#128; issue #99). +- `same_objects` is a one-to-one maximum matching; `mask_relationships.json` is report + version 2 (#131; issue #67). +- `different_sizes` and `different_plane_counts` follow the confirmed roles (#131; issue + #74). +- The conversion checks memory and free space before writing and refuses 64-bit integer + images (PP-11, #124; issues #75, #76). The run reads the copies themselves (issue #79). +- Masks one pixel high or wide are read as 2-D `YX` masks (#63). +- Step 1 keeps an unreadable `experiment_design.csv` untouched (#61). + +### Removed + +- `convert_tiffs` and `zarr_export` (OME-Zarr copies in `/ome_zarr`): + `ConversionSettings`, `ConversionEstimate`, `estimate_conversion`, `OmeZarrWriter`, + `ome_zarr_relative_path`. Also the CLI `--convert-tiff`, `--chunk-size`, `--compression`, + `--compression-level` and `--pyramid-levels`, and the notebook's conversion card (PP-2, + #66). +- The deprecated stand-ins of the code moved to `morphobricks_normalization` (`_moved`, + `csv_io`, `dtypes`, `histogram`, `normalization`, `pooling`, `records` and their + top-level names) (PP-2, #66). +- `check_declared_axes` and `declared_axes_transposed` (PP-2, #66). +- `find_masks(ome_zarr_labels=)` (PP-5, #111). +- `run_experiment(force_rebuild=)` and `--force-rebuild` (PP-7, #113). +- `notebook_options.PYRAMID_OPTIONS` (PP-10b, #121). +- The backup-and-restore publication protocol (PP-12, #132). +- Compatibility of `PhysicalSizes` pickles across versions is not guaranteed (owner + decision, design v0.4 change log). + +## 0.3.4 and earlier + +See the release notes of the tags and [docs/plan.md](docs/plan.md) §12, questions 33 to 41. diff --git a/README.md b/README.md index 4130c46a..07175b0b 100644 --- a/README.md +++ b/README.md @@ -51,9 +51,15 @@ Pre-alpha. Implemented and tested: and tells how the kinds relate (inside, grouped by, same objects, with the counts behind it, such as cells with two nuclei) and their hierarchy, in `mask_relationships.json` (`mask_scan`, command `masks`). A mask is 2D, or has depth (Z) and/or time (T) planes - named in its own metadata, one per image plane; a 2D mask of a Z-stack or time-lapse is - refused, projections included (`mask_planes_mismatch`), and the manifest records each - mask's `axes` and `shape` + named in its own metadata (or stated once per mask type), one per image plane; a 2D mask + of a Z-stack or time-lapse is refused (`mask_planes_mismatch`) unless its type is stated + as one mask shared by every plane (`--mask-axes Cells=YX`); masks are checked as stored + and never converted, and the manifest records each mask's axes, roles and size +- Reading a run from another brick: `RunInputs.open(output_folder)` gives the FOVs, pixel + sizes, planes, masks, design and image type the run recorded, and refuses a run whose + files changed or that misses files (`run_inputs`) +- Interrupted runs resume, and results are published in one crash-consistent step + (`run.staging`, `run.resume`, `run.generation`) - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb)) and a command line (`morphobricks-preprocessing check` / `run` / `masks` / `calibrate` / `design` / `convert`) @@ -68,6 +74,10 @@ removed in v0.4.0). Optional (the `zarr` extra): OME-Zarr images as input (`zarr_input`). **New users: start with the [user guide](docs/user_guide.md).** +**Upgrading from v0.3:** v0.4.0 is a breaking release. Every v0.3 run must be run again, and +bricks read runs through `RunInputs`. See the [migration guide](docs/migration-0.3-to-0.4.md) +and the [CHANGELOG](CHANGELOG.md). + ## Installation ```bash @@ -100,7 +110,7 @@ LabConstrictor's needs ipywidgets 8). Locally Step 0 prints the Python version, platform and working folder. No shell or line magics are used; pip runs with the kernel's Python. 3. When the package is missing (always on a fresh Colab runtime), Step 0 installs the release - named by `RELEASE_TAG` (currently `v0.4.0rc3`), never the moving `main` branch, and prints + named by `RELEASE_TAG` (currently `v0.4.0rc4`), never the moving `main` branch, and prints which source and version it uses. A checkout around the notebook, or an installed package, is used first. 4. `current_version` and `notebook_name` in Step 0 are the names LabConstrictor's version check @@ -108,8 +118,9 @@ LabConstrictor's lists the latest version. At each release, `RELEASE_TAG`, `current_version`, the index and the tag named in the Step 0 Colab callout move to the new tag before it is tagged ([plan](docs/plan.md) §10, release checklist); `tests/test_notebook_file.py` checks that - they agree, that the rules above hold, and runs the notebook in a local kernel. The Colab - run itself is checked by hand. + they agree, that the rules above hold, and runs the notebook in a local kernel. Until the + major release the notebook is verified headless only; the Colab run is checked by hand on + the major release with [docs/colab-release-check.md](docs/colab-release-check.md). 5. On Colab, Step 0 installs the `notebook` and `zarr` extras; locally it installs `notebook`. 6. Steps 0 and 1 have *Colab users* / *JupyterLab users* callouts. On Colab, keep the data and the output folder on Google Drive; the [user guide](docs/user_guide.md) §1 explains why, and diff --git a/docs/colab-release-check.md b/docs/colab-release-check.md new file mode 100644 index 00000000..2c642aea --- /dev/null +++ b/docs/colab-release-check.md @@ -0,0 +1,100 @@ +# Colab hand check for the major release + +**Until this check is done, the notebook is verified headless only.** The tests +(`tests/test_notebook_file.py`, `tests/test_notebook_*.py`) run the whole notebook in a local +kernel and drive its widgets without a browser, and they fake `google.colab`. No release +candidate of v0.4.0 has been run by hand on Google Colab. + +The owner runs this check by hand on Google Colab **once, for the major release (v0.4.0)**, +after the round-2 data-flow audit and the matrix run (owner, 2 November 2026). Release +candidates are not checked on Colab. The list collects the hand-check lists of PR #118 +(PP-8d, Colab readiness), PR #123 (PP-10d, Describe the masks) and PR #127 (PP-11b, Make +named-axis copies). Items were updated for later changes: PP-10a replaced the stale-lock code +cell with a button, and PP-12 replaced the `.run-*` backup folders with `.publish` and +`.staging`. + +Use a dataset on Google Drive with a masks folder, and a second small experiment of +Y/X-swapped OME-TIFFs for section F. Note the tag, the date and any failure next to each item. +Report a failure as an issue before the final tag. + +## A. Step 0 on a fresh runtime (PR #118) + +1. Open the notebook from the release tag on a fresh runtime and run Step 0. + - [ ] The Drive authorisation window appears and the mount succeeds. + - [ ] Step 0 prints "Using release v0.4.0 from GitHub" and installs + `morphobricks-preprocessing[notebook,zarr]`. + - [ ] The green ready card shows the expected package version. + - [ ] No "Restart session" request appears. If one does, restart, run Step 0 again and + note it. +2. - [ ] Run Steps 1–4. Every widget renders and reacts: buttons, dropdowns, accordions, the + preview image, sliders, progress bars and matplotlib plots. If anything is blank, try + without `enable_custom_widget_manager()` and report it. + +## B. Step 1 (PR #118) + +3. - [ ] **Browse…** shows the "could not be opened … Copy path" message. + - [ ] A pasted **/content/drive/MyDrive/…** path checks fine. + - [ ] The default output folder lands on Drive next to the experiment. + +## C. Step 3 on the full dataset (PR #118) + +4. - [ ] The run finishes. + - [ ] On Drive, the output folder has `plane_histograms.csv`, `run_manifest.json`, + `experiment_design.csv` and the report (`discovery_report.txt` and `.json`). + - [ ] There are no "(1)" duplicate files, and no leftover `.publish`, `.staging` or + `...tmp` files. + - [ ] There is no lock file (`.morphobricks-preprocessing.lock`) after the run. +5. - [ ] Run Step 3 again in the same session. Everything is unchanged and nothing is read. +6. - [ ] Disconnect the runtime (Runtime › Disconnect and delete runtime) and open a new one. + Run Steps 0–3 again on the same Drive folders. The rows are reused (unchanged). This + shows the Drive times are stable across mounts. +7. Optional: stop a run part-way (interrupt, or disconnect the runtime). + - [ ] The next check names the lock holder (`output_locked`), and **Remove the lock of a + stopped run** clears it. + - [ ] The next run resumes: the report lists the images read before the stop as + `resumed`. +8. - [ ] Optional: an experiment with OME-Zarr images on Drive checks and runs (the `zarr` + extra is installed). + +## D. Describe the masks (PR #123) + +9. - [ ] In Steps 1–3 with an experiment and a masks folder on Drive, before **Compute + histograms**, the card shows a greyed button: "Available when the output folder holds a + run that recorded masks". +10. - [ ] After the run, the help line says "Describes the run in … written …, of …: N mask + types from …", and the button is enabled. +11. - [ ] Click it: a spinner, a progress bar per image, then a green bar. The tables are + labelled as suggestions, and `mask_relationships.json` appears in the output folder. +12. - [ ] Run Step 3 again: the card clears. +13. - [ ] **Reopen** the notebook (Runtime › Restart session) and choose the same folders in + Step 1 without running. The button is enabled, names the earlier run, and works. +14. - [ ] Move the masks folder on Drive and click: a red card says the masks folder does not + exist, and nothing is written. +15. - [ ] The tables scroll inside the card, with no horizontal page scrollbar. + +## E. Make named-axis copies (PR #127) + +16. - [ ] Run Step 0, then Step 1 on an experiment of Y/X-swapped OME-TIFFs on Drive, and + press **Check folder**. +17. - [ ] In Step 2, swap the Y and X roles. The red axis check appears, with the **Make + named-axis copies (optional)** card under it. Its button is disabled. +18. - [ ] Press **Show preview** and **Yes, the preview looks right**, then **Yes, write the + copies with these roles**. +19. - [ ] Press **Browse…**: the amber card names **Copy path**. Paste a Drive folder inside + the experiment and check that it is refused. Then paste one next to the experiment. +20. - [ ] Check the plan lists and the space, then press **Make named-axis copies**. The bar + moves, the green card shows the record path, and the pixel-size note lists the copies. +21. - [ ] Press **Check the copies in step 1**. Step 1 shows the copy folder, the same output + folder and a clean check. The roles are suggested from the copies. +22. - [ ] Confirm the labels, preview, roles and image type, then run Step 3. The report says + "named-axis copies of …". +23. - [ ] Optional: disconnect the runtime during a conversion, reconnect, and convert into + the same folder. The lock card appears. Press **Remove the lock of a stopped run**, then + convert again. + +## After the check + +- Record the result (tag, date, items passed, failures with their issue numbers) in the + release PR of v0.4.0. +- Then update this file's first paragraph, the README's "Notebook" section and plan §10 + (release checklist) to say which release was checked on Colab. diff --git a/docs/contract.md b/docs/contract.md index 60186a3c..f2c1db21 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -279,6 +279,18 @@ the bricks' tests. | Notebook masks (v0.4.0, PP-10c) | Step 1 has an optional masks folder; **Check folder** pairs it with `preflight(masks_folder=, mask_axes=)`, the masks folder and statements being those the run would use (`run.planned.planned_masks`, from `resolve_inputs`: given, else recorded). The **Masks** card shows each mask type (`mask_review.mask_type_rows`), the images without one and the masks' issues (`masks.issues`, and `mask_checks.mask_issues` once the roles are confirmed, as `run`). Mask axes are asked per type only where needed (`mask_review.mask_axes_questions`: `mask_not_2d`, `invalid_mask_axes`, `mask_axes_contradict_header`, and `mask_planes_mismatch` of 2-D masks, answered `YX`), with only the statements `masks.stated.mask_roles` accepts for every mask of the type; nothing is preselected or applied until **Check the masks** (`run.preflight.remasked`, which keeps the other answers). The run is given the same `masks_folder` and `mask_axes`; a masks folder not given reuses the recorded one, and **Drop the recorded masks** gives `masks_folder=False`. Step 3 is gated on masks without errors and on a pairing that matches the masks folder and statements chosen. `folder_step(show_output=False)` shows neither and pairs no masks folder. The mask step (`mask_relationships.json`) is step 3's **Describe the masks** (next row). | | Notebook mask step (v0.4.0, PP-10d) | Step 3's **Describe the masks (optional)** card. Its button is enabled when step 1's output folder holds a v2 manifest that recorded masks (`run.mask_step.recorded_masks(output_folder)`, a `RecordedMasks` from the manifest's `experiment_root`, `written_at` and `masks` keys): after a run in this session, or for an earlier run in a reopened notebook. The help line names the run it describes (output folder, when written, experiment folder, masks folder and mask types). The output folder is the one step 1 shows (default: next to the experiment folder); the experiment folder typed in step 1 only locates it, and the recorded one is read. A click runs the `masks` command's step with the run's recorded inputs (`prepare_mask_step(None, output_folder)`, the path the CLI's `masks --output` takes: experiment folder, masks folder, mask-axis statements, exclusions) and the default thresholds (the notebook offers no threshold; the command line's options change them), with a progress bar per image, then `describe_masks`, which reads the masks and writes `mask_relationships.json` holding the lock throughout. The card shows the mask types, the relationships with their counts (`mask_text.mask_type_rows`, `mask_relationship_rows`), masks of the same objects and the hierarchy (`mask_hierarchy_lines`), labelled as suggestions to check, and the path written. A refused step changes nothing: `experiment_not_recorded`, errors found since the run (an experiment or masks folder moved or removed, a mask changed; shown with their messages), no masks, masks that cannot be used (a red card), and `output_locked` (the lock card with **Remove the lock of a stopped run**). A new run clears the card. `folder_step(show_output=False)` hides the card; the pinned embedding API is unchanged (the click handler is private). | +### 9.3 Calibration (`calibration.csv`, v0.4.0, design v0.4 §3) + +| Topic | Rule | +|---|---| +| Source per axis | For X, Y and Z, an image's size comes from its header (`header`) or from a statement in the output folder's `calibration.csv` (`stated`, or `pixels`: deliberately uncalibrated). The time step comes from the header only. An axis with neither is uncalibrated: no value, no source, never a default. A Z or T size is recorded only for an image whose roles include Z or T (v0.4.0, plan §12 question 53): a max projection's header may still give the z-step of its stack, and that is not a size of the projection (a singleton role of an axis the file lacks, `missing_as_singleton`, does not count). | +| File | UTF-8 CSV, columns `target,axis,size,unit`, one statement per line. `target`: `.` for the whole experiment, or a folder (a condition or a repeat) or an image, relative to the experiment folder with `/`, compared exactly (backslashes, a leading `./` and a trailing `/` are normalised; an absolute path, a drive, or an empty, `.` or `..` component elsewhere is refused). `axis`: `X`, `Y` or `Z`. `size`: a positive finite number in plain ASCII decimal notation (`0.65`, `1e-3`; not `0,65` or `1_0`); empty with the unit `pixels`. `unit`: a length unit of decision D-a2 in any spelling `space_unit` reads (written back as its OME-NGFF name), or `pixels`. The reader (`read_calibration`) is strict: other columns, a value missing or extra, outer whitespace, a target outside the experiment folder, an unknown axis or unit, a size that is not a positive number or not a positive finite length in micrometres once converted (`1e308 m`, issue #93), or two statements for one target and axis are refused naming the line (`CalibrationFormatError`). A UTF-8 byte-order mark and Windows line endings are read. | +| Precedence | The most specific statement covering an image applies: the image's own, then the deepest folder's, then the whole experiment's. Statements are per role: Y and X are stated and recorded on their own, and nothing exchanges them. | +| Writing | A run reads `calibration.csv` and never writes, replaces or removes it, so it survives `full`, stopped and failed runs. The user writes it by hand, with `write_calibration(calibration, output_folder)` (holding the output folder's lock; `OutputLockedError` while a run holds it) or with the CLI `calibrate` (`--size TARGET AXES SIZE UNIT`, `--pixels TARGET AXES`, `--remove TARGET AXES`; without them it prints the statements), which reads, changes and writes the file under the lock (`run.calibrate.update_calibration`), so two changes never overwrite each other. | +| Checks | `calibration_unreadable` (error: the file cannot be read; left as it is); `calibration_target_not_found` (error: a target that is neither a folder nor an image of the experiment, excluded images included); `calibration_differs_from_header` (warning, naming each image and axis with both sizes: a stated size not the header's by the D-a rule, or `pixels` where the header gives a size; z-steps only for images with more than one depth slice; the stated size is used); `calibration_axis_absent` (warning: a statement covers images without that axis, for example a Z size for the whole experiment with 2D images; the images are named, and the size is not recorded for them). The §11 pixel-size checks run on the resolved sizes; a header size in an unsupported unit is not reported on an axis that is stated. `check` and the notebook's folder check (`preflight`) use the same sizes. | +| Notebook (v0.4.0, PP-10) | Step 1's **Pixel sizes** card shows each image's X, Y, Z and T sizes with their source (`header`, `stated`, `pixels`, or uncalibrated) and states or removes statements per target, axes and unit with `calibration_entry` and `update_calibration` (under the lock; no second rule set), then reads the file again (`run.preflight.recalibrated`) without resetting the answers given. Before the axis roles are confirmed it shows the sizes the folder check found; after, those fitted to the confirmed roles (`fit_calibration`: no Z or T size for an image without that axis), as the run records them. The calibration messages are shown on the card. | +| Recording | Each manifest entry's `pixel_sizes` records the resolved sizes with their source (§9); the manifest records `calibration_sha256`; the report shows each image's sources and the statements, and its text counts the Z and T sizes of the images with that axis. Changing the calibration recomputes no row: the next `restart` records the new sizes, reuses every row and names the change (§9.2). `RunInputs` refuses a `calibration.csv` changed, removed or added since the run. | + ### 9.4 Staging, resume and publication (v0.4.0, PP-12, design v0.4 §1) | Topic | Rule | @@ -292,18 +304,6 @@ the bricks' tests. | What readers see | **Through the manifest** (`RunInputs`): exactly one generation, the old or the new one, never a mix: each file the manifest names is read by its SHA-256 from the output folder or, while a publication is unfinished, from `.publish` (`run.generation.published_path`), and the read is repeated if the manifest changed meanwhile (`read_stable`). **A plain reader of `plane_histograms.csv` alone** always reads one whole table of one generation (each move is one rename): the old one until it is moved. A reader that opens several output files without the manifest can pair files of two generations while a publication's step 3 runs or after a kill in it, until the next holder of the lock finishes it: read several files through `RunInputs`, or check `csv_sha256` and `experiment_design_sha256`. | | Windows | The same protocol. `os.replace` replaces a file atomically on NTFS too, but fails while another program has the file open: at the commit nothing has changed then (the run raises, the earlier generation is kept); after it, `PublicationIncompleteError` as above. Folders are not flushed (Windows cannot open them); the kill tests (SIGKILL at every step) run on POSIX only. | -### 9.3 Calibration (`calibration.csv`, v0.4.0, design v0.4 §3) - -| Topic | Rule | -|---|---| -| Source per axis | For X, Y and Z, an image's size comes from its header (`header`) or from a statement in the output folder's `calibration.csv` (`stated`, or `pixels`: deliberately uncalibrated). The time step comes from the header only. An axis with neither is uncalibrated: no value, no source, never a default. A Z or T size is recorded only for an image whose roles include Z or T (v0.4.0, plan §12 question 53): a max projection's header may still give the z-step of its stack, and that is not a size of the projection (a singleton role of an axis the file lacks, `missing_as_singleton`, does not count). | -| File | UTF-8 CSV, columns `target,axis,size,unit`, one statement per line. `target`: `.` for the whole experiment, or a folder (a condition or a repeat) or an image, relative to the experiment folder with `/`, compared exactly (backslashes, a leading `./` and a trailing `/` are normalised; an absolute path, a drive, or an empty, `.` or `..` component elsewhere is refused). `axis`: `X`, `Y` or `Z`. `size`: a positive finite number in plain ASCII decimal notation (`0.65`, `1e-3`; not `0,65` or `1_0`); empty with the unit `pixels`. `unit`: a length unit of decision D-a2 in any spelling `space_unit` reads (written back as its OME-NGFF name), or `pixels`. The reader (`read_calibration`) is strict: other columns, a value missing or extra, outer whitespace, a target outside the experiment folder, an unknown axis or unit, a size that is not a positive number or not a positive finite length in micrometres once converted (`1e308 m`, issue #93), or two statements for one target and axis are refused naming the line (`CalibrationFormatError`). A UTF-8 byte-order mark and Windows line endings are read. | -| Precedence | The most specific statement covering an image applies: the image's own, then the deepest folder's, then the whole experiment's. Statements are per role: Y and X are stated and recorded on their own, and nothing exchanges them. | -| Writing | A run reads `calibration.csv` and never writes, replaces or removes it, so it survives `full`, stopped and failed runs. The user writes it by hand, with `write_calibration(calibration, output_folder)` (holding the output folder's lock; `OutputLockedError` while a run holds it) or with the CLI `calibrate` (`--size TARGET AXES SIZE UNIT`, `--pixels TARGET AXES`, `--remove TARGET AXES`; without them it prints the statements), which reads, changes and writes the file under the lock (`run.calibrate.update_calibration`), so two changes never overwrite each other. | -| Checks | `calibration_unreadable` (error: the file cannot be read; left as it is); `calibration_target_not_found` (error: a target that is neither a folder nor an image of the experiment, excluded images included); `calibration_differs_from_header` (warning, naming each image and axis with both sizes: a stated size not the header's by the D-a rule, or `pixels` where the header gives a size; z-steps only for images with more than one depth slice; the stated size is used); `calibration_axis_absent` (warning: a statement covers images without that axis, for example a Z size for the whole experiment with 2D images; the images are named, and the size is not recorded for them). The §11 pixel-size checks run on the resolved sizes; a header size in an unsupported unit is not reported on an axis that is stated. `check` and the notebook's folder check (`preflight`) use the same sizes. | -| Notebook (v0.4.0, PP-10) | Step 1's **Pixel sizes** card shows each image's X, Y, Z and T sizes with their source (`header`, `stated`, `pixels`, or uncalibrated) and states or removes statements per target, axes and unit with `calibration_entry` and `update_calibration` (under the lock; no second rule set), then reads the file again (`run.preflight.recalibrated`) without resetting the answers given. Before the axis roles are confirmed it shows the sizes the folder check found; after, those fitted to the confirmed roles (`fit_calibration`: no Z or T size for an image without that axis), as the run records them. The calibration messages are shown on the card. | -| Recording | Each manifest entry's `pixel_sizes` records the resolved sizes with their source (§9); the manifest records `calibration_sha256`; the report shows each image's sources and the statements, and its text counts the Z and T sizes of the images with that axis. Changing the calibration recomputes no row: the next `restart` records the new sizes, reuses every row and names the change (§9.2). `RunInputs` refuses a `calibration.csv` changed, removed or added since the run. | - ## 10. Converting TIFF images to OME-Zarr (removed in v0.4.0) Up to v0.3, a run could convert every TIFF image to OME-Zarr in `/ome_zarr` and @@ -523,13 +523,13 @@ instead of keeping copies. | Plain-TIFF resolution (D-c) | `read_physical_sizes` | A TIFF's resolution tags give a pixel size only with OME or ImageJ metadata and a unit; a plain TIFF (scanner or image-editor dpi) is uncalibrated, never converted from inches or centimetres. | | Labels (D-b) | `check_labels(pixels, name)`, `LARGEST_LABEL` (`masks`) | A mask holds whole-number labels from 0 (background) to `LARGEST_LABEL` = 2,147,483,647: MorphoBricks labels are int32 in every brick. A negative label, a label above the limit, or a pixel type that is not integer or boolean is refused with a `ValueError` naming the mask and the value. `read_mask`, `piece_counts`, `pair_counts` and `scan_masks` call it. Since D86 the limit is int32; before, this brick refused only labels above 2^63 − 1 (int64), so a `uint32` or wider mask with a label of 3,000,000,000 is now refused. | | Header axes (D-d) | `header_axes(meta)`, `named_axes(meta)`, `series_axes_from_metadata(series)`, `PLACEHOLDER_AXES`, `OPEN_AXIS` (`metadata`) | The axes an image's own metadata names, as `tifffile` reads them with length-1 axes left out (`tifffile` keeps Y and X even of length 1, so an image one pixel high is `YX`): OME, ImageJ and `tifffile` "shaped" metadata (`{"axes": "CYX"}`, decision D-d), a plain 2D `YX` (or `YXS` RGB) TIFF, and an OME-Zarr image's OME-NGFF axes. None when an axis is a placeholder (`I`: a plain stack of pages; `Q`: shaped without axes, or an OME-NGFF axis of another type), or the image does not open: the file does not say which dimension is depth, time or channel. `named_axes` gives the rule position by position, with `?` (`OPEN_AXIS`) for a dimension the metadata does not name: a placeholder, and, in a plain stack of pages without axis metadata (`IYX`, `QYX`, `QQYX`; `TiffMetadata.axes_from_metadata` False, from `series_axes_from_metadata`; True, the strict reading, for a header built by hand), the `Y` and `X` too, which only say where each page's rows and columns are stored (a (Y, X, C) array written without metadata reads as `QYX`). `header_axes` gives axes exactly when nothing is open. The axis check (§2) compares confirmed roles with `named_axes`. | -| Declared axes (removed) | `check_declared_axes`, `declared_axes_transposed` | **Removed in v0.4.0** (spec revision 4): they accepted a declaration with Y and X swapped and read it transposed, which v0.4 forbids. Bricks take no axis declarations; preprocessing's axis check (§2) compares the confirmed roles with the header. Former text, for reference: | **Deprecated in v0.4.0, removed in v0.5.0** (they warn with a `DeprecationWarning`): bricks take no axis declarations and read the files of a preprocessing run whose axes were checked (§2); nothing is read transposed. Until then: axes a brick or user declares for an image or mask (`image_axes="CYX"`) may name what the header leaves open, never contradict it: one letter per header axis (any case), roles X, Y, Z, C, T each at most once, with `Y` and `X` declared when the header has them (a plain stack too); an axis the metadata names (`named_axes`) keeps its letter, except that the named `Y` and `X` may be declared swapped (header `CYX` declared `CXY`, `ZYX` declared `ZXY`): the image is read transposed, and `declared_axes_transposed` returns True so the caller can record it (owner decision of 31 October 2026; it raises `ValueError` for a declaration that does not fit); any other disagreement is refused; a dimension it leaves open may be declared any role, so a plain `QYX` stack may be declared `YXC` as well as `CYX` (since v0.3.4; before, its positional `Y` and `X` were binding and a placeholder could not be declared X or Y), while an ImageJ `ZYX` declared `CYX`, an OME `CYX` declared `YXC` or a shaped `CYX` declared `YXC` are refused; the RGB samples `S` may be declared C when the header has no C; any other header axis (tile `R`, ...) cannot be declared. Otherwise the error `declared_axes_mismatch` naming the image and both axes (`unreadable_tiff` or `unreadable_ome_zarr` when the header does not open). | +| Declared axes (removed) | `check_declared_axes`, `declared_axes_transposed` | **Removed in v0.4.0** (spec revision 4): they accepted a declaration with Y and X swapped and read it transposed, which v0.4 forbids. Bricks take no axis declarations; preprocessing's axis check (§2) compares the confirmed roles with the header. | | Mask files | `read_mask_file(path, z=, t=)` (`masks`) | Reads one plane of a TIFF mask or of an OME-Zarr label image (`.zarr/labels/`) given by its path, with `read_mask`'s rules (§15): its axes are its own metadata's, Y and X plus at most a Z and a T axis named so, length-1 axes other than Y and X left out (a mask one pixel high or wide is read as a 2D array, plan §12 question 44); any other or unnamed dimension (a plain stack, `IYX`; a `C` axis) is refused, the message giving the header's axes and shape and the dimension that is not named, never taken for depth or time; `z` and `t` are required for a mask with that axis; integer or boolean pixels; labels checked with `check_labels`. A `ValueError` names the file and the problem. | | Identity (D-e) | `FOV_KEY`, `check_identity_text(value, field)`, `check_plane_index(value, axis)`, `check_identity_case(keys)`, `MISSING_TEXT`, `IdentityError` (`discovery`) | `FOV_KEY` = (`condition`, `biological_repeat`, `technical_repeat`, `position_id`), the position key of §1. `check_identity_text` accepts a label or `position_id` that is text without leading or trailing whitespace and not `""`, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case (`MISSING_TEXT`); `check_plane_index` accepts None or an integer of at least 0, a Python `int` or a NumPy integer (returned as a plain `int`), and refuses everything else (bool, float, `"1"`) without converting it. Both raise `IdentityError` (a `ValueError`) naming the field and the value. Discovery applies the text rule to every folder and image name (`invalid_name`) and refuses names of one level that differ only in case anywhere in the experiment (`case_conflict`, §1). `position_id` keeps its own scope: unique within its technical repeat, ignoring case (`duplicate_position_id`). `check_identity_case` applies the same case rule to identities a brick reads from a table, one key per row in `FOV_KEY` order (or the three labels): a label of one level spelled two ways anywhere in the run, or two `position_id` values of one technical repeat equal ignoring case, raise `IdentityError` naming both spellings; the same key may repeat (several planes of one position). | | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`); from v0.4.0 file runs read through `RunInputs` (§9.1), which also checks masks. A version 1 manifest fails every image with `manifest_before_v040` alone (design v0.4 §6: every v1 run is redone). Otherwise each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | | Masks named as images (D-i) | `mask_name_issues(leaves, mask_names=())`, `images_named_as_masks(leaves, suffixes=MASK_NAME_SUFFIXES)`, `mask_folder_part(path)`, `MASK_NAME_SUFFIXES`, `MASK_FOLDER_WORDS` (`discovery`) | The screen MorphoRelate's `init` ran (`project_image_names`), now discovery's warnings `image_named_as_mask` and `mask_folder_in_experiment` (§1) for every brick. Names are compared in lower case across the whole experiment; the longest `position_id` owns a file, an image of the same folder first. `mask_names` adds the caller's expected mask names (`cells` for `f1_cells.tif`) to `MASK_NAME_SUFFIXES`; `discover_experiment` gives none. A folder counts as a masks folder when a word of its name (`CellLabels` is `Cell` and `Labels`) is in `MASK_FOLDER_WORDS`. `images_named_as_masks` returns the `(image, owner)` pairs of that detection for the caller's own name endings (`["_cells"]`), for a brick that acts on the files itself (MorphoRelate leaves out the images named with a declared mask name). | | Pixel sizes (D-a) | `same_pixel_size(a, b, roles=("X", "Y"))`, `pixel_size_groups(sizes, roles=)`, `PIXEL_SIZE_TOLERANCE` (`physical_sizes`) | Two pixel sizes are the same when `|a - b| <= 1e-3 * max(a, b)` on every compared axis, in micrometres; one rule for a declared size against a header and for two headers. `same_pixel_size` takes `PhysicalSizes` (converted with `micrometers`) or sizes already in micrometres, one per role, and returns None when either is unknown (no size, or a unit that cannot be converted), never True. The rule is not transitive (1.0 ≈ 1.0009 ≈ 1.0018, but 1.0 ≠ 1.0018), so there is no per-image grouping key: `pixel_size_groups` links images by single linkage, so images of two groups always differ while the extremes of one group may drift; unknown sizes are left out; the largest group first. A size that cannot be converted to a positive finite length in micrometres (`1e308 m`) is unknown, never infinite, and an infinite or NaN value is never the same as any other (issue #93; before, `inf` matched every size). The folder check's `different_pixel_sizes` uses both (§11). | -| Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order (a column named twice or without a name is refused naming it, as by the lenient reader, §12); one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking change for the next release:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | +| Experiment design | `read_confirmed_design(path, expected_sha256=)`, `ConfirmedDesign`, `DesignCheckError` (`design`) | The one reader of `experiment_design.csv` (§12) for a brick that pools biological repeats (formerly CellPatcher `load_experiment_design` and MorphoFeature `ExperimentDesign.read`). It takes the file as this brick writes it and fills nothing in: UTF-8 without a byte-order mark; the header exactly the §12 columns in order (a column named twice or without a name is refused naming it, as by the lenient reader, §12); one value per column in every row (an unclosed quote, a missing or extra value, or a blank line is refused); no value with outer whitespace; `control` and `confirmed` exactly `true` or `false`; counts written as whole numbers of at least 0; `schema_version` 1; `confirmed_on` empty or `YYYY-MM-DD`; `condition`, `biological_repeat` and `repeat_group` pass `check_identity_text` and do not differ only in case (`check_identity_case`); no folder twice, one `control` per condition, at least one row; every group confirmed with one folder per condition (`design_issues`); with `expected_sha256` (the manifest's `experiment_design_sha256`) the file must be that one. The SHA-256 is of the bytes parsed. `ConfirmedDesign` gives `path`, `sha256`, the `design`, read-only `repeat_groups` (`{(condition, biological_repeat): repeat_group}`, the normalization brick's input), `group_members()` (`{repeat_group: {condition: biological_repeat}}`), `conditions`, `controls` and `groups`. Refusals raise `DesignCheckError` (a `ValueError`) with its `issues`: `missing_experiment_design`, `unreadable_experiment_design`, `invalid_experiment_design`, `repeat_group_conflict`, `design_not_confirmed` or `changed_experiment_design`, each an error naming the file (or the group's folders). This brick's step 1 and `check` read the file being edited with the lenient `load_design(output_folder, repeats)` (§12), which returns `(design, changes)`: the saved design brought up to date with the folders, or the default design when there is no file. A saved file that cannot be read raises `DesignCheckError` with the one error `unreadable_experiment_design`, whose message is the same "The experiment design cannot be read: …" as here followed by the lenient reader's refusal, naming the line and column where it can (the `DesignFormatError` is its `__cause__`); no default design is returned and the file is not touched (owner decision of 31 October 2026, plan §12 question 43). **Breaking in v0.4.0:** up to v0.3.4 `load_design` returned a third value, `problem`, the reason as text, together with the default design. | ## 17. Named-axis copies (`convert`, v0.4.0, design v0.4 §5) diff --git a/docs/migration-0.3-to-0.4.md b/docs/migration-0.3-to-0.4.md new file mode 100644 index 00000000..112a7801 --- /dev/null +++ b/docs/migration-0.3-to-0.4.md @@ -0,0 +1,339 @@ +# Migrating from v0.3.4 to v0.4.0 + +v0.4.0 makes preprocessing the one place where an experiment's files, axes, masks and pixel +sizes are checked and recorded. Every later brick reads that record through `RunInputs` +instead of discovering, pairing or calibrating files itself. The rule behind every change is +the same: **nothing is guessed, nothing is silent, and nothing is reused when its identity is +uncertain.** + +This guide lists every breaking change from v0.3.4. Each item gives **what changed**, **why** +(the scientific rule) and **what to do**. The first section is for people who run +preprocessing (notebook, command line or Python). The second is for brick authors. The full +rules are in the [contract](contract.md) and the [user guide](user_guide.md). The decisions +are in the [plan](plan.md), §12 questions 45 to 69. The complete list of changes is in the +[CHANGELOG](../CHANGELOG.md). + +**In short, for users:** install v0.4.0 and run preprocessing again on every experiment, +into the same output folder. The first run recomputes every row. It asks again for the axis +roles of plain stacks, and it stops on the new errors listed below. Fix them, then run the +later bricks on the new run. + +## For users + +### 1. Every v0.3 run must be run again (manifest v1 → v2) + +- **What changed.** `run_manifest.json` is version 2 (contract §9). Each image records its + `roles`, where they come from (`axes_from`: `header` or `confirmed`) and its pixel sizes by + role, with a unit and a source. Each mask records its `size`, `mtime_ns`, `roles`, + `axes_from` and whether it is a stated shared 2-D mask. The manifest also records the SHA-256 + of `calibration.csv` and, for a run on named-axis copies, of the conversion record. +- **Why.** A v1 manifest does not say which axis is which, or whether the user or the header + decided it. A row computed under unknown roles cannot be trusted as the same row. +- **What to do.** Run preprocessing again into the same output folder. A v1 manifest gives + **no reusable row and no recorded input**: the run recomputes every row, and you give the + axis roles, masks and exclusions again. The report says that the earlier manifest was + version 1. The later bricks refuse a v1 run (`RunInputs.open` raises `RunInputsError`). + +### 2. Axis checks: swaps and contradictions are errors; plain stacks are confirmed per file + +- **What changed.** + - Confirmed roles that swap the Y and X the header names (`axes_yx_swapped`), contradict + the header (`axes_contradict_header`) or put X before Y in a plain stack + (`axes_x_before_y`) are **errors**. Such an image is never read transposed or under a + contradicted declaration. + - A plain stack (the header names no axes, `axes_not_named`, a warning) takes the roles you + confirm. The confirmation covers only the files it was given for, at their recorded + fingerprint. A new or changed plain stack is `plain_stack_not_confirmed` until you + confirm it again (CLI `--axes`). + - An image whose own header names X before Y (for example ImageJ `ZXY`) is still read, in + the canonical Y-before-X order, because its metadata names the axes. +- **Why.** A guessed or contradicted axis gives wrong pixels with no error. The roles must + come from the header or from an explicit answer. +- **What to do.** Correct the roles to match the header. If the header itself is wrong, make + named-axis copies (§9 below) and run preprocessing on them. + +### 3. Recorded inputs, modes and the output folder's lock + +- **What changed.** + - `force_rebuild=True` / `--force-rebuild` are **removed**. They are replaced by + `mode="restart"` (the default: reuse the rows of unchanged files) and `mode="full"` + (recompute every row and check). Both keep your confirmations: design, axis and + mask-axis statements, stated calibration and image type. + - One output folder belongs to one experiment. A later run takes every input you do not + give from the earlier v2 manifest: experiment folder, axis roles, exclusions, masks + folder and mask axes, image type, design, bins and histogram settings, intensity limits + and `missing_as_singleton`. `run --output OUT` alone reruns the recorded experiment + (without one: `experiment_not_recorded`). A value you give that differs is used, and the + report names the change and its effect. Masks are dropped only with `--no-masks` + (`masks_folder=False`). Exclusions are cleared only with `--no-exclude` (`exclude=[]`). + - A run holds an exclusive lock file, `.morphobricks-preprocessing.lock`, in the output + folder. A second run is refused with `output_locked`, naming the computer, process ID and + start time. A stale lock is removed only when you ask: `--remove-stale-lock` (every + command), `remove_stale_lock()`, or the notebook's **Remove the lock of a stopped run**. + The lock of a process still running on the same computer is never removed. On Google + Drive the lock cannot guard a folder across computers or sessions, so use one session per + output folder. +- **Why.** A silent change of input, or two runs writing one folder, gives a table that + describes neither run. +- **What to do.** Replace `force_rebuild=True` with `mode="full"` and `--force-rebuild` with + `--mode full`. Use one output folder per experiment. Read the report's `input_changes` + after a run. + +### 4. The report, written on every run + +- **What changed.** Every run that holds the lock writes `discovery_report.json` and + `discovery_report.txt`, also when it stops. The report has `status` (`stopped` or + `written`), the SHA-256 of the manifest it describes, and what the run reused, recomputed, + resumed and replaced, with the reasons. `check` writes the same report (`status: + "stopped"`, `step: "check"`) and, when the output folder holds no design, a prefilled + `experiment_design.csv` (nothing confirmed). +- **Why.** A run that stops must still say why, and what it kept. +- **What to do.** Read the report after every run. Bricks never read it. + +### 5. Pixel sizes: `calibration.csv` and the `calibrate` command + +- **What changed.** For each image, the X, Y and Z sizes come from the header, or from a + statement in the output folder's `calibration.csv` (`target,axis,size,unit`; target `.`, + a folder or an image; a length unit or `pixels` for deliberately uncalibrated). The most + specific statement applies. The run reads this file and never writes it. You write it with + `morphobricks-preprocessing calibrate`, `write_calibration()`, the notebook's **Pixel + sizes** card or by hand. Each axis is recorded with its source (`header`, `stated` or + `pixels`). A Z or T size is recorded only for an image that has that axis. A size that is + not a positive finite length in micrometres is refused when stated, and unknown (with + `invalid_physical_calibration`) when it comes from a header. +- **Why.** Pixel sizes belong to the experiment, not to each brick's configuration. A size + nobody gave is uncalibrated, never a default. +- **What to do.** Move the pixel sizes you gave to bricks (run files, FOV-table columns, + configs) into `calibration.csv`. Changing it recomputes no row; the next run records the + new sizes. `RunInputs` refuses a `calibration.csv` changed, added or removed since the run. + +### 6. Experiment design: the `design` command and stricter readers + +- **What changed.** + - New command `design --output OUT`. It prints the design and what is still unconfirmed, + and changes it without editing the CSV: `--repeat-group CONDITION/REPEAT=GROUP`, + `--control` / `--no-control CONDITION`, `--confirm GROUP`, `--confirm-all`. Python: + `edit_design` with `DesignEdit`. Nothing is confirmed unless you ask. + - `design_not_confirmed` stops a `restart` that has new unconfirmed repeats. + - Every design reader refuses a header that names a column twice or has a column without + a name (`DesignFormatError`). + - Technical-repeat folders whose images are all excluded are not counted. + - In the notebook, changing the output folder reloads that folder's design; nothing is + carried over. + - `load_design` returns `(design, changes)` and raises `DesignCheckError` for a file it + cannot read (in v0.3.4 it returned a third value, `problem`). +- **Why.** Pooling biological repeats is a scientific claim. It must be an explicit, recorded + answer for the output folder it belongs to. +- **What to do.** Confirm the design with `design` or the notebook. Remove duplicate + columns from hand-edited files. Update callers of `load_design` to unpack two values. + +### 7. Masks + +- **What changed.** + - `image_named_as_mask` is an **error**: an image named like a mask of another image + (`_mask`, `_masks`, `_cp_masks`, `_labels`, `_label`, `_seg`, `_segmentation`) is not + analysed as an image. `_cp_masks` and `_masks` are new mask suffixes. + - Masks are checked **as stored** and never converted or reordered. A mask whose header, + or stated axes, put X before Y is refused (`mask_layout`, with a `numpy.swapaxes` hint). + - A plain-stack mask needs its axes stated once per mask type: `--mask-axes Cells=TYX`, or + **Mask axes** in the notebook. + - A 2-D mask of an image with several Z or T planes is refused (`mask_planes_mismatch`) + unless its type is stated `YX` (`--mask-axes Cells=YX`): one mask shared by every plane. + - OME-Zarr `labels/` are always used as masks when present. `find_masks(ome_zarr_labels=)` + is **removed**. + - The notebook has a **Masks folder** field and a **Describe the masks** button (the + `masks` step). + - `same_objects` in `mask_relationships.json` is a one-to-one matching (report version 2). +- **Why.** A mask read under the wrong axes describes other objects. Objects must be joined + by their recorded pairing, never by guessing a layout. +- **What to do.** Rename or exclude images named like masks. Save masks with their axes + named (or state them per type). Save X-before-Y masks again with `numpy.swapaxes`. Remove + `ome_zarr_labels=` from `find_masks` calls. + +### 8. OME-Zarr input + +- **What changed.** + - An OME-NGFF 0.1 or 0.2 store is `unreadable_ome_zarr`: its metadata never name the + axes, and the order its specification fixes is not assumed. A 0.3, 0.4 or 0.5 store + without `axes` is unreadable too. + - Every OME-Zarr image is matched by the roles its axes declare. A confirmed role that is + not the declared one is `ome_zarr_axes_mismatch` (OME-Zarr images are never converted: + correct the roles). + - A malformed scale gives unknown sizes with `invalid_physical_calibration` (a warning). + - An OME-Zarr image's fingerprint adds the SHA-256 of its file inventory (each file's path, + size and time). **Images recorded by v0.4.0rc3 or earlier have no inventory: they count + as changed and are recomputed, and `RunInputs` refuses such a run.** + - Links inside a store follow the same containment rule as the experiment (§10 below). +- **Why.** The axes and the scale must come from the metadata. A chunk changed inside a store + must not leave an old row in place. +- **What to do.** Re-save OME-NGFF 0.1 and 0.2 stores as OME-NGFF 0.4 or later with their + axes. Run preprocessing again after upgrading, also from an rc. + +### 9. Conversion: `convert_tiffs` removed, `convert` added + +- **What changed.** + - **Removed:** `convert_tiffs` and the `zarr_export` module (OME-Zarr copies in + `/ome_zarr`): `ConversionSettings`, `ConversionEstimate`, `estimate_conversion`, + `OmeZarrWriter`, `ome_zarr_relative_path`; the manifest's `convert_tiffs`, + `conversion` and `ome_zarr_path` (ignored when an old manifest holds them); the CLI + `--convert-tiff`, `--chunk-size`, `--compression`, `--compression-level` and + `--pyramid-levels`; the notebook's "(Optional) Image conversion" card; and + `notebook_options.PYRAMID_OPTIONS` (nothing replaces it). + - **Added:** `convert EXPERIMENT --to FOLDER --axes ROLES` (`convert_experiment`; the + notebook's **Make named-axis copies** card). It is only for images with axis errors or + unnamed axes. They are written as named-axis OME-TIFF with your confirmed roles, pixel for + pixel. Every other image is copied byte for byte. Masks are never converted. + - The copies keep the file names and folders, so `source_relative_path`, `plane_id` and the + FOV key stay the same. A relabelled axis gets an unknown size, never a guessed one. The + only exception: a Y/X swap keeps the size when X and Y are exactly equal. + - The copies are written as `.partial` files and moved into place together. The conversion + record, `.morphobricks-conversion.json` (version 2), is written last, with the source and + copy fingerprints and each copy's SHA-256. A version 1 record (v0.4.0rc3 or earlier) is + refused: convert again into a new folder. +- **Why.** The run reads every image through its own readers, so the OME-Zarr copies served + no brick. A file whose header is wrong must be fixed in a new, recorded copy, never read + transposed. +- **What to do.** Stop passing the removed options. The old `/ome_zarr` folder is not + touched; delete it yourself if you no longer need it. For images with axis errors, run + `convert`, then run preprocessing on the copy folder with the **same output folder**: the + design, image type and calibration are kept, and every row is recomputed. A changed copy + stops the run (`copy_changed_after_conversion`). + +### 10. Links, the `--log` file and exit codes + +- **What changed.** + - A symbolic link (or a Windows junction) in the experiment or masks folder is followed + only when its real path stays inside that folder. Otherwise it is `link_outside_folder`. + One file reached by two paths, hard links included, is `linked_twice`. A loop is + `link_loop`. + - `--log` is checked before the file is opened. It is refused inside the experiment, the + masks folder or the folder of copies, and as or inside a protected output file (the + table, manifest, reports, design, `calibration.csv`, `mask_relationships.json`, lock, + conversion record, `.publish`, `.staging`) or the `--design` file. + - Every command exits with status **1 when it prints an error** and 0 otherwise. Warnings + never change the status, and options that cannot be parsed exit with 2. `check` and + `design` exited with 0 on some errors before. +- **Why.** A link that leaves the experiment brings in images that are not part of it, and + one file seen twice is counted twice. A log must never overwrite a result. Scripts must be + able to trust the exit status. +- **What to do.** Copy images into the experiment rather than linking to them from + elsewhere. Write logs next to the output folder. In scripts, check the exit status. + +### 11. Staging, resume and crash-consistent publication + +- **What changed.** + - Each image's rows are staged in the output folder's `.staging` as soon as the image is + read. A run that was interrupted is resumed by the next run: staged rows are reused only + when their row key, pixel type, schema version and package versions still match and the + image still has its fingerprint. Anything else is discarded and reported. + - Publication is crash-consistent. The new files are written into `.publish`, then one + `os.replace` of `run_manifest.json` commits them, then they are moved into place. The + next holder of the lock finishes or discards an interrupted publication. The v0.3 + backup-and-restore files are gone. + - An image that changes during the run is `source_changed_during_run`, and nothing is + published. + - A run replaces or removes only `plane_histograms.csv`, `run_manifest.json`, + `experiment_design.csv` (with a design), the reports, `mask_relationships.json`, + `.publish` and `.staging`. It never touches `calibration.csv`, source images or masks. + - New output files get your umask's permissions; existing files keep theirs. +- **Why.** A reader must see the old run or the new one, never a mix. A row is reused only + when it is certainly the same row. +- **What to do.** Nothing, except: read the outputs through `RunInputs` (bricks), or check + `csv_sha256` before pairing files by hand. Files written `0600` by earlier versions keep + that mode until you change it. + +### 12. Other changes users see + +- Mixed pixel types stay an error. MorphoRelate projects with mixed dtypes must be saved + again in one type. +- `--memory-limit` must be a whole number of bytes above 0 (`1.5GiB` is accepted, `1.5` + is refused). +- `different_sizes` and `different_plane_counts` compare the images by their confirmed roles + once the roles are confirmed. +- The notebook runs on Google Colab and in JupyterLab. It installs the pinned release + (`RELEASE_TAG`), never `main`. + +## For brick authors + +### 13. Read runs only through `RunInputs` + +Bricks read a preprocessing run through +`RunInputs.open(output_folder, experiment_root=None, masks_root=None)` (contract §9.1). They +take no axis or calibration arguments for file runs, never transpose, and never call +`find_masks`, `discover_experiment` or `position_id` for file runs. For in-memory arrays a +brick keeps only a canonical layout argument (Y before X, no swaps). Without a clean run a +brick refuses and gives the preprocessing command; there are no standalone exceptions. + +**`RunInputs` gives:** + +- `fovs()`: `Fov` records with the FOV key, `plane_id`s, axes, roles, `axes_from`, channel + names, dtype, `significant_bits` and the Z, T and C counts. They come from the manifest and + the table and are never parsed from names. +- `pixel_sizes(fov)`: the value, unit and source per role, or unknown. +- `read_plane(fov, z, t)`: a `(C, Y, X)` array of the stored values. `read_image(fov)`: a lazy + `(T, Z, C, Y, X)` `LazyImage`. Both are in canonical order. `z` and `t` must be integers + (a `plane_id` is a `TypeError`). +- `mask_types()` and `read_mask(fov, type, z=None, t=None)`: the recorded pairing only, read + as stored with `check_labels` (labels 0 to int32 max). A stated shared 2-D mask serves + every plane. +- `histograms()`, `design()` (the strict `read_confirmed_design`, checked against the + manifest's SHA-256) and `image_type()`. +- `RunInputs` is picklable for worker processes. `morphobricks_preprocessing.testing. + make_clean_run(folder, ...)` writes a small clean run for tests. + +**`RunInputs.open` refuses** (`RunInputsError`), naming the cause and the fix: + +- a version 1 manifest; +- a table, design or `calibration.csv` whose SHA-256 is not the manifest's (also a + `calibration.csv` added since the run); +- an image or mask changed or gone since the run (every read checks the fingerprint again); +- a mask recorded without axes (a v0.4.0rc1 run); +- an OME-Zarr image recorded without its inventory (v0.4.0rc3 or earlier); +- **files the run did not record** (`UnrecordedFilesError`, with `files.images` and + `files.masks`): images discovery would analyse, and masks it would pair with a recorded + image, that the run neither recorded nor excluded. There is no opt-out. The fix is to run + preprocessing again, or exclude the images; +- a `masks_root` given for a run that recorded no masks folder; +- a run on named-axis copies whose conversion record changed or disappeared + (`conversion_record_sha256`), whose record is version 1, or whose copies do not match the + record's fingerprints and SHA-256. The copies are read once, at `open`, to check them; +- a run holding OME-Zarr images when the `zarr` extra is missing (install + `morphobricks-preprocessing[zarr]`), and a recorded OME-Zarr image whose `labels` group + cannot be read; +- links that leave the experiment or masks folder, as discovery refuses them. + +`unrecorded_files(manifest, experiment_root, masks_root, output_folder=None)` gives the same +list without raising. + +### 14. Removed APIs + +| Removed | Use instead | +|---|---| +| `convert_tiffs=`, `zarr_export` (`ConversionSettings`, `ConversionEstimate`, `estimate_conversion`, `OmeZarrWriter`, `ome_zarr_relative_path`) | nothing (images are read directly); `convert_experiment` for axis errors | +| The stand-ins of the code moved to the normalization brick: `morphobricks_preprocessing.{csv_io,dtypes,histogram,normalization,pooling,records}` and their top-level names (`PlaneRecord`, `PlaneHistogram`, `compute_plane_histogram`, `compute_block_histogram`, `read_plane_histograms`, `write_plane_histograms`, `group_records`, `PooledHistogram`, `PooledStatistics`, `pooled_statistics`, `NormalizationParameters`, `percentile_range`, `min_max`, `z_score`, `robust_z_score`, `default_bin_count`, `make_plane_id`, `SINGLE`, `OUTPUT_FILENAME`, `NoValidPixelsError`, `UnsupportedDtypeError`) | import them from `morphobricks_normalization` | +| `check_declared_axes`, `declared_axes_transposed` | nothing. They accepted a Y/X swap and read it transposed, which v0.4 forbids. Read through `RunInputs` | +| `find_masks(..., ome_zarr_labels=)` | omit it: label images are always used | +| `run_experiment(force_rebuild=)`, CLI `--force-rebuild` | `mode="full"`, `--mode full` | +| `notebook_options.PYRAMID_OPTIONS` | nothing | +| CLI `--convert-tiff`, `--chunk-size`, `--compression`, `--compression-level`, `--pyramid-levels` | nothing | +| Pickled `PhysicalSizes` from v0.3 | not supported (owner decision, design v0.4 change log, 1 November 2026). The class gained `invalid` (issue #94). A v0.3.4 pickle loads today with `invalid` None, but this is not guaranteed. Do not keep pickles across versions: read the metadata again, or use `RunInputs.pixel_sizes` | + +Superseded in the bricks (each brick's own migration PR): CellPatcher `image_axes`/`mask_axes` +for file runs, `image_read_transposed`, its own mask pairing and name parsing; MorphoFeature's +run-file axes and calibration, FOV-table pixel-size columns and `inputs.read_transposed`; +MorphoRelate's own mask pairing and calibration config; fiberfeatures' `spacing_cols`; and +the brick-side swap refusals. + +### 15. Caches keyed on the run + +- A `restart` that changes nothing leaves `plane_histograms.csv` byte-identical, so a brick + cache keyed on its SHA-256 (the manifest's `csv_sha256`) survives. +- Key caches on the manifest's SHA-256 and the file fingerprints, never on paths alone. +- After upgrading from v0.3 or from a release candidate up to v0.4.0rc3, the table is + recomputed. Caches keyed on the old table must not be reused. + +### 16. The `zarr` extra + +Install `morphobricks-preprocessing[zarr]` in every brick that may open a run holding +OME-Zarr images. Without it, `RunInputs.open` refuses such a run and names the extra. It +never skips the images or their label images. diff --git a/docs/plan.md b/docs/plan.md index bfe88331..22e649cb 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -602,6 +602,30 @@ API, CLI). The notebook and CLI only call the API. atomic write's temporary file is open leaves that hidden `...tmp` file in the output folder (on main too, for the provisional report); the lock could remove them. +- **PP-13 (docs and the migration guide)** *(done, this PR; design v0.4 §8)*: + - [docs/migration-0.3-to-0.4.md](migration-0.3-to-0.4.md): every breaking change from + v0.3.4 for users and for bricks (what changed, why, what to do), linked from the README + and the user guide §7; + - [CHANGELOG.md](../CHANGELOG.md) for 0.4.0 (breaking, added, fixed, removed); + - docs pass: README (masks, `RunInputs`, resume), user guide §7 (upgrading), contract + (§9.3 before §9.4; the removed declared-axes text dropped), the notebook's step 3 note on + resuming; + - [docs/colab-release-check.md](colab-release-check.md): the Colab hand checks of PRs + #118, #123 and #127 in one list, for the major release; + - version 0.4.0rc4 (pyproject and the notebook pin), for the data-flow audit and the bricks' + migration; the final 0.4.0 bump comes after the audit. + - **Follow-ups recorded for the audit (coordinator, 2 November 2026; not changed here):** + (a) `pair_counts` breaks an exact-half tie by the lower parent index, so `inside` and + `parents_with_N_children` can depend on label numbering when a child lies exactly half in + each of two parents (the `same_objects` counts do not, question 67); (b) masks are not + bound to a fingerprint during a run: they are fingerprinted when the manifest is written + (PP-12 follow-up (a)); (c) a run killed while an atomic write's temporary file is open + leaves a hidden `...tmp` in the output folder, for example after the + provisional report (PP-12 follow-up (b)); (d) `length_units.to_micrometers` refuses NumPy + scalars other than `float64` (for example `np.float32`, `np.int64`) instead of + converting them; (e) the notebook's axis card does not show the size and plane-count + warnings recomputed under the confirmed roles (question 68): they appear only in the + run's result. - **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release candidates, in the release PR before the tag is pushed: - set `version` in `pyproject.toml`; @@ -609,8 +633,17 @@ API, CLI). The notebook and CLI only call the API. `current_version` to the tag without "v", the same version in `notebooks/notebook_latest_versions.yaml`, and the tag named in the Step 0 Colab callout (`tests/test_notebook_file.py` checks that the four agree); - - run the notebook by hand on Google Colab with the PR #118 checklist (the tests run it - only in a local kernel). + - add the tag's changes to [CHANGELOG.md](../CHANGELOG.md), and any breaking change to the + [migration guide](migration-0.3-to-0.4.md); + - run the full gate (ruff check and format, pytest with `.[dev,notebook,zarr]`, + `tools/check.py --rules`) and wait for CI on every platform; + - **for the major release v0.4.0 only** (owner, 2 November 2026): after the round-2 + data-flow audit and the matrix run, run the notebook by hand on Google Colab with + [docs/colab-release-check.md](colab-release-check.md) (PRs #118, #123 and #127). Release + candidates are not checked on Colab: until then the notebook is verified headless only + (the tests run it in a local kernel). + - v0.4.0rc4 (PP-13): version and pin set; the owner tags it. The data-flow audit and + MorphoFeature's migration run against it. ## 11. Tests and acceptance criteria diff --git a/docs/user_guide.md b/docs/user_guide.md index 94e6e1d7..60db9486 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -21,7 +21,7 @@ run **Step 0**. It detects local Jupyter or Google Colab and checks these option 2. **Installed package:** otherwise, use the package already available to this kernel. 3. **GitHub installation:** if the package is missing, install it with the notebook dependencies from the project's GitHub repository, at the release the notebook names - (`RELEASE_TAG` in Step 0, for example `v0.4.0rc3`), never the changing `main` branch. + (`RELEASE_TAG` in Step 0, for example `v0.4.0rc4`), never the changing `main` branch. Missing notebook dependencies are installed into the same Python environment as the kernel. A local checkout is installed in editable mode when it needs dependencies; @@ -927,6 +927,12 @@ Run the notebook or `run` again after adding, changing or removing images: - changing only the image type reads nothing again: the histograms are the same for both types, and only the manifest records the new type. +**After upgrading from v0.3** (or from v0.4.0rc3 or earlier with OME-Zarr images): the +first run recomputes every row. A v0.3 manifest (version 1) gives no reusable row and no +recorded input, so give the axis roles, masks and exclusions again; an OME-Zarr image +recorded without its file inventory counts as changed. Later bricks refuse such older runs +until they are run again. The [migration guide](migration-0.3-to-0.4.md) lists every change. + Run the brick again after adding images or masks before using a later brick: later bricks read the images and masks this brick's run recorded, and refuse an experiment holding images or masks the run did not record (they name them). @@ -1065,7 +1071,7 @@ The experiment design is optional in Python: without `design=`, the run neither nor writes one. To record it as the notebook does (`load_design` returns the saved design brought up to date with the folders, or the default design when there is no file yet; a saved file that cannot be read raises `DesignCheckError` instead, never a default design; -breaking change for the next release, plan §12 question 43: until v0.3.4 it returned a +breaking in v0.4.0, plan §12 question 43: until v0.3.4 it returned a third value, `problem`, with the default design): ```python diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 580ab353..b3d03ac7 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -90,7 +90,7 @@ "appears when everything is ready.\n", "\n", "> **Colab users:** Step 0 connects your Google Drive (allow access in the window that opens)\n", - "> and installs MorphoBricks preprocessing **v0.4.0rc3** from GitHub, which takes a minute or\n", + "> and installs MorphoBricks preprocessing **v0.4.0rc4** from GitHub, which takes a minute or\n", "> two. Keep your experiment folder and your results on Google Drive: in this notebook your Drive\n", "> is the folder **/content/drive/MyDrive**. Everything else on the runtime is wiped when it\n", "> disconnects; then run Step 0 again.\n", @@ -130,12 +130,12 @@ "\n", "# The notebook's version and name, as LabConstrictor's version check reads them\n", "# (notebooks/notebook_latest_versions.yaml lists the latest version).\n", - "current_version = \"0.4.0rc3\"\n", + "current_version = \"0.4.0rc4\"\n", "notebook_name = \"MorphoBricks_Preprocessing\"\n", "# The release installed from GitHub when the package is missing (Google Colab, or a\n", "# downloaded notebook). Set to the new tag, with current_version, before each release is\n", "# tagged, so a tagged notebook installs its own release (docs/plan.md, release checklist).\n", - "RELEASE_TAG = \"v0.4.0rc3\"\n", + "RELEASE_TAG = \"v0.4.0rc4\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "# The package's extras to install; on Colab also zarr, to read OME-Zarr images.\n", "EXTRAS = \"notebook\"\n", @@ -534,7 +534,10 @@ "> 🛡️ The table and manifest are written only when the whole run succeeds; until then the earlier\n", "> results stay as they are. The run writes its report, **discovery_report.txt**, either way:\n", "> **Show the report** under the status box shows it. A run that stops says so: *no table or\n", - "> manifest was written*.\n", + "> manifest was written*. If a run is interrupted (a stopped kernel, a lost Colab session), press\n", + "> **Compute histograms** again with the same settings: the images already read are not read\n", + "> again while they are unchanged (the report lists them as *resumed*). If the card says the\n", + "> output folder is locked, press **Remove the lock of a stopped run** first.\n", "\n", "After the run, **Intensity checks** compares the images using the table just written (no image is\n", "read again, nothing more is written):\n", diff --git a/notebooks/notebook_latest_versions.yaml b/notebooks/notebook_latest_versions.yaml index 667f3f3a..dcfd199f 100644 --- a/notebooks/notebook_latest_versions.yaml +++ b/notebooks/notebook_latest_versions.yaml @@ -1,3 +1,3 @@ # Latest version of each notebook, read by LabConstrictor's version check by notebook_name. # Equal to the notebook's current_version and its RELEASE_TAG without the "v" (README, "Notebook"). -"MorphoBricks_Preprocessing": "0.4.0rc3" +"MorphoBricks_Preprocessing": "0.4.0rc4" diff --git a/pyproject.toml b/pyproject.toml index 8242cfc7..8c590c7b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "morphobricks-preprocessing" -version = "0.4.0rc3" +version = "0.4.0rc4" description = "MorphoBricks preprocessing brick: per-plane intensity histograms and normalization methods for microscopy TIFFs." readme = "README.md" license = "MIT"