Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -834,13 +834,14 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
tests skip without zarr; CI's Python 3.11 jobs pin zarr 2.18 and 3.13 jobs get zarr 3.
- **The notebook file is tested too:**
- valid nbformat 4, with no outputs;
- code cells hidden and titled `# @title <N>. …` (1. Install, 2. Mount Google Drive, 3. Run preprocessing);
- code cells hidden as Colab forms (`"cellView": "form"`) and titled `# @title <N>. …` (1. Install, 2. Mount Google Drive, 3. Run preprocessing);
- Colab-only code and `/content` paths only inside the Colab guard; no magics;
- the GitHub install pinned to `RELEASE_TAG`, equal to `current_version` and the
version index;
- exactly three code cells, in order: 1 installs and reports versions, 2 mounts Google Drive
on Colab (a one-line note elsewhere), 3 displays only widgets built by `notebook_ui` and,
run out of order, names the cell to run first (`tests/test_notebook_setup.py`);
run before cell 1, names cell 1 (`tests/test_notebook_setup.py`); every stop of a cell is
one line without a traceback (the cell's `CellStop`, tested in a real kernel);
- the buttons it names exist;
- its cells execute, by `exec` and in a real kernel (`nbclient`, `tests/test_notebook_file.py`;
cell 1's alternatives and the cells out of order in `tests/test_notebook_setup.py`).
Expand Down Expand Up @@ -874,8 +875,13 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t
- Outputs are stripped.
- Three code cells (owner, 10 November 2026, as ZeroCostDL4Mic and NucleiSky): **1. Install**,
**2. Mount Google Drive**, **3. Run preprocessing**, each under a Markdown heading that
explains it. Cell 3 checks that cell 1 ran (`SETUP_RECORD`) and, on Colab, that Drive is
mounted, and otherwise stops naming the cell to run; no cell runs another.
explains it. Cell 3 checks that cell 1 ran (`SETUP_RECORD`), and otherwise stops naming
cell 1; on Colab without Drive mounted it opens the app after one line saying that only
files on the runtime are visible (owner, v0.5.1). No cell runs another.
- A cell that cannot go on raises its own `CellStop` (a `RuntimeError` whose
`_render_traceback_` returns the message), so Colab and JupyterLab show one line and no
traceback (owner, v0.5.1). A stop caused by another error (the interface import, the Drive
mount) names that error on the same line; an error that is not a stop keeps its traceback. Cell 1 checks the Python version before `import tomllib`.
- Cell 1 is a standalone standard-library bootstrap: nearby checkout, installed package,
or GitHub installation of the release `RELEASE_TAG`, with missing requirements
installed using the kernel's Python (`subprocess`, never `!pip`); on Colab with the
Expand Down Expand Up @@ -1051,14 +1057,14 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb

---

## 11. Status (current state, v0.4.0)
## 11. Status (current state, v0.5.1)

- **Released:** v0.4.0 (5 November 2026), with the normalization brick pinned at v0.3.4. The
- **Released:** v0.5.1 (10 November 2026), with the normalization brick pinned at v0.4.0. The
per-release history is in [CHANGELOG.md](CHANGELOG.md); the milestones PP-0 to PP-13 are in
[docs/archive/plan-milestones-v0.1-v0.4.md](docs/archive/plan-milestones-v0.1-v0.4.md); the older
status log of this file is in
[docs/archive/agents-status-history-v0.1-v0.3.md](docs/archive/agents-status-history-v0.1-v0.3.md).
- **Unreleased:** see the changelog's "Unreleased" section.
- **Unreleased:** none yet; later changes go in a new "Unreleased" section of the changelog.
- **Scope:** 2-D per plane (a Z-stack is refused as `z_stack_not_supported`); TIFF and OME-Zarr
input read directly (TIFF to OME-Zarr conversion and the `_moved.py` re-exports were removed);
optional masks, the mask step and the notebook's **Describe the masks**; named-axis copies
Expand Down
103 changes: 102 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,108 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature, on b
are in [docs/plan.md](docs/plan.md) §12. (The v0.3 to v0.4 migration guide was removed in
v0.5.1: no run before v0.5.1 is read.)

## Unreleased (v0.5.1)
## v0.5.1 (10 November 2026)

The first release in which every run is described, with the shared request resolver for the
other bricks and the seven-step notebook. There is no backward compatibility before the first
pipeline release (MorphoBricks-Spec ADR 0002 AR10): older run folders, records and drafts are
refused, never converted. Normalization is pinned at v0.4.0.

**Breaking**

- **Every run needs a published experiment description** (`morphobricks.toml`, saved by the
notebook's *Check and save*). The run, the mask step, `check`/`run`/`masks` and
`RunInputs.open` refuse an output folder without one with `A_DESCRIPTION_REQUIRED`, before
anything is read or written.
- **Older files are refused, not converted**: run manifests of versions 1 to 3 (every run of
v0.5.0 or earlier: run preprocessing again, which reads every image again), conversion
records other than version 3 (convert the originals again), answer drafts saved under the
earlier Spec pin (`spec_commit_mismatch`), and hand-edited manifests missing a key.
- **The legacy `experiment_design.csv` migration is gone**: the file is an output only and
nothing reads it back; the `design` command and `run --design` are removed.
- **Normalization v0.4.0**: `PlaneRecord.position_id` is required, a blank `position_id` in a
table row is refused, and tables of schema versions 1 to 3 are refused.

**New**

- **`RunInputs.resolve_request`** (resolver contract version 1): one interpreter of a described
run for every analysis route, channels by canonical ID per position, every refusal at once,
coverage per biological repeat and the provenance.
- **`Fov.image_type`**: each position's declared image type, `"rgb"` or `"channels"`.
- **Lab-tool links**: `run --lab-links FILE` (and `--confirm-repeat-codes`), recorded in the
run manifest only.
- **The seven-step notebook**: Images → Axes → Channels → Masks → Design → Check and save →
Run, with a quick scan and an example picture before the headers are read, progress for every
wait, one column and no box that scrolls on its own. Three code cells shown as Colab forms;
a cell that cannot go on stops with one line, without a traceback.

**Fixes**

- The notebook output is no longer flooded with debug messages from ImageJ TIFFs.
- *Is compared against* keeps its dropdown; lists of files draw no empty bullets; every
disabled action says beside it what it waits for.
- On Python 3.10, cell 1 says "Select a Python 3.11 or newer kernel" instead of failing on
`import tomllib`.

### Migration from v0.5.0

**Users.** Open the v0.5.1 notebook, describe the experiment (the seven steps) and run again
into the same output folder: the run reads every image again and replaces the v0.5.0 output.
Nothing in an old `experiment_design.csv` is carried over; describe the design in *Design*. On
the command line, `check`, `run` and `masks` need the published description: write it in the
notebook (`describe` shows what is still open). A `.morphobricks-conversion.json` written before
v0.5.1 is refused: run `convert` again into a new folder.

**Bricks** (each is unaffected until its pin moves to v0.5.1):

| v0.5.0 | v0.5.1 |
|---|---|
| A run without a description | Refused (`A_DESCRIPTION_REQUIRED`, `DescriptionRequiredError`); tests use `testing.make_clean_run` (described) or publish a `morphobricks.toml` before `run_experiment` |
| `run_experiment(design=...)`, `run --design`, the `design` command, `edit_design`, `write_check_outputs`/`CheckOutputs` | Removed: the design is derived from the description |
| `read_confirmed_design`, `ConfirmedDesign`, `DesignCheckError`, `RunInputs.design()` | Removed: `Fov.repeat_group` or `RunInputs.resolve_request` |
| `RunInputs.image_type()` | Removed: `Fov.image_type` (or `FovSelection.image_type`) |
| `RunInputs.description()` and its siblings could be None | Never None |
| `preflight`/`remasked`/`masks.find_masks(with_description=...)`, `run.preflight.description_in_use` | Removed: the name refusals always apply |
| `STRICT_MANIFEST_VERSION`, `DESCRIBED_MANIFEST_VERSION`, `LEGACY_MANIFEST_VERSION`, `version_for` | Removed: `MANIFEST_VERSION` is `"4"`, the only version read |
| `source_fingerprint` | Removed: `inventory.source_signature` |
| `GroupSummary.plane_count` | Removed: `xyc_plane_count` or `channel_histogram_count` |
| `discovery.TIFF_SUFFIXES`, `discovery.SYSTEM_NAMES` | Moved to `image_names` (`is_tiff_name` and `is_system_name` still import from `discovery`) |
| The design editor: `design.ExperimentDesign`, `DesignChanges`, `default_design`, `load_design`, `read_experiment_design`; the module `design_edit` (`DesignEdit`, `DesignEditError`, `apply_design_edit`, `parse_repeat_group`); `run.design_file` (`design_for_run`, `write_design`, `write_design_template`); `report.design_summary`, `design_table`; the module `cli_design` | Removed: the design rows are the read-only `DesignTable`, derived from the description |
| `draft.migrate_legacy_design`, `LegacyMigrationRefused`, `MigrationNote`, `Draft.origin`, `Draft.legacy_missing`, `Draft.notes`, `Draft.unresolved_controls` | Removed |
| `PreprocessingNotebook(describe=True)` | `PreprocessingNotebook()`: no `describe` argument; an embedding brick must let the user describe and save before the run |
| `PreprocessingNotebook.design`; the widgets `group_confirmations`, `groups_box`, `controls_box`, `control_checks`, `design_change` | `.pairing`; the widgets are removed |
| `welcome()`: a page with a *What next?* box | A one-line `VBox` (name, version, ready) |
| `results_step()` | After `steps()`, an empty hidden box (*Compare groups* is in the run step); alone, the page it was |
| `check_candidate(..., output)` | No `output` argument |
| `PlaneRecord(...)` (normalization v0.3.4) | `PlaneRecord(..., position_id=...)`, keyword-only and required |

Added for bricks: `Request`, `MaskRequest`, `Capabilities`, `ResolvedRequest`, `FovSelection`,
`MaskSelection`, `PlaneSelection`, `Refusal`, `RepeatCoverage`, `Provenance`,
`RESOLVER_VERSION`, `VolumeNotSupportedError`; `LabLinks`, `LabLinksError`, `read_lab_links`,
`RunMode(lab_links=...)`; `make_described_run(..., mask_images=...)` and
`testing.described_images`; `progress=` callbacks of `discover_experiment`, `masks.find_masks`
and `run.preflight.remasked`. The pinned surface is `tests/test_brick_api_surface.py` (contract
§9.1). The details are in the sections below.

### Release v0.5.1: the shipped notebook

- `RELEASE_TAG` `v0.5.1`, `current_version` `0.5.1`, and `RELEASE_REF` the v0.5.1 preparation
commit `d023ff9`; cell 3 calls `PreprocessingNotebook()` (`test_notebook_file.RELEASE_CALLS` records
the pair).
- Each code cell is a Colab form with its code hidden (`"cellView": "form"`, AGENTS.md §7).
- A cell that cannot go on (cell 3 before cell 1, a refused checkout, a failed install or Drive
mount, Python 3.10) stops with its message on one line and no traceback: it raises the cell's
own `CellStop`, a `RuntimeError` whose `_render_traceback_` is the message (shown so by
IPython, so in Colab and JupyterLab; tested in a real kernel). A stop caused by another error
names it on that line, as in "… its notebook interface could not load (ImportError: …)" or
"Google Drive could not be connected (ValueError: …)", so the one line never hides what
failed; any other error keeps its full traceback (tested in a real kernel).
- **Cell 3 on Colab without Drive opens the app** after one line: "Google Drive is not mounted:
only files on this Colab runtime are visible, and they are deleted when the runtime stops."
(It stopped and named cell 2 before.)
- Cell 1 checks the Python version before `import tomllib`, so Python 3.10 gets the "Python
3.11 or newer" line.
- The introduction describes the seven-step app in short.

### The legacy `experiment_design.csv` migration is removed (inventory PP-03)

Expand Down
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ from there.

## Status

Pre-alpha; the current release is v0.5.0 (what is planned next is in
Pre-alpha; the current release is v0.5.1 (what is planned next is in
[docs/roadmap.md](docs/roadmap.md)). Implemented and tested:

- Folder discovery and validation, TIFF and OME-Zarr header inspection without reading pixels
Expand Down Expand Up @@ -57,7 +57,7 @@ v0.5.1 is refused until it is run again; nothing is converted. See the [CHANGELO
## Installation

```bash
pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0"
pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.1"
```

For development, from a clone:
Expand All @@ -67,7 +67,7 @@ pip install -e ".[dev]"
```

The optional `notebook` extra installs the widget dependencies for the notebook
(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0"`).
(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.1"`).
The notebook's first code cell (**1. Install**) can set this up automatically: it uses a compatible nearby
checkout or installed package, and installs missing requirements into the active kernel. A standalone
download installs from GitHub if needed (internet access and Git required).
Expand All @@ -85,9 +85,10 @@ LabConstrictor's
code cells, each under a heading that explains it, as in the lab's ZeroCostDL4Mic and
NucleiSky notebooks: **1. Install MorphoBricks preprocessing** (prepares the environment and
reports versions), **2. Mount Google Drive** (Colab only; elsewhere a one-line note) and
**3. Run preprocessing** (explicitly displays the complete preprocessing UI). Cell 3 run out
of order says which cell to run first and never runs it for the user. Users never edit
code, so LabConstrictor may hide it.
**3. Run preprocessing** (explicitly displays the complete preprocessing UI). Cell 3 run
before cell 1 says in one line to run cell 1 and never runs it for the user; a cell that
cannot go on stops with one line, without a traceback. Users never edit code: each code
cell is a Colab form with its code hidden, and LabConstrictor may hide it.
2. Colab-only setup stays inside `if "google.colab" in sys.modules:` and mounts Google Drive.
The UI uses only core ipywidgets supported by both ipywidgets 7.7 and 8; it does not force an
ipywidgets upgrade/restart or enable Colab's custom-widget manager. Locally the same notebook
Expand Down Expand Up @@ -223,7 +224,7 @@ ruff format .
pytest
```

Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.5.0 is in [docs/roadmap.md](docs/roadmap.md).
Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.5.1 is in [docs/roadmap.md](docs/roadmap.md).

## License

Expand Down
Loading
Loading